Cleanup of old notifications, revisions and activities #18297
Replies: 4 comments 5 replies
-
For everyone using MySQL. I've just created a gist with an SQL transaction which does the cleanup (use at your own risk!). |
Beta Was this translation helpful? Give feedback.
-
Thoughts/feelings/concerns on my proposed first step? :) #18105 |
Beta Was this translation helpful? Give feedback.
-
Disabling them could be an option ? |
Beta Was this translation helpful? Give feedback.
-
+1 I realy appreciate this idea. I think a first step/alternative would be to have an activity auto remove schedule. Since we often dont want to have every activity logged (since GPDR). |
Beta Was this translation helpful? Give feedback.
-
Summary
Directus does currently not allow automatic or manual (via UI or webhook) deletion of old notifications, revisions (including flow logs) and activities. But I guess for most users storing all of these old items is unncessary and just clutters the database. We should provide a cleanup option to reduce the space requirements.
Motivation
Most users proabably don't need the full history of activities, but since Directus saves every event (login, flow execution, edit, notification etc.), the space requirements for these not needed items can grow to a significant size over time. A lot of programs use log rotation to counter this problem but we don't have something like that in Directus right now.
Also, if a flow is executed very often, the log history in the sidepanel gets cluttered quite fast.
One could argue that you can already cleanup the database by executing custom SQL queries. But this feels a bit cumbersome, especially since Directus advertises with the catchword "no-code".
Detailed Design
Note: Even though flow logs are also stored in the
directus_revisions
table I use the term "revisions" in this post to refer to all revision entries excluding flow execution logs. This is because flows still have normal edit revisions and I want to distinguish between them and flow execution logs.I suggest an (optional) automatic cleanup feature: We could add a new group in the project settings with the following fields:
For revisions, flow logs, activities:
An item is deleted when it's older than the calculated date
$NOW - minimal duration to keep
and there are at leastminimal number of items to keep
newer items of the same type (e.g. revisions for the same item, flow logs for the same flow, ...). Activities related to saved revisions / flow logs are also saved even when they normally would have been deleted.For notifications:
A notification is deleted when it's longer in the notification archive than
minimal duration to keep
. Maybe in this move it makes sense to rename "archive" to "recycle bin".I'd imagine that the cleanup can be triggered manually via a button or webhook or an internal CRON job. The later could be configured via environment variables.
We could also think about adding a manual delete button for activities, logs, revisions etc. as this shares a lot of the logic.
Requirements List
Must Have:
Should Have:
Could Have:
Drawbacks
Alternatives
Impact of not doing this: People may be challenged with high storage requirements after some time or they have trouble writing SQL queries for manual cleanup.
Adoption Strategy
Should be no breaking change. At least when we don't configure an automatic cleanup per default. However a small databse migration could be necessary to adjust a foreign key constraint (see #13481).
Unresolved Questions
See #13481
Beta Was this translation helpful? Give feedback.
All reactions