Repository navigation
Limit number of versions per object #4
Description
Will help keeping the DB size under control and keep stuff manageable.
Some options could include limiting on time (eg last 2 months) as well as a fixed number of deltas per object. Or a combination.
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
Activity
CharlesMx commentedon Jun 14, 2012
Voting for this one - makes sense and can easily be followed with business rules
amdbuilder commentedon Jan 13, 2013
I would include two settings to toggle unlimited retention and versions to retain. In some cases the versions to keep are more important than the length of time.
SnowCreative commentedon Jan 22, 2014
YES PLEASE!! Tables are too huge without this.
exside commentedon Mar 14, 2016
+1 from here as well
degoya commentedon Mar 23, 2016
+1
bwente commentedon Jun 10, 2016
+1
zaigham commentedon Nov 23, 2016
👍 💯
Peeet93 commentedon Oct 16, 2018
@Mark-H is there any plan for releasing this feature?
Mark-H commentedon Oct 16, 2018
Eventually, yeah. This and a few other issues, as well as a nicer UI, is what I'd like to do in a v3 some day.
jdaehne commentedon Jan 4, 2022
@Mark-H would it help to support this functionality with a "bounty"? I have a client website that needs the functionality to limit the snapshots/backups/versions to a number of items. Because this VersionX-tables get humungus at the moment. :-)
I would be glad to support this with some "bounty".
Mark-H commentedon Jan 4, 2022
With a bounty I'd be happy to add it to Murray's to do list which would certainly speed things up.
Still need to decide on the actual way to go about this as ideally it wouldn't just be "keep the last X versions" but rather it should sooner discard near duplicates than major revamps of a page, so then that might need to be tracked or processed on demand without saving a resource taking much longer.
jdaehne commentedon Jan 4, 2022
@Mark-H that sounds very good. Please let me know the approximate effort / amount, then I will clarify this with the customer and would be happy to be able to take care of it.
Is there a bounty service / github-way you prefer? Otherwise i am happy to just transfer the money via bill...
Mark-H commentedon Sep 21, 2022
To clarify what I want for this: the full history should always remain. That means merging versions past a cut-off time (e.g. 10 edits two months ago can be combined into a single one) but any approach that is only X versions or X dates is insufficient and won't be accepted.
Mark-H commentedon Sep 26, 2022
As I've just finished quoting this feature for a potential client, I figured I'd share the specification as I imagine it will need to work here as well. It ends up being a pretty much full rewrite, so say hello to the roadmap for VersionX 3.
It's still a full history, but with a much lower disk usage requirement as full copies are replaced with deltas (changesets) that get merged after some time.
Feedback is welcome, and if anyone wants to help fund the development, please do let me know.
The total estimate is 22 hours of work, which at normal €90/hr rate is €1980. Depending on a few factors, including scheduling, I may be able to offer a lower hourly open source rate (€65/hr) to make it more affordable (€1430) and I'll also need to check how much we've gotten in donations for VersionX to deduct from that number.
--
Development of VersionX 3.0 which incorporates a different data model that maximises the
version history available, while minimising the data usage required.
Remodel the database to a single table that stores deltas (“changesets”) rather than full
copies of each version.
a. Deltas contain the before and after value of individual fields.
b. This will merge all different tables for resources and elements that exist today, to
simplify the code. There is a lot of unnecessary code duplication in VersionX 2.
c. The table will have various common identifiers to allow easy filtering on specific
objects (resources/elements), who made a change, what time the change was made
(and as described below, what time period it was merged for) etc.
Estimate: 4 hours
Implement a cron job that merges similar deltas in a relatively short time frame. This will
leave a single before and after value for the edited fields within the specified time frame.
a. As before, each individual save that has any changed value will be stored. This
means that continued editing of an object can have a lot of versions.
b. Deltas older than a week will be merged to a single delta for a given hour.
This leaves at most 168 unique deltas per object per week, or 8736 per year.
c. Deltas older than a month will be merged to a single delta for a given day.
This leaves at most 7 unique deltas per object per week, or 364 per year
d. Deltas older than 3 months will be merged to a single delta for a given week.
This leaves at most 1 unique delta per object per week, or 52 per year.
e. Deltas older than 18 months will be merged to a single delta for a given month.
This leaves at most 1 unique delta per object per month, or 12 per year.
f. Deltas older than 5 years will be merged to a single delta for a given year.
g. The oldest delta (object creation) will never be removed, though it may be merged in
accordance with the above rules.
Estimate: 3 hours
The snapshot creation during the installation will be updated to create a single full delta with
all data, to start the history.
Estimate: 2 hours
When upgrading from v2 to v3, historic data will not be immediately imported as many sites
will have lots of data. Instead, a command line utility will be provided that can be manually
executed to parse through the full history, and create deltas for the changes.
Estimate: 3 hours
The VersionX UI will be rebuilt from scratch with a more visual comparison of the changes
that were made, allowing the user to scroll through changes rather than having to select a
specific version to compare.
a. Where possible, changes will be highlighted inline. This will help with, for example,
ContentBlocks being stored as a massive json structure.
b. Instead of restoring a version to a specific version, the UI will allow (1) a specific
delta to be undone, (2) or to undo all deltas up to a specific delta. This offers more
granular restore capabilities.
c. The Versions tab on resources/elements will be replaced with a History tab that
shows the deltas in a scrollable list.
Estimate: 8 hours
VersionX will provide a new internal API to allow custom objects to hook into the versioning
it provides. For example, ContentBlocks or other third party components will be able to
implement versioning in its CMP with just a few lines of code.
Estimate: 2 hours
4 remaining items
Mark-H commentedon Oct 31, 2022
Received a €200 commitment from Christoph (support ticket 33713).
Almost half-way to the target and getting this work scheduled :)
Direct contributions are welcome via the modmore donate form (any amount). Alternatively to make a commitment that will be billed when the work is done (€150 min because of manual billing) send me an email via support@modmore.com.
jdaehne commentedon Nov 2, 2022
hi @Mark-H in case you missed it, but we also did a "bounty source" of 250 $ because you linked it in the beginning of this issue. If you prefer a direct donation - i can try to cancel it.
Mark-H commentedon Nov 3, 2022
Good that you mentioned that @jdaehne, I did not get a notification of that. I disabled the automatic BountySource integration years ago as it wasn't getting used, but the issues still have the automatically added links...
They used to do something weird about cancelling. If possible, direct would be preferred, but if it's too much work or they bring up cancellation fees or whatever: don't worry about and I'll try and claim it once the work is done. I know you're good for it and we can work out logistics later. Thanks for the support. 💙
That brings down the remaining to €525!
Mark-H commentedon Jan 22, 2023
Made a little sketch to show my ideas for the new scrollable changes UI, and how the way deltas get merged over time affect what the user sees.
Obviously this is not a design, just a sketch to communicate the idea a bit more visually than the wall of text from earlier.
This would likely be in a tab on the resource/element.
I haven't fully decided what to do about the dedicated VersionX manager pages. Perhaps that'll show a similar scrollable list of changes, but across all resources/elements instead. There should be plenty space in the title bar to add the type and title of whatever it was that was edited. Add in some filters at the top.
We're still short of the funding goal to make this happen, but do have some time blocked out in February to work on this so hopefully we can get a few more people to pitch in 🤞
Current funding status:
Mark-H commentedon Jan 22, 2023
As an extra perk for supporters (besides, you know, VersionX 3 itself!) we'll make a list of supporters in the readme and in the 3.0 release announcement when it's complete.
All contributions of €10+ will get a text mention, €100+ will get it linked to your personal or company website, and €200+ will also get a small logo (~150px). €500 will get a bigger logo (~300 px).
Mark-H commentedon Jan 23, 2023
Received some more donations:
= €231 left until the funding goal 🚀
Mark-H commentedon Jan 23, 2023
Steffan from webaix just donated the remaining €231. Funding goal met, whoo! 🚀
Thanks everyone. We'll get started on this feature in the coming weeks, all progress will of course be public in the repository.
Mark-H commentedon Jun 25, 2023
There's a beta available in https://github.com/modmore/VersionX/tree/3.x/_packages, planning to get the release properly available this coming week 🎉
zaigham commentedon Jun 26, 2023
@Mark-H I have downloaded the beta2 zip, but upon upload it gives me an error:
This file does not appear to be a transport package.I tried this on MODX
v3.0.3-plMark-H commentedon Jun 26, 2023
It worked fine here @zaigham (also on 3.0.3), perhaps after downloading it got unzipped or renamed?
zaigham commentedon Jun 26, 2023
Surprisingly, I tried to upload the file via Media Browser, but it just silently failed. Then I used cPanel's File Manager and uploaded it, and then it showed up in the packages just fine.
So probably some edge case with my setup. Thanks @Mark-H for verifying.
Mark-H commentedon Aug 9, 2023
The 3.0 beta is available from the modmore provider now. We'll also release it on modx.com when we had more user feedback and have some marketing going out for it soon.
robliberty commentedon Feb 14, 2025
This is marked as Closed, but the limiting feature isn't available yet, that I can see?
Mark-H commentedon Feb 14, 2025
With 3.0, VersionX can automatically merge changesets to keep a full history but limit data storage. It's not a "Max x versions" implementation but it deals with the root problem, which is why I closed this.