Cache is not invalidated for relative date/time queries

[yantar92]

Consider a simple query like (ts-active :to today) and the following headline:

* test
SCHEDULED: <2021-06-21 Mon>

If today is Sunday, 20th, the query will correctly return nil. However, if the file containing the headline is unchanged, and we run the query again next day (Monday, 21st), the return value will still be nil. It should not be.

[alphapapa]

Thanks, that's a good catch.

For my own understanding: It's happening because the today argument is replaced in the --query-predicate function, which only applies inside the byte-compiled predicate, which is not part of the cache key (IIRC, there are some issues with hashing lambdas that makes them unsuitable for keys, at least before 27.1). Probably, the argument replacing should be done in the query normalizers. That seems like too large of a change for a bugfix release, so I'll do this in 0.6.

[yantar92]

Probably, the argument replacing should be done in the query normalizers.

Sounds reasonable. However, it may not be as simple as just putting the current (or relative) time/date into the normalised query. Consider a custom predicate matching all the tasks scheduled today, but no later than 1 hour after current time. If we simply put current date+time as one of the arguments in the normalised query, the cache will always be invalid. Maybe it is better to introduce a new cache invalidation mechanism based on time, not just on the query sexp? Something like (org-ql-defpred test (time) "" :cached `(:query t :time ,(ts-adjust 'hour 1 (ts-now))) :normalizers ... :body ...)

[alphapapa]

Consider a custom predicate matching all the tasks scheduled today, but no later than 1 hour after current time. If we simply put current date+time as one of the arguments in the normalised query, the cache will always be invalid.

IIUC, the cache should never used used for such a query--or more specifically, the cache would only be valid for up to one second, anyway, depending on the timestamp resolution--so running a new query every time would be correct.

[alphapapa]

@yantar92 That required a lot more work than I expected, due to moving some of the argument processing into query normalization and changing some internal macros (as well as unexpected, inexplicable test failures only on CI, but I digress). All the tests pass (including new ones I added to check for this problem). Please let me know how it works for you.

Regarding the custom predicate issue: Again, if I understand that specific example properly, I think that won't be a problem. But if I misunderstood it, or if there are other issues with custom predicates and cache invalidation, please open another issue and we can work on it.

Thanks for your help. I'm surprised it took this long for this issue to be noticed.

[alphapapa]

P.S. If you haven't seen #143, please take a look, as I'd like your input on the proposed changes.

[yantar92]

@yantar92 That required a lot more work than I expected

Thanks!

Thanks for your help. I'm surprised it took this long for this issue to be noticed.

I suspect that such kinds of queries are more common for agenda-like views. However, caching never works when refreshing agenda on master without #203. That's why the issue was never captured.

[alphapapa]

I suspect that such kinds of queries are more common for agenda-like views. However, caching never works when refreshing agenda on master without #203. That's why the issue was never captured.

I'm not exactly sure what you mean by "when refreshing agenda." Do you mean your own, agenda-like custom query views? Do you mean that, without merging #203, those queries' results are never cached? Or do you mean that something else in your config modifies the text properties and causes the org-ql cache to be invalidated?

(I'm reopening this issue to ensure that I'll see your response; notifications for closed issues tend to get lost in my notifications list.)

[alphapapa]

Retargeting this for 0.7. 0.6 has been delayed for too long.