Repository navigation
fix: decompress whole-file cache downloads made by cat - #2296
Open
adenzhou1350 wants to merge 3 commits into
Open
adenzhou1350 wants to merge 3 commits into
adenzhou1350 wants to merge 3 commits into
Conversation
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Contributor
Author
|
Merged current master in 2e6cca5 and resolved the changelog conflict, preserving both sets of entries. Local cache tests: 184 passed / 20 optional skips; pinned Ruff 0.14.3 check/format and diff checks pass. The new Windows CI failure is six GitHub-backend tests receiving HTTP 403 rate-limit responses, with 2462 tests passed: https://github.com/fsspec/filesystem_spec/actions/runs/38847997967/job/116802751467 . The cache regression tests are passing. Could a maintainer rerun that Windows job when convenient? The job-rerun API requires repository admin rights, which this account does not have. |
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cache-level
compression=is documented to decompress files on download. A coldfilecache.cat()orsimplecache.cat()bypasses that path, saves the compressed bytes as a complete cache entry, and returns them. A lateropen()then also returns compressed bytes. Opening the file first gives the expected contents, so the result depends on access order.Use the existing decompression download helper for compressed cache misses in
cat(). The default, uncompressed bulk-download path stays unchanged. This does not add async decompression or compressed-download progress reporting.Tests cover gzip, bz2 and inferred compression; cold and warm caches; scalar, list and glob inputs; persistent cache reload; and unchanged raw-byte behavior without cache-level compression. A separate regression checks that calling
cat()cannot change what a subsequentopen()reads, without assuming anything aboutcat()'s own return value.Validation:
test_cached.py+test_caches.py: 387 passed / 15 skipped on Linux. The skips require optionalpyftpdlib. Source-file identities remained unchanged after execution.PermissionErrorin the existing concurrenttest_partial_download_invisiblecontrol, and an earlier run had a warm-cache file-replacement error. Ten fresh matched diagnostic pairs passed both arms but did not reproduce or explain those intermittent failures. Failed reports are retained; the initial missingpytest-mockfixture errors were resolved in a private test-support directory.AI-assisted change. No full-repository, cloud-provider or performance claim.