Skip to content

fix: decompress whole-file cache downloads made by cat - #2296

Open
adenzhou1350 wants to merge 3 commits into
fsspec:masterfrom
adenzhou1350:codex/filecache-cat-compression-20261015
Open

adenzhou1350 wants to merge 3 commits into
fsspec:masterfrom
adenzhou1350:codex/filecache-cat-compression-20261015

Conversation

@adenzhou1350

Copy link
Copy Markdown
Contributor

Cache-level compression= is documented to decompress files on download. A cold filecache.cat() or simplecache.cat() bypasses that path, saves the compressed bytes as a complete cache entry, and returns them. A later open() then also returns compressed bytes. Opening the file first gives the expected contents, so the result depends on access order.

Use the existing decompression download helper for compressed cache misses in cat(). The default, uncompressed bulk-download path stays unchanged. This does not add async decompression or compressed-download progress reporting.

Tests cover gzip, bz2 and inferred compression; cold and warm caches; scalar, list and glob inputs; persistent cache reload; and unchanged raw-byte behavior without cache-level compression. A separate regression checks that calling cat() cannot change what a subsequent open() reads, without assuming anything about cat()'s own return value.

Validation:

  • Linux CPython 3.12, complete commit-exact source exports and identical selected tests: unchanged production 24 failed / 34 passed, candidate 58 passed. Both imported module paths and source hashes were checked.
  • Complete test_cached.py + test_caches.py: 387 passed / 15 skipped on Linux. The skips require optional pyftpdlib. Source-file identities remained unchanged after execution.
  • Windows CPython 3.14: the same focused 58 cases pass. Full Windows qualification is not claimed: a broader run had a PermissionError in the existing concurrent test_partial_download_invisible control, and an earlier run had a warm-cache file-replacement error. Ten fresh matched diagnostic pairs passed both arms but did not reproduce or explain those intermittent failures. Failed reports are retained; the initial missing pytest-mock fixture errors were resolved in a private test-support directory.
  • Project-pinned Ruff 0.14.3 check/format and diff checks pass.

AI-assisted change. No full-repository, cloud-provider or performance claim.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
@adenzhou1350

Copy link
Copy Markdown
Contributor Author

Merged current master in 2e6cca5 and resolved the changelog conflict, preserving both sets of entries. Local cache tests: 184 passed / 20 optional skips; pinned Ruff 0.14.3 check/format and diff checks pass. The new Windows CI failure is six GitHub-backend tests receiving HTTP 403 rate-limit responses, with 2462 tests passed: https://github.com/fsspec/filesystem_spec/actions/runs/38847997967/job/116802751467 . The cache regression tests are passing. Could a maintainer rerun that Windows job when convenient? The job-rerun API requires repository admin rights, which this account does not have.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant