Skip to content

Fix Pyodide HTTP request body serialization and Blob conversion - #2293

Open
adenzhou1350 wants to merge 7 commits into
fsspec:masterfrom
adenzhou1350:codex/fix-pyodide-json-body
Open

adenzhou1350 wants to merge 7 commits into
fsspec:masterfrom
adenzhou1350:codex/fix-pyodide-json-body

Conversation

@adenzhou1350

@adenzhou1350 adenzhou1350 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

The Pyodide shim serializes data in its json= branch. Real browser checks also exposed Python→JS proxy conversion of byte bodies and Blob options.

This fixes that boundary:

  • Serialize the supplied JSON value, including falsy values.
  • Convert Blob parts/options with pyodide.ffi.to_js and Object.fromEntries.
  • Consume finite chunk iterators used by HTTPFileSystem.put_file().
  • Preserve byte bodies and reject simultaneously supplied data/JSON.

Validation

Current head: 1b3510995181c93a7af981bf3c17082d31016648, merged with master 941e6f9. Resolved the changelog-only conflict while retaining the upstream DBFS fix; the entry now sits beside the existing Pyodide entry. The HTTP implementation and tests are unchanged.

  • Windows CPython 3.14: complete fsspec/implementations/tests/test_http_sync.py, 72 passed, rerun after the latest merge.
  • Same shim test selection on pinned master 4c15b0e: 14 failed, 8 passed, with no collection errors. This includes JSON and iterator regressions.
  • Pyodide 314.0.7 / CPython 3.14.2 workers: 19 real Blob/XHR checks passed on source 4902f55; the HTTP shim and its test file are unchanged at the current head. Includes JSON/falsy values, byte bodies, iterators, and 8 actual put_file() → cat_file() round trips across POST/PUT × file/BytesIO × binary/empty payloads. Bytes, callback counts, and caller-owned stream lifetime match expectations.
  • Applicable repository pre-commit hooks and git diff --check pass after the latest merge.

The browser checks verify imported source path/hash. Native mocks check the conversion protocol; the separate worker checks exercise JavaScript behavior. The earlier partial repair passed direct-request checks but failed actual iterator uploads; the final change covers those callers too. The latest merge imports the upstream DBFS retry fix, without changing this PR's HTTP implementation; its eight offline unit cases also pass on Windows.

Uploads remain buffered as a Blob, not incrementally streamed. No large-upload memory, cloud-service, performance, full-project or latest hosted-CI pass is claimed. Current-head Windows CI reports 4 GitHub-backend test failures, each 403 Client Error: rate limit exceeded; 2,657 tests passed, 315 skipped and 10 xfailed. This is not a passing Windows/full-CI result, and no test was disabled or workflow rerun requested here. Other matrix jobs were still in progress at inspection.

AI-assisted implementation and validation; signed off by Xucheng Zhou.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
@adenzhou1350

Copy link
Copy Markdown
Contributor Author

@martindurant Could you review this follow-up to the Pyodide shim? Real worker/XHR checks reproduce the JSON, byte-body and MIME-type issues; the same checks pass after explicit conversion. The native HTTP-sync test file also passes (65 tests). Thanks!

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
@adenzhou1350

adenzhou1350 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

The latest Python 3.13 job failed only six GitHub-backend tests, each reporting 403 Client Error: rate limit exceeded from the GitHub API. The job reports 3059 passed, 40 skipped and 2 xfailed; the other ten check runs, including lint, passed. Could a maintainer retry this failed job once the API limit resets? Contributor permissions do not allow me to rerun upstream workflows. Log: https://github.com/fsspec/filesystem_spec/actions/runs/39013522626/job/117340819702

Update (2026-10-20): the current-head Windows job also failed only four GitHub-backend tests with HTTP 403 rate-limit errors: test_github_open_lfs_file, test_github_cat, test_github_ls, and test_github_rm. It reports 2657 passed, 315 skipped and 10 xfailed. This is another external GitHub API-limit failure, not a new HTTP-sync/Pyodide assertion failure. Latest Windows job. Please handle the affected CI once API capacity is available; no retry-only source push was made.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
@adenzhou1350

Copy link
Copy Markdown
Contributor Author

Merged current master (4c15b0e) and resolved the changelog conflict. The merged version passes all 72 HTTP-sync tests, 19 real Pyodide/Blob/XHR checks, and applicable pre-commit hooks.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant