Skip to content

[Python] Wait for AsyncWrapper work before teardown - #40507

Draft
bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/python-async-teardown
Draft

bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/python-async-teardown

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

AsyncWrapper.teardown() can close the wrapped DoFn's resources while background work still uses them. A bounded DirectRunner pipeline reports DONE, then a delayed write fails with ValueError: I/O operation on closed file.

This change tracks work owned by each wrapper and waits for it before tearing down the wrapped DoFn. Asyncio tasks stay tracked through cancellation cleanup, and the shared executor and event loop remain available to other wrappers.

The AsyncWrapper tests and DirectRunner reproductions passed locally in both thread-pool and asyncio modes. Teardown has no timeout, so hung processing or cleanup can delay shutdown. Pending-output delivery and timer behavior are unchanged.

Run the Python unit suite with:

./gradlew :sdks:python:test-suites:tox:py312:testPython312

  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for contribution guidelines.

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.26%. Comparing base (76f6e6d) to head (93fa2ec).
⚠️ Report is 141 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40507      +/-   ##
============================================
+ Coverage     56.16%   56.26%   +0.09%     
  Complexity     2288     2288              
============================================
  Files          1124     1124              
  Lines        178336   178365      +29     
  Branches       1489     1487       -2     
============================================
+ Hits         100170   100354     +184     
+ Misses        75645    75490     -155     
  Partials       2521     2521              
Flag Coverage Δ
python 79.74% <100.00%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant