Skip to content

fix: use cibuildwheel's CA bundle in cross venvs - #3020

Merged
henryiii merged 4 commits into
pypa:mainfrom
henryiii:fix/cross-venv-pip-cert
Oct 22, 2026
Merged

henryiii merged 4 commits into
pypa:mainfrom
henryiii:fix/cross-venv-pip-cert

Conversation

@henryiii

@henryiii henryiii commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

馃 AI text below 馃

Follow-up to #3017 and #3019. On Android, iOS, and Pyodide, pip runs in a cross venv with a patched sys.platform. truststore chooses its backend from platform.system(), so it uses the OpenSSL backend instead of the macOS or Windows one. This backend reads OpenSSL's default CA file. On macOS hosts, that file is /etc/ssl/cert.pem (for python-build-standalone, and for a python.org framework build that has no cert.pem). That file still has the compromised eMudhra root (emSign ECC Root CA - G3). Apple blocks the bad intermediates through revocation in the Security framework, but this PEM file does not get those blocks.

This PR sets two variables to cibuildwheel's certifi bundle, only in the environments that have a patched sys.platform:

  • SSL_CERT_FILE: truststore then loads this file instead of the host's CA file. PIP_CERT alone is not sufficient here, because with truststore it only adds certs. With pip 26.2.2 and a PIP_CERT that has one unrelated CA, pip still connects to PyPI.
  • PIP_CERT: for pip without truststore (for example, pip < 24.2 pinned by the user). This path uses requests, which ignores SSL_CERT_FILE.

Scope per platform:

  • Android: only android_env (before-build, build, before-test, test install). build_env has the real sys.platform, so environment setup, repair, and the test runner do not get the variables.
  • iOS: the full cross venv environment (before-build, build, repair, and test). This venv patches the platform for every Python process, so the scope cannot be smaller without a separate host environment.
  • Pyodide: only the test venv (before-test, test install, test run). The patch is only in pyodide venv. The build environment is a normal host venv.

Values set by the user take priority.

uv also reads SSL_CERT_FILE in its default mode, so this also covers the uv frontend on Android (uv < 0.13.4 has the bad root in its bundled webpki store).

The Android and iOS test projects check in setup.py that both variables are set. The Pyodide test checks them in before-test. I ran the Pyodide tests locally: they fail without the fix and pass with it. Android and iOS were verified locally in the comments below.

@henryiii henryiii changed the title fix: use cibuildwheel's CA bundle for pip in cross venvs fix: use cibuildwheel's CA bundle in cross venvs Oct 21, 2026

@agriyakhetarpal agriyakhetarpal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm that the Pyodide fix makes sense at least (both SSL_CERT_FILE and PIP_CERT), but would like the Android/iOS code overall to get additional eyes, if possible, as there might be context that could be missed. Thanks!

@henryiii

henryiii commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

I was able to verify Android locally with an agent. My /etc/ssl/cert.pem is still vulnerable so it's easy to verify that SSL_CERT_FILE was needed. I left PIP_CERT just in case someone pins an old pip, though it's not otherwise needed when cross compiling (since pip is new enough to have the bundled certs). Without the fix, android was still able to use the bad certs. Wasn't able to verify iOS, but I think it should be the same.

@agriyakhetarpal

Copy link
Copy Markdown
Member

I have Xcode installed and can verify with iOS and my agent. Will run in a while.

@agriyakhetarpal

Copy link
Copy Markdown
Member

iOS checks out, and the reasoning seems sound. I tested with Python 3.14 and iOS 27.1 Simulator (24A94425).

@agriyakhetarpal

Copy link
Copy Markdown
Member

@henryiii here's what my agent found on your PR after it completed the iOS runner. Some good follow-ups here, I think:

Still reachable after PR 3020

  1. OpenSSL's default cert directory on Linux hosts. When SSL_CERT_FILE is set, truststore's OpenSSL backend calls set_default_verify_paths, which loads both the file and the default directory. On macOS the directory is empty, so the fix is clean here. A python-build-standalone host on Debian or Ubuntu has /etc/ssl/certs as its directory, full of hashed symlinks to the distro store, and truststore's own check accepts a directory like that. The system CAs then get loaded alongside certifi, and a stale ca-certificates package reopens the hole. Setting SSL_CERT_DIR to an empty directory next to SSL_CERT_FILE would close it. This follows from the truststore source and OpenSSL semantics rather than a run on a Linux box.
  2. before-all on all three platforms runs with the host environment, not the cross venv, so neither variable is set there. Host pip on macOS and Windows uses the operating system store and is fine. Anything else doing TLS from a python-build-standalone host Python, such as urllib or a Meson wrap download, reads /etc/ssl/cert.pem, which still trusts the root on this machine.
  3. Java in the Android test phase. CPython's Android runner drives Gradle and sdkmanager, which fetch the Gradle distribution, Maven dependencies, and SDK components over Java's own TLS stack. Java reads the JDK cacerts keystore and ignores SSL_CERT_FILE. The Temurin 17 on this Mac still holds all four emSign roots including ECC G3, added under JDK-8319187, and I found no OpenJDK change removing it. Only the test step is exposed, the wheels are not. The fix is a JDK update or a custom truststore, nothing cibuildwheel can do.
  4. requests-based tools inside the cross venv. requests ignores SSL_CERT_FILE and uses REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, or certifi. pyodide-build fetches its cross-build environment with requests. The certifi pin in the constraints file covers this from 4.3.2 on, but a user constraints file pinning an older certifi, or a stale bundle in one of those two variables inherited from the host shell, reopens it.

Covered by the PR without it saying so

  • The uv frontend on Android. cibuildwheel does not pin uv, and uv before 0.13.4 ships the root in its bundled webpki store. I checked uv 0.12.10 with a bundle holding one unrelated CA in SSL_CERT_FILE, and the PyPI handshake failed with an unknown issuer, then succeeded with certifi. So uv honours the variable in its default mode and the PR covers it. Worth a line in the PR description, or a case in the Android test.
  • emsdk and the Android runner's own downloads use curl on macOS, which goes through the Keychain, and urllib elsewhere under the same environment, so SSL_CERT_FILE applies.

Small notes

  • Ordering is right: the user's environment option is applied after the setdefault calls, so a user override still wins.
  • Node in the Pyodide test phase uses its own bundled roots, but only matters if tests reach the network.

Sources: Google blog on the Chrome Root Store removal, uv advisory GHSA-hq85-5cpj-mr62, OpenJDK 8u402 release notes adding the eMudhra roots.


I'm happy to take point 4 :D

@henryiii

Copy link
Copy Markdown
Contributor Author

On 1, guessing this is for android or pyodide? iOS can't be built from Linux. :) On 2, I don't know what we should do with host. I'd rather think I'd treat it like running something before cibuildwheel runs, and leave it up to the user/host. On 3, it says we can't fix that. On 4, up to you. If someone pins older packages, we don't have to try to always fix that, but we can attempt to if you'd like.

@joerick joerick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a bit confused here... I don't believe we're actually doing anything wrong by trusting the OS's cert file here. Isn't it the user or Apple's job to fix their /etc/ssl/cert.pem rather than us?

Setting SSL_CERT_FILE to certifi might fix a security hole today, but could easily open another one in the future. And if it does, that becomes our responsibility.

@henryiii

Copy link
Copy Markdown
Contributor Author

It just consistently uses our cert file, rather than ignoring it cross compiling. When you are not cross compiling, /etc/ssl/cert.pem is not used today. Though I see your point, one could argue users should patch /etc/ssl/cert.pem.

@henryiii

Copy link
Copy Markdown
Contributor Author

Also, this is still only making pip use certifi, not other tools, before-build, etc.

@joerick

joerick commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

I just reread the description and I see that by doing the sys.platform overriding we hitting a peculiar edge case where tools are getting confused, inviting this behaviour.

I still think that we're into the realm of patching things that aren't our responsibility here, but since the sys.platform monkeying is in our purview I can see the argument.

It would be better IMO if android env tweak is only applied to android_env, since the other env has a correct sys.platform.

Also, this is still only making pip use certifi, not other tools, before-build, etc.

SSL_CERT_FILE is used quite widely I believe. e.g. curl will read it. And this is setting the variables when the venv is created. So I think it will affect before-build and repair (FWIW) too.

@freakboy3742 freakboy3742 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The iOS and Android changes generally make sense to me; the only caveat I'd flag is the same one picked up by @joerick - that the environment being modified is the build environment, which is then inherited by the test/runtime environment (android_env on Android, test_env on iOS).

That means we're overriding the certificates on the build machine; that means system certificate stores are being overridden. That won't matter for most users/builds, as the updated certifi certs will be just as accurate as the system (if not more so) - but if the user is in an environment that has a corporate VPN cert, they might not be able to run pip installs.

On Android, iOS, and Pyodide, pip runs with a patched sys.platform, so
truststore uses its OpenSSL backend. On macOS hosts this reads
/etc/ssl/cert.pem, which still has the compromised eMudhra root. Set
PIP_CERT to our certifi bundle in these environments.

Assisted-by: ClaudeCode:claude-opus-5-5
With truststore (pip 24.2+), PIP_CERT only adds certs, so the host's
OpenSSL CA file was still trusted. SSL_CERT_FILE replaces it. Keep
PIP_CERT for pip without truststore.

Assisted-by: ClaudeCode:claude-opus-5-5
build_env has the real sys.platform, so it does not need them.

Assisted-by: ClaudeCode:claude-opus-5-5
Only pip in the pyodide venv has a patched sys.platform.

Assisted-by: ClaudeCode:claude-opus-5-5
@henryiii
henryiii force-pushed the fix/cross-venv-pip-cert branch from 5a649f4 to 7c86eeb Compare October 21, 2026 23:00
@henryiii

henryiii commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Tightened it up on Android/Pyodide. iOS I can't really tighten up without a refactor to split the environment.

before-build

I meant before-all, sorry. before-all tends to run non-pip things, before-build often is where you run pip.

馃 AI text below (could have mistakes) 馃

This table shows where SSL_CERT_FILE and PIP_CERT are set.

Step Android iOS Pyodide
before-all no no no
set up the build environment no yes no
before-build yes yes no
build yes yes no
repair no yes no
before-test yes yes yes
test install yes yes yes
test run no yes yes

On Android, "test run" is the Gradle runner, which runs in build_env. The variables are set with setdefault, so values that a user or the host already set always win.

@ryanking13 ryanking13 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't verified locally but the Pyodide-side change makes sense to me.

@henryiii
henryiii dismissed freakboy3742鈥檚 stale review October 22, 2026 13:16

Limited (at least on android)

@henryiii

Copy link
Copy Markdown
Contributor Author

Got an okay from @joerick on Discord, so going to put this in. Anyone can work around it by setting the variables like SSL_CERT_FILE manually.

@henryiii
henryiii merged commit 0d511ac into pypa:main Oct 22, 2026
45 checks passed
@henryiii
henryiii deleted the fix/cross-venv-pip-cert branch October 22, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants