Repository navigation
Provide 3.13t & 3.14t images #1082
Description
Activity
I think #947 (comment) is still the most reflective of our current state (the build matrix is large, the compilation is heavy, and this is essentially asking us to double it all for 3.13+):
We could shrink the impact by doing "clever" things like only supporting "nogil" on a limited set of architectures, a limited set of base distributions, etc, but ultimately any choice we make there is definitely going to cause some amount of friction (both for us in maintaining and explaining the line, and for users who get to navigate the line).
We could also consider just including both in the default images, but I doubt that's going to be a popular choice either (and doesn't decrease the compilation burden, just obfuscates it). Ultimately it would be really interesting if this could be an entirely runtime determined value (or some way of having two binaries compiled concurrently and using linking cleverly so that they share most of the final binary size in shared libraries), but that's probably also a tall order.
Technically, it is possible to re-enable the GIL on the free-threaded build by setting the environment variable
PYTHON_GIL=1. I don't know if it's intended for something like this.This is really interesting (because then while we'd still have a large matrix, half of it would be simply
FROM python:freethreading[what a mouthful 😭] +ENV PYTHON_GIL 1), but does make me wonder if this is actually really supported/recommended usage -- if it is, shouldn't core be adjusted to always compile with freethreading enabled and install the binary aspython3.14twith apython3.14symlink or small script wrapper that automatically enables the classic GIL mode? Then we'd have the best of both worlds with no real overhead and only a single image. 🤔Reacted by Dave Johansen, Vlad Gerasov, Javier Gallego Fernández and CanDWhile I understand the effort and maintenance burden needed for releasing free threading images, i can hardly understand the problem of job matrix size and compilation. I suppose Microsoft spent a lot more money for sponsoring developer time even last years efforts in "faster cpython" project and they have free runners for OSS projects, not to say how much resources something like brew takes, while Dockerhub cpython images have 1B+ docker pulls which means pretty much infinite bandwidth and uncountable money related.
I really cannot get what 80 additional jobs does anything bad for anybody, or even additional 1000 jobs, as long as the actions and GitHub itself can handle it? But of course, I might not be aware of something serious here.
And, let's be honest, people who desperately need free-threading probably can compile and build their own images, that is not big problem. The problem is, that virtually nobody will do it just for a sake of doing it without official images and end goal is for whole language to migrate (that is how i interpret https://peps.python.org/pep-0703/ and https://peps.python.org/pep-0779/). Python image is being run in uncountable CI/CD jobs, the effort of changing tag to free-threading-someting is very little so it would for sure encourage people to migrate, and if it work seamlessly, probably we would see more of it in production.
Thanks!
Reacted by Walter A. Boring IV and Alexey ShokovPerhaps as a compromise, free-threaded builds could be made available just for:
- 3.14+ (given that in 3.13 it was still marked as experimental, and anyone wanting to try out free-threading should really be doing so with the latest fixes/changes)
- The newer of the distro/OS versions - so Alpine 2.22 but not 2.21, Debian trixie but not bookworm, and so on. (Which seems like it would be reasonable given that someone wanting to try out a cutting edge Python feature is probably fine with being on the latest distro version?)
Doing that would add these 4 variants (and the various tags that re-use the same images):
3.14t-trixie3.14t-slim-trixie3.14t-alpine3.223.14t-windowsservercore-ltsc2025
So would increase the total variants from 40 to 44 (multiplied by the various architectures) - so an increase of 10%.
(The Python 3.9 EOL will then reduce it back down to 40 variants)Albeit for every year that goes by before free-threaded builds become enabled by default, the total builds would increase slightly - though (a) they could still be kept to just the latest distro/OS versions to minimise that, (b) hopefully it will only be a couple of years before the free-threaded project plays out one way or another?
Reacted by Edgar Ramírez Mondragón, Colin Gravill, Zanie Blue, Alexey Shokov, Joshix-1 and Iskander GabaReacted by Tianon Gravi, Petr Přikryl, Oleksandr Baltian, James Gillard, Burak Kadron, Nathan Goldbaum, Laurie O, Max Beauchez, udo-hf, Walter A. Boring IV and 3 moreOne other concern/design decision I see mentioned in the prior issue (#947) is that of naming.
It looks like since that discussion other tools have begun to standardise around the
tsuffix (eg3.14t):- uv: https://docs.astral.sh/uv/concepts/python-versions/#free-threaded-python
- GitHub Actions's
actions/setup-python: https://github.com/actions/setup-python?tab=readme-ov-file#basic-usage - pyenv: pyenv/pyenv@3ced1c4
(The
tsuffix is also what the free-threaded Wheel tag name uses, as well as thepython3.14tbinary name)Reacted by yosifkit, Tianon Gravi, Rafał Safin, Luca | Serenita and Dave JohansenThis is really interesting (because then while we'd still have a large matrix, half of it would be simply
FROM python:freethreading[what a mouthful 😭] +ENV PYTHON_GIL 1), but does make me wonder if this is actually really supported/recommended usage -- if it is, shouldn't core be adjusted to always compile with freethreading enabled and install the binary aspython3.14twith apython3.14symlink or small script wrapper that automatically enables the classic GIL mode? Then we'd have the best of both worlds with no real overhead and only a single image. 🤔Hmm, that wouldn't work unfortunately.
The challenge with a single binary approach is that free-threaded Python uses a different ABI, it's not just a runtime toggle. Extensions built for gil-enabled Python (e.g. cp314 wheels) are incompatible with free-threaded Python (e.g. cp314t wheels) and vice versa.
The
PYTHON_GILenvironment variable can control GIL behavior within a free-threaded build, but it doesn't make apython3.14tbinary ABI-compatible with standardpython3.14. You'd still need separate wheel ecosystems regardless of whether the GIL is enabled at runtime.Reacted by Tianon Gravi, Dave Johansen, Oleksandr Baltian, Antoine Pitrou and Edgar Ramírez MondragónReacted by Tianon GraviYeah, I realized that after re-reading the other thread and implementing the idea. 😞 ❤️
I'm not familiar with the details of the differences between the builds, but there is this notice when I run the free-threading build with some libraries, so is the GIL enabled on a free-threading build functionally the same as a standard build? If so, could we just have a single build and have
PYTHON_GIL=1set by default for backwards compatibility?<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'newrelic.packages.wrapt._wrappers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.Yeah, I really don't understand how that automatic fallback can work if they're not ABI compatible. 🤔
so is the GIL enabled on a free-threading build functionally the same as a standard build?
No, it still uses the free-threading ABI which is not the same as the regular ABI.
There are discussions to have a unified ABI in the future, it might land in 3.15.
Reacted by Tianon Gravi
Since #947 has been closed by the OP, I thought I'd reopen this here, since the free-threaded build is now a fully-supported build as of PEP 779.
Is there an appetite for providing 3.14t (and maybe also 3.13t, but certainly not necessary) images? If yes, we can certainly offer support with doing the necessary changes for that.