Repository navigation
Downloading pre-built Ruby Versions at runtime #44
Description
Activity
Ping @ioquatix @eregon @tenderlove
Sorry for the ping...
A Ruby build zip is +/- 10 MB. One thought is to have the Ruby community build Rubies and host them as packages in some repo, then provide an action that downloads and installs them...
I was thinking something along the same line.
We could have a separate GitHub action that just builds various Ruby versions for all virtual environments available in GitHub Action, for instance usingruby-buildorruby-install.
Then this action could just download the result from that action's repository release downloads.cc @clupprich this might interest you too (as the author of https://github.com/clupprich/ruby-build-action).
I think a more convenient caching/building mechanism would be quite nice for actions, instead of a cache per repository.Reacted by Benoit TigeotYes, avoiding having consumers build it every time would be great. Hosted machines are a fresh VM every job.
Building ruby, making it globally available and then having the setup-ruby pull the binaries just in time would offer the ability to pin to any version flexibly.
Things to note:
- The actions/virtual-environments images effectively do this not as flexibly. I believe they only lay down a couple versions into the cache which is why I believe the defaults push a major version binding. They do it at factory disk creation (not users build time) which is an advantage but it can't possibly lay down all versions.
- If you build and publish ruby binaries to a globally available location it would be nice to have a json file with all the metadata so the action can do semver and match versions. Nodejs does this and the setup-node action uses it.
- self-hosted runners will benefit from job to job if downloaded into the tool cache by tool, version, platform
- The recommended pattern in the tool actions is to match the semver supplied against the cache dir first and then against the cloud. This adds resiliency for availability / load. e.g. nodejs was having download issues and it affected all workflows since the factory disk tool cache didn't have enough versions (12 was published).
- If you have a globally available set of bits, a CDN would be best. Nodejs didn't have a CDN as of the time 12 was released and the load caused issues.
That would mean you have to create, manage a CDN. That is unless you can lean on another packaging service like GPR etc. for a universal type package of bits. One other option would be to use releases in the ruby repo and attach binaries (see actions/runner where we do that with most recent release). The binary packages in a release are backed by CDN.
Hopefully, all that makes sense. Let me know how I can help.
Reacted by Benoit Tigeot and Denis TuzhikThanks for the response. I've assumed that binaries/builds would be stored in a repo's releases.
I've maintained a Ruby master MinGW for a few years for CI use. I've had it running as a cron job, three times a day. One question about using GH repo releases for storage.
The current actions/upload-release-asset isn't really clear about how to update an item to an existing release.
Will there be issues with a repo's Actions job trying to download the binary when the cron job is updating the release build binary? Maybe I should ask the question there...
I made a quick prototype (just a proof of concept at this point) to build Ruby versions for each GitHub virtual environment: https://github.com/eregon/ruby-install-builder/releases/tag/builds
Then I think downloading that and unpacking that for the same virtual environment should just work.
One potential issue is virtual environments/images do change over time, so that might break things if e.g., a new libssl version is used.
It's really easy for Linux and macOS, but probably quite a bit different for Windows, where I think we should likely use MSP-Greg's Windows Ruby builds.
I worked a bit more on this idea, and https://github.com/eregon/use-ruby-action now works with MRI, JRuby and TruffleRuby, with exact versions, on Ubuntu and macOS.
It just downloads and extracts an archive, so it's just a couple seconds to set it up.Would an approach like that make sense for actions/setup-ruby?
Reacted by Bryan MacFarlane, Peter Solnica, Samuel Williams, Patrik Ragnarsson, Jordan Phillips, Benoit Tigeot and Pablo Borlaf Mena@eregon - thanks, I'll have a look today.
Reacted by Samuel WilliamsI reviewed
use-ruby-actionandruby-install-builderand I think we could adopt them. They seem well put together, have good separation of concerns, and should be reliable and "scalable" (i.e. support different versions of Ruby) into the future.The only thing I'd suggest is perhaps supporting more Ruby versions (trivial) and perhaps having a suggested template for users to follow, not sure if this exists or not. Ideally, the default template should be super simple - I wish users don't need to specify Ruby versions, but instead could write something like: all supported releases which right now would be MRI 2.4 - 2.7, JRuby and TruffleRuby, rather than mucking around with environments. Because most gems should just use this by default.
Reacted by Benoit Daloze and Peter SolnicaSorry, my bad,I just saw https://github.com/eregon/use-ruby-action#usage which shows the usage - so my only point is perhaps just some way to make it easier for users to have a default build matrix which includes all current released versions of Ruby across all latest supported OS distributions.
The matrix needs to list all combinations explicitly for the workflow to understand it AFAIK, so I think it can't really be much better than:
https://github.com/eregon/ruby-install-builder/blob/f967c81fdf7097dc33fbf6ac5adeab551ae9edaa/.github/workflows/build.yml#L32-L34
Copying here for convenience:matrix: os: [ 'ubuntu-16.04', 'ubuntu-18.04', 'macos-latest' ] ruby: [ 'ruby-2.4.9', 'ruby-2.5.7', 'ruby-2.6.5', 'ruby-2.7.0', 'truffleruby-19.3.0', 'jruby-9.2.9.0' ] # This also works and is a fair bit shorter: ruby: [ '2.4.9', '2.5.7', '2.6.5', '2.7.0', 'truffleruby', 'jruby' ]
We could accept
ruby-2.4andruby-2.4.x(setup-rubysupports this) of course.What about adding -head variants?
What follows is just my quick opinion.
First, I think testing against MRI master is of little value, it regularly breaks and there is no CI before pushing. Also, features and changes onmasterare often reverted/tweaked/fixed, so it would cause a lot of failures for CI of gems.
I think JRuby doesn't really recommend testing againstmaster, the CI is not always green either.
For TruffleRuby I think it could be useful because all tests must pass before merging tomaster, but we would first need nightly builds of TruffleRuby (truffleruby/truffleruby#1483, building TruffleRuby entirely from source is more challenging than downloading an existing release and just recompiling the openssl extension).In practice it's probably not too hard since I think
ruby-installsupports buildingruby-head. Not sure aboutjruby-head, there is https://www.jruby.org/nightly but that doesn't give a single link.JFYI, re ruby master, ruby-loco is set up that only the most recent passing build is used when downloading. All test suites are run from the install folder, and I also check that bin files are working correctly, or at least reporting a version (rake, gem, bundler, etc).
Granted, API changes are a different matter...
36 remaining items
It would complicate things, but we could leave
actions/setup-rubyto specifically use only versions in the toolcache, and makeeregon/ruby-install-buildersomething likeactions/install-ruby. Just a thought, no strong opinion...@MSP-Greg I think that would cause confusion. My comment was about building Rubies and using them is a fairly different thing, which is why I split it in two repositories for my account. But it doesn't matter, we can use the same repository and two different branches or so.
It would be awesome if I could also point it at a specific ruby version, e.g
git@github.com:ioquatix/ruby#thread-selector. Is that going to be possible?It would be awesome if I could also point it at a specific ruby version, e.g
git@github.com:ioquatix/ruby#thread-selector. Is that going to be possible?That's a different feature (and it would be hard to make it efficient). The pre-built Rubies mentioned here are only for official releases.
Just like I think a builder for-headversions doesn't belong here.If you really want this, I think that repo should build its own releases and then maybe we could have a way to specify a release URL to download some Ruby from.
- added a commit that references this issue
on Jan 17, 2020 Re MSYS2, see actions/runner-images#30. I posted a sample script for installation of MSYS2. Installs it, updates it, installs a few compile tools, both 64 & 32. Installs to what I believe is considered the normal location,
C:/msys64.It doesn't have all the packages needed to build Ruby, but it has all the compile tools. I can certainly add more packages, I'm not sure what opinion is on that.
Having a more complete version in one location would be much better than the current out-of-date versions in four locations.
Off-topic, but Ruby compiles fine with VS2019. Obviously there's issues with things like fork, signals, UNIXSockets, some thread issues, etc. A few years ago, ruby/ruby ran almost no tests on Windows builds. Now, MinGW & mswin (msvc) are fully tested.
Thanks, Greg
- added 2 commits that reference this issue
on Jan 18, 2020 I updated the ADR with another option that would work across self hosted and other platforms and only cost more on first cache miss. It also aligns with the only officially supported distribution mechanism
Reacted by Jordan PhillipsFYI, eregon/use-ruby-action has moved to ruby/setup-ruby.
Which means that action (and the builder repos) are now part of the official Ruby organization on GitHub.Reacted by Patrik Ragnarsson, David Rodríguez, Benoit Tigeot, Ben Pickles, Lars Kanis, Pierre-Morgan Gate, YAMADA Tsuyoshi, Masatoshi Iwasaki, Marat Radchenko, Denis Tuzhik and 1 more@eregon - Awesome. I'll follow up soon
I'm ok to close...
@bryanmacfarlane
With #42 closed, I thought I'd continue here.
If one views this repo's purpose as accessing pre-built Rubies located in 'hostedtoolcache', then it is functioning as expected.
For quite a while, issues have been posted that imply that what's contained in 'hostedtoolcache' is not meeting users needs.
Given that, and given that GitHub employs several people active in the Ruby community, are there any plans for changing the current setup?
The Ruby community can certainly come up with solutions, but some at point, we/they will have some needs from Actions to make it function reasonably well. An example would be MSYS2 for Windows extension gem CI, and also building Ruby itself. Installing MSYS2 from scratch is time consuming.
If this isn't the palace to discuss this, please suggest where...