Repository navigation
Run unmodified rootfs images: guest injected as an OCI image volume, plus second-runtime layers - #70
Conversation
a76974d to
e9fdc59
Compare
Jaana Dogan (rakyll)
left a comment
There was a problem hiding this comment.
Left a comment. After it's addressed, LGTM.
| // "<base>-<first 12 hex digits of the digest>" in the base's atespace; it | ||
| // is created on first use and reused afterwards, and the response reports | ||
| // it. | ||
| string task_image = 4; |
There was a problem hiding this comment.
Want to call this guest_image? Task is a very overloaded term in AI systems at this point and mean unit execution boundary for Google systems.
There was a problem hiding this comment.
Renamed in 2a495b1. The field is now rootfs_image (field 4, same number). The CLI flags are --rootfs-image, the Python argument is rootfs_image, and the docs moved to docs/rootfs-images/.
I did not use guest_image because that name already means the injected ate-env-guest image: --guest-image, and the guest volume at /ate. The image here is the root filesystem of the environment, thus rootfs_image.
There was a problem hiding this comment.
I also updated the main readme with this cool feature, which I think is a great differentiator for Substrate (run another imagemounted as a read-only OCI image volume at /ate and started from there, so the rootfs image is never rebuilt)
|
Rebase checklist for after #77 (Substrate bump to
|
e9fdc59 to
6dbcd40
Compare
|
Rebased onto
|
"Task" is overloaded (review on #70): in AI systems it names a unit of work, and at Google an execution boundary. The image is the environment's root filesystem, which the design doc already calls it, so name it that on the wire and everywhere the name shows: - proto: CreateEnvironmentRequest.rootfs_image (field 4, same number as the unreleased task_image, so wire-compatible) - CLI: `ate-env create --rootfs-image`, `ate-env manifest template --rootfs-image`, ROOTFS_IMAGE in the runtime-layers example - Python: Client.create(rootfs_image=...); stubs regenerated - Go: RootfsImage / GetRootfsImage; docs and comments follow guest_image was suggested as well, but that name already means the injected ate-env-guest image (--guest-image, the `guest` volume) and would invert the vocabulary.
|
Live run of this branch on
|
Until now a template's container image had to be the ate-env-guest image itself, so running a task image meant rebaking it with the guest as the entrypoint, once per image. Substrate's image volumes allow the other split: the task image stays unmodified and the guest is mounted into it read-only and started from the mount. Two ways to get such a template: - ate-env manifest template --task-image <repo@sha256:...> writes a template whose container is the task image, with the guest image as a volume at /ate and the command re-rooted to /ate/ko-app/ate-env-guest. --workspace passes -workspace to the guest in either mode. - CreateEnvironment gains an image field. ate-env-api takes the request's template (default-template by default) as the base, derives "<base>-<12 hex digits of the digest>" from it on first use and reuses it for later environments on the same image; the response reports it. A derived name already taken by a template that runs something else is a FailedPrecondition, as is a missing base; an unpinned image, a derived name over 63 characters, or a base command that is not an absolute path to the guest is an InvalidArgument. Concurrent creates racing on one derived template both succeed. The derivation lives in internal/apiservice/imagetemplate.go and accepts either kind of base, guest-as-image or already injected, so a second runtime can reuse it. internal/ate gains Get/CreateActorTemplate and the fake control plane stores templates. The CLI gains create --image, the Python client create(image=), and the full-stack e2e suite a create-from-image test gated on ATE_ENV_TASK_IMAGE that checks the guest comes from the volume, the rootfs is the image's, and a second environment reuses the derived template. docs/task-images/README.md is the how-to (both flows, what an image needs, errors, cleanup) and docs/task-images/DESIGN.md the design (derivation rules, naming and idempotency, failure semantics, security, alternatives, future work). Stubs regenerated: Go with buf and protoc-gen-go v1.36.11, Python with grpcio-tools 1.61.3, keeping the committed license headers.
A task-image template carries the guest as a read-only image volume; an agent framework's own in-sandbox daemon, such as OpenSandbox's execd, is one more volume plus a process that has to start next to the guest. The guest is the container's only process, so it is the one that starts it. ate-env-guest gains -sidecar (repeatable: an absolute binary path and its arguments, run without a shell, output logged with the binary's name as prefix, restarted with a backoff that doubles from one second to thirty and resets after a minute of stable running) and -sidecar-readyz (repeatable: URLs that must answer 2xx once before /readyz reports ready, so Substrate's wakeup probe admits traffic only when every runtime is serving). Bad -sidecar values fail at start rather than loop. ate-env manifest template gains --layer name=image@sha256:...=/mount, --sidecar and --sidecar-readyz, which become image volumes, mounts and guest flags on the template. Template derivation recognizes the guest volume by name rather than as "an image volume", so a base with layers still gets the guest added or kept correctly and environments created on demand with --image inherit the second runtime. Only a command that starts with the guest binary is re-rooted under the mount. docs/task-images/RUNTIMES.md describes the model, the flags, who can reach the second runtime from where (processes inside the environment now; outside callers still only reach port 80 through the router, which is the next piece), suspend and restart behavior, and the limits. examples/runtime-layers is a runnable demo with a static busybox as the stand-in runtime: httpd on 44772 as the sidecar under both a python and a debian task image, the second one created from an image on the layered base. The first attempt with the dynamically linked busybox failed on the debian image with a glibc version error, which is why the docs make the self-contained-layer rule and its failure signature explicit.
…rived-template GC Spell out on CreateEnvironment how id, template and atespace default, that one atespace resolves the template and places the actor, how an image request derives and reuses its template, and which inputs yield InvalidArgument versus FailedPrecondition. Move derived-template garbage collection to the top of the design's future work and record why it is not tied to environment deletion: the template is shared by every environment on the image and holds the golden snapshot.
Review asked for a less ambiguous name now that runtime layers are also images on the template. The field is new in this branch, so the rename is free: proto field task_image (same number), Go GetTaskImage, CLI `ate-env create --task-image` to match `manifest template --task-image`, Python Client.create(task_image=), and the docs and examples. Generated Go and Python stubs regenerated; the Go file keeps main's descriptor path.
After the rebase onto #77 the fixtures and assertions in imagetemplate_test.go and server_test.go still named the old template fields. Use wakeupProbe / ContainerWakeupProbe and snapshotConfig / SnapshotConfig, and set preferredFidelity on the base fixture so the derived-template equality check also covers the field the server now requires.
"Task" is overloaded (review on #70): in AI systems it names a unit of work, and at Google an execution boundary. The image is the environment's root filesystem, which the design doc already calls it, so name it that on the wire and everywhere the name shows: - proto: CreateEnvironmentRequest.rootfs_image (field 4, same number as the unreleased task_image, so wire-compatible) - CLI: `ate-env create --rootfs-image`, `ate-env manifest template --rootfs-image`, ROOTFS_IMAGE in the runtime-layers example - Python: Client.create(rootfs_image=...); stubs regenerated - Go: RootfsImage / GetRootfsImage; docs and comments follow guest_image was suggested as well, but that name already means the injected ate-env-guest image (--guest-image, the `guest` volume) and would invert the vocabulary.
The client maps FAILED_PRECONDITION to FailedPreconditionError since #65, a sibling of RpcError rather than a subclass, so the negative test written before that change no longer matched the exception it got. Assert the typed error and the message; drop the unused grpc and RpcError imports.
7fc518c to
c4758ba
Compare
…dependabot - env.proto: three of the four TODO(jbd) notes describe work that is done (the data plane lives in guest.proto, the REST lifecycle endpoints are gone, docs were updated); only ListEnvironments remains. client.go: the two atespace TODOs are done, atespace is a parameter everywhere. - Python stubs regenerated for the proto comment change. The generator is grpcio-tools 1.84, which stamps GRPC_GENERATED_VERSION 1.84.0 and makes the import-time check require grpcio >= 1.84; the client's and the nemo-gym integration's floors move from 1.83 to 1.84 to match. - docs/release.md says rootfs-image templates, the name since #70. - .github/dependabot.yml: weekly grouped updates for the Go module, the two Python projects and the workflow actions. The Substrate module is excluded: its bumps change the generated API and go through the bump procedure, not an automated PR.
…dependabot - env.proto: three of the four TODO(jbd) notes describe work that is done (the data plane lives in guest.proto, the REST lifecycle endpoints are gone, docs were updated); only ListEnvironments remains. client.go: the two atespace TODOs are done, atespace is a parameter everywhere. - Python stubs regenerated for the proto comment change. The generator is grpcio-tools 1.84, which stamps GRPC_GENERATED_VERSION 1.84.0 and makes the import-time check require grpcio >= 1.84; the client's and the nemo-gym integration's floors move from 1.83 to 1.84 to match. - docs/release.md says rootfs-image templates, the name since #70. - .github/dependabot.yml: weekly grouped updates for the Go module, the two Python projects and the workflow actions. The Substrate module is excluded: its bumps change the generated API and go through the bump procedure, not an automated PR.
…dependabot - env.proto: three of the four TODO(jbd) notes describe work that is done (the data plane lives in guest.proto, the REST lifecycle endpoints are gone, docs were updated); only ListEnvironments remains. client.go: the two atespace TODOs are done, atespace is a parameter everywhere. - Python stubs regenerated for the proto comment change. The generator is grpcio-tools 1.84, which stamps GRPC_GENERATED_VERSION 1.84.0 and makes the import-time check require grpcio >= 1.84; the client's and the nemo-gym integration's floors move from 1.83 to 1.84 to match. - docs/release.md says rootfs-image templates, the name since #70. - .github/dependabot.yml: weekly grouped updates for the Go module, the two Python projects and the workflow actions. The Substrate module is excluded: its bumps change the generated API and go through the bump procedure, not an automated PR.
…dependabot - env.proto: three of the four TODO(jbd) notes describe work that is done (the data plane lives in guest.proto, the REST lifecycle endpoints are gone, docs were updated); only ListEnvironments remains. client.go: the two atespace TODOs are done, atespace is a parameter everywhere. - Python stubs regenerated for the proto comment change. The generator is grpcio-tools 1.84, which stamps GRPC_GENERATED_VERSION 1.84.0 and makes the import-time check require grpcio >= 1.84; the client's and the nemo-gym integration's floors move from 1.83 to 1.84 to match. - docs/release.md says rootfs-image templates, the name since #70. - .github/dependabot.yml: weekly grouped updates for the Go module, the two Python projects and the workflow actions. The Substrate module is excluded: its bumps change the generated API and go through the bump procedure, not an automated PR.
…dependabot - env.proto: three of the four TODO(jbd) notes describe work that is done (the data plane lives in guest.proto, the REST lifecycle endpoints are gone, docs were updated); only ListEnvironments remains. client.go: the two atespace TODOs are done, atespace is a parameter everywhere. - Python stubs regenerated for the proto comment change. The generator is grpcio-tools 1.84, which stamps GRPC_GENERATED_VERSION 1.84.0 and makes the import-time check require grpcio >= 1.84; the client's and the nemo-gym integration's floors move from 1.83 to 1.84 to match. - docs/release.md says rootfs-image templates, the name since #70. - .github/dependabot.yml: weekly grouped updates for the Go module, the two Python projects and the workflow actions. The Substrate module is excluded: its bumps change the generated API and go through the bump procedure, not an automated PR.
…rom images, run the tests in CI The provider has not imported since #65 (2026-10-06): it used OutputSource, ProcessStatus, Env.stream_outputs and Env.wait, all removed by the guest service redesign, and #69 merged it a day later without a rebase. Nothing caught it because the integration's tests are not in CI and the project does not resolve with uv (its ate-env-client dependency is unpublished). - provider: Env.start_process returns a Process; output is collected from Process.output(follow=True), whose final message carries the exit state, with Process.wait as the fallback; the client-side timeout kills through Process.kill. The guest-side deadline is the process's own `timeout` instead of a coreutils `timeout` wrapper, so task images need no `timeout` binary; zero and sub-second values floor to 1 s as before. A killed process now reports 137 rather than coreutils' 124. - templates: an image with no `image_templates` entry is sent as `rootfs_image` on the base template (#70); the API derives and reuses a template per image. The "no ActorTemplate mapped" error is gone; a tag instead of a digest is rejected by the API. - tests/fakes.py speaks the current proto (Process, ProcessOutput with an exit message, SignalProcess, ReadFileResponse/WriteFileRequest) and records the guest timeout and rootfs_image for the assertions. - pyproject: a [tool.uv.sources] path source for ate-env-client so `uv run --extra test pytest` works from a checkout; the `gym` extra is marked python_version >= 3.13, which nemo-gym 0.7.0 requires. - ci.yaml installs the integration and runs its tests next to the client's. - README: the derived-template flow first, the stale CRD-style template (snapshotsConfig, workerSelector) gone, the `timeout` binary requirement gone. 35 tests pass from a clean pip install and under uv; the live e2e is unchanged and skips without SUBSTRATE_E2E_API_URL.
…rom images, run the tests in CI (#82) * integrations/nemo-gym: port to the current client, derive templates from images, run the tests in CI The provider has not imported since #65 (2026-10-06): it used OutputSource, ProcessStatus, Env.stream_outputs and Env.wait, all removed by the guest service redesign, and #69 merged it a day later without a rebase. Nothing caught it because the integration's tests are not in CI and the project does not resolve with uv (its ate-env-client dependency is unpublished). - provider: Env.start_process returns a Process; output is collected from Process.output(follow=True), whose final message carries the exit state, with Process.wait as the fallback; the client-side timeout kills through Process.kill. The guest-side deadline is the process's own `timeout` instead of a coreutils `timeout` wrapper, so task images need no `timeout` binary; zero and sub-second values floor to 1 s as before. A killed process now reports 137 rather than coreutils' 124. - templates: an image with no `image_templates` entry is sent as `rootfs_image` on the base template (#70); the API derives and reuses a template per image. The "no ActorTemplate mapped" error is gone; a tag instead of a digest is rejected by the API. - tests/fakes.py speaks the current proto (Process, ProcessOutput with an exit message, SignalProcess, ReadFileResponse/WriteFileRequest) and records the guest timeout and rootfs_image for the assertions. - pyproject: a [tool.uv.sources] path source for ate-env-client so `uv run --extra test pytest` works from a checkout; the `gym` extra is marked python_version >= 3.13, which nemo-gym 0.7.0 requires. - ci.yaml installs the integration and runs its tests next to the client's. - README: the derived-template flow first, the stale CRD-style template (snapshotsConfig, workerSelector) gone, the `timeout` binary requirement gone. 35 tests pass from a clean pip install and under uv; the live e2e is unchanged and skips without SUBSTRATE_E2E_API_URL. * nemo-gym: name the guest deadline on stderr, exact nemo-gym marker - exec(): a process the guest kills at its deadline comes back as a plain SIGKILL (exit 137) with nothing on stderr. The provider now appends "exec timed out after Ns (killed by the guest)" when the exit code is 137 and the deadline has elapsed; the exit code stays 137. The 1 s floor on the guest deadline is a named constant; the guest takes any positive duration, the floor is the provider's choice. - The fake guest honours the recorded deadline, and a test covers the path. - gym extra: python_full_version >= '3.13.14', which is what nemo-gym 0.7.0 declares on PyPI; the 3.13 marker let 3.13.0-13 attempt an impossible install. - Two blank lines between tests restored (E302/E305). * nemo-gym: say why the fake sends exit_code 0 while running, document the gym extra's Python floor Answers the review on #82: Process.exit_code is a proto3 int32 with no unset value and state is the discriminator, so the fake keeps mirroring the wire format and now says so. The gym extra resolves only on Python >= 3.13.14 (what nemo-gym 0.7.0 on PyPI declares); the provider, tests and CI run on >= 3.10 without it, which the README install section now states.
What
Run any digest-pinned OCI image as an ate-env environment without rebuilding it. The rootfs image stays as published; the
ate-env-guestis mounted into it as a read-only OCI image volume at/ateand started from there. Two commits:1. Rootfs images (
3f943a8)CreateEnvironmentgains animagefield.ate-env-apitakes the request's template (defaultdefault-template) as the base, derives a template named<base>-<12 hex of the digest>next to it on first use, reuses it afterwards, and reports it in the response. Concurrent first creates on the same image both succeed.ate-env manifest template --rootfs-imagewrites such a template up front;--workspacepasses-workspaceto the guest.ate-env create --rootfs-image, Pythonclient.create(rootfs_image=).InvalidArgumentbefore anything is created; missing base or a derived name held by a template running another image areFailedPrecondition.internal/apiservice/imagetemplate.goand accepts either kind of base (guest as image, or already injected).2. A second runtime as a layer (
a76974d)ate-env-guestgains-sidecar(start and supervise an extra runtime from a layer: no shell, output logged with a prefix, restart with backoff 1s→30s) and-sidecar-readyz(URLs the guest's/readyz, Substrate's wakeup probe, waits for once).ate-env manifest templategains--layer name=image@sha256:...=/mount,--sidecar,--sidecar-readyz. Layers survive derivation, socreate --rootfs-imageon a layered base inherits the second runtime.This is step 1 of the runtime-injection plan: the mechanism every injected runtime needs. The intended first foreign runtime is OpenSandbox's
execd, which is built to be injected this way.Docs
docs/rootfs-images/README.md: how-to, both flows, what an image needs, errors, cleanup.docs/rootfs-images/RUNTIMES.md: second runtime as a layer, OpenSandbox specifics, reachability, limits.docs/rootfs-images/DESIGN.md: derivation rules, naming and idempotency, failure semantics, security, alternatives, future work.examples/runtime-layers: runnable demo with a static busybox as the stand-in runtime.README.mdquickstart: registering a rootfs-image template, creating from an image on demand, and the layered template with its sidecar, created from and queried inside the environment.Testing
go test ./...: 12 packages. Covers derivation from both base kinds (guest image, injected), naming limits, every rejection, server create / reuse / race / failure paths against the fake control plane, manifest layers and sidecars, and the guest's sidecar supervisor (restart with backoff, stop on shutdown, readiness gate once per URL). 19 new Go tests.clients/pythonunit tests: 73 pass (6 new).tests/e2e/test_full_stack.pygains two negative tests (unpinned image, missing base template) and a create-from-image test gated onATE_ENV_ROOTFS_IMAGEthat checks the guest comes from the volume and not the image, the rootfs is the image's, files round-trip, and a second environment reuses the derived template.glottman-sandbox-test-1(Substratec2052e45, v0.4.0-83, 10 gVisor workers on c4-standard-8) with the images built from this branch, the ones now in the README. Details in the results comment.ate-env-apirolled to the branch image; base templaterootfs-baseregistered fromate-env manifest template, golden ready after ~20 s.ate-env create --rootfs-imageof an unmodifiedpython:3.12-slim: derived template present at once, guest serving after 5 s on first use,python3 --versionanswered from the image's own rootfs with the guest under/ate; suspend, resume on first traffic, delete all OK; a second create on the same image served after 4 s from the derived golden.examples/runtime-layers/demo.sh: layered template, environment serving in 3 s with the sidecar's readiness gating the actor, the second runtime answered from inside, then anode:22-slimenvironment created on the same base inherited the layer through its derived template; exit 0, 24 s total.tests/e2e/test_full_stack.pyagainst the same stack: 5 passed.busybox:1.37failed on a debian image with a glibc version error, exactly as the supervisor and fail-fast readiness are meant to surface it. The docs state the self-contained-layer rule and that signature.hack/verify/boilerplate.shpasses. Go stubs viabuf generate(protoc-gen-go v1.36.11); Python stubs via grpcio-tools 1.83 (protoc 7.35.1), with the license header prepended by hand becausegen-protos.shdoes not add it.Known gaps, by design
ate-env-apiproxies only the guest. An endpoint lookup that names a runtime's port is the next piece and belongs with the OpenSandbox backend.CreateEnvironmentresolves the template in the environment's atespace and ignorestemplate.atespace.Relation to other PRs
mainafter Bump Substrate to f03327bd: follow the current API and stop sending ate-target-actor twice #77 (Substratef03327bd): the template builder emitswakeupProbe,snapshotConfig.preferredFidelity, and WorkerPoolsandboxClasses; derived templates inherit all of it. Redesign guest ProcessService and FileSystemService #65 and integrations/nemo-gym: NeMo Gym sandbox provider built on the Python client #69 are inmainand accounted for.rootfs_imageafter review (image→task_image→rootfs_image; same field number throughout).guest_imagewas not used because it already names the injected guest.cmd/ate-env/manifest.go; see the note there about the upstream snapshot fields.image_templatesmap.