Skip to content

fix: drop docker/docker and the chrismellard ACR helper, bump buildkit to v0.31.2 - #38

Open
stirby wants to merge 10 commits into
mainfrom
stirby/drop-docker-docker
Open

stirby wants to merge 10 commits into
mainfrom
stirby/drop-docker-docker

Conversation

@stirby

@stirby stirby commented Oct 8, 2026 •

Copy link
Copy Markdown

Removes github.com/docker/docker and github.com/chrismellard/docker-credential-acr-env from the module graph and bumps buildkit to v0.31.2. These are the kaniko-side fixes for the remaining envbuilder advisories, consumed by coder/envbuilder#535.

Generated by Coder Agents on behalf of @stirby.

Advisories fixed

Advisory Module How
GO-2026-4858, GO-2026-4859, GO-2026-6255 moby/buildkit v0.16.0 → v0.31.2
GO-2026-4883, GO-2026-4887 docker/docker Module removed (see below)
GO-2026-6225 chrismellard/docker-credential-acr-env Helper ported in-tree with an anchored registry hostname check
GO-2026-6253 (in envbuilder) moby/go-archive v0.3.3

govulncheck ./... (reachable, non-stdlib): 46 on main, 27 on this branch, none introduced. The 27 remaining are in go-git, containerd, grpc, x/crypto and x/text. They are out of scope here, and envbuilder already overrides them with newer versions; its source scan is 0.

Changes

  • docker/docker removed:
    • pkg/archive → github.com/moby/go-archive (+ compression).
    • pkg/system xattr helpers → pkg/util/xattr_linux.go, with a non-Linux stub.
    • builder/dockerfile.BuildArgs → pkg/dockerfile/internal/buildargs, copied verbatim with its tests.
    • Healthcheck types → moby/docker-image-spec.
  • buildkit v0.31.2:
    • Dockerfile parsing now passes a linter, which fixes a nil-pointer panic on # check= directives.
    • ParseCommands rejects stage instructions.
    • Instruction flags that kaniko does not implement (COPY/ADD --exclude, COPY --parents, ADD --unpack, RUN --security=insecure, RUN --device) now fail with a clear "not supported by kaniko" error. Previously they failed at parse time as unknown flags, so builds that worked before still work.
  • ACR helper:
    • pkg/creds/acr ports chrismellard's helper, with attribution.
    • The registry hostname regex is now anchored: ^(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\.)+azurecr\.(?:io|cn|de|us)$. The original regex was unanchored, so a host like evil.azurecr.io.attacker.com received an AAD token exchanged from the environment's service principal.
    • cmd/docker-credential-acr-env builds the binary in deploy/Dockerfile.
  • go-containerregistry v0.19.1 → v0.21.7 (forced by buildkit). v0.21 limits concurrent blob pulls per image (4) and holds a slot until the reader is closed. Two hangs followed from that:
    • GetFSFromLayers deferred every layer's Close until return, so images with more than 4 layers deadlocked. Each layer is now extracted in a helper that closes its reader.
    • v0.21.6 tarball.Write never closes layer readers, which hung multi-stage builds (FROM <stage>) and COPY --from=<image>. v0.21.7 fixes it.
    • Both have regression tests against an in-memory registry. Each test fails (times out) without its fix.
  • Toolchain: go 1.26.9. Buildkit requires at least 1.25.9, and Go 1.25 reached end of life on Aug 19, so the module moves to the current release. Builder image golang:1.26, workflows go-version: 1.26. The certs stage moves to debian:bookworm-slim, because bullseye-security now returns 404 and breaks the image build on main too.
    • The unit test workflow runs sudo env "PATH=$PATH" make test. Plain sudo used the runner's older Go, which auto-downloaded go1.25.9 and then failed with no such tool "covdata" (reproduced locally).

Behavior changes for users

  • Minimum dependency versions rise (AWS/Azure SDKs, cobra, x/*), as do the Go requirements for anyone importing kaniko.
  • Heredoc parsing now follows buildkit v0.31. Kaniko still drops RUN heredoc bodies, as it did before.
  • go-containerregistry v0.21 hardening:
    • *.local registries no longer get an automatic plain-HTTP fallback. Only localhost, *.localhost and loopback IPs do. Use --insecure/--insecure-registry for an in-cluster HTTP registry.
    • Token realms, blob redirects and upload locations that point at a different private IP literal are rejected.
    • Retry backoff is now 1s/3s/9s, and 429 responses are retried.

Validation

  • hack/boilerplate.sh, hack/gofmt.sh, go vet (non-integration packages) and the full unit suite as root all pass.
  • deploy/Dockerfile builds. The resulting binaries contain no docker/docker or chrismellard modules. The ACR helper rejects a lookalike host.
  • End to end through envbuilder built from coder/envbuilder stirby/kaniko-moby (pins this branch):
    • Builds the coder/coder main devcontainer (64-layer base image) to INIT. The previous pin deadlocked at layer 5.
    • A multi-stage Dockerfile with FROM <stage> and COPY --from=golang:1.25-bookworm builds. The previous pin deadlocked in "Storing source image".
    • # check= Dockerfiles build. COPY --exclude fails with the new error, matching 1.3.0, which also fails.
    • Test template on dogfood: https://dogfood.cdr.dev/templates/coder/kirby-envbuilder-rollup
  • CI: all checks pass, including every integration group (layers, misc, run, k8s).
  • Integration fixtures: Dockerfile_test_issue_2049 moves from debian:bullseye-20220328 to debian:bookworm, and Dockerfile_test_issue_1039 from ubi7 to ubi8 with the el7 version pins dropped. Both old bases' package repos are gone, so the plain docker build reference step failed before kaniko ran; main fails the same way. ubi8 keeps the /lib -> usr/lib symlinks that 1039 covers.
Decision log
  • Port instead of upgrade: docker/docker and chrismellard were ported in-tree rather than upgraded. Neither module has a fixed version: the moby fixes exist only in github.com/moby/moby/v2 betas, and chrismellard is unmaintained. The ported code is small and copied verbatim except for the hostname check.
  • x/crypto not bumped: out of scope here. The Go 1.26 bump removes the earlier blocker (v0.56+ requires Go 1.26), so it can follow separately.
  • grpc not bumped: GO-2026-6443/6061/6348 are out of scope.
  • Unsupported flags rejected explicitly: this keeps v0.16 behavior (an error) instead of silently ignoring the flags, which buildkit v0.31's parser would otherwise allow.
  • .local change documented, not shimmed: it is an upstream security hardening, and restoring the plain-HTTP fallback would undo it.
  • Reviews: independent subagent reviews covered security/correctness, buildkit parser behavior, and go-containerregistry v0.19 → v0.21. All actionable findings are addressed above.
  • Not addressed, documented only: README credsStore "acr" naming, and no NOTICE file for the ported Apache-2.0 code (attribution is in the file headers).

stirby added 5 commits October 8, 2026 15:54
docker-credential-acr-env matches registry hosts with an unanchored
regular expression, so a host such as evil.azurecr.io.attacker.com is
treated as Azure Container Registry and receives an AAD refresh token
exchanged from the environment's service principal. Upstream has no fix.

Port the helper into pkg/creds with an anchored expression. It still
uses the module's token and registry packages for the exchange, so
behavior for real ACR and MCR hosts is unchanged.
Fixes GO-2026-4858, GO-2026-4859 and GO-2026-6255 (buildkit),
GO-2026-6253 (moby/go-archive) and removes docker/docker, which carries
GO-2026-4883 and GO-2026-4887 with no fixed release.

- pkg/archive: use github.com/moby/go-archive v0.3.3 and its compression
  package. Constants and detection are identical to docker v27.3.1.
- pkg/system xattr helpers: port Lgetxattr and Lsetxattr from docker
  v27.3.1 onto golang.org/x/sys/unix, with a !linux stub.
- builder/dockerfile BuildArgs: port unchanged, with its tests, into
  pkg/dockerfile/internal/buildargs. This also removes the docker daemon
  packages from the build.
- api/types/container.HealthConfig: buildkit now exposes the
  moby/docker-image-spec HealthcheckConfig type directly.

buildkit v0.31.2 raises the minimum versions of go-containerregistry
(v0.21.6), the AWS and Azure storage SDKs, cobra and golang.org/x/*.
- Parse with linter.New instead of a zero-value or nil linter. buildkit
  now applies "# check=" comments to the linter for each instruction,
  which panicked on Dockerfiles that parsed with v0.16.
- Reject COPY/ADD --exclude, COPY --parents, ADD --unpack,
  RUN --security=insecure and RUN --device. buildkit v0.16 failed to
  parse them; v0.31 accepts them, and kaniko does not implement them, so
  builds would otherwise silently differ (COPY --exclude would copy the
  excluded files).
The standalone docker-credential-acr-env in the kaniko images was still
installed from github.com/chrismellard/docker-credential-acr-env, so it
kept GO-2026-6225, and scanners flagged the module in the executor.

- Move the fixed helper to pkg/creds/acr and port the module's token and
  registry packages, dropping the module from go.mod.
- Tighten the hostname check to DNS labels followed by an ACR domain and
  accept a trailing dot.
- Build cmd/docker-credential-acr-env into the images from this repo.
- Bump golang-jwt/jwt/v4 to v4.5.2 (GO-2024-3250, GO-2025-3553), used by
  the Azure token exchange.
- Build images with golang:1.25 (go.mod requires 1.25.9) and pin CI to
  Go 1.25.
- Use debian:bookworm-slim for the certs stage; bullseye security
  packages now return 404 and fail the image build.
go-containerregistry v0.21 holds a pull limiter slot for every open remote
blob reader. GetFSFromLayers deferred each layer's Close until the function
returned, so images with more layers than the limit (4) deadlocked during
extraction.

Also bump go-containerregistry to v0.21.7, which closes layer readers in
tarball.Write. v0.21.6 hangs the same way when saving a stage or an
external COPY --from image with more than four layers as a tarball.
stirby added 2 commits October 8, 2026 19:35
sudo resets PATH, so root ran the runner's older Go, which downloaded the
go 1.25.9 toolchain from go.mod. That switched toolchain fails with
'no such tool "covdata"' for packages without tests under -cover.
debian:bullseye-20220328 apt sources and the ubi7 yum repos no longer
resolve, so the docker reference build fails before kaniko runs. Use
debian:bookworm and ubi8 (which keeps the /lib -> usr/lib symlinks issue
1039 covers), and drop the el7 package version pins.

@BobbyHo BobbyHo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One follow-up from #39, which I closed in favour of this PR.

The ported helper still asks Azure AD for a token with the Azure Resource Manager audience and hands that to the registry's exchange endpoint. The maintained fork, osscontainertools/docker-credential-acr v0.9.1, asks for a token scoped to https://containerregistry.azure.net instead and uses the same exchange call, so the registry accepts it. With the anchored host check the token now only reaches real ACR hosts, so this is no longer a leak. A registry-scoped token is still worth much less than a management-plane one if that check ever fails or an ACR endpoint is compromised, so it seems worth carrying over.

Two one-line changes below. The second is needed because the client-credentials branch uses clientCredentialsConfig.Resource, which also defaults to the ARM endpoint, so the resource argument is ignored on that path today.

Comment thread pkg/creds/acr/token.go Outdated
Comment thread pkg/creds/acr/token.go Outdated

@Lisa-Fiander Lisa-Fiander left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you confirm kaniko is only used as a library through envbuilder, and the kaniko executor image built from deploy/Dockerfile is never published or used?

Comment thread deploy/Dockerfile Outdated
@BobbyHo

BobbyHo commented Oct 9, 2026

Copy link
Copy Markdown

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

@coder-agents-review

coder-agents-review Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Chat: Review posted | View chat
Requested: 2026-10-09 15:04 UTC by @BobbyHo

Review history
  • R1 (2026-10-09), COMMENT. Review

deep-review v0.13.0 | Round 1 | b20ff58..a81f425

Last posted: Round 1, no findings, COMMENT. Review

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Dropped by orchestrator (requester limited review to P0-P2) pkg/creds/acr/token.go:46 ACR token selection and AAD exchange functions have 0% test coverage R1 Netero No
CRF-2 P4 Dropped by orchestrator (requester limited review to P0-P2) pkg/creds/acr/token.go:80 Missing AZURE_TENANT_ID reports "failed to get client id" R1 Netero No
CRF-3 P4 Dropped by orchestrator (requester limited review to P0-P2) pkg/creds/acr/helper.go:58 Add and Delete return "list is unimplemented" R1 Netero No
CRF-4 P4 Dropped by orchestrator (requester limited review to P0-P2) pkg/dockerfile/dockerfile_test.go:209 Test failure message hides command count R1 Netero No
CRF-5 Nit Dropped by orchestrator (requester limited review to P0-P2) pkg/creds/acr/registry.go:60 Doc comment names parseRegistryName instead of getRegistryURL R1 Netero No
CRF-6 Note Dropped by orchestrator (informational; requester limited review to P0-P2) pkg/commands/commands.go:120 checkUnsupportedFlags covers every flag buildkit v0.31.2 adds or un-gates R1 Netero No
CRF-7 Note Dropped by orchestrator (informational; requester limited review to P0-P2) pkg/util/fs_util_test.go:1454 Both pull-limit regression tests fail without their fixes (verified) R1 Netero No
CRF-8 OOS Dropped by orchestrator (unchanged line; requester limited review to P0-P2) README.md:682 credsStore "acr" does not match binary docker-credential-acr-env R1 Netero No
CRF-9 OOS Dropped by orchestrator (fails at base too; requester limited review to P0-P2) pkg/util/fs_util.go:688 pkg/util does not build for GOOS=windows R1 Netero No

Law analysis

  • Round: R1
  • Head SHA: a81f425
  • Effective LOC: +1484 -208 (32 files)
  • Verdict: Split into 5 vertical slices (A: ACR helper port with anchored host check; B: EOL base image repairs; C: ggcr v0.21.7 + layer-reader close fix + Go 1.25; D: docker/docker import removal; E: buildkit v0.31.2 parser changes). Order: A and B any time, C, then D, then E.
  • Enforcement: Mandatory. Trigger: the ACR concern carries credential risk and builds and passes tests alone on base b20ff58 (verified by Law); the PR also holds independent risk domains (registry client behavior, Dockerfile parser semantics).

Contested and acknowledged

Round log

Round 1

Netero + Law. Law: Split, Mandatory, so the panel did not run. Netero: 1 P3, 3 P4, 1 Nit, 2 Notes, 2 out-of-scope; all dropped from posting because the requester (IC_kwDOJPpmn88AAAABapuxow) limited the review to P0-P2. Panel planned for the next round (full panel: ging-go, kurapika, ryosuke, takumi, killua, melody, pariston, mafuuu, bisky, gon, leorio, mafu-san, wildcard chopper). Reviewed against b20ff58..a81f425.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR removes docker/docker and the chrismellard ACR helper and bumps buildkit, go-containerregistry and Go. The first pass found no P0, P1 or P2 issues; lower-severity items are not posted because the review was limited to P0 to P2. The full review panel has not run.

The panel review is blocked until the PR is split. The ACR hostname check (GO-2026-6225) decides which hosts receive tokens derived from the Azure service principal, and here it sits in a diff where most lines are buildkit, go-containerregistry and docker/docker migrations. Applying only the pkg/creds, cmd and tools/tools.go changes to b20ff58 builds and passes go test ./pkg/creds/... with buildkit v0.16.0 and Go 1.24.6, so it does not need the rest of the PR.

Proposed split, each PR with its own tests:

  1. ACR helper port with the anchored hostname check: pkg/creds/acr, cmd/docker-credential-acr-env, the go install line in deploy/Dockerfile, README. No dependencies.
  2. EOL base images: the debian:bookworm-slim certs stage and the issue 1039 and 2049 fixtures. No dependencies.
  3. go-containerregistry v0.21.7 with the GetFSFromLayers close fix and both pull-limit tests, plus the Go 1.25 toolchain (go.mod, builder image, workflows, sudo env "PATH=$PATH").
  4. docker/docker removal: go-archive, the xattr port, the buildargs copy, healthcheck types. After 3, because go-archive v0.3.3 requires Go 1.25.
  5. buildkit v0.31.2 parser changes: linter, ParseCommands, unsupported-flag rejection. After 3, because buildkit v0.31.2 requires go-containerregistry v0.21.6 or later, which hangs on images with more than 4 layers without the close fix.

coder/envbuilder#535 can pin the top of the stack. If the PR is instead merged without squashing, commits ac3ca5a, b36b6fe and 8e293b6 pin go-containerregistry v0.21.6 without the close fix, so git bisect across them hangs on images with more than 4 layers.

🤖 This review was automatically generated with Coder Agents.

BobbyHo and others added 3 commits October 9, 2026 09:14
- Go 1.25 reached end of life on Aug 19 and gets no further security fixes
- go.mod, builder image, and both workflows move to 1.26
- Importers such as envbuilder must build with Go 1.26.9 or newer

@BobbyHo BobbyHo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants