Every rules_img strategy that reads blobs out of Bazel's remote cache — lazy push, CAS registry push, BES push and compact-layer reconstruction — talks to the cache over the Remote Execution API (REAPI) and its ByteStream service. This page describes what happens when those requests fail or stall, and the environment variables that tune it.
Transient failures are retried, the way Bazel's own remote cache client retries
them. Blobs are content-addressed and every read is idempotent, so retrying is
always safe: the digest validates whatever comes back. A read that the server
tears down mid-transfer is resumed at the offset it reached, so no bytes are
transferred twice, and an interrupted upload continues from the committed_size
the server reports.
UNAVAILABLE, INTERNAL, ABORTED, UNKNOWN, DEADLINE_EXCEEDED,
RESOURCE_EXHAUSTED and a CANCELLED that is not our own cancellation are
retried with exponential backoff and jitter. NOT_FOUND is a cache miss, not a
failure: it falls through to the next blob source (a registry, the disk cache)
instead. Everything else, and anything that is not a gRPC status at all, is
permanent.
When the server sends a RetryInfo error detail, the delay it asks
for is used instead of the computed backoff (bounded by
IMG_REAPI_RETRY_MAX_DELAY), as the REAPI specification asks clients to do.
The attempt budget counts consecutive failures: a transfer that is making forward progress gets its full budget back, so a large blob on a lossy link is not killed by an attempt counter.
| Variable | Effect |
|---|---|
IMG_REAPI_RETRY_MAX_ATTEMPTS |
Attempts per operation, including the first (default 6). 1 disables retrying |
IMG_REAPI_RETRY_BASE_DELAY |
Backoff before the first retry, doubling from there (default 250ms) |
IMG_REAPI_RETRY_MAX_DELAY |
Cap on a single wait, including one the server asked for (default 5s) |
These are the remote-cache counterpart of the IMG_REGISTRY_RETRY_* variables,
which govern container registry traffic.
| Variable | Effect |
|---|---|
IMG_REAPI_RPC_TIMEOUT |
Deadline for one attempt of a unary call — FindMissingBlobs, BatchReadBlobs, BatchUpdateBlobs, GetCapabilities. Like Bazel's --remote_timeout (default 60s, 0 disables) |
IMG_REAPI_IDLE_TIMEOUT |
How long a blob download may receive no data before it is torn down and resumed. Like Bazel's --remote_grpc_download_idle_timeout (default 60s, 0 disables) |
Bulk transfers get an inactivity limit rather than a deadline on the whole call, because a deadline cannot tell a slow-but-progressing transfer from a hung one. A download that stops producing data is resumed from the offset it reached instead of blocking the deploy forever.
Uploads are deliberately left without an idle limit: unlike a read, a write cannot be resumed for free.
Every connection the img tool opens to the remote cache accepts gRPC messages up
to 100 MiB, above grpc-go's own 4 MiB default. Without that, an ordinary blob read
can fail:
rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4194309 vs. 4194304)
The remote execution API places no bound on the size of a ReadResponse, and
GetCapabilities exposes no field from which a client could derive one, so a server
is free to divide a blob into whatever chunks it likes, and is within spec when it
does. Bazel raises the same limit to Integer.MAX_VALUE; 100 MiB is the figure
bazelbuild/remote-apis-sdks uses.
Treat 100 MiB as a compromise rather than a safety bound. It applies per call, and
img deploy pools one connection per job, so reads in flight together can hold a
multiple of it.
| Variable | Effect |
|---|---|
IMG_REAPI_MAX_RECV_MSG_SIZE |
Largest gRPC message the tool will accept, in bytes (default 104857600, 100 MiB). Values below grpc-go's own 4 MiB default are ignored with a warning |
The limit covers every call on those connections, including to the
local blob cache endpoint, which is not a
remote execution API service despite the variable's IMG_REAPI_ prefix.
Blobs small enough to batch are read and written with BatchReadBlobs and
BatchUpdateBlobs instead of being streamed. max_batch_total_size_bytes from
GetCapabilities says how large a batch the server accepts.
That figure bounds the blob payload rather than the serialized message: the specification calls it the "maximum total size of blobs" and names the message size limit of the transport as a separate constraint. A request for exactly the declared limit is therefore within the server's stated budget, yet once per-blob digest, status and length-delimiter fields are added the message is over it. 4 MiB is the default limit for what a gRPC server will accept, so the server refuses the request.
img holds back 1 KiB of the budget for framing, subtracted from whichever limit is
lower: what the server declares, or the 4 MiB it assumes of the transport. A server
declaring less than 4 MiB gets the same treatment, because it may be enforcing a
lower transport limit to match.
This matters on the upload path, for BatchUpdateBlobs and for the chunks a streamed
upload sends: the client's own receive limit does nothing about a message the server
will not accept.
img deploy does not ask the server for its capabilities and uses a fixed 2 MiB
budget regardless. The figures above apply to the BES and CAS registry paths.
Reconstructing a compact layer means reading every file the layer's index references out of the cache. A layer of many small files is thousands of such references — often tens of kilobytes each — and reading them one at a time costs one round trip per file, which on a high-latency link dominates everything else.
So the reference table is read as a list rather than a blob at a time.
Consecutive references small enough to batch are fetched with a single
BatchReadBlobs request, filled up to the payload budget above, so the cost is a
round trip per few megabytes instead of per file. References too large for a batch
keep using ByteStream, one at a time. Blobs are fetched as the layer is written out,
so a reconstruction never holds more than one batch in memory however long the list
is.
The 1 KiB of framing headroom above is a fixed reservation, and a read batch holding many blobs spends far more than that: every blob in the response carries its own digest, status and length delimiters. A batched read therefore charges each blob its framing as well as its bytes against the budget, and caps the number of blobs in one request, so the message stays inside the limit however small the blobs are.
References the deploy can satisfy without the network — from the layer's input directory (shipped by the eager strategies) or from the disk cache — are still read straight from disk, and only the gaps between them go to the remote cache. Whatever the batch fetches is written into the disk cache on the way through, so a later deploy finds it there.
Batching alone still leaves the link idle while a layer is written out: fetch a batch, write it, fetch the next. So the batches after the one being delivered are already in flight, spread across the connection pool, and the round trip for the next one is paid while the last is still being read.
Reading ahead costs memory, and the ceiling is shared by every reader in the process rather than applied per reader: a deploy reconstructs as many layers at once as it has jobs, and a per-reader limit would multiply by the job count on exactly the machines where that hurts. A reader holding nothing is always allowed one batch, whatever the ceiling says, so a busy deploy slows a reader's read-ahead down but never stops it making progress.
| Variable | Effect |
|---|---|
IMG_REAPI_PREFETCH_BYTES |
Prefetched blob data held across the whole process, in bytes (default 67108864, 64 MiB). 0 turns read-ahead off, leaving one batch in flight at a time |
A blob too large to batch is streamed rather than read ahead: a gRPC stream nobody is reading would sit idle until the idle timeout tore it down.
A single gRPC connection multiplexes every request onto one TCP connection, which
caps bulk-download throughput on high-latency links. img deploy therefore opens
a pool of connections and spreads reads across them, mirroring Bazel's
--remote_max_connections.
| Variable | Effect |
|---|---|
IMG_REAPI_MAX_CONNECTIONS |
Connections to the remote cache (default: the deploy's --jobs, i.e. the host CPU count) |
Retries make use of the pool: an attempt that failed on one connection is retried on the next, so a connection the server poisoned is not immediately reused.
Connections ping the server after 5 minutes with a stream in flight and no data arriving, and are closed if the ping goes unacked for 20 seconds — so a peer that disappeared mid-transfer is noticed instead of leaving a read blocked forever. The read is then resumed from the offset it reached.
5 minutes is not a tuning choice: it is the minimum gRPC servers enforce by
default (keepalive.EnforcementPolicy.MinTime). A client that pings more often
collects strikes and, after three, is sent
GOAWAY(ENHANCE_YOUR_CALM, "too_many_pings"), which drops the whole connection
and fails every stream on it. If you see that in a server log alongside a burst of
UNAVAILABLE, an intermediary — not rules_img — is pinging too eagerly.
The 20-second timeout matches Bazel's default. On Linux it is also applied as
TCP_USER_TIMEOUT, so it bounds how long any unacknowledged write may go before
the kernel drops the connection — not just an unacked keepalive ping. That makes a
short value costly on a congested link, where it would tear down connections that
are merely slow.
ByteStream.Write is resumable by design: after a failure the client asks
QueryWriteStatus how much the server committed and continues from there. How far
that gets depends on whether the bytes can be reproduced.
- A source that can seek — a file, an in-memory buffer — is rewound to the server's committed offset. These uploads resume from anywhere.
- A source that cannot (an HTTP request body being relayed into the cache by the CAS registry) is resumable only as far as the chunk still held in memory, which covers a stream the server tore down while it was being sent. A failure that leaves the server behind older, already-discarded data is reported to the caller, who still has the source and can start over.
A server that does not implement QueryWriteStatus (it may answer
UNIMPLEMENTED) makes the upload start over from offset 0 under a fresh upload
id, so the server never sees two different prefixes written under one resource
name.
Retries are printed on stderr as they happen, up to 20 per process:
WARNING: remote cache: reading blob 4f8a…9c1 (1258291 bytes) from the remote cache
failed (rpc error: code = Unavailable desc = unavailable); retrying in 268ms (attempt 2/6)
An operation that exhausts its budget says so, which distinguishes a give-up from a first-attempt failure:
giving up reading blob 4f8a…9c1 (1258291 bytes) from the remote cache after 6 attempts over 7.9s: …
img deploy summarizes the retries alongside its blob statistics:
blob transfers: 3 from disk, 0 from disk cache, 0 from container registry, 5 from remote cache, 0 from compact stream
remote cache blobs: 2 from local cache (48.1 MiB), 3 fetched (210.4 MiB), 1 deduplicated, 0 evicted
remote cache requests: 12 retried (Internal 1, Unavailable 11)
- Push Strategies — the strategies that use the remote cache, and the local blob cache that fronts it
- Credential Helpers — authenticating remote cache
traffic, and scoping a helper to the cache with
credential_helper_remote_cache