Decide behaviour when a zone goes offline but its MeshZoneAddress remains #17877

When a zone CP goes offline, its MeshZoneAddress stays on Global and keeps being synced to every other zone. Nothing withdraws it and, by default, nothing at the data plane ejects it either, so other zones keep dialing an ingress that may be gone. We need to decide what the intended behaviour is.

Current behaviour

MeshZoneAddress is the only source of publicly reachable coordinates for a remote zone. fillRemoteMeshServices (pkg/xds/topology/outbound.go:158-231) builds remote endpoints from it, gating only on three things: the MZA carries a kuma.io/zone label, the zone is not the local one, and mTLS is enabled. There is no liveness check — not on ZoneInsight, not on the zone's ingress Dataplanes.

The MZA itself survives a zone outage in both environments:

• Kubernetes: MeshZoneAddressReconciler deletes the MZA when the ingress Service loses Ready endpoints (pkg/plugins/runtime/k8s/controllers/meshzoneaddress_controller.go:70-78), but that delete lands in the zone store and needs KDS to reach Global. With the zone CP offline there is no stream, so Global keeps the stale MZA and keeps fanning it out.
• Universal: the MZA is operator-authored. Nothing creates it, nothing deletes it, nothing ties it to zone proxy liveness.

Two fallbacks that look like they should help, but do not:

• fillMeshMultiZoneServices skips remote services whose Spec.State != Available (pkg/xds/topology/outbound.go:142), but MeshService state is only recomputed by the owning zone (pkg/core/resources/apis/meshservice/status/updater.go:132). An offline zone stops updating, so its services stay frozen at Available.
• Outlier detection exists only through MeshCircuitBreaker (pkg/plugins/policies/meshcircuitbreaker/plugin/xds/configurer.go:85) and active probing only through MeshHealthCheck. Neither ships as a default policy, so Envoy treats the dead ingress as healthy indefinitely.

Impact

Locality awareness is on by default (pkg/plugins/policies/meshloadbalancingstrategy/plugin/v1alpha1/plugin.go:262) and priorityLocal=0 / priorityRemote=1 (pkg/xds/topology/outbound.go:681-682) map directly onto Envoy LocalityLbEndpoints.Priority (pkg/xds/envoy/endpoints/v3/endpoints.go:84-91). The blast radius therefore depends on how the destination is addressed.

The MZS case is the dangerous one precisely because it hides: the dead zone is only reached once local endpoints degrade, so failover walks into a black hole at exactly the moment it is supposed to help.

What exists today

Zone.enabled: false on Global already withdraws a zone's SyncedAcrossZones resources, MZA included. The predicate is at pkg/kds/context/context.go:327-336 and the behaviour is e2e-proven in test/e2e_env/multizone/zonedisable/zone_disable.go:176-196 — the disabled zone's MZA disappears from the other zone and traffic collapses to local only. It works, it just needs a human to notice and run it.

ZoneWatch already tracks per-zone liveness and emits ZoneWentOffline (pkg/kds/mux/zone_watch.go:112-119), but that event only tears down KDS streams (pkg/kds/mux/zone_sync.go:292, pkg/kds/service/server.go:179). No resource or propagation decision is derived from it.

Options to evaluate

1. Automatic withdrawal driven by zone liveness — extend the context.go:327-336 predicate to also skip zones whose ZoneInsight has been offline beyond a threshold, fed from ZoneWatch rather than an inline Get (the insight row is heavy and that predicate runs per resource per sync). Nothing is deleted, Global stays the source of truth, and reconnect restores propagation automatically. Needs to be opt-in and default off: a Global/zone partition is indistinguishable from a dead zone, so a false positive would withdraw a healthy zone.
2. Ship a default outlier detection posture for cross-zone clusters so Envoy ejects a dead ingress passively, independent of any control plane decision.
3. Make MeshService.State decay when the owning zone is offline, so the existing State != Available gate starts firing instead of being frozen.
4. Do nothing automatic and document Zone.enabled: false as the supported operator response, accepting the failover black hole.

Options 1 and 2 are complementary rather than exclusive — one fixes propagation, the other fixes the data path.

Out of scope

Deleting the stale rows on Global. That is a separate gap: deleting a Zone cascades nothing, and zone-originated resources are read-only on Global. Confirmed empirically that DELETE returns 405 for both ZoneIngress and MeshZoneAddress on a Global CP, because IsReadOnly is true for any type lacking ProvidedByGlobalFlag (pkg/core/resources/model/resource.go:347). Withdrawal fixes routing; purging rows is its own problem and deserves its own issue.

Supporting documentation

Context from the zone proxy GC investigation in #17296. Not caused by that work — MeshZoneAddress carries cross-zone routing and is a different resource from the zone proxy Dataplane that #17296 collects.