[Architecture Review] State store durability: restart / multi-instance / fencing / cross-store failure tests #5339

Part of #5296 (Architecture Review).

Q3 from the 2026-09-07 review questions: does each state-store contract have the intended durable implementation, and are restart, multi-instance, fencing, and cross-store failure scenarios covered by executable tests rather than documentation only?

Context

The unified state control plane landed in #5301 Sub-PR A / B / C / D:

What needs verification (Q3 specifics)

For each store:

1. Durable implementation is the one documented in the contract (not an InMemory* stub in production code).
2. Restart scenario — kill the runtime, restart, confirm state is recovered and behavior matches pre-crash.
3. Multi-instance scenario — two runtime instances sharing the same Meta / RocksDB; confirm the store handles concurrent writers (CAS retry, epoch fencing).
4. Fencing scenario — stale partition owner (pre-[Architecture Review][P0] PARTITION_OWNED_PULL delivery topology (reopens #5300 second mode) #5309) tries to write; confirm the write is rejected (not silently overwritten).
5. Cross-store failure — one store's backend (Meta) goes down; the runtime either fails fast or degrades gracefully, with the failure visible in metrics / logs.

Specific test scenarios needed (executable, not doc-only)

• OffsetStore restart: persist offset 100, kill runtime, restart, read offset = 100.
• SubscriptionStore multi-instance: two instances concurrently register / unregister the same subscription; CAS retry should converge.
• SessionStore fencing: instance A registers agent X, then is paused; instance B takes over; A's subsequent writes for X are rejected.
• DeadLetterStore restart: produce an event that fails 3 times, kill runtime, restart, confirm the event is in DLQ (not lost).
• TaskStore cross-store failure: Meta goes down mid-submit; the TaskStore surfaces the failure rather than silently dropping the task.
• DeliveryStateStore flush: sub-second flush path; confirm the state survives a kill -9.

Acceptance

• For each of the 6 stores above, a JUnit/Testcontainers test exists in eventmesh-runtime and passes in CI.
• The test for each scenario above (restart / multi-instance / fencing / cross-store failure) is referenced in this issue's evidence table.
• Any store still backed by an InMemory* implementation in production code is either replaced or marked as experimental with a deprecation comment.
• A state-store-failure-matrix.md (or section in docs/) documents the expected failure mode per backend (Meta, RocksDB).

Depends on / Blocks

• Depends on: [Architecture Review][P0] Unified state control-plane stores #5301 (DONE), [Architecture Review][P0] PARTITION_OWNED_PULL delivery topology (reopens #5300 second mode) #5309 (PARTITION_OWNED_PULL — for fencing)
• Blocks: [Architecture Review] Consolidate EventMesh boundaries and production-grade consistency #5296 closure (Q3 must be answered)