perf(relay): wake operator-listener delivery on commit

[baxen]

Why

Operator-listener deliveries (the push path kgoose uses to wake remote agents) reached the listener 0.25–2s after the mention committed. The delivery worker only found new outbox rows on its idle poll, which backs off from 250ms to 2s. A claimed batch also had to finish completely before the worker claimed again, so one slow endpoint (up to the 5s timeout) held up every mention queued behind it.

What changed

• Wake on commit. enqueue_mentions_in_transaction runs pg_notify when it adds outbox rows. Postgres sends the notification only when the event transaction commits and drops it on rollback, so workers wake only for rows they can already claim. Each delivery worker LISTENs on a dedicated one-connection pool, so it never takes a serving-pool slot. Every pod with a worker gets the wake, and the existing SKIP LOCKED claim decides which pod delivers.
• Idle poll stays as the fallback. It still finds rows whose notification was lost (for example while the listener was reconnecting) and scheduled retries.
• No batch head-of-line blocking. The worker keeps up to 10 deliveries in flight in a JoinSet and claims again whenever a slot frees, rather than join_all on a batch of 10.

Measured

Local relay. Each latency runs from POST /events accepted to the listener receiving the delivery, with 15 mentions spaced 2.5–4s apart so the worker sits at its idle ceiling:

	median	p90	max
main	871 ms	1773 ms	1947 ms
this branch	1.9 ms	2.7 ms	3.9 ms

I also killed the LISTEN backend with pg_terminate_backend mid-run. sqlx reconnected on a new backend right away, and the next mentions were still about 2ms.

Tests

• buzz-db Postgres tests: a committed mention wakes a listener, and a rolled-back one does not.
• buzz-relay Postgres test: a committed mention wakes the real run_wake_listener.
• buzz-relay unit tests drive the production run_delivery_loop:
  • a wake delivers before any idle poll fires
  • a stalled delivery does not hold up a later row
  • in-flight deliveries stay at 10
• I checked that each new test fails under the matching mutation: no wake branch, batch semantics, no in-flight cap, no pg_notify, no notify_one.

Tradeoffs worth a look

• A transaction that issues NOTIFY takes a database-wide lock at commit. Only event inserts that actually enqueue listener rows pay that, which today means mentions of registered agents.
• LISTEN needs a direct session. bbpci points DATABASE_URL straight at the Aurora writer, so that works. A transaction-pooling proxy in front would silently fall back to the poll.
• A half-open LISTEN socket that never errors would also silently fall back to the poll. sqlx has no TCP keepalive here. Possible follow-up: a timeout with a heartbeat, plus a metric.

🤖 Generated with Claude Code