Observed while dogfooding (Ace Canary 0.0.16/0.0.17, two Macs on the tailnet): opening a channel shows "Loading chat" for 20–60 s, on the peer host and locally.
Evidence on natembp's Ace Helper (pid 64359):
- Gateway request durations rose steadily from ~250 ms (14:39Z) to ~6 s (14:50Z), starting right after the peer
nate16-1 connected (14:38:57Z), including local-only requests.
- The helper sat at 83–97% CPU;
sample shows the main thread busy in synchronous openat/read/close/access/kill/getdirentries — the catalog listing (local()).
- One authenticated loopback WebSocket, one
channels request: handshake 9.4 s, then 45 unsolicited identical channels frames (1 unique) in 0.6 s, ~1.97 MB (~75 frames/s). Loopback /health timed out at 3 s/8 s.
Cause: a peer's channels frame makes GatewayClient emit, peers.watch calls broadcast(), and broadcast() sends to every socket, including the inbound socket of that same peer's client. Both hosts echo unchanged listings to each other forever, re-reading the catalog for each frame. Every new trigger adds another circulating frame, so latency keeps growing until the helper restarts.
Fix: peer and directory changes go to the owner's app sockets only; only this host's own catalog changes go to peers. Local listings are computed once per broadcast.
Observed while dogfooding (Ace Canary 0.0.16/0.0.17, two Macs on the tailnet): opening a channel shows "Loading chat" for 20–60 s, on the peer host and locally.
Evidence on natembp's Ace Helper (pid 64359):
nate16-1connected (14:38:57Z), including local-only requests.sampleshows the main thread busy in synchronousopenat/read/close/access/kill/getdirentries— the catalog listing (local()).channelsrequest: handshake 9.4 s, then 45 unsolicited identicalchannelsframes (1 unique) in 0.6 s, ~1.97 MB (~75 frames/s). Loopback/healthtimed out at 3 s/8 s.Cause: a peer's
channelsframe makesGatewayClientemit,peers.watchcallsbroadcast(), andbroadcast()sends to every socket, including the inbound socket of that same peer's client. Both hosts echo unchanged listings to each other forever, re-reading the catalog for each frame. Every new trigger adds another circulating frame, so latency keeps growing until the helper restarts.Fix: peer and directory changes go to the owner's app sockets only; only this host's own catalog changes go to peers. Local listings are computed once per broadcast.