[aleroot]

After getting curious about this #4689 I decided to setup the seam and see exactly what happens.

The main problem is: the GPU scheduler doesn't know the spin is waiting for something. I observed a waiting encoder fail to start until a later, running encoder finished. This happens in two cases:

• it is waiting on an MTLFence on the same queue;
• it is any new encoder, while another queue has a kernel running.

When that running encoder is the spin, nothing can finish. PR #4689 only stops the spin from getting ahead of work on its own queue. What's needed is: a spin may start only after all GPU work encoded before it, on every queue, has finished. Within one thread, work is encoded in graph order. So anything the CPU (or another stream) must finish before the spin can exit was encoded before the spin.

The patch attached works because of a Metal behaviour I tested first (sig_order.mm): a command-buffer-level encodeSignalEvent fires only after all earlier command buffers on that queue have finished, including untracked ones still running. A second queue waiting on it started at about 28 ms, when the long kernel ended.

Limits

• Other threads' streams aren't covered. The loop only sees streams in the current thread's map. Thread-unsafe global streams are skipped on purpose, because touching them from here could race.
• A later-work stall remains. A running spin can still hold back later work on other queues until it exits. By the graph-order reasoning above, that should only slow things down, not deadlock, but I haven't measured it.

---

• ☑️ I understand it is strictly prohibited to use AI to write PR description
• AI usage disclosure: use my own handcrafted app, Lampo in coding agent mode with Qwen 3.8 27B to prepare test environment and the seam.

[nastya236]

A later-work stall remains. A running spin can still hold back later work on other queues until it exits.

Not sure that i understand this sentence. Could you clarify what is other queues? Other cpu queues?

[aleroot]

Im am sorry I did not explain myself correctly. What I meant was a Metal MTLCommandQueue, so basically other queues really means other GPU streams.

In a nutshell: GPU stream A spins, then GPU stream B next kernel waits for it...

To clarify this cannot cause a deadlock, at least I could not reproduce in my seam, it is just a delay in this specific case.

[aleroot]

I have created a small reproduction example for you:

import time
import mlx.core as mx

D = 4096
W = [mx.random.normal((D, D)) / D**0.5 for _ in range(8)]
x = mx.ones((512, D))
mx.eval(W, x)

s2 = mx.new_stream(mx.gpu)

for n in range(20):
    h = x
    for w in W:
        h = mx.tanh(h @ w)
    out = mx.add(h, 1, stream=s2)
    tic = time.perf_counter()
    mx.eval(out)
    print(f"step {n}: {(time.perf_counter() - tic) * 1e3:.0f} ms", flush=True)

run this script with: MLX_METAL_FAST_SYNCH=1 python repro.py

I have run this on an M4 Pro base model 24GB of RAM.

Notes

uncapped, the stuck step hangs forever, and killing the hung process left my Mac unstable once...