Windows ASan: electric-fence guard pages + dealloc interception

[kkuehl]

Problem

libafl_frida's AsanRuntime did not detect heap memory errors on Windows for two compounding reasons:

1. Byte-granular shadow is not available on Windows. The runtime's normal approach relies on a fixed 16 TiB shadow mapping plus per-access shadow checks; on Windows that mapping cannot be committed, so the instrumented memory accesses have nothing to validate against.

2. The Windows heap entry points are forwarders / CFG stubs, and the Stalker bypasses Interceptor. RtlAllocateHeap is a forwarder to ntdll!RtlAllocateHeap; RtlFreeHeap is a 5-byte forwarder stub whose target is a CFG push [rip+...]; jmp [rip+...] dispatch stub. Under the Stalker, a recompiled call [__imp_HeapAlloc] / call [__imp_HeapFree] gets re-resolved back to the original export, so Interceptor::replace on those exports is never reached. Allocations/deallocations therefore bypass the ASan hooks entirely.

Changes

Electric-fence allocator (Windows only)

On Windows the allocator now places each allocation so its data ends on a page boundary with an immediately-following PAGE_NOACCESS guard page (reused blocks have that guard flipped back to PAGE_READWRITE). The first byte of a heap overflow or a write-after-free therefore faults in hardware, replacing the byte-granular shadow for the object's own bounds.

Do not emit the shadow check on Windows

With the guard page doing overrun/use-after-free detection, the byte-granular shadow check is redundant and actively harmful: on a fault it re-enters handle_trap, which blocks on the allocator lock and hangs resolving symbols. Emission is now gated to non-Windows targets.

Raw detour on the resolved RtlAllocateHeap

Because the Stalker re-routes call [__imp_HeapAlloc] to the original export (bypassing the Interceptor trampoline), the runtime patches a plain absolute jump (mov rax, imm64; jmp rax) over RtlAllocateHeap's real entry, resolving any forwarder stub to its real target first. This guarantees every allocation lands on the fence.

Dealloc interception in the Stalker transformer

RtlFreeHeap cannot be patched (forwarder -> CFG dispatch stub), so the free path is fixed in the transformer instead: when it emits call [rip+disp] targeting the target image's __imp_HeapFree IAT slot, it emits a direct call to the runtime's free replacement. HeapFree then reaches Allocator::release, which flags double-free (metadata already released) and makes the freed page inaccessible for use-after-free.

Avoid the dbghelp deadlock in report_error

Error reporting captured a resolved Backtrace and printed at full verbosity; resolving symbols through dbghelp while holding the allocator lock deadlocks. It now captures Backtrace::new_unresolved() and prints at Verbosity::Minimal.

Locking in handle_trap

handle_trap now try_locks the allocator and clones out the metadata it needs, rather than blocking, since the interrupted in-process executor may still hold the allocator lock; this avoids a deadlock and a poisoned mutex.

Windows import enumeration

enumerate_imports now yields each import's own IAT slot (it could previously reuse one slot), so the HeapAlloc/HeapFree IAT rewrites address the correct slots.

Shutdown observer

A FridaHelperShutdownObserver deinitializes the runtimes on engine shutdown.

[kkuehl]

Dependency note: this change calls Module::enumerate_imports, which is not yet in a released crates.io frida-gum. It is being added upstream in frida/frida-rust#249 (the fix lives on the fork branch fix/windows-module-imports, which also carries the Windows import-slot reconstruction this relies on). Building this PR currently requires pointing frida-gum at that branch until #249 lands.

[s1341]

All in all it looks good. My only worry is that by getting rid of the "byte-granular" shadow, and relying only on guard pages, we will miss some errors. I don't know why you say that it doesn't work. In my initial testing it did, so far as I recall.