Busy Waiting

For a loop guarding more than one word: the monitor arms a single line, so a store to another word goes unseen and the cap bounds how late it is noticed. Selects the timed instruction - WFET, WRS.STO. A spin waiter ignores the tag.

For a loop guarding a single word: the monitor covers every wake source, so a timeout would only wake the core to learn nothing and waste power. Selects the untimed instruction - WFE, WRS.NTO.

Works everywhere, cheap nowhere.

A spin waiter ignores the watched word - the caller's loop already re-checks its own condition, so this only needs to emit one backoff hint per turn. The monitored waiters - arm64_wfet_t, x86_tpause_t, and riscv64_wrs_t - use the word to sleep the core until that line changes.

Every waiter takes the atomic being watched, the value last seen, and the calling thread's index. A spin functor ignores the first two; a monitored functor arms a hardware address monitor on the watched line and sleeps the core until it changes. The thread index is carried for a future adaptive backoff, and dropped by every waiter today.

Every policy takes the address of the line being handed away, claimed or read next, plus a tag choosing the direction. A policy for an ISA with no matching instruction implements the overload as a no-op, so callers never branch - the emptiness compiles away.

The C++ standard would recommend using std::hardware_destructive_interference_size alignment, as well as std::atomic_flag::notify_one and std::this_thread::yield APIs, but our solution is better despite being more primitive.

A std::atomic<bool> is used rather than std::atomic_flag so the flag has an address a monitored waiter can arm: the lock loop is test-and-test-and-set, spinning on a plain load, so a monitored micro_yield sleeps the core until unlock's store to that line wakes it. On every platform we target std::atomic<bool> is lock-free, so the atomic_flag guarantee buys nothing. Compatible with STL unique locks.

UMONITOR arms an address-range monitor on the watched line; UMWAIT drops the logical processor into an optimized sleep until any agent's store to that line, an interrupt, or the TSC deadline. The store pushes the waiter awake as an event, so a fork-join handoff is observed rather than polled, and on an SMT core the sibling reclaims the freed pipeline slots for the duration.

There are several older ways to wait on x86, but they may require different privileges:

• MONITOR and MWAIT in SSE - used for power management, require ring 0 privilege.
• MWAITX in MONITORX ISA on AMD - used for power management, requires ring 0 privilege.
• TPAUSE in WAITPKG - a blind timed pause that watches no address, so a store never wakes it early. This waiter replaces it: UMWAIT is strictly better given a word to watch.

The UMWAIT control selects the sleep depth: bit 0 = 1 picks C0.1, shallow and fast-waking; bit 0 = 0 picks C0.2, deeper and slower. A fork-join barrier resolves in tens of nanoseconds, so we ask for C0.1. Neither state lowers voltage: like ARM's WFE, this clock-gates and saves dynamic power only, and never releases the core to the scheduler.

The policy differs only in what each wait-bound tag selects. Capped two-word waits - lock and capacity gates whose wakes are rare - sleep in the deeper C0.2 state with a quarter-microsecond deadline, trading ~100 ns of extra wake latency for pipeline resources the sibling hyper-thread reclaims meanwhile. Uncapped single-word waits - serialized publication convoys where every nanosecond of wake latency lands on the critical chain - keep the shallow C0.1 state.

Motivated by a 128-thread convoy workload where the shallow-everywhere policy of x86_tpause_t measured 13-16% behind plain std::this_thread::yield at full occupancy, while leading at the 7/8-occupancy operating point - the saturated sibling is the tool for the former regime.

Grace, Graviton 3 and 4 and the Neoverse N-series lack WFET, so this is their only way to sleep on a word. The store that clears the monitor also sets the event register, so a store landing between the re-check and the WFE ends the wait at once instead of being lost. Capped loops keep YIELD, since the kernel's event stream bounds a WFE only to about 100 micro-seconds, far coarser than the cap.

LDAXR arms this core's exclusive monitor on the watched word; WFET then places the core into light sleep until any core's store to that line clears the monitor as an event, or the timeout expires. The store pushes the waiter awake through the cache-coherence transaction, rather than the waiter polling for it - on an Apple M5 this wakes in ~42ns, faster than a bare YIELD spin, while the core is clock-gated. The timeout is the belt: a fork-join loop can watch two independent lines while the monitor arms only one, and the cap bounds how late a change to the other line is noticed.

WFET clock-gates the core to save dynamic power. It does not lower voltage, so it saves no static leakage power, and it does not release the core to the OS scheduler: the thread stays runnable and no sibling can be placed on it. A real retention or power-down C-state needs firmware - PSCI CPU_SUSPEND via SMC at EL3. This is a power hint for a busy core.

LR.W / LR.D arms a reservation on the watched word - the same reservation an LR and SC pair would use - and reads its current value; WRS.STO "Wait-for-Reservation-Set, Short TimeOut" then parks the hart until any other hart's store breaks the reservation as an event, an interrupt arrives, or an implementation-bounded timeout elapses. The store pushes the waiter awake, exactly like AArch64's LDAXR and WFET, and the bounded timeout is the same belt for a loop that guards two words while the reservation covers one.

RISC-V has LR only at word and double-word width, so a 1- or 2-byte watch falls back to the Zihintpause spin - which the pool never needs, as its loops watch the 32-bit mood and the register-width epoch.

MSVC reaches them through __prefetch2, whose prfop immediates 0b10000 and 0b00010 spell PST-L1-KEEP and PLD-L2-KEEP - the same encodings the asm arm emits.

A monitored waiter needs only the address and observed bit pattern, so both forms route here.

Where inline assembly is available the UMONITOR opcode is hand-encoded so no header is pulled in; MSVC has no inline assembly and calls the _umonitor intrinsic from <immintrin.h> instead.

Inline assembly hand-encodes the opcode to avoid an include, while MSVC calls the _umwait intrinsic from <immintrin.h> with sleep_state and deadline.

It only ever needs the address and the bit pattern, so both std::atomic objects and in-place std::atomic_ref-owned words route here.