# Busy Waiting > ForkUnion for C++ Here's what "busy waiting" looks like in C++: ```cpp while (!has_work_to_do()) std::this_thread::yield(); ``` On Linux, the `std::this_thread::yield()` translates into a `sched_yield` system call, which means context switching to the kernel and back. Instead, you can replace the `standard_yield_t` wrapper with a platform-specific micro-wait instruction, which is much cheaper. Those instructions, like [`WFET` on Arm](https://developer.arm.com/documentation/ddi0602/2025-03/Base-Instructions/WFET--Wait-for-event-with-timeout-), generally hint the CPU to transition to a low-power state. | Wrapper | ISA | Instruction | Privileges | | :---------------- | :----------- | :------------------ | :--------- | | `x86_pause_t` | x86 | `PAUSE` | R3 | | `x86_tpause_t` | x86+WAITPKG | `UMONITOR`+`UMWAIT` | R3 | | `arm64_yield_t` | AArch64 | `YIELD` | EL0 | | `arm64_wfe_t` | AArch64 | `LDAXR`+`WFE` | EL0 | | `arm64_wfet_t` | AArch64+WFXT | `WFET` | EL0 | | `riscv64_pause_t` | RISC-V | `PAUSE` | U | | `riscv64_wrs_t` | RISC-V+Zawrs | `WRS.STO`/`WRS.NTO` | U | No kernel calls. No futexes. Works in tight loops. The `UMWAIT`, `WFE`, `WFET`, and `WRS` wrappers go a step further than a spin hint: they sleep the core until a store to the watched line, capped by a timeout only where the loop guards more than one word, so the pool parks a worker in a low-power state rather than burning the core. `spin_mutex` takes the same wrappers, while the default `spin_mutex_t` yields to the scheduler and Rust's `SpinMutex` spins on `core::hint::spin_loop` - a futex-free `std::mutex` alternative, and the lock used in the NUMA example above. On WebAssembly threads the waiter is `wasm_wait_t`: a 32- or 64-bit watch sleeps in `memory.atomic.wait` until a `wasm_atomic_ref` releasing write notifies or 1 µs passes, so pair the two; uncapped waits and 1- or 2-byte watches spin. `#include ` `#include ` ## mood_t Defines the mood of the thread-pool, whether it is busy or about to die. ```cpp enum mood_t ``` - `grind_k` = `0`: That's our default ;). - `nap_k`: Stays on the core, parked on this word by the waiter's monitor; no syscall to wake. - `sleep_k`: Off the core, blocked in the kernel on this word; one syscall to wake. - `die_k`: The thread is about to die, we must exit the loop peacefully. ## dummy_lambda_t Placeholder type for Parallel Algorithms. ```cpp struct dummy_lambda_t ``` ## wait_capped_t Compile-time tag: a monitored wait bounded by a timeout, so it re-checks on its own. ```cpp struct wait_capped_t ``` For a loop guarding more than one word: the monitor arms a single line, so a store to another word goes unseen and the cap bounds how late it is noticed. Selects the timed instruction - `WFET`, `WRS.STO`. A spin waiter ignores the tag. See also: [wait_uncapped_t](#wait_uncapped_t) ## wait_uncapped_t Compile-time tag: a monitored wait with no timeout, woken only by the store it watches. ```cpp struct wait_uncapped_t ``` For a loop guarding a single word: the monitor covers every wake source, so a timeout would only wake the core to learn nothing and waste power. Selects the untimed instruction - `WFE`, `WRS.NTO`. See also: [wait_capped_t](#wait_capped_t) ## standard_yield_t The portable busy-wait: hands the core back to the scheduler. ```cpp struct standard_yield_t ``` Works everywhere, cheap nowhere. A spin waiter ignores the watched word - the caller's loop already re-checks its own condition, so this only needs to emit one backoff hint per turn. The monitored waiters - `[arm64_wfet_t](#arm64_wfet_t)`, `[x86_tpause_t](#x86_tpause_t)`, and `[riscv64_wrs_t](#riscv64_wrs_t)` - use the word to sleep the core until that line changes. - `capability_k` `capabilities_t` ## is_wait_functor Whether `yield_type_` is a valid waiter, callable as: `yield(watched, observed, thread_index)`. ```cpp template struct is_wait_functor ``` Every waiter takes the atomic being watched, the value last seen, and the calling thread's index. A spin functor ignores the first two; a monitored functor arms a hardware address monitor on the watched line and sleeps the core until it changes. The thread index is carried for a future adaptive backoff, and dropped by every waiter today. - `value` `bool` ## demote_line_t Tag for pushing a just-written line away, toward the LLC or the coherency point. ```cpp struct demote_line_t ``` ## promote_line_t Tag for pulling a line toward this core with write intent, ahead of an atomic claim. ```cpp struct promote_line_t ``` ## prefetch_line_t Tag for pulling a line into this core's L2 with read intent, a few loads ahead of its use. ```cpp struct prefetch_line_t ``` ## standard_cache_hints_t The do-nothing cache-hints policy - the default, and the fallback for every ISA gap. ```cpp struct standard_cache_hints_t ``` - `capability_k` `capabilities_t` See also: preferred_cache_hints_t in `capabilities.hpp`, which picks the per-ISA emitters where `FORKUNION_WITH_DEMOTE_CACHE_LINES` and `FORKUNION_WITH_PROMOTE_CACHE_LINES` hold. ## is_cache_hints_functor Whether `hints_type_` is a valid cache-hints policy: callable with every line tag. ```cpp template struct is_cache_hints_functor ``` Every policy takes the address of the line being handed away, claimed or read next, plus a tag choosing the direction. A policy for an ISA with no matching instruction implements the overload as a no-op, so callers never branch - the emptiness compiles away. - `value` `bool` ## spin_mutex A trivial minimalistic lock-free "mutex" implementation over a single `std::atomic`. ```cpp template class spin_mutex ``` The C++ standard would recommend using `std::hardware_destructive_interference_size` alignment, as well as `std::atomic_flag::notify_one` and `std::this_thread::yield` APIs, but our solution is better despite being more primitive. A `std::atomic` is used rather than `std::atomic_flag` so the flag has an __address__ a monitored waiter can arm: the lock loop is test-and-test-and-set, spinning on a plain load, so a monitored `micro_yield` sleeps the core until `unlock's` store to that line wakes it. On every platform we target `std::atomic` is lock-free, so the `atomic_flag` guarantee buys nothing. Compatible with STL unique locks. See also: Compatible with STL unique locks: [https://en.cppreference.com/w/cpp/thread/unique_lock.html](https://en.cppreference.com/w/cpp/thread/unique_lock.html) ## x86_pause_t On x86, hints a spin-wait so the core neither burns issue slots nor trips memory-order speculation. ```cpp struct x86_pause_t ``` - `capability_k` `capabilities_t` ## x86_tpause_t On x86 `WAITPKG`, a monitored wait built on `UMONITOR` and `UMWAIT`. ```cpp struct x86_tpause_t ``` `UMONITOR` arms an address-range monitor on the watched line; `UMWAIT` drops the logical processor into an optimized sleep until __any__ agent's store to that line, an interrupt, or the TSC deadline. The store __pushes__ the waiter awake as an event, so a fork-join handoff is observed rather than polled, and on an SMT core the sibling reclaims the freed pipeline slots for the duration. There are several older ways to wait on x86, but they may require different privileges: - `MONITOR` and `MWAIT` in SSE - used for power management, require ring 0 privilege. - `MWAITX` in `MONITORX` ISA on AMD - used for power management, requires ring 0 privilege. - `TPAUSE` in `WAITPKG` - a blind timed pause that watches __no__ address, so a store never wakes it early. This waiter replaces it: `UMWAIT` is strictly better given a word to watch.The `UMWAIT` control selects the sleep depth: bit 0 = 1 picks __C0.1__, shallow and fast-waking; bit 0 = 0 picks __C0.2__, deeper and slower. A fork-join barrier resolves in tens of nanoseconds, so we ask for C0.1. Neither state lowers voltage: like ARM's `WFE`, this clock-gates and saves dynamic power only, and never releases the core to the scheduler. - `capability_k` `capabilities_t` Warnings: The `UMONITOR` and `UMWAIT` opcodes are hand-encoded and, unlike the AArch64 path, untested on `WAITPKG` silicon in this tree; [`capability_x86_tpause_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_x86_tpause_k) gates them. ## x86_tpause_saturated_t Sibling of `[x86_tpause_t](#x86_tpause_t)` for saturated hosts - every logical core busy, SMT siblings competing for pipeline slots. ```cpp struct x86_tpause_saturated_t ``` The policy differs only in what each wait-bound tag selects. Capped two-word waits - lock and capacity gates whose wakes are rare - sleep in the deeper C0.2 state with a quarter-microsecond deadline, trading ~100 ns of extra wake latency for pipeline resources the sibling hyper-thread reclaims meanwhile. Uncapped single-word waits - serialized publication convoys where every nanosecond of wake latency lands on the critical chain - keep the shallow C0.1 state. Motivated by a 128-thread convoy workload where the shallow-everywhere policy of `[x86_tpause_t](#x86_tpause_t)` measured 13-16% behind plain `std::this_thread::yield` at full occupancy, while leading at the 7/8-occupancy operating point - the saturated sibling is the tool for the former regime. - `capability_k` `capabilities_t` ## arm64_yield_t On Arm, hints the core to release its pipeline slot to a sibling hardware thread. ```cpp struct arm64_yield_t ``` - `capability_k` `capabilities_t` ## arm64_wfe_t On AArch64 without `FEAT_WFxT`, a monitored wait for uncapped loops: `LDAXR` arms the exclusive monitor and `WFE` sleeps until a store to the line clears it. ```cpp struct arm64_wfe_t ``` Grace, Graviton 3 and 4 and the Neoverse N-series lack `WFET`, so this is their only way to sleep on a word. The store that clears the monitor also sets the event register, so a store landing between the re-check and the `WFE` ends the wait at once instead of being lost. Capped loops keep `YIELD`, since the kernel's event stream bounds a `WFE` only to about 100 micro-seconds, far coarser than the cap. - `capability_k` `capabilities_t` ## arm64_wfet_t On AArch64, a monitored wait built on the `WFET` "Wait For Event, Timed" instruction. ```cpp struct arm64_wfet_t ``` `LDAXR` arms this core's exclusive monitor on the watched word; `WFET` then places the core into light sleep until __any__ core's store to that line clears the monitor as an event, or the timeout expires. The store __pushes__ the waiter awake through the cache-coherence transaction, rather than the waiter polling for it - on an Apple M5 this wakes in ~42ns, faster than a bare `YIELD` spin, while the core is clock-gated. The timeout is the belt: a fork-join loop can watch two independent lines while the monitor arms only one, and the cap bounds how late a change to the __other__ line is noticed. `WFET` clock-gates the core to save __dynamic__ power. It does __not__ lower voltage, so it saves no __static__ leakage power, and it does __not__ release the core to the OS scheduler: the thread stays runnable and no sibling can be placed on it. A real retention or power-down C-state needs firmware - `PSCI CPU_SUSPEND` via `SMC` at EL3. This is a power hint for a busy core. - `capability_k` `capabilities_t` Notes: `WFET` is hand-encoded with `.inst` because `target("arch=armv8-a+wfxt")` breaks Apple Clang and neither compiler defines a `WFxT` feature macro. It is reached only when [`capability_arm64_wfet_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_arm64_wfet_k) reports `FEAT_WFxT` at runtime. ## riscv64_pause_t On RISC-V, the `Zihintpause` spin-wait hint. ```cpp struct riscv64_pause_t ``` - `capability_k` `capabilities_t` ## riscv64_wrs_t On RISC-V `Zawrs`, a monitored wait built on `LR` + `WRS.STO`. ```cpp struct riscv64_wrs_t ``` `LR.W` / `LR.D` arms a reservation on the watched word - the same reservation an `LR` and `SC` pair would use - and reads its current value; `WRS.STO` "Wait-for-Reservation-Set, Short TimeOut" then parks the hart until __any__ other hart's store breaks the reservation as an event, an interrupt arrives, or an implementation-bounded timeout elapses. The store __pushes__ the waiter awake, exactly like AArch64's `LDAXR` and `WFET`, and the bounded timeout is the same belt for a loop that guards two words while the reservation covers one. RISC-V has `LR` only at word and double-word width, so a 1- or 2-byte watch falls back to the `Zihintpause` spin - which the pool never needs, as its loops watch the 32-bit mood and the register-width epoch. - `capability_k` `capabilities_t` Notes: Like ARM's `WFE`, this clock-gates and saves dynamic power only; it never releases the hart to a scheduler. Its sibling `WRS.NTO` "No TimeOut" waits unbounded for one-word loops. Notes: `WRS.STO` is __manually__ encoded as `.4byte 0x01d00073` so the surrounding build needs no `-march=...+zawrs` assembler support; `LR` is plain `A-extension`, present on any core that would carry `Zawrs`. Warnings: Hand-encoded and not exercised on `Zawrs` silicon in this tree; [`capability_riscv64_wrs_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_riscv64_wrs_k), the kernel's `hwprobe` word, gates its selection at runtime. ## x86_cache_hints_t x86 cache hints: `CLDEMOTE` toward the LLC, `PREFETCHW` for write-intent promotion, `PREFETCHT1` for read-intent prefetching into L2. ```cpp struct x86_cache_hints_t ``` - `capability_k` `capabilities_t` Notes: Both live in hint or reserved-NOP space, so neither can fault on any x86-64 part; whether `CLDEMOTE` actually bites is reported by [`capability_x86_cldemote_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_x86_cldemote_k) - detected, never dispatched on. Hand-assembled so stock toolchains need no `-mcldemote` / `-mprfchw`; MSVC encodes the same bytes through `_mm_cldemote` and `_m_prefetchw`, no `/arch` needed. ## arm64_cache_hints_t AArch64 cache hints: `DC CVAC` cleans to the coherency point, `PRFM PSTL1KEEP` promotes, `PRFM PLDL2KEEP` prefetches into L2. ```cpp struct arm64_cache_hints_t ``` - `capability_k` `capabilities_t` Notes: There is no demote on Arm - the clean is the nearest thing: the next claimer's snoop finds a clean line instead of forcing a dirty intervention, at the price of a memory write. The clean is EL0-legal only where the kernel sets `SCTLR_EL1.UCI`; Linux does, and the `FORKUNION_WITH_DEMOTE_CACHE_LINES` gate requires `FORKUNION_OS_LINUX_` on this architecture. The persistence-targeted `DC CVAP` and `DC CVADP` are deliberately absent: UNDEFINED without `FEAT_DPB` or `FEAT_DPB2`, and they buy a NUMA hand-off nothing. ## arm64_prefetch_cache_hints_t AArch64 promotion and prefetching only, for kernels that keep `SCTLR_EL1.UCI` clear - Windows and the BSDs do, so an EL0 `DC CVAC` traps there and the demote stays a no-op. ```cpp struct arm64_prefetch_cache_hints_t ``` MSVC reaches them through `__prefetch2`, whose prfop immediates `0b10000` and `0b00010` spell PST-L1-KEEP and PLD-L2-KEEP - the same encodings the asm arm emits. - `capability_k` `capabilities_t` Notes: Mirrors `[riscv64_cache_hints_t](#riscv64_cache_hints_t):` both are `PRFM` hints that cannot fault anywhere. ## riscv64_cache_hints_t RISC-V promotion and prefetching only: `prefetch.w` and `prefetch.r` are `ORI x0, ...` hints that cannot fault, with or without Zicbop silicon. ```cpp struct riscv64_cache_hints_t ``` - `capability_k` `capabilities_t` Notes: The `cbo.clean` demote is deliberately a no-op here: it raises illegal-instruction unless the kernel set `senvcfg.CBCFE`, which only `hwprobe` can attest at runtime - so it belongs to a runtime-dispatch tier behind [`capability_riscv64_zicbom_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_riscv64_zicbom_k), not a compile-time policy. ## riscv64_cbo_cache_hints_t RISC-V cache hints where `hwprobe` attested Zicbom: `cbo.clean` writes the dirty block back toward another cache or memory, and `prefetch.w` promotes with write intent. ```cpp struct riscv64_cbo_cache_hints_t ``` - `capability_k` `capabilities_t` Notes: Never selected at compile time - `cbo.clean` raises illegal-instruction unless the kernel set `senvcfg.CBCFE`, which only the [`capability_riscv64_zicbom_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#capability_riscv64_zicbom_k) runtime bit can attest - so this functor is reachable exclusively through the C ABI's runtime cascade. ## os_wait_on_word Blocks the calling thread in the kernel while `word` holds `observed`. ```cpp template void os_wait_on_word( word_type_ const *word, word_type_ observed, std::size_t const wake_up_periodicity_micros) noexcept; ``` - `word` (in): The 32-bit word to wait on. - `observed` (in): The value last read, so a change before the wait arms ends it at once. - `wake_up_periodicity_micros` (in): Longest wait in microseconds, or 0 to wait for the wake. Notes: May return early, so callers loop on their own condition. See also: [os_wake_all_on_word](#os_wake_all_on_word) ## os_wake_all_on_word Wakes every thread blocked on `word` by [`os_wait_on_word`](#os_wait_on_word). ```cpp template void os_wake_all_on_word(word_type_ const *word) noexcept; ``` - `word` (in): The 32-bit word they wait on. ## watched_address The address of a waited word, a `std::atomic` object or a bare `std::atomic_ref-owned` slot. ```cpp template void const * watched_address( std::atomic const & watched) noexcept; ``` A monitored waiter needs only the address and observed bit pattern, so both forms route here. ## x86_arm_address Arms this core's address-range monitor on the line holding [`watched_address`](#watched_address). ```cpp void x86_arm_address(void const *watched_address) noexcept; ``` Where inline assembly is available the `UMONITOR` opcode is hand-encoded so no header is pulled in; MSVC has no inline assembly and calls the `_umonitor` intrinsic from `` instead. ## x86_arm_monitor Arms this core's address-range monitor on `watched`, reporting whether to wait. ```cpp template bool x86_arm_monitor( std::atomic const & watched, value_type_ const observed) noexcept; ``` ```cpp template bool x86_arm_monitor( value_type_ const *watched, value_type_ const observed) noexcept; ``` Same, for a word owned through [`std::atomic_ref`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#atomic_ref) rather than a `std::atomic` object. ## x86_umwait_until Sleeps in `sleep_state` until `deadline` as a TSC value, an interrupt, or a store to the monitored line. ```cpp void x86_umwait_until( std::uint64_t const deadline, std::uint32_t const sleep_state) noexcept; ``` Inline assembly hand-encodes the opcode to avoid an include, while MSVC calls the `_umwait` intrinsic from `` with `sleep_state` and `deadline`. ## arm64_arm_monitor Arms this core's exclusive monitor on the word at [`watched_address`](#watched_address) and reports whether to enter the wait. ```cpp template bool arm64_arm_monitor( void const *watched_address, value_type_ const observed) noexcept; ``` It only ever needs the address and the bit pattern, so both `std::atomic` objects and in-place `std::atomic_ref-owned` words route here. ## wait_capped_k The canonical `[wait_capped_t](#wait_capped_t)` value to pass as a wait tag. ```cpp wait_capped_t wait_capped_k; ``` ## wait_uncapped_k The canonical `[wait_uncapped_t](#wait_uncapped_t)` value to pass as a wait tag. ```cpp wait_uncapped_t wait_uncapped_k; ``` ## os_waits_on_words_k Whether this platform can block a thread on a word; [`sleep`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#sleep) falls back to [`nap`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#nap) otherwise. ```cpp bool os_waits_on_words_k; ``` ## demote_line_k Canonical `[demote_line_t](#demote_line_t)` value, mirroring the [`wait_capped_k`](#wait_capped_k) tag convention. ```cpp demote_line_t demote_line_k; ``` ## promote_line_k Canonical `[promote_line_t](#promote_line_t)` value, mirroring the [`wait_uncapped_k`](#wait_uncapped_k) tag convention. ```cpp promote_line_t promote_line_k; ``` ## prefetch_line_k Canonical `[prefetch_line_t](#prefetch_line_t)` value, beside [`demote_line_k`](#demote_line_k) and [`promote_line_k`](#promote_line_k). ```cpp prefetch_line_t prefetch_line_k; ``` ## x86_umwait_shallow_c01_k `UMWAIT` sleep-depth control: bit 0 = 1 selects the shallow, fast-waking C0.1 state. ```cpp std::uint32_t x86_umwait_shallow_c01_k; ``` ## x86_umwait_deeper_c02_k `UMWAIT` sleep-depth control: bit 0 = 0 selects the deeper C0.2 state - slower to wake, but ceding more of the shared core's pipeline resources to the SMT sibling meanwhile. ```cpp std::uint32_t x86_umwait_deeper_c02_k; ```