# ForkUnion

> NUMA-aware fork-join parallelism

## Introduction

On the hot path it makes __zero__ [heap allocations](#memory-allocations), __zero__ [system calls](#locks--mutexes), __zero__ [CAS loops](#atomics--cas), and suffers no [false-sharing](#alignment--false-sharing) of cache-lines.
So dispatch latency stays flat into the hundreds of cores, precisely where task-queue runtimes like Rayon collapse and even OpenMP begins to slip.
It is also unique in parking idle workers on a __hardware address monitor__ — x86 `UMONITOR`/`UMWAIT`, Arm `WFE`/`WFET`, RISC-V `Zawrs` — so a worker light-sleeps on the exact cache-line it is waiting for and the silicon wakes it the instant another core writes there, with no hot spinning and no kernel futex.
That bet compounds as sockets multiply: Intel Xeon Platinum and NVIDIA Vera pack up to 8 sockets per node, where shared CAS thrashes the interconnect and private `fetch_add` cursors keep synchronization off it.

> __One parallel-for dispatch:__ ForkUnion is often __~3–6× faster than OpenMP__ and up to __~12–16× faster than Rayon and Taskflow__ on many-core servers — the fork-join tax, paid once per loop, cut by an order of magnitude.
> [Full tables ↓](#performance)

It is exhaustively tested for boundary-condition scheduling with miniaturized `uint8_t` indices, even runs on your big-endian IBM mainframe, and ships with `no_std` and Miri coverage.
The core is a C++ 20 library; the C 99, Rust, Zig, and Mojo APIs bind it, and all five can pin threads to [NUMA](https://en.wikipedia.org/wiki/Non-uniform_memory_access) nodes or individual cores and allocate node-local memory.
Despite being far more deeply tied to hardware and the OS than most alternatives, ForkUnion runs on __six operating systems__ — Linux, FreeBSD, Windows, macOS, Android, and iOS — including asymmetric compute and memory topologies.
Topology harvesting and thread placement work on all six; NUMA-local memory placement is implemented on Linux, FreeBSD, and Windows, and elsewhere allocation falls back to a single memory domain.
It also compiles to __WebAssembly__, where the pools are real in the two shapes that carry threads — Emscripten with `-pthread` over a shared memory, and `wasm32-wasip1-threads` under a runtime like Wasmtime — and collapse to the calling thread in a plain single-threaded module.

## Quickstart

__`ForkUnion`__ is dead-simple to use!
There is no nested parallelism, exception handling, or "future promises"; they are banned.
A pool runs one dispatch at a time and refuses any other with the `contended` status, including one from inside its own callbacks.
The thread pool itself has a few core operations:

- `spawn` to initialize worker threads, and
- `for_threads` to launch a blocking callback on all threads.

Higher-level APIs for index-addressable tasks are also available:

- `for_n` - for individual evenly-sized tasks,
- `for_n_dynamic` - for individual unevenly-sized tasks,
- `for_slices` - for contiguous runs of evenly-sized tasks, one per worker.

For additional flow control and tuning, following helpers are available:

- `nap` - to park workers on the CPU's address monitor, waking with no syscall,
- `sleep(microseconds)` - to block workers in the kernel for long idle periods, with 0 waiting for the next dispatch,
- `terminate` - to kill the threads before the destructor is called,
- `unsafe_for_threads` - to broadcast a callback without blocking, returning a generation token,
- `unsafe_join` - to block until the completion of a broadcasted generation,
- `is_complete` - to poll a generation token for completion without blocking.

Every dispatch is identified by a `generation` token.
On `caller_exclusive_k` pools you can dispatch, overlap your own work, poll `is_complete`, and join - the classic poll-then-join pattern.
On `caller_inclusive_k` pools the calling thread owes one slice of the work, which only runs inside `unsafe_join`, so completion can't be reached by polling alone.
The same rule shapes the RAII guard returned by `for_threads` and, in C++, the `for_n` family: the workers start at the guard's construction on every pool, and on inclusive pools the caller's own slice runs at `join` or destruction.

On `caller_exclusive_k` pools thread 0 may also fork from inside its own slice, at any depth: such a fork waits for every other slice of the one enclosing it, runs on all threads, and returns complete.
So `unsafe_for_threads` can hand a whole multi-step call to thread 0, which forks the rest of the pool as it goes, while the caller only polls a status word the call stores at its end.
That is how one manager thread keeps several exclusive pools busy with no queue and no extra thread.
A join on such a pool waits for whatever dispatch runs, since the call may end before its submitter stores the token.

In C++ it provides [NUMA extensions](https://www.unum.cloud/docs/forkunion/cpp.md#numa).
That includes the `colocated_pool` analog of the `flat_pool` and the `domain_allocator_t` for allocating memory in a specific memory domain.
Those are out-of-the-box compatible with the higher-level APIs.
Most interestingly, for Big Data applications, a higher-level `distributed_pool` class addresses and balances the work across all compute domains.

> `domain_allocator_t` is backed by `linux_numa_allocator_t`, `freebsd_numa_allocator_t`, or `windows_numa_allocator_t`.

## Alternatives & Differences

Most "thread-pools" are not thread-pools at all, but task-queues - think a `std::queue<std::function<void()>>` behind a `std::mutex`, executing each heap-allocated task on whatever core the OS scheduler picks.
That shape is what most alternatives are built around, and it is the wrong shape for a tight `parallel for`:

- Modern C++: [`taskflow/taskflow`](https://github.com/taskflow/taskflow), [`progschj/ThreadPool`](https://github.com/progschj/ThreadPool), [`bshoshany/thread-pool`](https://github.com/bshoshany/thread-pool)
- Traditional C++: [`vit-vit/CTPL`](https://github.com/vit-vit/CTPL), [`mtrebi/thread-pool`](https://github.com/mtrebi/thread-pool)
- Rust: [`tokio-rs/tokio`](https://github.com/tokio-rs/tokio), [`rayon-rs/rayon`](https://github.com/rayon-rs/rayon), [`smol-rs/smol`](https://github.com/smol-rs/smol)
- Zig: [`std.Io.Group`](https://ziglang.org/documentation/0.16.0/std/#std.Io.Group), which replaced the removed `std.Thread.Pool` in 0.16

__Reach for ForkUnion__ when you have data-parallel loops, bulk-synchronous phases, or NUMA-sharded scans, and you want them to dispatch in nanoseconds with neither an allocator nor the kernel on the hot path.
__Reach for something else__ when you need task graphs, async I/O, futures and promises, work that outlives its scope, or nested parallelism — Taskflow, Tokio, and oneTBB are built for exactly that, and ForkUnion deliberately bans it.

The trade-offs line up like this, verified against each project's own source and documentation:

| Property                        | ForkUnion                   | OpenMP             | Rayon                      | Taskflow              | oneTBB                |
| :------------------------------ | :-------------------------- | :----------------- | :------------------------- | :-------------------- | :-------------------- |
| Hot-path heap allocations       | __None__                    | ≈ None ¹           | Per spawned task ²         | Per task node         | Per task, pooled      |
| Syscalls to dispatch            | __None__                    | Spin, then futex   | Spin/yield, then futex     | Spin, then futex      | Spin, then futex      |
| Dynamic-schedule primitive      | __`fetch_add` cursors__     | CAS retry loop ³   | Chase-Lev CAS deque ⁴      | Chase-Lev CAS deque ⁴ | Chase-Lev CAS deque ⁴ |
| NUMA pinning & local allocators | __Yes__                     | Yes ⁵              | No                         | No ⁶                  | Partial ⁷             |
| Hardware timed-wait             | __Yes__                     | x86 only, opt-in ⁸ | No                         | No                    | No                    |
| Languages                       | C++ · C · Rust · Zig · Mojo | C · C++ · Fortran  | Rust                       | C++                   | C++                   |
| Design center                   | Fork-join loops             | Loops + tasks      | Stealing tasks + iterators | Task-graph DAG        | Stealing tasks        |

> ¹ OpenMP reuses a persistent thread team, so a repeated `parallel for` over a fixed-size team does not re-create it; the per-region worksharing descriptor is reused but not provably allocation-free on every path.
> ² Rayon keeps `join` and parallel-iterator jobs on the stack, but `scope`/`spawn` heap-allocate one job per task, and the Chase-Lev deque grows on the heap under deep nesting.
> ³ libgomp and libomp hand out `schedule(dynamic)` iterations with an atomic fetch-add plus a CAS retry loop.
> ⁴ Rayon, Taskflow, and oneTBB all steal through a Chase-Lev deque whose steal path is a CAS.
> ⁵ `OMP_PROC_BIND` / `OMP_PLACES` pin threads, and the OpenMP 5 allocator and memory-space API places memory.
> ⁶ Taskflow exposes only a manual `WorkerInterface` hook for affinity, nothing automatic.
> ⁷ oneTBB pins threads to a NUMA node via `task_arena` constraints, but does not place memory locally for you.
> ⁸ Only LLVM/Intel `libomp` issues x86 `UMONITOR` / `UMWAIT`, gated behind `KMP_USER_LEVEL_MWAIT` and off by default; there is no Arm `WFET` or RISC-V `Zawrs` path in any OpenMP runtime.

### Locks & Mutexes

Unlike `std::atomic`, `std::mutex` can fall into a system call, and it can be expensive to acquire and release.
Its implementations generally have 2 executable paths:

- the fast path, where the mutex is not contended, where it first tries to grab the mutex via a compare-and-swap operation, and if it succeeds, it returns immediately.
- the slow path, where the mutex is contended, and it has to go through the kernel to block the thread until the mutex is available.

On Linux, the latter translates to ["futex"](https://en.wikipedia.org/wiki/Futex) ["system calls"](https://en.wikipedia.org/wiki/System_call), which is expensive.

### Memory Allocations

C++ has rich functionality for concurrent applications, like `std::future`, `std::packaged_task`, `std::function`, `std::queue`, `std::condition_variable`, and so on.
Most of those, I believe, are unusable in Big-Data applications, where you always operate in memory-constrained environments:

- The idea of raising a `std::bad_alloc` exception when there is no memory left and just hoping that someone up the call stack will catch it is not a great design idea for any Systems Engineering.
- The threat of having to synchronize ~200 physical CPU cores across 2-8 sockets and potentially dozens of [NUMA](https://en.wikipedia.org/wiki/Non-uniform_memory_access) nodes around a shared global memory allocator practically means you can't have predictable performance.

As we focus on a simpler ~~concurrency~~ parallelism model, we can avoid the complexity of allocating shared states, wrapping callbacks into some heap-allocated "tasks", and other boilerplate code.
Less work - more performance.

### Atomics & CAS

Once you get to the lowest-level primitives on concurrency, you end up with the `std::atomic` and a small set of hardware-supported atomic instructions.
Hardware implements it differently:

- x86 is built around the "Total Store Order" (TSO) [memory consistency model](https://en.wikipedia.org/wiki/Memory_ordering) and provides `LOCK` variants of the `ADD` and `CMPXCHG`, which act as full-blown "fences" - no loads or stores can be reordered across it.
- Arm, on the other hand, has a "weak" memory model and provides a set of atomic instructions that are not fences, that match the C++ concurrency model, offering `acquire`, `release`, and `acq_rel` variants of each atomic instruction—such as `LDADD`, `STADD`, and `CAS` - which allow precise control over visibility and order, especially with the introduction of "Large System Extension" (LSE) instructions in Armv8.1.
- WebAssembly with a shared memory has its own `i32.atomic.rmw.add`, `i32.atomic.rmw.cmpxchg` and kin, which the engine compiles to the host's atomic instructions; a module without shared memory runs one thread, so its atomics compile to plain loads and stores.

In practice, a locked atomic on x86 requires the cache line in the Exclusive state in the requester's L1 cache.
This would incur a Read-for-Ownership coherence transaction if some other core had the line.
Both Intel and AMD handle this similarly.

It makes [Arm and Power much more suitable for lock-free programming](https://arangodb.com/2021/02/cpp-memory-model-migrating-from-x86-to-arm/) and concurrent data structures, but some observations hold for both platforms.
Most importantly, "Compare and Swap" (CAS) is a costly operation and should be avoided whenever possible.

On x86, for example, the `LOCK ADD` [can easily take 50 CPU cycles](https://travisdowns.github.io/blog/2020/07/06/concurrency-costs), being 50× slower than a regular `ADD` instruction, and an uncontended `LOCK CMPXCHG` costs about the same.
Once contention rises, the CAS falls behind, amplified by its "failure" rate, particularly when the value being compared has already changed.
That's why, in the "dynamic" mode, threads claim tasks with `fetch_add` on private per-thread cursors rather than a CAS loop.
A dispatch does take one compare-and-swap to claim the pool, and never retries it: a claim that loses is refused rather than repeated.

### Atomic References

`forkunion/atomics.hpp` replaces `std::atomic_ref` with references that spell each operation as the one instruction a CPU has for it, picked by capability bit at runtime rather than by the translation unit's `-march`.
Besides the standard operations, every reference offers the verbs the standard lacks:

- `fetch_add_if_at_most` and `fetch_sub_if_at_least`: one `cmpccxadd` on x86, a compare-exchange loop elsewhere;
- `fetch_max` and `fetch_min`: one `ldsmax`, `ldumax` or `amomax` on Arm and RISC-V;
- no-return `add`, `sub`, `set_bits`, `clear_bits` and `flip_bits`: `stadd` and kin on Arm, `amoadd` into `x0` on RISC-V, and RAO-INT `aadd` and kin for relaxed orders on x86 parts that implement it.

On an M5 Pro the Arm references run 5x faster than a baseline build's exclusive loops uncontended, and 3-4x faster with 18 threads on one word.
A no-return add may complete away from the core, so a relaxed one carries nothing to a later release: publish through it with `release`, as the `far` memory model in [`verification/`](https://github.com/ashvardanian/ForkUnion/tree/main/verification) checks.

### Alignment & False Sharing

The thread-pool needs several atomic variables to synchronize the state.
If those variables are located on the same cache line, they will be "falsely shared" between threads.
This means that when one thread updates one of the variables, it will invalidate the cache line in all other threads, causing them to reload it from memory.
This is a common problem, and the C++ standard recommends addressing it with `alignas(std::hardware_destructive_interference_size)` for your hot variables.

There are, however, caveats.
The `std::hardware_destructive_interference_size` is [generally 64 bytes on x86](https://stackoverflow.com/a/39887282), matching the size of a single cache line.
But in reality, on most x86 machines, [depending on the BIOS "spatial prefetcher" settings](https://www.techarp.com/bios-guide/cpu-adjacent-sector-prefetch/), will [fetch 2 cache lines at a time starting with Sandy Bridge](https://stackoverflow.com/a/72127222).
Because of these rules, padding hot variables to 128 bytes is a conservative but often sensible defensive measure adopted by Folly's `cacheline_align` and Java's `jdk.internal.vm.annotation.Contended`.

## Performance

Two benchmarks measure two different things, each against the same runtimes — __ForkUnion__, [__OpenMP__](https://www.openmp.org), [__Rayon__](https://github.com/rayon-rs/rayon), and [__Taskflow__](https://github.com/taskflow/taskflow) — on deliberately equal footing ¹.
N-body stresses the __dispatch path__: every task costs the same, so what is left over is scheduling latency.
Connected Components by label propagation stresses the __fork-join frequency__: one bandwidth-bound sweep per round until no label changes, so every round re-pays the dispatch-and-join tax.
Implementations live in `bench/nbody.{cpp,rs,zig,mojo}` and `bench/propagation.{cpp,rs,zig,mojo}`, and every binary generates __bit-identical inputs__ from the same counter-based SplitMix64 generator, so cells compare exactly across languages.

### N-Body — Dispatch Latency

Microseconds per iteration — µs ↓, lower is better — at `N=512` bodies on every logical core, as `static / dynamic`, one fixed 30-second window per cell.
This is where fork-join runtimes genuinely differ; ➕ marks C++, 🦀 marks Rust.

| Machine                    | ➕ ForkUnion | 🦀 ForkUnion | ➕ OpenMP |  🦀 Rayon | ➕ Taskflow |
| :------------------------- | -----------: | -----------: | --------: | --------: | ----------: |
| 18× Apple M5 Pro, macOS    |      32 / 37 |  __28 / 32__ | 115 / 137 | 222 / 339 |     83 / 94 |
| 24× Intel RPL, Windows     |    148 / 143 | __112 / 91__ | 360 / 627 | 151 / 151 |  812 / 1026 |
| 128× Intel SPR, Linux      |      54 / 86 |  __40 / 67__ | 115 / 226 | 483 / 739 |   666 / 694 |
| 192× AWS Graviton 5, Linux | 47 / __136__ | __42__ / 137 | 266 / 216 | 490 / 580 |   602 / 639 |

### Connected Components — Fork-Join Frequency

Billions of traversed edges per second — GTEPS ↑, higher is better — on a ~27M-edge "necklace" of R-MAT communities, on every logical core, as `static / dynamic`, one fixed 30-second window per cell.
Every pass converges in exactly 88 rounds - 88 fork-join dispatches - bit-identical in every cell and language; ➕ marks C++, 🦀 marks Rust.

| Machine                    |    ➕ ForkUnion |     🦀 ForkUnion |      ➕ OpenMP |    🦀 Rayon | ➕ Taskflow |
| :------------------------- | --------------: | ---------------: | -------------: | ----------: | ----------: |
| 18× Apple M5 Pro, macOS    | __29.2 / 23.5__ |      19.4 / 20.2 |    23.0 / 18.3 | 23.8 / 14.6 |  21.4 / 1.3 |
| 24× Intel RPL, Windows     |  10.0 / __7.2__ |        5.2 / 5.5 | __11.4__ / 0.1 |   9.4 / 5.2 |   6.6 / 0.7 |
| 128× Intel SPR, Linux      |     46.5 / 20.9 |  __65.2 / 31.4__ |     28.2 / 0.4 |   8.5 / 6.5 |  25.1 / 0.5 |
| 192× AWS Graviton 5, Linux |    192.6 / 77.6 | __258.7 / 87.3__ |     53.9 / 0.4 |   8.4 / 6.2 |  53.0 / 0.4 |

What the spread means - same binaries, same graph:

- __The static column is the fork-join tax, compounded 88 times per pass__: every runtime sweeps the identical bandwidth-bound round, so the entire spread is scheduling - dispatch latency, barrier cost, and thread placement - never arithmetic.
- __The dynamic column is the claim architecture, ~92M one-vertex claims per pass__: ForkUnion claims from private `fetch_add` cursors, Rayon steals halves through its CAS deques, and OpenMP's and Taskflow's shared queues collapse - a 128-thread contention effect for OpenMP, whose queue survives the M5 Pro's 18 threads, while Taskflow's collapse reproduces even there.
- __The two ForkUnion columns are the same pool underneath__: the scheduler, the graph, and the labels are bit-identical, so the gap between them - Rust ahead on SPR, behind on the M5 Pro - is compiler codegen of the compute loops, not the claims.

> ¹ Parity, deliberately enforced: identical `-O3` + `target-cpu=native` on all sides, no LTO anywhere, unchecked hot loops in Rust, identical kernels, a 1-second warm-up, and the same one-vertex dynamic grain in every runtime.
> The finer per-benchmark protocol - scheduling equivalents and page-placement controls - lives in the `bench/nbody.*` and `bench/propagation.*` headers, and every setting with its default is in the table below.

You can rerun these benchmarks with the following commands:

```bash
cmake -B build_release -D CMAKE_BUILD_TYPE=Release -D FORKUNION_BUILD_BENCH=ON
cmake --build build_release --config Release
# Rust examples: plain `cargo build` grants neither flag, so parity needs both set explicitly
RUSTFLAGS="-C target-cpu=native" CXXFLAGS="-O3 -march=native" cargo build --release --features benchmarks
# N-body: microseconds per iteration, sustained over a fixed window (FORKUNION_TIME_LIMIT, default 10s)
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_shared build_release/forkunion_nbody
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=taskflow_dynamic build_release/forkunion_nbody
# Connected components: traversal throughput on a necklace graph; FORKUNION_PROPAGATION_COMMUNITIES controls the rounds
FORKUNION_BACKEND=forkunion_static_shared build_release/forkunion_propagation
FORKUNION_BACKEND=rayon_dynamic target/release/forkunion_propagation
```

Every port reads the same environment variables once at start, and a value that does not parse prints one line and exits with status 1.
Leaving a variable unset or empty keeps its default.

| Variable                            | Default                   | Meaning                           |
| :---------------------------------- | :------------------------ | :-------------------------------- |
| `FORKUNION_SEED`                    | `42`                      | Seed of every draw, or `random`   |
| `FORKUNION_WARMUP`                  | `1s`                      | Untimed run before timing         |
| `FORKUNION_TIME_LIMIT`              | `10s`                     | Timed window, like `500ms`        |
| `FORKUNION_THREADS`                 | all cores                 | Threads; `0` also means all cores |
| `FORKUNION_BACKEND`                 | `forkunion_static_shared` | Backend, per benchmark header     |
| `FORKUNION_NBODY_COUNT`             | the thread count          | Bodies to simulate                |
| `FORKUNION_PROPAGATION_SCALE`       | `14`                      | `2^scale` vertices per community  |
| `FORKUNION_PROPAGATION_COMMUNITIES` | `64`                      | Communities strung on the ring    |
| `FORKUNION_PROPAGATION_EDGE_FACTOR` | `16`                      | Raw edges per vertex              |
| `FORKUNION_PROPAGATION_CHECK`       | `false`                   | Also converge serially to check   |

## Safety & Logic

There are only 3 core atomic variables in this thread-pool, plus 1 private claim cursor per worker for dynamically-stealing tasks.
Let's call every invocation of a `for_*` API - a "fork", and every exit from it a "join".

| Variable          | Users Perspective                    | Internal Usage                                          |
| :---------------- | :----------------------------------- | :------------------------------------------------------ |
| `mood`            | Spin, sleep, or stop the pool        | Tells workers to spin, sleep, or exit the loop          |
| `epoch`           | Odd while a fork runs, even if idle  | Tells workers to wake up on new forks                   |
| `threads_to_sync` | Contributors still running this fork | Claimed from zero by a dispatch, last one steps `epoch` |
| `claim.next`      | One worker's stealable cursor        | Next task in that worker's slice                        |

The next task for `for_n_dynamic` calls is drained by neighbors from `claim.next` in coprime order once their own slice runs dry, each thread overshooting a cursor by at most one.

### Why don't we need atomics for "total_threads"?

The only way to change the number of threads is to `terminate` the entire thread-pool and then `spawn` it again.
Either of those operations can only be called from one thread at a time and never coincide with any running tasks.
That is the caller's contract; the pool does not enforce it.

### Why don't we need atomics for a "job pointer"?

Only one dispatch holds the pool at a time: it claims the pool by exchanging `threads_to_sync` from zero, and any other dispatch, nested or concurrent, is refused.
During its update of the fork state, the workers are asleep, spinning on old values of `epoch` and `mood`.
They only wake up and access the new state once `epoch` turns odd, ensuring safety.

### How do we deal with overflows and `SIZE_MAX`-sized tasks?

The library avoids saturating arithmetic entirely.
To test the consistency of arithmetic, the C++ template class can be instantiated with a custom `index_t`, such as `std::uint8_t` or `std::uint16_t`.
In the former case, no more than 255 threads can operate, and no more than 255 tasks can be addressed, allowing us to easily test every weird corner case of [0:255] threads competing for [0:255] tasks.

### Why not reimplement it in Rust?

The original Rust implementation was a standalone library, but in essence, Rust doesn't feel designed for parallelism, concurrency, and expert Systems Engineering.
It enforces stringent safety rules, which is excellent for building trustworthy software, but realistically, it makes lock-free concurrent programming with minimal memory allocations too complicated.
Now, the Rust library is a wrapper over the C binding of the C++ core implementation.

## Testing & Benchmarking

Toolchain floors, never caps.
The header needs __C++20__, a C++23 consumer keeps its own standard, and the pre-compiled libraries build at C++20 regardless.
The C ABI needs __C99__, and the build tooling __CMake 3.28__, __Rust 1.100__, __Zig 0.16__, and __Mojo 1.2__.

To run the C++ tests, use the CMake presets CI uses; `cmake --list-presets` shows the ones this machine can build:

```bash
cmake --workflow --preset release               # configure, build, and run every test
cmake --workflow --preset asan                  # or `tsan`
```

For C++ debug builds, consider using the VS Code debugger presets or the following commands:

```bash
cmake --preset debug && cmake --build --preset debug
build_debug/forkunion_test_cpp20                # run a single test executable
```

To cross-compile for WebAssembly, pick the preset for the shape you need, and `ctest` runs the binaries through Node or Wasmtime:

```bash
cmake --preset wasm32_emscripten                # one thread, loads in any page
cmake --preset wasm64_emscripten                # shared memory and 64-bit addressing
cmake --preset wasm32_wasi_threads              # wasi-threads under Wasmtime
cmake --build --preset wasm64_emscripten && ctest --preset wasm64_emscripten
```

The wasm32 module has one thread, so its pool spawns caller-inclusive and only its C test runs there; the other two shapes run the C++20 and C tests, minus the stress cases.

To cross-compile for Android, point the `android_arm64` preset at the NDK's toolchain file and pick an ABI; the binaries only compile, since they need a device to run:

```bash
export CMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake
cmake --preset android_arm64 -D ANDROID_ABI=arm64-v8a -D ANDROID_PLATFORM=24    # or armeabi-v7a
cmake --build --preset android_arm64
```

Android builds need __NDK r28__ or newer, the first with libc++ 19 and its `std::atomic_ref`, and CI builds with the `dockcross/android-arm64` and `dockcross/android-arm` images.

Machine-specific settings, like a compiler path, belong in an untracked `CMakeUserPresets.json`.

To run static analysis:

```bash
sudo apt install cppcheck clang-tidy
cmake --build build_debug --target cppcheck     # detects bugs & undefined behavior
cmake --build build_debug --target clang-tidy   # suggest code improvements
```

No packages are needed for NUMA or Huge Pages, on any distribution.
The topology is read from `/sys/devices/system/node`, pages are placed with the `mbind` syscall, and huge pages are requested with `mmap(MAP_HUGETLB)` — all kernel interfaces, so a build picks up whatever the machine it runs on offers rather than whatever the machine it was compiled on had installed.
Neither `libnuma` nor `libhugetlbfs` is linked; `ld.hugetlbfs` remaps a program's own text segment, which is a different feature entirely.

Huge pages do have to be _reserved_ before they can be placed, which is a runtime knob rather than a build dependency:

```bash
cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages          # what exists now
echo 1024 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
```

To build with an alternative compiler, like LLVM Clang, use the following command:

```bash
sudo apt-get install libomp-18-dev clang-18 # OpenMP version must match Clang
cmake -B build_debug -D CMAKE_BUILD_TYPE=Debug -D CMAKE_CXX_COMPILER=clang++-18 -D FORKUNION_BUILD_TEST=ON
cmake --build build_debug --config Debug
build_debug/forkunion_test_cpp20
```

Or on macOS with Homebrew's LLVM Clang:

```bash
brew install llvm@20
cmake -B build_debug -D CMAKE_BUILD_TYPE=Debug -D CMAKE_CXX_COMPILER=$(brew --prefix llvm@20)/bin/clang++ -D FORKUNION_BUILD_TEST=ON
cmake --build build_debug --config Debug
build_debug/forkunion_test_cpp20
```

For Rust, use the following commands to verify no_std compatibility:

```bash
rustup toolchain install
cargo build --lib --no-default-features --release
cargo build --lib --no-default-features --features place-memory-on-domain --release
cargo doc --lib --no-default-features --no-deps
```

Verify the tests pass:

```bash
cargo test --lib --release
cargo test --doc --release
cargo test --lib --features place-memory-on-domain --release
```

Rust provides a lot of tooling for concurrency testing, like Miri and Loom.
Most of it, however, is not applicable in this case, as the core logic is implemented in C++.

To automatically detect the Minimum Supported Rust Version (MSRV):

```sh
cargo +stable install cargo-msrv
cargo msrv find --ignore-lockfile
```

---

For Zig, use the following commands:

```bash
zig build test --summary all               # run tests
zig build -Dplace-memory-on-domain=true    # require NUMA-aware allocations (Linux)
zig build -Dportable=true                  # STL thread pool only

# Run benchmarks from the `bench` directory
cd bench
zig build -Doptimize=ReleaseFast
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_shared ./zig-out/bin/forkunion_nbody
FORKUNION_BACKEND=forkunion_static_shared ./zig-out/bin/forkunion_propagation
```

They read the same environment variables as the C++ and Rust benchmarks, through `bench/harness.zig`.

---

For Mojo, every task compiles the C++ core first and installs it where `dlopen` will find it:

```bash
pixi run test                        # the binding's test suite, via `std.testing.TestSuite`
pixi run -e benchmarks nbody         # dispatch overhead, all-to-all
pixi run -e benchmarks propagation   # fork-join frequency, bit-identical to the sibling ports
```

The benchmarks sit in their own environment because their baseline needs `max`, which the binding itself never does.
Both take the same environment variables as the other ports, the four `forkunion_{static,dynamic}_{shared,replicated}` cells, and a `max_parallelize` baseline:

```bash
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_replicated pixi run -e benchmarks nbody
FORKUNION_BACKEND=max_parallelize pixi run -e benchmarks propagation
FORKUNION_PROPAGATION_CHECK=1 pixi run -e benchmarks propagation   # converge serially too, and fail on any disagreement
```

The suite's own per-test durations are not wall clock — a pool's workers busy-wait, and the runner's clock counts that — so time the process rather than trusting them.

```bash
pixi run -e portable test   # the STL thread pool only
pixi run -e numa test       # require NUMA-aware allocations
mojo format mojo/forkunion/*.mojo test/*.mojo bench/*.mojo   # width pinned in pyproject.toml
```