ForkUnion is dead-simple to use!
There is no nested parallelism, exception handling, or “future promises”; they are banned.
A pool runs one dispatch at a time and refuses any other with the contended status, including one from inside its own callbacks.
The thread pool itself has a few core operations:
spawn to initialize worker threads, and
for_threads to launch a blocking callback on all threads.
Higher-level APIs for index-addressable tasks are also available:
for_n - for individual evenly-sized tasks,
for_n_dynamic - for individual unevenly-sized tasks,
for_slices - for contiguous runs of evenly-sized tasks, one per worker.
For additional flow control and tuning, following helpers are available:
nap - to park workers on the CPU’s address monitor, waking with no syscall,
sleep(microseconds) - to block workers in the kernel for long idle periods, with 0 waiting for the next dispatch,
terminate - to kill the threads before the destructor is called,
unsafe_for_threads - to broadcast a callback without blocking, returning a generation token,
unsafe_join - to block until the completion of a broadcasted generation,
is_complete - to poll a generation token for completion without blocking.
Every dispatch is identified by a generation token.
On caller_exclusive_k pools you can dispatch, overlap your own work, poll is_complete, and join - the classic poll-then-join pattern.
On caller_inclusive_k pools the calling thread owes one slice of the work, which only runs inside unsafe_join, so completion can’t be reached by polling alone.
The same rule shapes the RAII guard returned by for_threads and, in C++, the for_n family: the workers start at the guard’s construction on every pool, and on inclusive pools the caller’s own slice runs at join or destruction.
On caller_exclusive_k pools thread 0 may also fork from inside its own slice, at any depth: such a fork waits for every other slice of the one enclosing it, runs on all threads, and returns complete.
So unsafe_for_threads can hand a whole multi-step call to thread 0, which forks the rest of the pool as it goes, while the caller only polls a status word the call stores at its end.
That is how one manager thread keeps several exclusive pools busy with no queue and no extra thread.
A join on such a pool waits for whatever dispatch runs, since the call may end before its submitter stores the token.
In C++ it provides NUMA extensions.
That includes the colocated_pool analog of the flat_pool and the domain_allocator_t for allocating memory in a specific memory domain.
Those are out-of-the-box compatible with the higher-level APIs.
Most interestingly, for Big Data applications, a higher-level distributed_pool class addresses and balances the work across all compute domains.
domain_allocator_t is backed by linux_numa_allocator_t, freebsd_numa_allocator_t, or windows_numa_allocator_t.
To integrate into your C++ project, either copy the c/ and include/ directories into your project and add both to the include path, add a Git submodule, or CMake.
The forkunion.hpp umbrella is a list of #includes; the implementation lives beside it in include/forkunion/, one header per concern, so a platform backend can be read or replaced on its own.
For a Git submodule, run:
#include <forkunion.hpp> // `flat_pool_t`#include <cstdio> // `stderr`#include <cstdlib> // `EXIT_SUCCESS`namespace fu = ashvardanian::forkunion;int main() { alignas(fu::default_alignment_k) fu::flat_pool_t pool; if (fu::failed(pool.spawn(fu::allowed_cores_count()))) { std::fprintf(stderr, "Failed to fork the threads\n"); return EXIT_FAILURE; } // Dispatch a callback to each thread in the pool pool.for_threads([&](std::size_t thread_index) noexcept { std::printf("Hello from thread # %zu (of %zu)\n", thread_index + 1, pool.threads_count()); }); // Execute 1000 tasks in parallel, expecting them to have comparable runtimes // and mostly co-locating subsequent tasks on the same thread. Analogous to: // // #pragma omp parallel for schedule(static) // for (int i = 0; i < 1000; ++i) { ... } // // You can also think about it as a shortcut for the `for_slices` + `for`. pool.for_n(1000, [](std::size_t task, fu::thread_in_domain_t) noexcept { std::printf("Running task %zu of 1000\n", task + 1); }); pool.for_slices(1000, [](fu::tasks_range_t range, fu::thread_in_domain_t) noexcept { std::printf("Running run [%zu, %zu)\n", range.first, range.first + range.count); }); // Like `for_n`, but each thread greedily steals tasks, without waiting for // the others or expecting individual tasks to have same runtimes. Analogous to: // // #pragma omp parallel for schedule(dynamic, 1) // for (int i = 0; i < 3; ++i) { ... } pool.for_n_dynamic(3, [](std::size_t task, fu::thread_in_domain_t) noexcept { std::printf("Running dynamic task %zu of 3\n", task + 1); }); return EXIT_SUCCESS;}
For advanced usage, refer to the NUMA section below.
Every kernel and ISA facility the library uses is detected by default.
CMake pins the four kernel facilities with an AUTO/ON/OFF tri-state like -D FORKUNION_WITH_PLACE_MEMORY_ON_DOMAIN=ON or -D FORKUNION_WITH_PLACE_THREADS_BY_AFFINITY=OFF.
Finer preprocessor gates - like the cache-line hints FORKUNION_WITH_DEMOTE_CACHE_LINES, which pushes a just-written line away from the core via x86 CLDEMOTE or Arm DC CVAC, and FORKUNION_WITH_PROMOTE_CACHE_LINES, which pulls a line in with write intent via x86 PREFETCHW, Arm PRFM PSTL1KEEP, or RISC-V prefetch.w - accept the same overrides as compile definitions.
Call fu_comptime_capabilities() to see what survived the build, fu_runtime_capabilities() to see what the machine underneath actually offers, and fu_name_capabilities() to render either mask as text - every binding exposes the same trio, Rust and Mojo spelling it runtime_capabilities() and Zig runtimeCapabilities with runtimeCapabilitiesString to render, alongside a version accessor: fu_version_major/_minor/_patch in C, version() in Rust, Zig, and Mojo, and the FORKUNION_VERSION_* macros in C++.
Every all-caps name starts with the full project name, FORKUNION_.
A trailing _ marks a name as internal: it may change in any release, and nothing outside this repository may define or test it.
A name without it is a public contract, either a switch you may set or a value you may read.
Family
Form
Example
ISA capability bit
FORKUNION_TARGET_<CAPABILITY>
FORKUNION_TARGET_X86_PAUSE
Dispatch mode
FORKUNION_RUNTIME_DISPATCH
Optional feature
FORKUNION_WITH_<FEATURE>
FORKUNION_WITH_TOPOLOGY
Permission for a liberty
FORKUNION_ALLOW_<LIBERTY>
FORKUNION_ALLOW_UNSAFE
Architecture fact
FORKUNION_ARCH_<ARCH>_
FORKUNION_ARCH_X8664_
Operating-system fact
FORKUNION_OS_<OS>_
FORKUNION_OS_LINUX_
Toolchain fact
FORKUNION_HAS_<FEATURE>_
FORKUNION_HAS_INLINE_ASM_
Architectures are spelled X8664, ARM64, RISCV64 and WASM.
Every name in these families is always defined, as 0 or 1, and tested with #if, never with defined(...).
Every enumerator takes its value from fu_status_t in forkunion/forkunion.h, so the C++ mirror and the ABI cannot drift - there is nothing to keep in step and nothing to assert.
Causes are collapsed on purpose where the remedy is the same, and split where it differs.
The call completed and any output holds a meaningful value.
unknown_k
fu_unknown_k
No reason was reported, or one this build does not name.
bad_alloc_k
fu_bad_alloc_k
An allocation or mapping failed; a smaller request may succeed.
capacity_exhausted_k
fu_capacity_exhausted_k
A fixed ceiling was reached, so unlike bad_alloc_k a smaller request will not help.
invalid_argument_k
fu_invalid_argument_k
Any argument fault - a zero count, an out-of-range index, a byte count that would wrap - as one code, because the remedy is the same.
config_mismatch_k
fu_config_mismatch_k
The handles or the pool kind cannot serve this call together.
already_spawned_k
fu_already_spawned_k
The pool is already spawned; terminate it first.
not_spawned_k
fu_not_spawned_k
The pool was never spawned; spawn it first.
thread_refused_k
fu_thread_refused_k
The OS declined to create a thread - a resource limit, or permissions.
topology_unavailable_k
fu_topology_unavailable_k
The machine could not be described; transient if its CPU set changed mid-probe.
permission_denied_k
fu_permission_denied_k
A privileged operation was declined.
unsupported_k
fu_unsupported_k
This build or this machine has no such facility.
library_missing_k
fu_library_missing_k
Binding-only: nothing to load, or a different major version.
symbol_missing_k
fu_symbol_missing_k
Binding-only: the loaded core is missing a symbol.
contended_k
fu_contended_k
The pool is running another dispatch: another caller's, or the one this call runs inside.
Notes
A refused affinity is not here. The pool still partitions work by domain when the kernel declines to pin; all_threads_pinned() is where that is reported.
caller_exclusivity_tDefines the in- and exclusivity of the calling thread in for the executing task.
This enum affects how the join is performed. If the caller is inclusive, 1/Nth of the call will be executed by the calling thread, as opposed to workers, and the join will happen inside of the calling scope.
One bit per facility, and two accessors ask two questions of the same bit. comptime_capabilities() reports whether the code for a facility was built:capability_place_huge_pages_on_domain_k there means we compiled the path that asks the kernel for them. runtime_capabilities() reports whether the facility is present on this machine: the same bit means the kernel is offering them now.
Neither implies the other. A binary that built capability_place_memory_on_domain_k runs well on a single-node box, where the runtime accessor never sets that bit; and a machine with four NUMA nodes reports none of them to a build that left the topology out. A facility is usable here only when both accessors agree: the honest set is comptime_capabilities() & runtime_capabilities().
Values
capabilities_unknown_k
0
No facility at all.
capability_x86_pause_k
1 << 0
The PAUSE spin hint, on every x86 since the Pentium 4: a short pipeline stall that keeps a busy-wait from flooding the load ports and eases the sibling hardware thread.
capability_x86_tpause_k
1 << 1
TPAUSE sleeps the core until a deadline rather than spinning - the WAITPKG feature, the CPUID leaf 7, subleaf 0, ECX bit 5, on Sapphire Rapids, Alder Lake and their successors.
capability_arm64_yield_k
1 << 2
The YIELD hint, on every AArch64: releases the pipeline to a sibling hardware thread and costs nothing where there is none.
capability_arm64_wfet_k
1 << 3
WFET sleeps the core until a deadline or an event on the monitored line - FEAT_WFxT, Armv8.7, read from ID_AA64ISAR2_EL1 on Linux and hw.optional.arm.FEAT_WFxT on Apple.
capability_riscv64_pause_k
1 << 4
The PAUSE spin hint of the Zihintpause extension - encoded as a FENCE every hart accepts, so it is reported on every RISC-V.
capability_riscv64_wrs_k
1 << 5
WRS.STO sleeps the hart until a reservation breaks or a short timeout - the Zawrs extension, attested by the kernel's hwprobe.
capability_os_threads_k
1 << 6
Own the raw OS thread handle instead of a std::thread - the substrate every thread lever below stands on.
capability_topology_k
1 << 7
Enumerate this machine's cores, compute domains, and memory domains - the root every placement needs.
capability_place_threads_by_affinity_k
1 << 8
Bind a thread to a set of cores, choosing where it runs.
capability_place_threads_by_core_class_k
1 << 9
Steer a thread onto a class of core at creation, choosing where it runs.
capability_reschedule_threads_by_class_k
1 << 10
Reclass a thread's scheduler to sleep or wake it, choosing when it runs.
capability_place_memory_on_domain_k
1 << 11
Place a buffer's pages on a chosen memory domain.
capability_place_huge_pages_on_domain_k
1 << 12
Place larger-than-base pages on a chosen memory domain.
capability_huge_transparent_pages_k
1 << 13
The kernel promotes base pages to huge pages on its own.
capability_colocate_pools_on_domain_k
1 << 14
Compile the domain-aware colocated_pool and distributed_pool.
capability_x86_cldemote_k
1 << 15
CLDEMOTE moves a just-written line from this core's private caches toward the shared LLC and retains it there.
capability_arm64_dc_cvac_k
1 << 16
DC CVAC cleans a dirty line to the coherency point, AArch64's nearest thing to a demote: the next claimer's snoop finds a clean line, forcing no dirty intervention.
capability_riscv64_zicbom_k
1 << 17
The kernel enabled user-mode Zicbom cache-block management - senvcfg.CBCFE - attested through hwprobe, the only sound signal, since a compile-time +zicbom proves nothing about the kernel.
capability_x86_cmpccxadd_k
1 << 18
CMPCCXADD, the conditional atomic add - CPUID leaf 7, subleaf 1, EAX bit 7, on Sierra Forest, Diamond Rapids and their successors: a bounded claim as one instruction instead of a compare-exchange loop.
capability_x86_raoint_k
1 << 19
RAO-INT aadd, aand, aor and axor - CPUID leaf 7, subleaf 1, EAX bit 3, on Grand Ridge: no-return atomics executed at the shared cache rather than pulling the line; weakly ordered, so relaxed only.
capability_arm64_lse_k
1 << 20
Armv8.1 FEAT_LSE:swp, cas, ldadd and the no-return st* forms - one instruction per read-modify-write instead of a load-exclusive loop.
capability_arm64_rcpc_k
1 << 21
Armv8.3 FEAT_LRCPC:ldapr, the RCpc acquiring load that needn't wait for the core's earlier release stores the way RCsc ldar may.
capability_riscv64_zacas_k
1 << 22
Zacas:amocas, compare-and-swap as one instruction instead of an lr and sc loop, attested by the kernel's hwprobe.
capability_riscv64_atomic_k
1 << 23
The A extension: lr, sc and the amo* read-modify-writes, the RISC-V baseline every Linux ABI guarantees, so the detector sets it unconditionally.
capability_arm64_prfm_k
1 << 24
The AArch64 PRFM prefetch hints, base ISA that can never fault, so the detector reports them on every AArch64 core.
capability_riscv64_zicbop_k
1 << 25
The Zicbop prefetches, prefetch.r and prefetch.w, encoded as ORI hints every hart accepts, so the detector reports them on every RISC-V core.
capability_riscv64_zabha_k
1 << 26
Zabha: the byte and halfword amo forms, and with Zacas the byte and halfword amocas, attested by the kernel's hwprobe.
This is the shape discriminator, not a capability: capability_colocate_pools_on_domain_k says the colocated and distributed shapes were compiled, while this says which one a pool is.
Values
unknown_k
0
No pool constructed - the shape is undecided until a spawn builds one.
flat_k
No domain awareness - one worker set over all cores, as if the machine were uniform.
colocated_k
Pinned to a single compute domain, so its workers share a cache and a memory domain.
distributed_k
Spans every compute domain, replicating hot state per memory domain.
limited_arrayA fixed-capacity array with inline storage, so it never allocates.
template <typename value_type_,
std::size_t capacity_>
class limited_array
Sized for the small, bounded lists a machine hands us - the huge page sizes of a NUMA node, the processor groups of a Windows box. Refuses to grow past capacity_k rather than truncating in silence, because a list quietly cut short is a topology quietly misreported.
Fields
capacity_k
std::size_t
See also
dynamic_array when the count is only known at runtime.
dynamic_padded_arrayA dynamic_array with elements at a caller-chosen stride, each on its own cache line.
template <typename object_type_,
typename allocator_type_>
class dynamic_padded_array
Move-only and owning like std::unique_ptr<T[]>, but it spaces objects by stride() bytes rather than sizeof(T) and honours over-alignment, so an alignas(128) pool cell never false-shares and never lands in under-aligned storage. It also accepts an allocate_at_least allocator, the way the NUMA backends hand back more bytes than asked.
See also
dynamic_array for the packed counterpart at natural alignment.
tasks_rangeA half-open [first, first + count) run of task indices, the "what work" of a dispatch.
Iterable, so a callback reads for (auto task : range) rather than rebuilding the bounds. The exclusive end is first + count; an idle thread receives a range with count == 0.
Fields
first
index_t
The first task index in the run.
count
index_t
How many tasks the run covers; zero means an idle thread.
indexed_splitSplits a range of tasks into fair-sized runs for each thread.
The first (tasks % threads) runs have size ceil(tasks / threads). The remaining tasks - (tasks % threads) runs have size floor(tasks / threads). Has the convenient added property that the difference between the largest and smallest run size is at most 1, which can be used in some ordering algorithms.
A single shared counter serializes an entire dispatch: only one core may own its cache line at a time, so no dispatch retires tasks faster than that line circulates. Handing every thread its own cursor turns the common claim into an uncontended read-modify-write on a line nobody else touches, which is roughly fifty times cheaper. The line is only shared once a thread runs dry and starts helping a neighbour, which is exactly when the extra cost is worth paying.
Pad these to a full cache line - two cursors sharing a line would reintroduce the very traffic the split exists to avoid.
Fields
next
std::atomic<index_type_>
Next task in this slice; only ever grows, and may overshoot end by threads.
end
index_type_
One past this slice's last task; written once before the dispatch, then read-only.
See also
invoke_for_n_dynamic hands each thread a slice; idle threads drain their neighbours'.
template <typename pool_type_,
typename fork_type_,
typename index_type_>
class invoke_for_n_dynamic
Scheduling Logic
Tasks are split into one contiguous slice per thread. A thread first drains its own slice, then walks the others in a coprime_permutation_range order and drains theirs, one task per claim. Claiming one task at a time is what preserves the makespan guarantee of greedy list scheduling: a thread can never be handed a batch of tasks that turn out to be expensive, because it is never handed a batch. Claiming from a private cursor is what makes that guarantee affordable.
Probing the neighbours in a coprime order rather than linearly keeps two drained threads from descending on the same victim, which would serialize them on one line for no reason.
Overflow Considerations
If we run a default for-loop at 1 Billion times per second on a 64-bit machine, then every 585 years of computational time we will wrap around the std::size_t capacity for the task index. In case we n + thread >= std::size_t(-1), a simple condition won't be enough. Alternatively, we can make sure, that each thread can do at least one increment of a cursor without worrying about the overflow. The way to achieve that is to preprocess the trailing threads of elements externally, before entering this loop!
That trailing reservation also bounds the cursors. Every thread visits a given slice exactly once - each thread's coprime walk is a permutation of all the slices, beginning with its own - and a visit overshoots by at most one increment, since drain_claim leaves the moment it reads >= end and its read-only probe skips already-drained slices without any increment at all. A cursor therefore never passes end + threads.
Two regimes bound that. When n > threads the last slice ends at n - threads, so no cursor passes n. When n <= threads every slice is empty and end == 0, so no cursor passes threads, which may exceed n but must be representable anyway for the pool to have spawned. Either way max(cursor) == max(n, threads), and no index type can wrap.
See also
invoke_distributed_for_n_dynamic, which applies the same trick one level up, across compute domains, so a thread exhausts local work before touching a remote node's memory.
dummy_pool_tA zero-setup thread-pool that runs every task on the calling thread.
A drop-in serial executor - it satisfies is_pool and is_unsafe_pool and offers the same scheduling surface as flat_pool, namely for_threads, for_n, for_n_dynamic and for_slices, but with one thread on one compute domain, no spawn, and no allocation. unsafe_for_threads runs the fork synchronously as thread 0; everything else composes through broadcast_join exactly as the real pools do. Useful as a serial baseline and as the default executor for the domain-aware containers.
broadcasted_noop_tDoes nothing on every thread, the default fork for a broadcast_join needing only a join.
dynamically resize: all threads must be stopped and re-initialized to grow/shrink.
re-enter: a dispatch from inside its own fork, or beside another caller's, is refused.
copy/move: the threads depend on the address of the parent structure.
handle exceptions: you must try-catch them yourself and return void.
stop early: assuming the user can do it better, knowing the task granularity.
overflow: as all APIs are aggressively tested with smaller index types.
This allows this thread-pool to be extremely lightweight and fast, without heap allocations and no expensive abstractions. It only uses std::thread and std::atomic, but avoids std::function, std::future, std::promise, std::condition_variable, that bring unnecessary overhead.
Repeated operations are performed with a "weak" memory model, to leverage in-hardware support for atomic fence-less operations on Arm and IBM Power architectures. Most atomic counters use the "acquire-release" model, and some going further to "relaxed" model.
A minimal example, similar to #pragma omp parallel in OpenMP:
#include <cstdio>// `std::printf`#include <cstdlib>// `EXIT_FAILURE`, `EXIT_SUCCESS`#include <forkunion.hpp>// `flat_pool_t`namespace fu = ashvardanian::forkunion;
int main() {
fu::flat_pool_t pool; // ? Alias to `fu::flat_pool<>` templateif (fu::failed(pool.spawn(fu::allowed_cores_count()))) return EXIT_FAILURE;
pool.for_threads([](std::size_t i) noexcept { std::printf("Hi from thread %zu\n", i); });
return EXIT_SUCCESS;
}
Unlike OpenMP, however, separate thread-pools can be created isolating work and resources. This is handy when some logic has to be split between "performance" & "efficiency" cores, between different NUMA nodes, between GUI and background tasks, etc. It may look like this:
#include <cstdio>// `std::printf`#include <cstdlib>// `EXIT_FAILURE`, `EXIT_SUCCESS`#include <forkunion.hpp>// `flat_pool_t`using fu = ashvardanian::forkunion;
int main() {
fu::flat_pool_t first_pool, second_pool;
if (!first_pool.spawn(2) || !second_pool.spawn(2, fu::caller_exclusive_k)) return EXIT_FAILURE;
auto broadcast = second_pool.for_threads([](std::size_t i) noexcept { poll_ssd(i); });
first_pool.for_threads([](std::size_t i) noexcept { poll_nic(i); });
broadcast.join(); // ! Wait for the second pool to finishreturn EXIT_SUCCESS;
}
Concurrency Model
Three roles interact with a pool:
the dispatcher - the one thread whose dispatch the pool admitted: it dispatches, polls, and joins, while any other dispatch, nested or concurrent, is refused with the zero token; that spawn and terminate overlap no dispatch is contractual and not enforced;
the contributors - threads executing one slice each per generation: all workers, plus the calling thread itself on caller_inclusive_k pools, whose slice runs inside unsafe_join;
the pollers - any threads calling is_complete, which is a read-only probe.
All synchronization is built from three cache-line-aligned atomics and no mutexes: one compare-and-swap per dispatch, one fetch_sub per contributor, and plain loads and stores:
epoch_ - the generation clock: odd while a fork is in flight, even when idle; stepped once by the dispatcher on dispatch and once by the last contributor on completion, so every generation advances it by exactly two, each step a store by the clock's only writer;
threads_to_sync_ - the countdown identifying the last contributor - the only thread allowed to make the completion step; zero exactly when no contributor runs, so a dispatch claims the pool by exchanging it from zero;
mood_ - the lifecycle switch between spinning, sleeping, and exiting workers.
Four synchronization edges keep the non-atomic fork state safe:
claim: the dispatcher exchanges the countdown from zero with an acquire, so the fork state it writes next follows every read of the previous one, then waits out the previous completion step, so a dispatch never races it and generation tokens are always odd;
publish: the dispatcher releases the dispatch step; contributors acquire it and see the fork state and the countdown;
completionchain: every contributor decrements the countdown with acq_rel, chaining each contributor's writes into the last one;
completionedge: the last contributor releases the completion step, so any acquire-load observing it sees all contributors' results - is_complete included.
Returns the number of threads in one NUMA-specific local compute_domain.
Returns
The thread count: 0 if uninitialized, 1 if only the main thread is used.
Notes
This API is not synchronized.
Shape parity with distributed_pool: generic callers - the C ABI's visit and the distributed invokers - call pool.threads_count(domain) on every pool kind.
caller_exclusivityReports if the current calling thread will be used for broadcasts.
Useful in batch processing when we clearly know the next task won't arrive for a while, so power can be saved at the price of syscalls in that dispatch. Falls back to nap where the kernel can't wait on a word.
It may also be used in a high-level Python or JavaScript library offloading some parallel operations to an underlying C++ engine, where latency is irrelevant.
Parameters
wake_up_periodicity_micros
in
Longest a worker stays blocked before re-checking, or 0 to leave the wake to the dispatch alone.
Notes
Callable only between tasks on a single thread; no synchronization runs.
for_slicesDistributes n similar-duration calls between threads in slices, not per index.
Designed for a "balanced" workload where all threads do roughly equal work, and fork runs n times, with each thread receiving a slice of consecutive tasks.
The callback reference, receiving the thread index as an argument.
Returns
A generation_t token identifying this dispatch, or the zero token when the pool refused it: another dispatch runs, another caller's or the one this call runs inside.
The claim cursor must not share a line with anything, or the dynamic scheduler reintroduces the very coherence traffic that giving each thread a private cursor exists to remove. Rather than allocate a second array beside std::thread, both live in one padded cell, so the pool still performs exactly one allocation - in spawn, never on a dispatch path.
Cells are indexed by threadindex, so on inclusive pools cell 0 belongs to the caller and holds no std::thread, costing a cell but giving claim and worker one index.
The worker thread; default-constructed, and left so for the caller's own cell.
Notes
Separation comes from the buffer's stride, not from an alignas on this type. A std::allocator only promises __STDCPP_DEFAULT_NEW_ALIGNMENT__, so over-aligning the cell would placement-new it into storage that cannot satisfy the request.
Structbroadcast_joinA synchronization point that waits for all threads to finish the last fork.6
The fork is dispatched at construction, so the workers start immediately and the caller may overlap its own work. The join call, or the destructor, waits for completion, and on caller_inclusive_k pools first runs the calling thread's own slice, so only caller_exclusive_k pools may poll is_complete before it.
The pool admits one dispatch at a time, so it refuses this one while another runs: another caller's, or the one whose fork this broadcast is created in. A refused broadcast runs nothing, joins nothing, keeps generation at zero, and is never retried, not even by the destructor.
You don't have to explicitly handle the return value and wait on it. According to the C++ standard, the destructor of the broadcast_join will be called in the end of the for_threads-calling expression.
The object is immovable: the pool holds a pointer to the fork_ member for the lifetime of the broadcast, so the object must never relocate. Guaranteed C++17 copy elision still allows returning it by value from for_threads.
Notes
You don't have to explicitly call any of the APIs, it's like std::jthread ;)
The fall-through sits after the switch rather than in a default: label, so adding an enumerator without a text here is a -Wswitch warning instead of silent prose.
capability_nameLower-case name of one capability bit, or nullptr for composite and unknown ones.
0 where the product would wrap, which every caller already refuses alongside a zero-element request - so the guard and the arithmetic cannot drift apart.
copy_bytesByte-wise memcpy of from into to, written as an explicit loop.
Not std::memcpy: the wait monitors that call this - arm64_wfet_t and riscv64_wrs_t - are pinned to a narrower target(...) than -march=native, where the fortified always_inlinememcpy cannot inline - "target specific option mismatch". Reading and writing through unsigned char keeps it well-defined.
min_ofThe smaller of first and second, without <algorithm> or the windows.hmin macro.
A counter-based generator instead of a stateful one: every draw is independent, so parallel consumers need no shared state, and a sequence is reproducible from indices alone. The same constants drive the benchmark generators in bench/, ported bit-identically to Rust and Zig.
dense_rankDense-ranks count items by an ascending integer key, writing each item's 0-based rank.
key(index) must read a stable source and assign(index, rank) write a different field, so ranking in place never corrupts a not-yet-ranked item whose key repeats. Used to turn raw CPU capacities into compute levels and raw memory tiers into memory levels.
Returns
The number of distinct keys, at least 1 when count > 0.
drain_claimDrains whatever is left of one thread's slice into fork - the shared core of both the flat and the distributed for_n_dynamic invokers, so its invariants live in one place.
Drains what is left of the slice thread's claim in pool, whether or not we own it.
A read-only probe first: a drained slice is skipped in Shared state - no dirtying add, no line migration - which is the entire cost of visiting an empty neighbour once a small dispatch runs dry. A live slice is promoted with write intent while the probe's branch resolves, claimed one task at a time - the makespan guarantee of greedy list scheduling - and demoted once our overshooting add has dirtied it, so the next visitor snoops the LLC instead of this core.
See also
The dynamic_claim overload above, where the probe and the overshoot invariants live.
cpu_capabilitiesThe instruction-level bits this CPU and its kernel offer: the spin hint, the monitored wait, the cache-line hint and the atomics of the ISA in hand.
Consult it before reaching for a domain-aware API. Without capability_colocate_pools_on_domain_k there is no colocated_pool to spawn and no linux_numa_allocator to construct, and a caller has no other way to tell that apart from a machine that merely has one compute domain.
These are the same facility bits runtime_capabilities reports, asked the other way: this says the code was compiled, that says the machine offers it, and a facility is usable only where both agree.
That however results into all kinds of ABI warnings with GCC, and suboptimal alignment choice, unless you hard-code –param hardware_destructive_interference_size=64 or disable the warning with -Wno-interference-size.
So the width is FORKUNION_DEFAULT_ALIGNMENT, picked from the target. Only alignas rests on this guess; a per-thread stride takes destructive_interference_bytes(), which asks the machine.
The cursors live in the pool, shared by every dispatch, so a refused one must not touch them. Matched by type rather than by a member name, so no user fork is ever asked to rewind.