# NUMA > ForkUnion for C++ Placing memory on a chosen domain is implemented on Linux via the `mbind` syscall, on FreeBSD via `domainset` policies, and on Windows via `VirtualAllocExNuma`, and degrades to a single domain elsewhere. The portable `domain_allocator_t` alias picks the right backend per platform - `linux_numa_allocator_t`, `freebsd_numa_allocator_t`, `windows_numa_allocator_t`, or a plain aligned fallback - each an STL-compatible allocator bound to one `memory_domain_id_t`. Let's say you are building a Big Data application — brute-forcing Vector Search with the [NumKong](/docs/numkong/) library — on a 2 dual-socket CPU system, the kind that powers production [USearch](https://github.com/unum-cloud/USearch/pulls) deployments. The first part of that program may be responsible for sharding the incoming stream of data between distinct memory regions. That part, in our simple example will be single-threaded: ```cpp #include // `std::vector` #include // `std::span` #include // `linux_numa_allocator`, `machine_topology_t`, `distributed_pool_t` #include // `nk_angular_f32`, `nk_f64_t` namespace fu = ashvardanian::forkunion; using floats_alloc_t = fu::linux_numa_allocator; constexpr std::size_t dimensions = 768; /// Matches most BERT-like models static std::vector first_half(floats_alloc_t(fu::memory_domain_id_t {0})); static std::vector second_half(floats_alloc_t(fu::memory_domain_id_t {1})); static fu::machine_topology_t machine_topology; static fu::distributed_pool_t distributed_pool; /// Dynamically shards incoming vectors across 2 nodes in a round-robin fashion. void append(std::span vector) { bool put_in_second = first_half.size() > second_half.size(); if (put_in_second) second_half.insert(second_half.end(), vector.begin(), vector.end()); else first_half.insert(first_half.end(), vector.begin(), vector.end()); } ``` The concurrent part would involve spawning threads adjacent to every memory pool to find the best `search_result_t`. The primary `search` function, in ideal world would look like this: 1. Each thread finds the best match within its "slice" of a compute domain, tracking the best distance and index in a local CPU register. 2. All threads in each compute domain atomically synchronize using a domain-local instance of `search_result_t`. 3. The main thread collects aggregates of partial results from all compute domains. That is, however, overly complicated to implement. Such tree-like hierarchical reductions are optimal in a theoretical sense. Still, assuming the relative cost of spin-locking once at the end of a thread scope and the complexity of organizing the code, the more straightforward path is better. A minimal example would look like this: ```cpp /// On each compute domain we'll synchronize the threads struct search_result_t { nk_f64_t best_distance {std::numeric_limits::max()}; std::size_t best_index {0}; }; inline search_result_t pick_best(search_result_t const& a, search_result_t const& b) noexcept { return a.best_distance < b.best_distance ? a : b; } /// Uses all CPU threads to search for the closest vector to the @p query. search_result_t search(std::span query) { bool const need_to_spawn_threads = distributed_pool.threads_count() == 0; if (need_to_spawn_threads) { fu::status_t const harvested = machine_topology.harvest(); assert(fu::succeeded(harvested) && "Failed to harvest NUMA topology"); assert(machine_topology.memory_domains_count() == 2 && "Expected exactly 2 NUMA nodes"); fu::status_t const spawned = distributed_pool.spawn(machine_topology); assert(fu::succeeded(spawned) && "Failed to spawn NUMA pools"); } search_result_t result; fu::spin_mutex_t result_update; // ? Lighter `std::mutex` alternative w/out futexes std::size_t const first_vectors = first_half.size() / dimensions; std::size_t const total_vectors = first_vectors + second_half.size() / dimensions; auto slices = distributed_pool.for_slices(total_vectors, [&](fu::tasks_range_t tasks, fu::thread_in_domain_t) noexcept { search_result_t thread_local_result; for (std::size_t const task : tasks) { bool const in_second = task >= first_vectors; auto const &shard = in_second ? second_half : first_half; std::size_t const local_index = in_second ? task - first_vectors : task; nk_f64_t distance; nk_angular_f32(query.data(), shard.data() + local_index * dimensions, dimensions, &distance); thread_local_result = pick_best(thread_local_result, {distance, task}); } // ! Still synchronizing over a shared mutex for brevity. std::lock_guard lock(result_update); result = pick_best(result, thread_local_result); }); slices.join(); return result; } ``` In a dream world, we would call `distributed_pool.for_n`, but there is no clean way to make the scheduling processes aware of the data distribution in an arbitrary application, so that's left to the user. The `for_slices` helper pairs each run with a `fu::thread_in_domain_t` naming its compute domain, but slices follow the thread count rather than the shard boundary, so the callback above picks the shard per task. For more flexibility around building higher-level low-latency systems, there are unsafe APIs expecting you to manually "join" the broadcasted calls: `unsafe_for_threads` returns a generation token, `is_complete` polls it without blocking, and `unsafe_join` blocks until that generation completes. The manual two-vector sharding above is what the __symmetric allocators__ automate: `symmetric_memory_allocator_t` - `fu_allocate_symmetric` in C - maps one virtual range striped a slice per memory domain, and Rust, Zig, and Mojo wrap it as `ShardedArray`, one shard per domain, and `ReplicatedArray`, a full copy per domain for read-mostly data. To place and coordinate pools by hand, `spawn_on` - `fu_pool_spawn_on` in C - pins a pool to a single compute domain, while `locate_thread_in` and `threads_count_in` map a global thread index to its domain and count the workers living there. `#include ` `#include ` ## allocation_result Back-ports the C++ 23 [`std::allocation_result`](#allocation_result). ```cpp template struct allocation_result ``` Unlike STL, also contains the page size. - `ptr` `pointer_type`: Pointer to the allocated memory, or nullptr if allocation failed. - `count` `size_type`: Number of elements allocated, or 0 if allocation failed. - `bytes` `size_type`: Total volume of memory allocated, in bytes. - `pages` `size_type`: Number of memory pages allocated. See also: [https://en.cppreference.com/w/cpp/memory/allocator/allocate_at_least](https://en.cppreference.com/w/cpp/memory/allocator/allocate_at_least) ## symmetric_allocation_result Result of a __symmetric__ allocation, one mapping with equal-stride slices per domain. ```cpp template struct symmetric_allocation_result ``` Slice __d__ begins at the byte address `ptr + d * stride_bytes` - see `slice` - and holds `count` elements. Every slice is a uniform distance `stride_bytes` apart - the CPU analog of a GPU symmetric heap. The stride is in bytes, not elements, because it is page-aligned and a page rarely divides `sizeof(value_type)`. - `ptr` `pointer_type`: Base of the mapping, or nullptr on failure. - `count` `size_type`: Usable elements per domain slice - what was requested. - `stride_bytes` `size_type`: Page-aligned __byte__ distance between consecutive slice bases; `>= count * sizeof(value_type)`. - `domains` `size_type`: Number of domain slices. - `bytes` `size_type`: Total mapped volume in bytes - `domains * stride_bytes`. - `pages` `size_type`: Number of memory pages mapped across all slices. ## linux_symmetric_allocator Allocator for a __symmetric__ mapping - one range striped across every memory domain. ```cpp template struct linux_symmetric_allocator ``` A sibling of `[linux_numa_allocator](#linux_numa_allocator)`, but instead of a block on a single node it returns a `[symmetric_allocation_result](#symmetric_allocation_result)` carrying the base pointer __and__ the per-slice `stride_bytes`, so slice `d` of element `i` lives at `ptr + d * stride_bytes + i * sizeof(value_type)`. It borrows the [machine_topology](https://www.unum.cloud/docs/forkunion/cpp/topology.md#machine_topology) it stripes across; hand it the one you harvested and let it outlive the whole mapping. ## freebsd_numa_allocator STL-compatible allocator bound to a single memory domain on FreeBSD. ```cpp template struct freebsd_numa_allocator ``` Mirrors `[linux_numa_allocator](#linux_numa_allocator)'s` huge-page ladder, but every placement is a thread-policy PREFER plus a first-touch rather than a per-mapping `mbind`, and larger pages are the `MAP_ALIGNED_SUPER` hint. See also: [linux_numa_allocator](#linux_numa_allocator) is the Linux counterpart; both satisfy the pool's allocator needs. ## freebsd_symmetric_allocator Allocator for a __symmetric__ mapping on FreeBSD: one range striped across every domain. ```cpp template struct freebsd_symmetric_allocator ``` See also: [linux_symmetric_allocator](#linux_symmetric_allocator) is the Linux counterpart, borrowing its [machine_topology](https://www.unum.cloud/docs/forkunion/cpp/topology.md#machine_topology). ## windows_numa_allocator STL-compatible allocator bound to a single memory domain on Windows, backed by `VirtualAllocExNuma`. ```cpp template struct windows_numa_allocator ``` Deliberately plainer than the Linux allocator: it exposes only [`allocate`](#allocate) and `deallocate`, so `[dynamic_padded_array](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#dynamic_padded_array)` takes its ordinary `allocate(total)` path rather than the sized [`allocate_at_least`](#allocate_at_least) one. Large pages are opt-in per allocator instance through the `large_pages` ctor flag; they need `SeLockMemoryPrivilege` first, see See also: [linux_numa_allocator](#linux_numa_allocator) is the Linux counterpart; both satisfy the pool's allocator needs. See also: `[windows_enable_lock_memory_privilege](#windows_enable_lock_memory_privilege)`, and round every request up to `GetLargePageMinimum()`. The pool constructs the allocator without them, since its own state is small; a caller wanting large pages for bulk data opts in explicitly. ## windows_symmetric_allocator Allocator for a __symmetric__ mapping on Windows - one reservation committed per node. ```cpp template struct windows_symmetric_allocator ``` Reserves a range and commits each equal-stride slice on its own node via `VirtualAllocExNuma`. Slice `d` of element `i` then lives at ptr + d × stride_bytes + i × sizeof(value_type). Base pages only - large pages cannot be reserved then committed in slices. Borrows the [machine_topology](https://www.unum.cloud/docs/forkunion/cpp/topology.md#machine_topology) it stripes across, which the caller must keep alive. See also: [linux_symmetric_allocator](#linux_symmetric_allocator) is the Linux counterpart. ## portable_aligned_allocator A heap-backed, cache-line-aligned stand-in for NUMA allocators, where no domains exist. ```cpp template struct portable_aligned_allocator ``` Mirrors `[linux_numa_allocator](#linux_numa_allocator)'s` interface so `domain_allocator_t` stays uniform: it ignores the memory domain and reports the base page size, letting the domain-aware code paths compile and run everywhere while degenerating to plain heap allocation. Every block is aligned to at least [`default_alignment_k`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#default_alignment_k), so the NUMA backends' page alignment and this fallback offer one alignment contract - over-aligned element types need no caller-side padding on any platform. ## portable_symmetric_allocator A heap-backed stand-in for the symmetric allocators, where domains cannot be bound. ```cpp template struct portable_symmetric_allocator ``` Mirrors their `allocate_at_least(size) → [symmetric_allocation_result](#symmetric_allocation_result)` shape so the containers stay uniform, but the one contiguous block is merely `default_alignment_k-aligned` - no `mbind`, no `VirtualAllocExNuma`. The slice count still comes from the [machine_topology](https://www.unum.cloud/docs/forkunion/cpp/topology.md#machine_topology), so on a machine the platform reports as single-domain the mapping is one stride wide and the same addressing `ptr + d * stride_bytes + i * sizeof(value_type)` holds everywhere. Borrows the topology; hand it the one you harvested and let it outlive the mapping. See also: [linux_symmetric_allocator](#linux_symmetric_allocator) / [windows_symmetric_allocator](#windows_symmetric_allocator) are the domain-pinning siblings. ## linux_numa_allocator STL-compatible allocator bound to a single memory domain, prioritizing huge pages. ```cpp template struct linux_numa_allocator ``` A light-weight, but high-latency BLOB allocator, tied to a specific NUMA node ID: every allocation is one `mmap` plus `mbind` syscall, aligned to at least a 4 KB page size. #### C++ 23 Functionality Whenever possible, the newer [`allocate_at_least`](#allocate_at_least) API reduces the number of reallocations. ## linux_numa_allocator ```cpp constexpr linux_numa_allocator() noexcept=default; ``` ```cpp constexpr linux_numa_allocator( memory_domain_id_t id, size_type paging = ram_page_size()) noexcept; ``` ```cpp template constexpr linux_numa_allocator( linux_numa_allocator const & o) noexcept; ``` ## allocate_at_least Allocates memory for at least `size` elements of `value_type`. ```cpp allocation_result allocate_at_least( size_type size, size_type page_size_bytes) noexcept; ``` ```cpp allocation_result allocate_at_least( size_type size) noexcept; ``` - `size` (in): The number of elements to allocate. - `page_size_bytes` (in): The size of the memory page to allocate, must be a multiple of `sizeof(value_type)`. ## allocate Allocates a memory for `size` elements of `value_type`. ```cpp value_type * allocate( size_type size, size_type page_size_bytes) noexcept; ``` ```cpp value_type * allocate(size_type size) noexcept; ``` Allocates memory for `size` elements of `value_type`. - `size` (in): The number of elements to allocate. - `page_size_bytes` (in): The size of the memory page to allocate, must be a multiple of `sizeof(value_type)`. ## operator== ```cpp template bool operator==( linux_numa_allocator const & o) const noexcept; ``` ## replicated_array One uninitialized replica of a sequence per memory domain - a domain-local buffer. ```cpp template struct replicated_array ``` For read-shared data - graph adjacency, body positions. The logical length is __n__; the footprint is `n * memory_domains_count()`. It owns memory and hands out slices - the caller fills every replica and keeps them coherent. Backed by one symmetric mapping, so replica `d` of element `i` lives at `base() + d * stride_bytes() + i * sizeof(value_type)`. ## replicated_array ```cpp replicated_array() noexcept=default; ``` ```cpp replicated_array(replicated_array && other) noexcept; ``` ```cpp replicated_array(replicated_array const &)=delete; ``` ## *_array ```cpp ~replicated_array() noexcept; ``` - `~replicated_array` - `~sharded_array` ## operator= ```cpp replicated_array & operator=(replicated_array && other) noexcept; ``` ```cpp replicated_array & operator=(replicated_array const &)=delete; ``` ## * Frees the mapping. ```cpp void reset() noexcept; ``` Elements are raw storage, so no destructors run. ## resize_uninitialized Allocates one uninitialized length `n` replica per memory domain; the caller first-touches them. ```cpp status_t resize_uninitialized( machine_topology_t const & topology, std::size_t n) noexcept; ``` ## at One element of the replica on `[memory_domain](https://www.unum.cloud/docs/forkunion/cpp/topology.md#memory_domain)`. ```cpp value_type_ & at( memory_domain_index_t memory_domain, std::size_t local_index) noexcept; ``` ```cpp value_type_ const & at( memory_domain_index_t memory_domain, std::size_t local_index) const noexcept; ``` ## on_memory_domain The whole replica living on `[memory_domain](https://www.unum.cloud/docs/forkunion/cpp/topology.md#memory_domain)`. ```cpp span_type on_memory_domain( memory_domain_index_t memory_domain) noexcept; ``` ```cpp view_type on_memory_domain( memory_domain_index_t memory_domain) const noexcept; ``` ## sharded_array A domain-local buffer - a sequence partitioned once across every memory domain. ```cpp template struct sharded_array ``` For datasets too large to replicate. The logical length is __n__; the footprint is `n`. Each domain owns a __contiguous__ logical segment of `segment() == ceil(n / memory_domains_count())` elements, so element __i__ lives at `{i / segment(), i % segment()}` - see [`location_of`](#location_of) - and a sequential scan of a domain's shard is sequential in memory. It owns memory and hands out slices; the caller fills them. Backed by one symmetric mapping whose uniform stride is that segment, so a shorter trailing shard leaves slots unused. ## sharded_array ```cpp sharded_array() noexcept=default; ``` ```cpp sharded_array(sharded_array && other) noexcept; ``` ```cpp sharded_array(sharded_array const &)=delete; ``` ## operator= ```cpp sharded_array & operator=(sharded_array && other) noexcept; ``` ```cpp sharded_array & operator=(sharded_array const &)=delete; ``` ## resize_uninitialized Allocates uninitialized storage for `n` elements in contiguous per-domain segments. ```cpp status_t resize_uninitialized( machine_topology_t const & topology, std::size_t n) noexcept; ``` ## segment The contiguous logical segment size per domain - `ceil(n / memory_domains_count())`. ```cpp std::size_t segment() const noexcept; ``` ## length_on_memory_domain How many elements the shard on `[memory_domain](https://www.unum.cloud/docs/forkunion/cpp/topology.md#memory_domain)` holds - a trailing shard may be shorter. ```cpp std::size_t length_on_memory_domain( memory_domain_index_t memory_domain) const noexcept; ``` ## location_of The memory domain and local index that store logical element `logical_index`. ```cpp location_t location_of(std::size_t logical_index) const noexcept; ``` Notes: Requires a non-empty array - `segment()` is the divisor and is zero when empty. ## logical_index_of The logical index of the element at `local_index` on `[memory_domain](https://www.unum.cloud/docs/forkunion/cpp/topology.md#memory_domain)` - inverse of [`location_of`](#location_of). ```cpp std::size_t logical_index_of( memory_domain_index_t memory_domain, std::size_t local_index) const noexcept; ``` ## at The single home of a logical element - freely mutable, unlike a replica. ```cpp value_type_ & at( memory_domain_index_t memory_domain, std::size_t local_index) noexcept; ``` ```cpp value_type_ const & at( memory_domain_index_t memory_domain, std::size_t local_index) const noexcept; ``` ## on_memory_domain The whole shard living on `[memory_domain](https://www.unum.cloud/docs/forkunion/cpp/topology.md#memory_domain)`. ```cpp span_type on_memory_domain( memory_domain_index_t memory_domain) noexcept; ``` ```cpp view_type on_memory_domain( memory_domain_index_t memory_domain) const noexcept; ``` ## location_t Where a logical element lives: which memory domain and its index within that shard. ```cpp struct location_t ``` - `memory_domain` `memory_domain_index_t` - `local_index` `std::size_t` ## linux_numa_bind Tries binding the given address range to a specific NUMA [`memory_domain_id`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#memory_domain_id). ```cpp status_t linux_numa_bind( void *ptr, std::size_t size_bytes, memory_domain_id_t memory_domain_id) noexcept; ``` ## linux_numa_allocate Tries allocating and binding uninitialized memory to a specific NUMA [`memory_domain_id`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#memory_domain_id). ```cpp void * linux_numa_allocate( std::size_t size_bytes, std::size_t page_size_bytes, memory_domain_id_t memory_domain_id) noexcept; ``` ## linux_symmetric_allocate Maps one range of `domains * stride_bytes` and binds slice `d` to `domain_ids[d]`. ```cpp void * linux_symmetric_allocate( machine_topology_t const & topology, std::size_t stride_bytes, std::size_t page_size_bytes) noexcept; ``` Unlike [`linux_numa_allocate`](#linux_numa_allocate), which places a block on a single node, this reserves __one__ contiguous virtual range and `mbinds` each equal-stride slice to its own node - a symmetric layout where the MMU then serves every slice from its local memory. ## freebsd_domain_restore Restores the calling thread's memory-domain policy saved by [`freebsd_domain_prefer`](#freebsd_domain_prefer). ```cpp void freebsd_domain_restore( void const *saved_set, int saved_policy, bool have_saved) noexcept; ``` Notes: A no-op where the save failed, so a refused `getdomain` never widens the policy. ## freebsd_first_touch First-touches every page in [ `ptr`, `ptr` + `size_bytes` ) so PREFER binds them. ```cpp void freebsd_first_touch(void *ptr, std::size_t size_bytes) noexcept; ``` Notes: Anonymous pages are already zero, so writing a zero changes nothing but the residency. ## freebsd_domain_prefer Points the calling thread's `domainset` at [`memory_domain_id`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#memory_domain_id) with a PREFER policy. ```cpp bool freebsd_domain_prefer( memory_domain_id_t memory_domain_id, void *saved_set, int *saved_policy) noexcept; ``` FreeBSD 12 removed `numa_setaffinity`, so there is no per-allocation `numa_alloc_onnode:` the memory domain is a __thread__ policy. The caller sets it, first-touches the region so its pages fault on the preferred domain, then restores the saved policy - the domainset analogue of Linux's per-mapping `mbind`. ## freebsd_domain_allocate Tries allocating and placing uninitialized memory on a specific NUMA [`memory_domain_id`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#memory_domain_id). ```cpp void * freebsd_domain_allocate( std::size_t size_bytes, std::size_t page_size_bytes, memory_domain_id_t memory_domain_id) noexcept; ``` Larger-than-base page sizes ask for `MAP_ALIGNED_SUPER` - an alignment hint, since FreeBSD's superpages are transparent and have no `MAP_HUGETLB-style` pool. It is best-effort: a failed super-aligned map retries with base pages so the allocator stays total, matching the Linux `nullptr-on-unsupported` contract. See also: [linux_numa_allocate](#linux_numa_allocate) is the Linux counterpart; FreeBSD places by thread policy, not `mbind`. ## freebsd_symmetric_allocate Maps `domains * stride_bytes` and PREFER-touches slice `d` on `domain_ids[d]`. ```cpp void * freebsd_symmetric_allocate( machine_topology_t const & topology, std::size_t stride_bytes, std::size_t page_size_bytes) noexcept; ``` See also: [linux_symmetric_allocate](#linux_symmetric_allocate) is the Linux counterpart; FreeBSD places slices by thread policy. ## windows_enable_lock_memory_privilege Enables `SeLockMemoryPrivilege` for the process, needed before large-page allocation. ```cpp status_t windows_enable_lock_memory_privilege() noexcept; ``` The account must first be granted "Lock pages in memory" - Local Security Policy / `SeLockMemoryPrivilege` - typically by an admin. Call once at start-up, then construct a `[windows_numa_allocator](#windows_numa_allocator)` with `large_pages = true`. Notes: This only __enables__ a privilege the account already holds. ## windows_numa_allocate Allocates uninitialized memory placed on a specific NUMA [`memory_domain_id`](https://www.unum.cloud/docs/forkunion/cpp/quickstart.md#memory_domain_id) on Windows. ```cpp void * windows_numa_allocate( std::size_t size_bytes, memory_domain_id_t memory_domain_id, bool large_pages = false) noexcept; ``` `VirtualAllocExNuma` reserves and commits a range whose pages the kernel will fault in on the requested node - the Windows analogue of Linux's `mbind`, folded into the allocation call. Large pages need `SeLockMemoryPrivilege`, which [`windows_enable_lock_memory_privilege`](#windows_enable_lock_memory_privilege) grants, and can still fail under memory fragmentation; a caller wanting a soft failure should retry with `large_pages` false. - `size_bytes` (in): Number of bytes to allocate. - `memory_domain_id` (in): The NUMA node to place the memory on. - `large_pages` (in): Request `MEM_LARGE_PAGES`; the size is rounded up to `GetLargePageMinimum()`. ## windows_symmetric_allocate Reserves one range of `domains * stride_bytes` and commits slice `d` on its own node. ```cpp void * windows_symmetric_allocate( machine_topology_t const & topology, std::size_t stride_bytes) noexcept; ``` The Windows analogue of `linux_symmetric_allocate:` one `VirtualAlloc(MEM_RESERVE)` for the whole range, then a per-slice `VirtualAllocExNuma(MEM_COMMIT, node)`, so each slice's pages fault in on its own node. Base pages only - large pages cannot be reserved and committed across two separate mapping steps.