Placing memory on a chosen domain is implemented on Linux via the mbind syscall, on FreeBSD via domainset policies, and on Windows via VirtualAllocExNuma, and degrades to a single domain elsewhere.
The portable domain_allocator_t alias picks the right backend per platform - linux_numa_allocator_t, freebsd_numa_allocator_t, windows_numa_allocator_t, or a plain aligned fallback - each an STL-compatible allocator bound to one memory_domain_id_t.
Let’s say you are building a Big Data application — brute-forcing Vector Search with the NumKong library — on a 2 dual-socket CPU system, the kind that powers production USearch deployments. The first part of that program may be responsible for sharding the incoming stream of data between distinct memory regions. That part, in our simple example will be single-threaded:
#include <vector> // `std::vector`
#include <span> // `std::span`
#include <forkunion.hpp> // `linux_numa_allocator`, `machine_topology_t`, `distributed_pool_t`
#include <numkong/spatial.h> // `nk_angular_f32`, `nk_f64_t`
namespace fu = ashvardanian::forkunion;
using floats_alloc_t = fu::linux_numa_allocator<float>;
constexpr std::size_t dimensions = 768; /// Matches most BERT-like models
static std::vector<float, floats_alloc_t> first_half(floats_alloc_t(fu::memory_domain_id_t {0}));
static std::vector<float, floats_alloc_t> second_half(floats_alloc_t(fu::memory_domain_id_t {1}));
static fu::machine_topology_t machine_topology;
static fu::distributed_pool_t distributed_pool;
/// Dynamically shards incoming vectors across 2 nodes in a round-robin fashion.
void append(std::span<float, dimensions> vector) {
bool put_in_second = first_half.size() > second_half.size();
if (put_in_second) second_half.insert(second_half.end(), vector.begin(), vector.end());
else first_half.insert(first_half.end(), vector.begin(), vector.end());
}The concurrent part would involve spawning threads adjacent to every memory pool to find the best search_result_t.
The primary search function, in ideal world would look like this:
search_result_t.That is, however, overly complicated to implement. Such tree-like hierarchical reductions are optimal in a theoretical sense. Still, assuming the relative cost of spin-locking once at the end of a thread scope and the complexity of organizing the code, the more straightforward path is better. A minimal example would look like this:
/// On each compute domain we'll synchronize the threads
struct search_result_t {
nk_f64_t best_distance {std::numeric_limits<nk_f64_t>::max()};
std::size_t best_index {0};
};
inline search_result_t pick_best(search_result_t const& a, search_result_t const& b) noexcept {
return a.best_distance < b.best_distance ? a : b;
}
/// Uses all CPU threads to search for the closest vector to the @p query.
search_result_t search(std::span<float, dimensions> query) {
bool const need_to_spawn_threads = distributed_pool.threads_count() == 0;
if (need_to_spawn_threads) {
fu::status_t const harvested = machine_topology.harvest();
assert(fu::succeeded(harvested) && "Failed to harvest NUMA topology");
assert(machine_topology.memory_domains_count() == 2 && "Expected exactly 2 NUMA nodes");
fu::status_t const spawned = distributed_pool.spawn(machine_topology);
assert(fu::succeeded(spawned) && "Failed to spawn NUMA pools");
}
search_result_t result;
fu::spin_mutex_t result_update; // ? Lighter `std::mutex` alternative w/out futexes
std::size_t const first_vectors = first_half.size() / dimensions;
std::size_t const total_vectors = first_vectors + second_half.size() / dimensions;
auto slices = distributed_pool.for_slices(total_vectors,
[&](fu::tasks_range_t tasks, fu::thread_in_domain_t) noexcept {
search_result_t thread_local_result;
for (std::size_t const task : tasks) {
bool const in_second = task >= first_vectors;
auto const &shard = in_second ? second_half : first_half;
std::size_t const local_index = in_second ? task - first_vectors : task;
nk_f64_t distance;
nk_angular_f32(query.data(), shard.data() + local_index * dimensions, dimensions, &distance);
thread_local_result = pick_best(thread_local_result, {distance, task});
}
// ! Still synchronizing over a shared mutex for brevity.
std::lock_guard<fu::spin_mutex_t> lock(result_update);
result = pick_best(result, thread_local_result);
});
slices.join();
return result;
}In a dream world, we would call distributed_pool.for_n, but there is no clean way to make the scheduling processes aware of the data distribution in an arbitrary application, so that’s left to the user.
The for_slices helper pairs each run with a fu::thread_in_domain_t naming its compute domain, but slices follow the thread count rather than the shard boundary, so the callback above picks the shard per task.
For more flexibility around building higher-level low-latency systems, there are unsafe APIs expecting you to manually “join” the broadcasted calls: unsafe_for_threads returns a generation token, is_complete polls it without blocking, and unsafe_join blocks until that generation completes.
The manual two-vector sharding above is what the symmetric allocators automate: symmetric_memory_allocator_t - fu_allocate_symmetric in C - maps one virtual range striped a slice per memory domain, and Rust, Zig, and Mojo wrap it as ShardedArray<T>, one shard per domain, and ReplicatedArray<T>, a full copy per domain for read-mostly data.
To place and coordinate pools by hand, spawn_on - fu_pool_spawn_on in C - pins a pool to a single compute domain, while locate_thread_in and threads_count_in map a global thread index to its domain and count the workers living there.
allocation_resultBack-ports the C++ 23 std::allocation_result.template <typename pointer_type_ = char,
typename size_type_ = std::size_t>
struct allocation_resultUnlike STL, also contains the page size.
ptr | pointer_type | Pointer to the allocated memory, or nullptr if allocation failed. |
|---|---|---|
count | size_type | Number of elements allocated, or 0 if allocation failed. |
bytes | size_type | Total volume of memory allocated, in bytes. |
pages | size_type | Number of memory pages allocated. |
symmetric_allocation_resultResult of a symmetric allocation, one mapping with equal-stride slices per domain.template <typename value_type_,
typename size_type_ = std::size_t>
struct symmetric_allocation_resultSlice d begins at the byte address ptr + d * stride_bytes - see slice - and holds count elements. Every slice is a uniform distance stride_bytes apart - the CPU analog of a GPU symmetric heap. The stride is in bytes, not elements, because it is page-aligned and a page rarely divides sizeof(value_type).
ptr | pointer_type | Base of the mapping, or nullptr on failure. |
|---|---|---|
count | size_type | Usable elements per domain slice - what was requested. |
stride_bytes | size_type | Page-aligned byte distance between consecutive slice bases; >= count * sizeof(value_type). |
domains | size_type | Number of domain slices. |
bytes | size_type | Total mapped volume in bytes - domains * stride_bytes. |
pages | size_type | Number of memory pages mapped across all slices. |
linux_symmetric_allocatorAllocator for a symmetric mapping - one range striped across every memory domain.template <typename value_type_ = char>
struct linux_symmetric_allocatorA sibling of linux_numa_allocator, but instead of a block on a single node it returns a symmetric_allocation_result carrying the base pointer and the per-slice stride_bytes, so slice d of element i lives at ptr + d * stride_bytes + i * sizeof(value_type). It borrows the machine_topology it stripes across; hand it the one you harvested and let it outlive the whole mapping.
freebsd_numa_allocatorSTL-compatible allocator bound to a single memory domain on FreeBSD.template <typename value_type_ = char>
struct freebsd_numa_allocatorMirrors linux_numa_allocator's huge-page ladder, but every placement is a thread-policy PREFER plus a first-touch rather than a per-mapping mbind, and larger pages are the MAP_ALIGNED_SUPER hint.
linux_numa_allocator is the Linux counterpart; both satisfy the pool's allocator needs.
freebsd_symmetric_allocatorAllocator for a symmetric mapping on FreeBSD: one range striped across every domain.template <typename value_type_ = char>
struct freebsd_symmetric_allocatorlinux_symmetric_allocator is the Linux counterpart, borrowing its machine_topology.
windows_numa_allocatorSTL-compatible allocator bound to a single memory domain on Windows, backed by VirtualAllocExNuma.template <typename value_type_ = char>
struct windows_numa_allocatorDeliberately plainer than the Linux allocator: it exposes only allocate and deallocate, so dynamic_padded_array takes its ordinary allocate(total) path rather than the sized allocate_at_least one. Large pages are opt-in per allocator instance through the large_pages ctor flag; they need SeLockMemoryPrivilege first, see
windows_enable_lock_memory_privilege, and round every request up to GetLargePageMinimum(). The pool constructs the allocator without them, since its own state is small; a caller wanting large pages for bulk data opts in explicitly.windows_symmetric_allocatorAllocator for a symmetric mapping on Windows - one reservation committed per node.template <typename value_type_ = char>
struct windows_symmetric_allocatorReserves a range and commits each equal-stride slice on its own node via VirtualAllocExNuma. Slice d of element i then lives at ptr + d × stride_bytes + i × sizeof(value_type). Base pages only - large pages cannot be reserved then committed in slices.
Borrows the machine_topology it stripes across, which the caller must keep alive.
linux_symmetric_allocator is the Linux counterpart.
portable_aligned_allocatorA heap-backed, cache-line-aligned stand-in for NUMA allocators, where no domains exist.template <typename value_type_ = char>
struct portable_aligned_allocatorMirrors linux_numa_allocator's interface so domain_allocator_t stays uniform: it ignores the memory domain and reports the base page size, letting the domain-aware code paths compile and run everywhere while degenerating to plain heap allocation. Every block is aligned to at least default_alignment_k, so the NUMA backends' page alignment and this fallback offer one alignment contract - over-aligned element types need no caller-side padding on any platform.
portable_symmetric_allocatorA heap-backed stand-in for the symmetric allocators, where domains cannot be bound.template <typename value_type_ = char>
struct portable_symmetric_allocatorMirrors their allocate_at_least(size) → symmetric_allocation_result shape so the containers stay uniform, but the one contiguous block is merely default_alignment_k-aligned - no mbind, no VirtualAllocExNuma. The slice count still comes from the machine_topology, so on a machine the platform reports as single-domain the mapping is one stride wide and the same addressing ptr + d * stride_bytes + i * sizeof(value_type) holds everywhere. Borrows the topology; hand it the one you harvested and let it outlive the mapping.
linux_symmetric_allocator / windows_symmetric_allocator are the domain-pinning siblings.
linux_numa_allocatorSTL-compatible allocator bound to a single memory domain, prioritizing huge pages.4template <typename value_type_ = char>
struct linux_numa_allocatorA light-weight, but high-latency BLOB allocator, tied to a specific NUMA node ID: every allocation is one mmap plus mbind syscall, aligned to at least a 4 KB page size.
Whenever possible, the newer allocate_at_least API reduces the number of reallocations.
linux_numa_allocatorconstexpr linux_numa_allocator() noexcept=default;constexpr linux_numa_allocator(
memory_domain_id_t id,
size_type paging = ram_page_size()) noexcept;template <typename other_type_>
constexpr linux_numa_allocator(
linux_numa_allocator<other_type_> const & o) noexcept;allocate_at_leastAllocates memory for at least size elements of value_type.allocation_result<value_type *, size_type> allocate_at_least(
size_type size,
size_type page_size_bytes) noexcept;allocation_result<value_type *, size_type> allocate_at_least(
size_type size) noexcept;size | in | The number of elements to allocate. |
|---|---|---|
page_size_bytes | in | The size of the memory page to allocate, must be a multiple of sizeof(value_type). |
An allocation_result holding the pointer and the element count, empty when the allocation fails or the size is not a multiple of sizeof(value_type).
allocateAllocates a memory for size elements of value_type.value_type * allocate(
size_type size,
size_type page_size_bytes) noexcept;value_type * allocate(size_type size) noexcept;Allocates memory for size elements of value_type.
size | in | The number of elements to allocate. |
|---|---|---|
page_size_bytes | in | The size of the memory page to allocate, must be a multiple of sizeof(value_type). |
An allocation_result holding the pointer and the element count, empty when the allocation fails or the size is not a multiple of sizeof(value_type).
operator==template <typename other_type_>
bool operator==(
linux_numa_allocator<other_type_> const & o) const noexcept;replicated_arrayOne uninitialized replica of a sequence per memory domain - a domain-local buffer.7template <typename value_type_,
typename allocator_type_ = symmetric_memory_allocator_t>
struct replicated_arrayFor read-shared data - graph adjacency, body positions. The logical length is n; the footprint is n * memory_domains_count(). It owns memory and hands out slices - the caller fills every replica and keeps them coherent. Backed by one symmetric mapping, so replica d of element i lives at base() + d * stride_bytes() + i * sizeof(value_type).
replicated_arrayreplicated_array() noexcept=default;replicated_array(replicated_array && other) noexcept;replicated_array(replicated_array const &)=delete;*_array× 2operator=replicated_array & operator=(replicated_array && other) noexcept;replicated_array & operator=(replicated_array const &)=delete;*× 2Frees the mapping.resize_uninitializedAllocates one uninitialized length n replica per memory domain; the caller first-touches them.status_t resize_uninitialized(
machine_topology_t const & topology,
std::size_t n) noexcept;atOne element of the replica on memory_domain.value_type_ & at(
memory_domain_index_t memory_domain,
std::size_t local_index) noexcept;value_type_ const & at(
memory_domain_index_t memory_domain,
std::size_t local_index) const noexcept;on_memory_domainThe whole replica living on memory_domain.span_type on_memory_domain(
memory_domain_index_t memory_domain) noexcept;view_type on_memory_domain(
memory_domain_index_t memory_domain) const noexcept;sharded_arrayA domain-local buffer - a sequence partitioned once across every memory domain.10template <typename value_type_,
typename allocator_type_ = symmetric_memory_allocator_t>
struct sharded_arrayFor datasets too large to replicate. The logical length is n; the footprint is n. Each domain owns a contiguous logical segment of segment() == ceil(n / memory_domains_count()) elements, so element i lives at {i / segment(), i % segment()} - see location_of - and a sequential scan of a domain's shard is sequential in memory. It owns memory and hands out slices; the caller fills them. Backed by one symmetric mapping whose uniform stride is that segment, so a shorter trailing shard leaves slots unused.
sharded_arraysharded_array() noexcept=default;sharded_array(sharded_array && other) noexcept;sharded_array(sharded_array const &)=delete;operator=sharded_array & operator=(sharded_array && other) noexcept;sharded_array & operator=(sharded_array const &)=delete;resize_uninitializedAllocates uninitialized storage for n elements in contiguous per-domain segments.status_t resize_uninitialized(
machine_topology_t const & topology,
std::size_t n) noexcept;segmentThe contiguous logical segment size per domain - ceil(n / memory_domains_count()).std::size_t segment() const noexcept;length_on_memory_domainHow many elements the shard on memory_domain holds - a trailing shard may be shorter.std::size_t length_on_memory_domain(
memory_domain_index_t memory_domain) const noexcept;location_ofThe memory domain and local index that store logical element logical_index.location_t location_of(std::size_t logical_index) const noexcept;Requires a non-empty array - segment() is the divisor and is zero when empty.
logical_index_ofThe logical index of the element at local_index on memory_domain - inverse of location_of.std::size_t logical_index_of(
memory_domain_index_t memory_domain,
std::size_t local_index) const noexcept;atThe single home of a logical element - freely mutable, unlike a replica.value_type_ & at(
memory_domain_index_t memory_domain,
std::size_t local_index) noexcept;value_type_ const & at(
memory_domain_index_t memory_domain,
std::size_t local_index) const noexcept;on_memory_domainThe whole shard living on memory_domain.span_type on_memory_domain(
memory_domain_index_t memory_domain) noexcept;view_type on_memory_domain(
memory_domain_index_t memory_domain) const noexcept;location_tWhere a logical element lives: which memory domain and its index within that shard.struct location_tmemory_domain | memory_domain_index_t | |
|---|---|---|
local_index | std::size_t |
linux_numa_bindTries binding the given address range to a specific NUMA memory_domain_id.status_t linux_numa_bind(
void *ptr,
std::size_t size_bytes,
memory_domain_id_t memory_domain_id) noexcept;True if binding succeeded, false otherwise.
linux_numa_allocateTries allocating and binding uninitialized memory to a specific NUMA memory_domain_id.void * linux_numa_allocate(
std::size_t size_bytes,
std::size_t page_size_bytes,
memory_domain_id_t memory_domain_id) noexcept;A pointer to the allocated memory, or nullptr when the allocation failed or the page size stays unsupported.
linux_symmetric_allocateMaps one range of domains * stride_bytes and binds slice d to domain_ids[d].void * linux_symmetric_allocate(
machine_topology_t const & topology,
std::size_t stride_bytes,
std::size_t page_size_bytes) noexcept;Unlike linux_numa_allocate, which places a block on a single node, this reserves one contiguous virtual range and mbinds each equal-stride slice to its own node - a symmetric layout where the MMU then serves every slice from its local memory.
nullptr if the mapping or any slice binding failed, or the page size is unsupported.
freebsd_domain_restoreRestores the calling thread's memory-domain policy saved by freebsd_domain_prefer.void freebsd_domain_restore(
void const *saved_set,
int saved_policy,
bool have_saved) noexcept;A no-op where the save failed, so a refused getdomain never widens the policy.
freebsd_first_touchFirst-touches every page in [ ptr, ptr + size_bytes ) so PREFER binds them.void freebsd_first_touch(void *ptr, std::size_t size_bytes) noexcept;Anonymous pages are already zero, so writing a zero changes nothing but the residency.
freebsd_domain_preferPoints the calling thread's domainset at memory_domain_id with a PREFER policy.bool freebsd_domain_prefer(
memory_domain_id_t memory_domain_id,
void *saved_set,
int *saved_policy) noexcept;FreeBSD 12 removed numa_setaffinity, so there is no per-allocation numa_alloc_onnode: the memory domain is a thread policy. The caller sets it, first-touches the region so its pages fault on the preferred domain, then restores the saved policy - the domainset analogue of Linux's per-mapping mbind.
true when the prior domainset landed in saved_set / saved_policy for restore.
freebsd_domain_allocateTries allocating and placing uninitialized memory on a specific NUMA memory_domain_id.void * freebsd_domain_allocate(
std::size_t size_bytes,
std::size_t page_size_bytes,
memory_domain_id_t memory_domain_id) noexcept;Larger-than-base page sizes ask for MAP_ALIGNED_SUPER - an alignment hint, since FreeBSD's superpages are transparent and have no MAP_HUGETLB-style pool. It is best-effort: a failed super-aligned map retries with base pages so the allocator stays total, matching the Linux nullptr-on-unsupported contract.
A pointer to the allocated memory, or nullptr when the allocation failed or the page size stays unsupported.
linux_numa_allocate is the Linux counterpart; FreeBSD places by thread policy, not mbind.
freebsd_symmetric_allocateMaps domains * stride_bytes and PREFER-touches slice d on domain_ids[d].void * freebsd_symmetric_allocate(
machine_topology_t const & topology,
std::size_t stride_bytes,
std::size_t page_size_bytes) noexcept;nullptr if the mapping failed or the page size is unsupported.
linux_symmetric_allocate is the Linux counterpart; FreeBSD places slices by thread policy.
windows_enable_lock_memory_privilegeEnables SeLockMemoryPrivilege for the process, needed before large-page allocation.status_t windows_enable_lock_memory_privilege() noexcept;The account must first be granted "Lock pages in memory" - Local Security Policy / SeLockMemoryPrivilege - typically by an admin. Call once at start-up, then construct a windows_numa_allocator with large_pages = true.
true if the privilege is now held by the process token.
This only enables a privilege the account already holds.
windows_numa_allocateAllocates uninitialized memory placed on a specific NUMA memory_domain_id on Windows.void * windows_numa_allocate(
std::size_t size_bytes,
memory_domain_id_t memory_domain_id,
bool large_pages = false) noexcept;VirtualAllocExNuma reserves and commits a range whose pages the kernel will fault in on the requested node - the Windows analogue of Linux's mbind, folded into the allocation call. Large pages need SeLockMemoryPrivilege, which windows_enable_lock_memory_privilege grants, and can still fail under memory fragmentation; a caller wanting a soft failure should retry with large_pages false.
size_bytes | in | Number of bytes to allocate. |
|---|---|---|
memory_domain_id | in | The NUMA node to place the memory on. |
large_pages | in | Request MEM_LARGE_PAGES; the size is rounded up to GetLargePageMinimum(). |
nullptr if allocation failed, the size is zero, or NUMA memory is unavailable.
windows_symmetric_allocateReserves one range of domains * stride_bytes and commits slice d on its own node.void * windows_symmetric_allocate(
machine_topology_t const & topology,
std::size_t stride_bytes) noexcept;The Windows analogue of linux_symmetric_allocate: one VirtualAlloc(MEM_RESERVE) for the whole range, then a per-slice VirtualAllocExNuma(MEM_COMMIT, node), so each slice's pages fault in on its own node. Base pages only - large pages cannot be reserved and committed across two separate mapping steps.
nullptr if the reservation or any slice commit failed.
Member types13linux_numa_allocator::value_type | value_type_ |
|---|---|
linux_numa_allocator::size_type | std::size_t |
linux_numa_allocator::propagate_on_container_move_assignment | std::true_type |
replicated_array::value_type | value_type_ |
replicated_array::allocator_type | allocator_type_ |
replicated_array::span_type | std::span<value_type_> |
replicated_array::view_type | std::span<value_type_ const> |
replicated_array::symmetric_allocator_type | typename std::allocator_traits<allocator_type_>::template rebind_alloc<value_type_> |
sharded_array::value_type | value_type_ |
sharded_array::allocator_type | allocator_type_ |
sharded_array::span_type | std::span<value_type_> |
sharded_array::view_type | std::span<value_type_ const> |
sharded_array::symmetric_allocator_type | typename std::allocator_traits<allocator_type_>::template rebind_alloc<value_type_> |