# NUMA

> ForkUnion for C

Placing memory on a chosen domain is implemented on Linux via the `mbind` syscall, on FreeBSD via `domainset` policies, and on Windows via `VirtualAllocExNuma`, and degrades to a single domain elsewhere.
The portable `domain_allocator_t` alias picks the right backend per platform - `linux_numa_allocator_t`, `freebsd_numa_allocator_t`, `windows_numa_allocator_t`, or a plain aligned fallback - each an STL-compatible allocator bound to one `memory_domain_id_t`.

Let's say you are building a Big Data application — brute-forcing Vector Search with the [NumKong](/docs/numkong/) library — on a 2 dual-socket CPU system, the kind that powers production [USearch](https://github.com/unum-cloud/USearch/pulls) deployments.
The first part of that program may be responsible for sharding the incoming stream of data between distinct memory regions.
That part, in our simple example will be single-threaded:

```cpp
#include <vector> // `std::vector`
#include <span> // `std::span`
#include <forkunion.hpp> // `linux_numa_allocator`, `machine_topology_t`, `distributed_pool_t`
#include <numkong/spatial.h> // `nk_angular_f32`, `nk_f64_t`

namespace fu = ashvardanian::forkunion;
using floats_alloc_t = fu::linux_numa_allocator<float>;

constexpr std::size_t dimensions = 768; /// Matches most BERT-like models
static std::vector<float, floats_alloc_t> first_half(floats_alloc_t(fu::memory_domain_id_t {0}));
static std::vector<float, floats_alloc_t> second_half(floats_alloc_t(fu::memory_domain_id_t {1}));
static fu::machine_topology_t machine_topology;
static fu::distributed_pool_t distributed_pool;

/// Dynamically shards incoming vectors across 2 nodes in a round-robin fashion.
void append(std::span<float, dimensions> vector) {
    bool put_in_second = first_half.size() > second_half.size();
    if (put_in_second) second_half.insert(second_half.end(), vector.begin(), vector.end());
    else first_half.insert(first_half.end(), vector.begin(), vector.end());
}
```

The concurrent part would involve spawning threads adjacent to every memory pool to find the best `search_result_t`.
The primary `search` function, in ideal world would look like this:

1. Each thread finds the best match within its "slice" of a compute domain, tracking the best distance and index in a local CPU register.
2. All threads in each compute domain atomically synchronize using a domain-local instance of `search_result_t`.
3. The main thread collects aggregates of partial results from all compute domains.

That is, however, overly complicated to implement.
Such tree-like hierarchical reductions are optimal in a theoretical sense.
Still, assuming the relative cost of spin-locking once at the end of a thread scope and the complexity of organizing the code, the more straightforward path is better.
A minimal example would look like this:

```cpp
/// On each compute domain we'll synchronize the threads
struct search_result_t {
    nk_f64_t best_distance {std::numeric_limits<nk_f64_t>::max()};
    std::size_t best_index {0};
};

inline search_result_t pick_best(search_result_t const& a, search_result_t const& b) noexcept {
    return a.best_distance < b.best_distance ? a : b;
}

/// Uses all CPU threads to search for the closest vector to the @p query.
search_result_t search(std::span<float, dimensions> query) {

    bool const need_to_spawn_threads = distributed_pool.threads_count() == 0;
    if (need_to_spawn_threads) {
        fu::status_t const harvested = machine_topology.harvest();
        assert(fu::succeeded(harvested) && "Failed to harvest NUMA topology");
        assert(machine_topology.memory_domains_count() == 2 && "Expected exactly 2 NUMA nodes");
        fu::status_t const spawned = distributed_pool.spawn(machine_topology);
        assert(fu::succeeded(spawned) && "Failed to spawn NUMA pools");
    }

    search_result_t result;
    fu::spin_mutex_t result_update; // ? Lighter `std::mutex` alternative w/out futexes

    std::size_t const first_vectors = first_half.size() / dimensions;
    std::size_t const total_vectors = first_vectors + second_half.size() / dimensions;

    auto slices = distributed_pool.for_slices(total_vectors,
        [&](fu::tasks_range_t tasks, fu::thread_in_domain_t) noexcept {

        search_result_t thread_local_result;
        for (std::size_t const task : tasks) {
            bool const in_second = task >= first_vectors;
            auto const &shard = in_second ? second_half : first_half;
            std::size_t const local_index = in_second ? task - first_vectors : task;

            nk_f64_t distance;
            nk_angular_f32(query.data(), shard.data() + local_index * dimensions, dimensions, &distance);
            thread_local_result = pick_best(thread_local_result, {distance, task});
        }

        // ! Still synchronizing over a shared mutex for brevity.
        std::lock_guard<fu::spin_mutex_t> lock(result_update);
        result = pick_best(result, thread_local_result);
    });
    slices.join();
    return result;
}
```

In a dream world, we would call `distributed_pool.for_n`, but there is no clean way to make the scheduling processes aware of the data distribution in an arbitrary application, so that's left to the user.
The `for_slices` helper pairs each run with a `fu::thread_in_domain_t` naming its compute domain, but slices follow the thread count rather than the shard boundary, so the callback above picks the shard per task.
For more flexibility around building higher-level low-latency systems, there are unsafe APIs expecting you to manually "join" the broadcasted calls: `unsafe_for_threads` returns a generation token, `is_complete` polls it without blocking, and `unsafe_join` blocks until that generation completes.

The manual two-vector sharding above is what the __symmetric allocators__ automate: `symmetric_memory_allocator_t` - `fu_allocate_symmetric` in C - maps one virtual range striped a slice per memory domain, and Rust, Zig, and Mojo wrap it as `ShardedArray<T>`, one shard per domain, and `ReplicatedArray<T>`, a full copy per domain for read-mostly data.
To place and coordinate pools by hand, `spawn_on` - `fu_pool_spawn_on` in C - pins a pool to a single compute domain, while `locate_thread_in` and `threads_count_in` map a global thread index to its domain and count the workers living there.

`#include <forkunion/forkunion.h>`

## fu_allocate_at_least_on_domain_id

Allocates memory in `memory_domain_id` with the largest suitable page size.

```c
fu_status_t fu_allocate_at_least_on_domain_id(
    fu_memory_domain_id_t memory_domain_id,
    size_t minimum_bytes,
    size_t *allocated_bytes,
    size_t *bytes_per_page,
    void **memory_out);
```

Prefers the largest available huge-page size to minimize TLB pressure; the actual size may exceed the request due to page alignment, so always read `allocated_bytes`. No topology handle is needed - the id fully identifies the domain.

```
size_t memory_domain_index = 0;
fu_memory_domain_id_t domain = -1;
if (fu_local_memory_of(topology, compute_domain, &memory_domain_index) != fu_success_k) return;
if (fu_memory_domain_id_at_index(topology, memory_domain_index, &domain) != fu_success_k) return;
void *pointer = NULL; size_t actual_bytes = 0, page = 0;
if (fu_allocate_at_least_on_domain_id(domain, 1u << 20, &actual_bytes, &page, &pointer) == fu_success_k)
    fu_free_on_domain_id(domain, pointer, actual_bytes);
```

- `memory_domain_id` (in): Target memory domain, from [`fu_memory_domain_id_at_index`](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_memory_domain_id_at_index).
- `minimum_bytes` (in): Minimum number of bytes to allocate, must be > 0.
- `allocated_bytes` (out): Receives the actual allocation size - at least `minimum_bytes` - must not be NULL.
- `bytes_per_page` (out): Receives the page size used for the allocation, must not be NULL.
- `memory_out` (out): Receives the pointer to the allocated memory.

Notes: This API is __thread-safe__ and can be called from any thread.

Notes: The pointer is aligned to at least the cache-line default, so over-aligned element types up to that width need no caller-side padding.

See also: [fu_free_on_domain_id](#fu_free_on_domain_id), [fu_memory_domain_id_at_index](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_memory_domain_id_at_index).

Statuses: - [`fu_invalid_argument_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_invalid_argument_k) A NULL output pointer.

## fu_allocate_on_domain_id

Allocates exactly `bytes` in `memory_domain_id`.

```c
fu_status_t fu_allocate_on_domain_id(
    fu_memory_domain_id_t memory_domain_id,
    size_t bytes,
    void **memory_out);
```

- `memory_domain_id` (in): Target memory domain, from [`fu_memory_domain_id_at_index`](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_memory_domain_id_at_index).
- `bytes` (in): Number of bytes to allocate, must be > 0.
- `memory_out` (out): Receives the pointer to the allocated memory.

Notes: This API is __thread-safe__. Unlike [`fu_allocate_at_least_on_domain_id`](#fu_allocate_at_least_on_domain_id), it does not over-allocate for page optimization - use it for standard-allocator compatibility.

Notes: The pointer is aligned to at least the cache-line default, matching [`fu_allocate_at_least_on_domain_id`](#fu_allocate_at_least_on_domain_id).

See also: [fu_free_on_domain_id](#fu_free_on_domain_id), [fu_allocate_at_least_on_domain_id](#fu_allocate_at_least_on_domain_id).

Statuses: - [`fu_invalid_argument_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_invalid_argument_k) A NULL `memory_out`.

## fu_free_on_domain_id

Releases memory allocated in `memory_domain_id`.

```c
void fu_free_on_domain_id(
    fu_memory_domain_id_t memory_domain_id,
    void *pointer,
    size_t bytes);
```

- `memory_domain_id` (in): The memory domain the memory was allocated in.
- `pointer` (in): Pointer to the memory to release, must not be NULL.
- `bytes` (in): Number of bytes to release; must match the `allocated_bytes` from allocation.

Notes: This API is __thread-safe__. A mismatched `bytes` is undefined behavior.

See also: [fu_allocate_at_least_on_domain_id](#fu_allocate_at_least_on_domain_id), [fu_allocate_on_domain_id](#fu_allocate_on_domain_id).

## fu_allocate_symmetric

Allocates one __symmetric__ mapping - `bytes_per_domain` striped across every domain.

```c
fu_status_t fu_allocate_symmetric(
    fu_topology_t topology,
    size_t bytes_per_domain,
    size_t *stride_bytes,
    size_t *memory_domains_count,
    size_t *total_bytes,
    size_t *bytes_per_page,
    void **memory_out);
```

Prefers the largest available huge-page size to minimize TLB pressure, so the stride may exceed `bytes_per_domain` - always read `stride_bytes`.

```
size_t stride = 0, domains = 0, total = 0, page = 0;
void *base = NULL;
if (fu_allocate_symmetric(topology, 1u << 20, &stride, &domains, &total, &page, &base) == fu_success_k)
    fu_free_symmetric(base, total);
```

- `topology` (in): Machine topology from [`fu_topology_new`](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_topology_new).
- `bytes_per_domain` (in): Minimum usable bytes per domain slice; the slice stride is page-rounded up from it.
- `stride_bytes` (out): The page-aligned byte distance between slices; must not be NULL.
- `memory_domains_count` (out): Receives the number of domain slices, must not be NULL.
- `total_bytes` (out): Receives the whole mapping size to hand back to [`fu_free_symmetric`](#fu_free_symmetric), must not be NULL.
- `bytes_per_page` (out): Receives the page size used, may be NULL.
- `memory_out` (out): Receives the base pointer of the mapping.

Notes: This API is __thread-safe__. Slice __d__ begins at `base + d * *stride_bytes` and is bound to its own memory domain; a machine with no NUMA API collapses the mapping to a single heap-backed memory slice. Every slice starts zeroed.

See also: [fu_free_symmetric](#fu_free_symmetric), [fu_local_memory_of](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_local_memory_of), [fu_memory_domains_count](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_memory_domains_count).

Statuses: - [`fu_invalid_argument_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_invalid_argument_k) A NULL handle or a NULL output pointer.

## fu_free_symmetric

Releases a symmetric mapping from [`fu_allocate_symmetric`](#fu_allocate_symmetric).

```c
void fu_free_symmetric(void *base, size_t total_bytes);
```

- `base` (in): Base pointer returned by [`fu_allocate_symmetric`](#fu_allocate_symmetric), must not be NULL.
- `total_bytes` (in): The `total_bytes` the allocation reported; mismatches are undefined.

Notes: This API is __thread-safe__. The size is required - Linux unmaps the whole range.

See also: [fu_allocate_symmetric](#fu_allocate_symmetric).

## fu_pool_spawn_near_memory_domain

Spawns `threads` workers across every compute domain local to one memory domain.

```c
fu_status_t fu_pool_spawn_near_memory_domain(
    fu_topology_t topology,
    fu_pool_t pool,
    fu_memory_domain_id_t memory_domain_id,
    size_t threads,
    fu_caller_exclusivity_t exclusivity);
```

Sits between the extremes: [fu_pool_spawn_on](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_pool_spawn_on) binds one compute domain and [fu_pool_spawn](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_pool_spawn) binds them all, but a memory domain often feeds several - P/E clusters on one node, or several CCXs sharing one NUMA node - and this is the spawn that engages exactly those. Workers report their compute domain by its topology index, so `fu_local_memory_of-style` lookups stay valid. Where the placement capability is masked off, falls back to [fu_pool_spawn](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_pool_spawn) on single-domain machines and reports [`fu_unsupported_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_unsupported_k) on NUMA ones. It returns once every worker started across the domains local to `memory_domain_id`.

- `topology` (in): Machine topology from [`fu_topology_new`](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_topology_new); read only during this call.
- `pool` (in): Pool handle, must not be NULL.
- `memory_domain_id` (in): The OS id of the memory domain whose local cores participate.
- `threads` (in): Worker count, must be > 0; split across the local compute domains.
- `exclusivity` (in): Whether the calling thread also executes tasks.

Notes: Not thread-safe; call once per pool.

See also: [fu_pool_spawn](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_pool_spawn), [fu_pool_spawn_on](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_pool_spawn_on), [fu_memory_domain_id_at_index](https://www.unum.cloud/docs/forkunion/c/topology.md#fu_memory_domain_id_at_index).

Statuses: - [`fu_invalid_argument_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_invalid_argument_k) A NULL handle, an unnamed exclusivity, `threads` of 0, or a `memory_domain_id` no compute domain is local to.
- [`fu_unsupported_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_unsupported_k) Placement is masked off on a machine with several memory domains, or a caller-exclusive or multi-thread pool in a build without OS threads.
- [`fu_already_spawned_k`](https://www.unum.cloud/docs/forkunion/c/quickstart.md#fu_already_spawned_k) While the pool still holds workers.