# Batches and Arenas

> UStore for C

### Executors, Blocking and Deferred Calls

An executor owns the threads that run calls and the scratch they use, and a `NULL` executor runs a call on the calling thread.
Its kind picks where calls compute: a RAM or NVMe executor serves every engine on host threads, and a VRAM executor serves the reads and scans an engine computes on its one device, refusing changes, the control plane and every call no device tower serves yet with `ustore_status_unsupported_k`.
`ustore_executor_for_ram` spawns its threads, caller-exclusive, so its first thread can serve a call in the caller's place.
Its `ram` options name the memory domains it serves, NUMA nodes as the operating system numbers them, where `NULL` derives every domain the process may allocate on and an unknown one is refused with `ustore_status_invalid_argument_k`.
Each thread keeps its scratch on its local domain when the options name it, and on the first named domain otherwise, and the call's own scratch and every arena the executor makes live where its first thread's does.
Zero threads take 7/8 of the cores local to those domains, rounded down and at least one, leaving the rest to blocking calls on their callers' threads, and to the background flushes and compactions of RocksDB and LevelDB.
ForkUnion spawns the threads near every domain or near one alone, so naming several domains but not all spreads them over the whole machine.
`ustore_executor_on_pool` borrows a ForkUnion pool another library owns, which must outlive the executor, and places its scratch and arenas by the same `ram` options; a dispatch on a pool that runs another is refused, so a call finding the pool running its lender's dispatch finishes `ustore_status_contended_k` instead of forking over the lender.
`ustore_executor_for_nvme` spawns the threads of the NVMe engine, and its `nvme` options tell every database `ustore_database_open_nvme` opens on it how to move bytes: the transport, the channels and their buffers, read once at the open.
It may borrow an executor of `ustore_executor_for_vram`, whose CUDA device and scratch limit the compactions take, and which answers `ustore_status_unsupported_k` in a library built without CUDA.
That executor queues every call on one stream of its device, its own or one the caller lends through `stream`, from the calling thread, which waits on it there, so a deferred call is final when it returns.
A call blocks by default and returns with its outcome final, while `ustore_completion_deferred_k` returns once the call is submitted:

```c
int32_t const memory_domains[1] = {0};
ustore_executor_for_ram(&(ustore_executor_for_ram_t){
    .executor = &executor,
    .outcome = &outcome,
    .ram = {memory_domains, 1},
});

ustore_outcome_t outcomes[2] = {{0}};
for (size_t batch = 0; batch != 2; ++batch) { // one call per executor at a time, so two executors overlap two reads
    reads[batch].executor = executors[batch], reads[batch].outcome = &outcomes[batch], reads[batch].arena = arenas[batch];
    reads[batch].completion = ustore_completion_deferred_k;
    ustore_blobs_read(&reads[batch]);
}
while (ustore_outcome_status(&outcomes[0]) == ustore_status_deferred_k ||
       ustore_outcome_status(&outcomes[1]) == ustore_status_deferred_k)
    do_other_work(); // polling alone finishes both
```

Submission copies the argument struct, so the caller may reuse or drop it at once, while everything it points to, the outcome, inputs, outputs and arena alike, stays borrowed until the call is final.
A deferred call's status reads `ustore_status_deferred_k` until it finishes; the library then stores the message and the final status with release ordering and touches nothing the struct points to after that.
The caller reads the status with acquire ordering, polling through `ustore_outcome_status` or waiting through `ustore_outcome_wait`, and never loads the field itself.
An executor serves one call at a time, and nothing queues: a call on an executor serving another, from any thread, finishes at once with `ustore_status_contended_k`.
A blocking call on an executor runs on the calling thread, and a deferred call is delegated to the executor's first thread, which serves it in the caller's place, so polling alone finishes it.
Inside a call every fork of the executor's pool splits the call's batch over the other threads, in both modes.
A call ends in one order: it releases its arena, cursor and transaction and its counts on the handles it names, then the executor, and stores its status last, so a caller seeing it final may submit again or free at once.
On a borrowed pool, whose lender keeps its dispatches and joins, and on a caller-inclusive pool, as a build without OS threads spawns, a deferred call runs at submission and is final when it returns.
`ustore_executor_drain` waits until no call holds the executor, then parks its threads or keeps them spinning.
A call is refused with `ustore_status_contended_k` rather than waited for when its executor, arena, transaction or cursor serves another call, or when it drains its executor from a callback of that executor's own call.
Freeing a database, transaction, snapshot, cursor or arena waits until every executor call naming it ends, so a binding may drop its handle before those calls finish, but never from a callback of one of them.
A progress callback never waits on a call its own executor has not finished, and polls the status instead.
[The C++ wrappers](/docs/ustore/cpp/) and the Rust and Python bindings choose the executor and the completion by receiver, as `database.on(executor)` and `database.deferred(executor)`, so blocking calls read as they would without executors; the Design sections of `rust/README.md` and `python/README.md` state the rules every binding keeps.
A call stops early only when its progress callback returns `ustore_cancel_k`, as [Progress and Cancellation](#progress-and-cancellation) describes.

#### Inclusive and Exclusive Executors

A pool is caller-inclusive when the thread that spawned or lent it counts as one of its threads, and caller-exclusive when it only dispatches and waits.
An inclusive pool needs its caller to run anything, so a deferred call on it runs at submission and is final when it returns.
An exclusive pool, which every executor UStore spawns is, delegates a deferred call to its own first thread, one call at a time, and forks the other threads for it.
So one manager thread in `int main()` can overlap a USearch RAM executor, a UStore NVMe executor and a UForm VRAM executor, giving each one deferred call and polling the three statuses, with no thread of its own per engine.

### Progress and Cancellation

Scans, cursor pulls, saves and compactions take an optional `progress` callback and the `progress_state` handed to it.
Each reports in the units its verb consumes: finished tasks for a scan, one unit for a pull, bytes of keys and values for a save, and bytes compacted where the engine measures them, as RocksDB does, or collections elsewhere.
A callback never runs concurrently with itself, however many threads do the work, never runs after the call is final, and never sees `finished` decrease:

```c
static ustore_continuation_t within_budget(ustore_size_t finished, ustore_size_t total, void* state) {
    ustore_size_t const budget = *(ustore_size_t const*)state;
    fprintf(stderr, "saved %llu of %llu bytes\n", (unsigned long long)finished, (unsigned long long)total);
    return finished <= budget ? ustore_continue_k : ustore_cancel_k;
}
```

```c
ustore_size_t budget = 1ull << 30;
char const path[] = "/var/backups/ustore";
ustore_snapshot_save(&(ustore_snapshot_save_t){
    .snapshot = snapshot,
    .outcome = &outcome,
    .path = {(uint8_t const*)path, sizeof(path) - 1},
    .progress = within_budget,
    .progress_state = &budget,
});
```

Returning `ustore_cancel_k` is the only way to stop a call, which then answers `ustore_status_cancelled_k`:

- scan and pull outputs stay `NULL`, and a cancelled pull leaves its cursor where it was;
- a cancelled save removes what it wrote, as a failed one does;
- a cancelled compaction stops between collections, keeping what it compacted.

The callback runs on the thread that runs the call, the caller's for a blocking call and one of the executor's for a deferred one, so a binding's callback enters its language's runtime there.

`#include <ustore/types.h>`

## ustore_outcome_t

Outcome of one call: a status to branch on, and a static message that is never freed and is `NULL` exactly when the status is [`ustore_status_success_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_success_k).

```c
struct ustore_outcome_t
```

- `status` `ustore_status_t`: What went wrong, [`ustore_status_success_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_success_k), or [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k) until a deferred call finishes.
- `message` `char const *`: Static detail the status cannot carry.

## ustore_completion_t

Whether a call returns final or once submitted to its executor.

```c
enum ustore_completion_t
```

- `ustore_completion_blocking_k` = `0`: Return with the final status stored and every output readable; the default.
- `ustore_completion_deferred_k` = `1`: Return once submitted, with the status [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k) until the call finishes; the caller waits or polls.

## ustore_continuation_t

Whether the call a progress callback watches carries on or unwinds.

```c
enum ustore_continuation_t
```

- `ustore_continue_k` = `0`: Carry on to the next unit of work.
- `ustore_cancel_k` = `1`: Stop, publish nothing, and report [`ustore_status_cancelled_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_cancelled_k).

## ustore_progress_callback_t

Periodic report of a long call's advance, and the caller's only way to stop it.

```c
typedef ustore_continuation_t(*) ustore_progress_callback_t(ustore_size_t finished, ustore_size_t total, void *state);
```

Counts are in the units the verb consumes: tasks for a scan, one pull for a cursor, bytes of keys and values for a save, and bytes or collections for a compaction. The callback never runs concurrently with itself, however many threads do the work, and never after the call stores its final status. Its arguments are the units finished so far, never decreasing within one call, the units the call finishes if it runs to the end, and the `progress_state` registered beside it.

## ustore_executor_t

An opaque handle to the threads or I/O rings that run calls, shared by any number of databases.

```c
typedef struct ustore_executor_handle_t * ustore_executor_t;
```

Its kind picks where calls compute: host threads for every engine, or one device for the reads and scans an engine computes there, where a change, the control plane and every call no device tower serves yet answer [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k), and an arena or cursor of another kind of executor [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k). Calls on one executor start in submission order, and a blocking call runs alone: after every call submitted before it finishes, and before any submitted after it starts. Databases stay freely concurrent. A blocking call runs on the calling thread, while one driver thread hands deferred calls in batches to the executor's threads. In a build without OS threads, an executor runs every call at submission, before returning. It owns the scratch and staging its calls need, placed where it computes and grown to a high-water mark, so a steady workload allocates once: a RAM executor keeps the scratch of each thread on the memory domain local to it.

An executor holds its deferred calls in a deferred table sized at creation by `deferred_limit`, so submitting never allocates, and a deferred submission that finds the table full stores [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) as its final status. A call leaves the table before its status turns final, so the table is full only of calls no caller has seen final. From inside a progress callback its own executor runs, [`ustore_executor_drain`](#ustore_executor_drain) and a blocking call on that executor report [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k), so nothing waits on itself. Frees cannot report, so [`ustore_executor_free`](#ustore_executor_free) is never called there, nor the free of a handle a deferred call names.

The library stages only to change layout on the side that computes, or to make pageable memory DMA-able, and never routes device bytes through the host:

| Case | Action |
|:-|:-|
| Device inputs on the executor's device | read in place |
| Device inputs needing padding or a cast | a device-to-device copy or a cast kernel |
| Pinned or managed host inputs | a direct asynchronous upload |
| Pageable host inputs | the pinned bounce buffer, then an upload |
| Outputs in an arena on the executor's device | the kernel writes them directly |

## ustore_mood_t

What an executor's threads or rings do once it goes idle.

```c
enum ustore_mood_t
```

- `ustore_mood_parked_k` = `0`: Parked, so the next call pays a wake-up.
- `ustore_mood_spinning_k` = `1`: Spinning, trading power for the next call's latency.

Spellings: ```
char const * ustore_mood_name(ustore_mood_t mood);
```

```
ustore_mood_t ustore_mood_from_name(ustore_bytes_view_t name);
```

## ustore_cpu_class_t

The CPU generation a database or an executor confines itself to, named as the sibling libraries name it.

```c
enum ustore_cpu_class_t
```

- `ustore_cpu_class_unknown_k` = `0`: Whatever the running CPU offers.
- `ustore_cpu_class_serial_k`: The portable floor every CPU runs.
- `ustore_cpu_class_intel_haswell_k`: x86 cores from Intel Haswell on.
- `ustore_cpu_class_intel_sierra_k`: x86 efficiency cores from Intel Sierra Forest on.
- `ustore_cpu_class_intel_skylake_k`: x86 server cores from Intel Skylake on.
- `ustore_cpu_class_intel_icelake_k`: x86 server cores from Intel Ice Lake on.
- `ustore_cpu_class_aws_g2_k`: Arm cores from AWS Graviton 2 on.
- `ustore_cpu_class_apple_m2_k`: Arm cores from Apple M2 on.
- `ustore_cpu_class_aws_g5_k`: Arm server cores from AWS Graviton 5 on.
- `ustore_cpu_class_alibaba_c930_k`: RISC-V cores with RVV, Zawrs and Zacas, from Alibaba's C930 on.
- `ustore_cpu_class_monoversioned_k`: The one class of a monoversioned build, whatever its floor implies or its preset vouches for.

Spellings: ```
char const * ustore_cpu_class_name(ustore_cpu_class_t cpu_class);
```

```
ustore_cpu_class_t ustore_cpu_class_from_name(ustore_bytes_view_t name);
```

## ustore_ram_options_t

The host memory domains a RAM executor serves, NUMA nodes as the operating system numbers them.

```c
struct ustore_ram_options_t
```

Names where the executor keeps memory and runs its threads, never where a database keeps its payload. The scratch of each thread lives on the domain local to that thread when the options name it, and on the first domain named otherwise, as does every arena the executor makes.

- `memory_domains` `int32_t const *`: Identifiers of the memory domains, copied by the call, where `NULL` derives every domain the process may allocate on, and an identifier no domain carries is refused with [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k).
- `memory_domains_count` `ustore_size_t`: Entries of `memory_domains`, where zero derives as `NULL` does.

## ustore_transport_t

How the NVMe engine moves bytes between its files and memory.

```c
enum ustore_transport_t
```

- `ustore_transport_automatic_k` = `0`: io_uring where the kernel offers it, Linux AIO where it offers only that, and blocking calls elsewhere.
- `ustore_transport_blocking_k` = `1`: Positioned reads and writes that block the polling thread, which every platform has.
- `ustore_transport_linux_uring_k` = `2`: io_uring on unbuffered files, refused where the kernel creates no ring.
- `ustore_transport_linux_aio_k` = `3`: Linux AIO on unbuffered files.
- `ustore_transport_linux_spdk_k` = `4`: SPDK's user-space driver on NVMe devices the process owns.

Spellings: ```
char const * ustore_transport_name(ustore_transport_t transport);
```

```
ustore_transport_t ustore_transport_from_name(ustore_bytes_view_t name);
```

## ustore_nvme_options_t

How the threads of an NVMe executor move bytes between the drives and memory, where every zero size or count keeps the engine's default.

```c
struct ustore_nvme_options_t
```

Every thread of the executor keeps a channel of its own, and the threads outside it share `shared_io_channels` more. A channel is one queue pair of `queue_depth` submissions over pinned request and streaming buffers.

- `transport` `ustore_transport_t`: How bytes move between the files and memory.
- `shared_io_channels` `ustore_size_t`: I/O channels the threads outside the executor share.
- `queue_depth` `ustore_size_t`: Submissions a channel holds at once.
- `request_buffer_bytes` `ustore_size_t`: Bytes of each request buffer of a channel.
- `request_buffers_count` `ustore_size_t`: Request buffers of each channel.
- `stream_buffer_bytes` `ustore_size_t`: Bytes of each streaming buffer of a channel.
- `stream_buffers_count` `ustore_size_t`: Streaming buffers of each channel.

## ustore_gds_t

How a VRAM executor moves bytes between the drives and its device through GPUDirect Storage.

```c
enum ustore_gds_t
```

- `ustore_gds_automatic_k` = `0`: Direct transfers where the `nvidia_fs` driver is loaded, and cuFile's compatibility mode elsewhere.
- `ustore_gds_direct_k` = `1`: Direct transfers, refused with [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) where the driver is absent.
- `ustore_gds_compatibility_k` = `2`: cuFile's compatibility mode, which bounces through host pages it manages.

Spellings: ```
char const * ustore_gds_name(ustore_gds_t gds);
```

```
ustore_gds_t ustore_gds_from_name(ustore_bytes_view_t name);
```

## ustore_gpu_vendor_t

The GPU vendors one binary drives at once, each through a runtime opened on first use.

```c
enum ustore_gpu_vendor_t
```

- `ustore_gpu_vendor_unknown_k` = `0`: No vendor, as an unrecognized spelling reads back.
- `ustore_gpu_vendor_nvidia_k` = `1`: NVIDIA, through the CUDA driver when one is installed.
- `ustore_gpu_vendor_amd_k` = `2`: AMD, through the HIP runtime when one is installed.

Spellings: ```
char const * ustore_gpu_vendor_name(ustore_gpu_vendor_t vendor);
```

```
ustore_gpu_vendor_t ustore_gpu_vendor_from_name(
    ustore_bytes_view_t name);
```

## ustore_gpu_class_t

The GPU generations UStore tells apart, the device twin of `[ustore_cpu_class_t](#ustore_cpu_class_t):` a class implies its vendor, and within a vendor a newer device takes the newest class it satisfies.

```c
enum ustore_gpu_class_t
```

- `ustore_gpu_class_unknown_k` = `0`: No class, which no listed device carries.
- `ustore_gpu_class_nvidia_ampere_k` = `1`: Ampere-class NVIDIA and every NVIDIA GPU without a class of its own - A100, A10 & RTX 30.
- `ustore_gpu_class_nvidia_ada_k` = `2`: Ada-class NVIDIA - L4, L40S & RTX 40.
- `ustore_gpu_class_nvidia_hopper_k` = `3`: Hopper-class NVIDIA - H100 & H200.
- `ustore_gpu_class_nvidia_blackwell_k` = `4`: Blackwell-class NVIDIA - B200, B300 & GB300.
- `ustore_gpu_class_nvidia_blackwellrtx_k` = `5`: Consumer Blackwell NVIDIA - RTX 50, RTX PRO 6000 & DGX Spark.
- `ustore_gpu_class_amd_cdna3_k` = `6`: CDNA3 AMD - MI300X & MI325X, gfx942.
- `ustore_gpu_class_amd_cdna4_k` = `7`: CDNA4 AMD - MI350X & MI355X, gfx950.
- `ustore_gpu_class_amd_cdna5_k` = `8`: CDNA5 AMD - MI400, gfx1250.

Spellings: ```
char const * ustore_gpu_class_name(ustore_gpu_class_t gpu_class);
```

```
ustore_gpu_class_t ustore_gpu_class_from_name(ustore_bytes_view_t name);
```

## ustore_gpu_device_t

One GPU as UStore numbers it: its position in the listing of [`ustore_gpu_devices`](#ustore_gpu_devices) is the ordinal `[ustore_executor_for_vram_t::device](#ustore_executor_for_vram)` names it by.

```c
struct ustore_gpu_device_t
```

- `vendor` `ustore_gpu_vendor_t`: The vendor whose runtime drives the device.
- `vendor_ordinal` `uint32_t`: The device's ordinal in its vendor's runtime, as `nvidia-smi` or `rocm-smi` list it.
- `gpu_class` `ustore_gpu_class_t`: The class of the device.
- `compute_units` `uint32_t`: Streaming multiprocessors or compute units.
- `memory_bytes` `uint64_t`: Device memory in bytes.
- `name` `char`: The vendor's own name for the device, null-terminated and truncated to fit.

## ustore_arena_t

Memory the library writes every output whose length scales with a call into, reused across calls.

```c
typedef struct ustore_arena_handle_t * ustore_arena_t;
```

Memory

Made by [`ustore_arena_init`](#ustore_arena_init) where its executor computes, and an executor whose device runs out of memory reports [`ustore_status_out_of_memory_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_out_of_memory_k) rather than falling back to the host. Each call given the arena rewinds it, which invalidates every output of the previous call, and refills it, growing only when the new outputs outsize everything it held, so a steady loop allocates once. Every output starts aligned to 64 bytes, as Arrow buffers do, so typed columns read in place. To keep one call's outputs alive past the next call, pass the next call another arena.

A `NULL` arena is refused with [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k), and so is a call whose inputs point into the arena it is about to rewind. An arena serves one call at a time, so a second call naming it finishes [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k), and a deferred one does so at submission. A call whose executor computes on host threads given an arena on a device, or on another device than the arena's, reports [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k). A blocking call borrows the arena until it returns, and a deferred one holds it until final. A call that fails, is cancelled, or has nothing to report leaves its outputs `NULL`.

## ustore_residency_t

Where an arena's outputs live, as [`ustore_arena_describe`](#ustore_arena_describe) reports it.

```c
enum ustore_residency_t
```

- `ustore_residency_unknown_k` = `0`: No residency, which only a zeroed description reads as.
- `ustore_residency_ram_k` = `1`: Host pages, which the calling thread reads in place.
- `ustore_residency_vram_k` = `2`: One device's memory, which only that device's code reads.

## ustore_outcome_wait

Blocks on an atomic wait until `outcome` is final, with acquire ordering.

```c
ustore_status_t ustore_outcome_wait(ustore_outcome_t *outcome);
```

Never wait on a call's own outcome from inside its progress callback: the call finishes only after the callback returns.

- `outcome` (in): An outcome passed to a call, read until its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).

## ustore_outcome_status

Reads the status of `outcome` without waiting, with the acquire ordering its final store pairs with, so a final status also publishes every output of the call.

```c
ustore_status_t ustore_outcome_status(ustore_outcome_t *outcome);
```

The one way to poll: the status word is atomic only by this contract, so a caller reading the field directly needs an atomic load its language may not offer on foreign memory.

- `outcome` (in): An outcome passed to a call.

Statuses: - [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k) The call is not final.

## ustore_executor_for_ram

Creates an executor of host threads for the RAM, RocksDB and LevelDB engines.

```c
void ustore_executor_for_ram(ustore_executor_for_ram_t *arguments);
```

The default thread count leaves an eighth of the cores to the driver thread, to blocking calls, which run on their callers' threads, and to the background flushes and compactions of RocksDB and LevelDB. The threads interleave across the compute domains local to the memory domains named, where ForkUnion spawns near them: every domain the process may allocate on, or one alone. Naming several but not all spreads them over the whole machine.

- `executor`: Output for the new executor, which the caller frees with [`ustore_executor_free`](#ustore_executor_free); `NULL` on failure.
- `outcome`: Output for the outcome, final on return.
- `ram`: The memory domains the executor serves; copied, not borrowed.
- `threads`: Threads to spawn, interleaved across the compute domains local to `ram`, where zero takes 7/8 of the cores those domains carry, rounded down and at least one.
- `deferred_limit`: Deferred calls the executor holds at once, sizing its deferred table, where zero takes a default.
- `cpu_class`: The CPU class the threads confine their waits and atomics to, where zero takes what the CPU offers.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `executor` output is `NULL`, a memory domain identifier names no domain, the CPU class is unknown, or ForkUnion refuses an argument.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The process may allocate on no memory domain, several threads are asked of a build without them, the CPU does not run the `cpu_class`, or the operating system denies ForkUnion an operation.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The `deferred_limit` is past 2^30.

## ustore_executor_on_pool

Creates an executor for the RAM, RocksDB and LevelDB engines over a ForkUnion pool it borrows, so several libraries in one process share one set of threads.

```c
void ustore_executor_on_pool(ustore_executor_on_pool_t *arguments);
```

A dispatch on a pool that runs another is refused, so a batch finding the pool running its lender's dispatch runs on the executor's own driver thread instead of forking over the lender.

- `executor`: Output for the new executor, which the caller frees with [`ustore_executor_free`](#ustore_executor_free); `NULL` on failure.
- `outcome`: Output for the outcome, final on return.
- `ram`: The memory domains the executor serves, keeping the scratch of each thread and its arenas there; copied, not borrowed.
- `pool`: The ForkUnion `fu_pool_t` to borrow; it must outlive the executor, and keeps its lender's mood.
- `deferred_limit`: Deferred calls the executor holds at once, sizing its deferred table, where zero takes a default.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `executor` output is `NULL`, a memory domain identifier names no domain, or ForkUnion refuses the `pool` or another argument.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The process may allocate on no memory domain, or the operating system denies ForkUnion an operation.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The `deferred_limit` is past 2^30.

## ustore_executor_for_nvme

Creates an executor for the NVMe engine, whose every thread submits and polls its own I/O ring.

```c
void ustore_executor_for_nvme(ustore_executor_for_nvme_t *arguments);
```

A database [`ustore_database_open_nvme`](https://www.unum.cloud/docs/ustore/c/quickstart.md#ustore_database_open_nvme) opens on it moves bytes as `nvme` names, which the open reads once, so the executor may be freed before the databases it opened. Where its merges run is the open's own `compactions`.

- `executor`: Output for the new executor, which the caller frees with [`ustore_executor_free`](#ustore_executor_free); `NULL` on failure.
- `outcome`: Output for the outcome, final on return.
- `nvme`: How the databases it opens move bytes; copied, not borrowed.
- `threads`: Threads to spawn, each owning one submission ring, where zero derives them from the cores.
- `deferred_limit`: Deferred calls the executor holds at once, sizing its deferred table, where zero takes a default.
- `cpu_class`: The CPU class the threads confine their waits and atomics to, where zero takes what the CPU offers.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `executor` output is `NULL`, the transport or the CPU class is unknown, or ForkUnion refuses an argument.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The process may allocate on no memory domain, several threads are asked of a build without them, the CPU does not run the `cpu_class`, or the operating system denies ForkUnion an operation.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The `deferred_limit` is past 2^30.

## ustore_gpu_devices

Lists every GPU this process can drive, NVIDIA devices first, then AMD, so the position of a device is the ordinal `[ustore_executor_for_vram_t::device](#ustore_executor_for_vram)` names it by.

```c
void ustore_gpu_devices(ustore_gpu_devices_t *arguments);
```

Each vendor's runtime opens on the first call, and a vendor whose runtime is absent lists no devices, so the call succeeds with a zero count on a machine without GPUs. Devices of no class UStore knows are left out. The listing is taken once per process and never changes after.

- `outcome`: Output for the outcome, final on return.
- `devices`: Output for up to `devices_capacity` devices, in UStore ordinal order; may be `NULL`.
- `devices_capacity`: Entries `devices` holds.
- `devices_count`: The __optional__ output for every device this process can drive, which may exceed `devices_capacity`.

## ustore_executor_for_vram

Opens one submission stream on `device`, or borrows the caller's, driven by one host thread.

```c
void ustore_executor_for_vram(ustore_executor_for_vram_t *arguments);
```

Every call queues on that stream in order, so one host thread keeps the device fed, and a multi-GPU caller makes one executor per device. Its calls read through an engine's device verbs, which refuse changes and the control plane, and every call no device tower serves yet, with [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k), and arenas it makes live on its device. [`ustore_database_open_nvme`](https://www.unum.cloud/docs/ustore/c/quickstart.md#ustore_database_open_nvme) takes one of an NVIDIA device as its `compactions`, reading its device and scratch limit once. Every build spawns one, as the vendor's runtime opens at the first call and the library links none.

- `executor`: Output for the new executor, which the caller frees with [`ustore_executor_free`](#ustore_executor_free); `NULL` on failure.
- `outcome`: Output for the outcome, final on return.
- `device`: The UStore ordinal of the device every call and arena of this executor uses, its position in the listing of [`ustore_gpu_devices`](#ustore_gpu_devices); an ordinal no device carries is refused with [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k).
- `stream`: The __optional__ borrowed stream on `device`, a `cudaStream_t` on NVIDIA and a `hipStream_t` on AMD, which must outlive the executor; `NULL` creates one.
- `scratch_limit`: The __optional__ ceiling on the device scratch a call or a merge takes; zero lets scratch grow as calls ask.
- `gds`: How reads and compactions move bytes between the drives and the device.
- `deferred_limit`: Deferred calls the executor holds at once, sizing its deferred table, where zero takes a default.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `executor` output is `NULL`, the `gds` mode is unknown, or ForkUnion refuses an argument.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The process may allocate on no memory domain, or the operating system denies ForkUnion an operation.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) No device carries the ordinal, its driver refuses a stream or an event, or [`ustore_gds_direct_k`](#ustore_gds_direct_k) names a device other than NVIDIA.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The `deferred_limit` is past 2^30.

## ustore_executor_drain

Waits for every call submitted before the drain began, so later submissions never delay it.

```c
void ustore_executor_drain(ustore_executor_drain_t *arguments);
```

- `executor`: The executor to drain.
- `outcome`: Output for the outcome: an executor-level failure, such as a lost device, never one of a drained call.
- `mood`: What the threads do whenever the executor next goes idle, where the last drain wins.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `mood` is unknown.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) A callback drains the executor running it.

## ustore_executor_free

Drains every call, then frees the executor's threads, rings and scratch, and hands a borrowed pool back to its lender; `NULL` is safe, and it cannot fail, so takes no outcome.

```c
void ustore_executor_free(ustore_executor_t executor);
```

Waits until every call running on the executor, blocking or deferred, is final. Arenas it made stay valid, each keeping its memory until freed.

- `executor` (inout): The executor to be freed; invalid afterwards.

## ustore_arena_init

Creates an empty arena where `executor` computes, so its placement is decided once and device code keeps its outputs on the device without asking.

```c
void ustore_arena_init(ustore_arena_init_t *arguments);
```

Nothing is allocated until a call writes into it. An arena may outlive the executor that made it: it keeps its own memory, and the device that memory lives on, until it is freed.

- `arena`: Output for the new arena, which the caller frees with [`ustore_arena_free`](#ustore_arena_free); `NULL` on failure.
- `executor`: The executor whose calls the arena serves, which decides where it lives, as a RAM executor places it on the first of its memory domains; `NULL` serves calls on the calling thread from host pages.
- `outcome`: Output for the outcome, final on return.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena` output is `NULL`.

## ustore_arena_free

Frees the arena and every output in it; `NULL` is safe, and it cannot fail, so takes no outcome.

```c
void ustore_arena_free(ustore_arena_t arena);
```

Waits until every deferred call naming the arena is final.

- `arena` (inout): The arena to be freed; invalid afterwards.

## ustore_arena_describe

Reports where an arena's outputs live, as the executor that made it decided, so a caller reads them as host memory or hands them to device code without remembering that executor.

```c
void ustore_arena_describe(ustore_arena_describe_t *arguments);
```

- `arena`: The arena to describe.
- `outcome`: Output for the outcome, final on return.
- `description`: Output for where the arena's outputs live.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) A `NULL` arena or description.