Executors

An executor owns the threads or device streams that run calls, the traversal scratch they walk with, and the pinned bounce buffer staging passes through. One executor serves any number of indexes, of any quantization and metric, since its scratch is sized from geometry, grown to the widest asked of it, and kept between calls. A VRAM executor’s scratch_limit caps each stream’s run, which every call’s walk scratch and outputs carve from, and a call needing more reports usearch_status_out_of_memory_k. One call holds an executor at a time, while a handle admits any number of concurrent callers: the executor serializes, never the index.

int32_t const sockets[2] = {0, 1};
usearch_executor_for_ram(&(usearch_executor_for_ram_t) {.executor = &executor,
                                                        .outcome = &outcome,
                                                        .ram = {sockets, 2}, // NUMA nodes whose cores it spawns on
                                                        .threads = 0});      // zero takes 7/8 of those cores

Threads interleave across the named domains, so a half-size executor takes half of every socket rather than all of one, and the default stops at 7/8 because full occupancy starves ordered publication. usearch_executor_for_vram opens one stream per device, or borrows the caller’s, and usearch_executor_for_nvme spawns one thread per submission ring. An application with its own ForkUnion pool lends it with usearch_executor_on_pool instead, though the engine refuses borrowed pools with usearch_status_unsupported_k for now. An executor’s kind must match the residency of every handle it drives, and a mismatch reports usearch_status_device_k.

A NULL executor is accepted by every verb and runs the call on the caller’s own thread, final when it returns whatever its completion, which suits size lookups and other short calls. The exception is a VRAM verb that takes an arena: that arena must be a GPU arena on the index’s first device, and it serves only calls on the executor that made it. A second call on a held executor finishes usearch_status_contended_k, so callers wanting real parallelism give each thread its own executor. usearch_executor_drain waits until no call holds the executor, then sets its threads’ mood until the next call: usearch_mood_parked_k, the default, lets idle threads sleep in the kernel, usearch_mood_spinning_k keeps them hot. A blocking verb called from inside a progress handler or predicate of a call on the same executor reports usearch_status_contended_k, and so does a drain on the thread running that call’s body; a drain from any other of its threads would wait on itself. usearch_executor_free waits out the call holding the executor before it releases the threads; host arenas the executor made stay valid until freed, and GPU arenas it made must be freed before it.

§ Completion & Deferred Calls

Every call reports through its outcome and names its completion in the same struct, so no verb has a second, asynchronous twin. usearch_completion_blocking_k, the default, returns with the final status stored and every output readable. usearch_completion_deferred_k returns once submitted, as an io_uring entry does, and the call finishes on its executor. A thread pool, a GPU stream or a remote server can serve the same verbs without changing a call site.

Submission copies the argument struct, but everything it points to stays borrowed until the outcome is final. One call holds an executor at a time, so calls overlap by running on different executors, each with its own outcome and arena:

usearch_matches_view_t matches[2];
usearch_outcome_t outcomes[2] = {{0}};
for (size_t shard = 0; shard != 2; ++shard)
    usearch_dense_search(&(usearch_dense_search_t) {
        .index = shards[shard],
        .executor = executors[shard],
        .outcome = &outcomes[shard],
        .completion = usearch_completion_deferred_k,
        .count = 32,
        .queries = {queries, 128 * sizeof(float), 128, usearch_scalar_f32_k},
        .matches = &matches[shard],
        .arena = arenas[shard],
        .wanted = 10});
while (usearch_outcome_status(&outcomes[0]) == usearch_status_deferred_k ||
       usearch_outcome_status(&outcomes[1]) == usearch_status_deferred_k)
    do_other_work(); // polling alone finishes both

The status reads usearch_status_deferred_k until the call is final, even for a call refused at submission. Then the library stores the message and the final status with release ordering, and touches nothing the struct names after that. The caller reads it with acquire ordering: usearch_outcome_status polls and usearch_outcome_wait blocks on an atomic wait, both returning the status. Those two are the only safe readers of a deferred outcome, since the status word is atomic only by their contract. Never wait on a call’s own outcome from inside its progress handler or predicate, since the call finishes only after they return. A call ends in one order: its arena and the counts of the handles it names, then the pool, then the executor, then the message and the status. So a caller that reads success may resubmit on the same executor or free a handle at once.

A call submitted while another holds the executor finishes at once with usearch_status_contended_k, and nothing queues, so submitting never allocates. Every *_free waits until each executor call naming its handle has ended, so a handle may be dropped at any moment, though never from a callback of a call naming it. Waits, drains and frees only wait: none of them runs another thread’s call.

§ Inclusive and Exclusive Executors

A caller-inclusive pool, a single thread or a WebAssembly build needs its caller to make progress, so a deferred call on it runs at submission and is final when the verb returns. A caller-exclusive pool, which usearch_executor_for_ram and usearch_executor_for_nvme spawn whenever they spawn more than one OS thread, delegates a deferred call to its own thread 0. That thread runs the call in the caller’s place and forks the other threads as the caller would, so the call finishes while the caller does something else, one call per executor at a time. Progress handlers and predicates of a delegated call therefore run on the pool’s threads. VRAM executors run every verb at submission, since each verb reads its device results back on the host before it ends. Together these let one manager thread in int main() overlap a RAM, an NVMe and a VRAM executor: it submits a deferred call to each and polls their outcomes.