An executor owns host threads or device streams, and the traversal scratch each of them walks with, grown to a high-water mark.
One executor serves any number of indexes, and its residency must match the one their handles were built for.
There is a constructor per residency, each taking only what its own hardware has, and the placement lists are the ones the indexes were built with, which keeps compute and memory domains in agreement.
let threads = Executor::for_ram(&[], None, 8)?; // eight workers, where zero takes 7/8 of the coreslet domains = Executor::for_ram(&[0, 1], None, 0)?; // threads interleaved across two NUMA nodeslet pinned = Executor::for_ram(&[], Some(CPUClass::IntelHaswell), 0)?; // waiters and atomics of one classlet rings = Executor::for_nvme(&["/mnt/nvme0"], None, 4)?; // one submission ring apiecelet streams = unsafe { Executor::for_vram(&[0, 1], None, &[], 0)? }; // one stream per device, scratch grown as askedlet shared = unsafe { Executor::on_pool(&[], None, pool)? }; // shares the threads another library spawnedlet local = Executor::default(); // owns nothing, runs on the calling threadthreads.drain(Mood::Spinning)?; // waits out the holding call, then keeps spinning
for_ram and for_nvme take a thread count, where zero derives one. for_vram takes none, since one host thread drives one stream per device, and may borrow your CUDA streams, one per device. for_vram and on_pool are unsafe, since the streams or the pool they borrow must outlive the executor; the engine wires neither borrowing yet and answers Status::Unsupported. Executor::default() owns nothing and runs every call handed to it on the calling thread.
One call holds an executor at a time, so two threads that genuinely overlap take one apiece; an executor is Send but not Sync, while index handles stay freely shared. drain waits until no call holds the executor and sets the Mood idle workers adopt, and dropping an executor drains it, unless a GPU arena it made still holds it, which then does so on its own drop.
The receiver decides where a verb runs and how it completes, while the verb and its arguments stay the same:
index.search(..) runs on the calling thread, through &DenseIndex.
index.on(&executor).search(..) blocks on the executor.
index.deferred(&executor).search(..) submits the call and answers a Deferred.
DenseVerbs writes every verb once for all three.
A blocking verb returns its Result directly, so nothing about asynchrony reaches a caller who never asks for it.
An On borrows the index and the executor, so neither it nor a Deferred it submits can outlive them.
A Deferred owns the inputs the library reads until the call is final, and borrows the index and the arena, as a blocking call does.
The waited result is the Matches a blocking search answers, borrowing the same arena. status answers Ok(()) once the call succeeded, Err(Status::Deferred) while it is not final, and its failure otherwise.
Inside an async block, .await polls the call’s status until it is final, so other tasks run meanwhile, while wait blocks the thread without spinning.
The crate starts no thread, so waiting off-thread goes through a scoped thread, as std::thread::scope(|scope| scope.spawn(move || call.wait()).join()) does.
Dropping an unfinished Deferred waits for the call, and a panic a handler raised resurfaces from wait, .await, or a drop outside unwinding.
The library reads a deferred call’s inputs after the verb returns, so a deferred receiver takes owned inputs: keys in a Vec or an array, rows in a NumKong Matrix or a 'staticMatrixView, a PathBuf, String or 'static path, and 'static configs, handlers and value lookups.
Lending &[Key] or &MatrixView fails to compile with “a deferred receiver needs owned inputs”, and a config or view borrowing a local fails as “does not live long enough”.
A deferred call allocates one box holding its outcome, argument struct and inputs until it is final.
The box is the crate’s rather than yours, since a Deferred leaked without a wait would leave the library writing into storage you had reused.
One call holds an executor at a time, so a second call finding it held finishes at once with Status::Contended, and nothing queues.
A deferred call on a caller-exclusive pool runs on the pool’s thread 0, so polling alone finishes it, and its progress handler and predicate run on the pool’s threads; an inclusive pool, a single thread, WebAssembly and a VRAM executor run it at submission.
Calls depending on each other are ordered by waiting on the first before submitting the second, since no call carries an ordering field.
API
TraitDenseVerbsEvery dense verb, written once for every receiver: &DenseIndex blocks on the calling thread, and the On that DenseIndex::on or DenseIndex::deferred makes runs on an executor.21
A blocking receiver answers what the call produced; a deferring one answers a
Deferred that borrows the index. Every configurable verb's config
converts into its Dense*Config, so the bare value is the whole call and
..DenseSearchConfig::from(10) reaches every knob.
Every verb whose output scales with the call borrows an Arena last as &mut Arena, blocking
or deferring alike, except save_buffer, whose config holds it, and its result borrows the arena
in turn.
fn insert<'config, Keys: AsRef<[Key]>, Rows: RowsView>(
self,
keys: Keys,
config: impl Into<DenseInsertConfig<'config, Rows>>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseInsert<'config, Keys, Rows>>
where
Self::Completion: Accepts<Keys> + Accepts<DenseInsertConfig<'config, Rows>>,
A batch is all-or-nothing and becomes visible at once. Every entry is claimed before the first
progress report, so a Break is ignored. Overlaps safely with concurrent searches on the
same handle, and the input scalar type is independent of the stored quantization - the
library converts on ingest.
*× 2Re-selects the neighbors of every stored entry against the finished graph, answering a RefineReport of the nodes re-selected and changed and the islands it re-linked.
fn refine<'config>(
self,
config: DenseRefineConfig<'config>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseRefine<'config>>
where
Self::Completion: Accepts<DenseRefineConfig<'config>>,
A cheap insertion pass followed by refinement beats one expensive pass, so a wide expansion
factor belongs here rather than in DenseVerbs::insert. On a RAM handle searches may run
throughout, on a VRAM one they may not, and insertions never may. A stop is a time budget:
the call still re-links its islands, leaving the graph correct, and answers
Status::Cancelled.
*× 2Audits how much of the graph a search reaches: forward reachability from as many landmarks as one search draws, swept level by level.
fn search<'config, 'arena, Queries: RowsView>(
self,
queries: Queries,
config: impl Into<DenseSearchConfig<'config>>,
arena: &'arena mut Arena,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseSearch<'config, 'arena, Queries>>
where
Self::Completion: Accepts<Queries> + Accepts<DenseSearchConfig<'config>>,
Never writes to the graph, so any number of these may overlap with concurrent insertions.
*× 2Late-interaction scoring of multi-row queries against multi-row entries, writing up to wanted keys per query, largest score first, in arena as one ragged run; zero keeps every key a query scored.
Every query row takes its best match within a key, and those maxima sum to that key's
score - so rows_per_query is the token count each query occupies, queries holds them
end to end, and wanted counts documents rather than rows. This is what values_per_key
above one exists for. Runs on an inner-product index alone.
fn multiply<'config, 'arena, Left: RowsView>(
self,
left: Left,
config: impl Into<DenseMultiplyConfig<'config>>,
arena: &'arena mut Arena,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseMultiply<'config, 'arena, Left>>
where
Self::Completion: Accepts<Left> + Accepts<DenseMultiplyConfig<'config>>,
The linear-algebra sibling of DenseVerbs::search, reporting every pair the traversal
scored rather than only the nearest. Pairs arrive in traversal order rather than sorted.
Runs on an inner-product index alone, since a product is a logit.
*× 2Softmax attention of every query row over every indexed key its walk scores, folded as Flash Attention folds key tiles, with no top-k cut, writing into arena as rows of the queries' own scalar type.
The index holds one layer's keys for one key-value head and no values: value_lookup
resolves every scored key to its value row, as wide as the keys and stored in the index's
own quantization. A key left without a row, or given one of another type or width, refuses the
call with Status::ShapeMismatch, since a missing row is a wrong softmax rather than a
smaller one. A VRAM walk resolves values on the device, where no Rust closure runs, so an
arena from a VRAM executor is refused with Status::Device. The outputs come back
already divided by the partition. Runs on an inner-product index alone, since a softmax
needs logits.
*× 2Absorbs every donor into this index, one after another, keeping their graphs rather than re-inserting their vectors and linking each in by walks across the appended part.
fn absorb<'config, Donors: IntoIterator<Item = DenseIndex>>(
self,
config: impl Into<DenseAbsorbConfig<'config, Donors>>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseAbsorb<'config>>
where
Self::Completion: Accepts<DenseAbsorbConfig<'config, Donors>>,
Readers of a RAM index may keep searching throughout, while a VRAM one admits no
overlapping search. The donors are consumed whether or not the absorb succeeds: a stop cuts
only the closing refinement, every donor already moved, and answers Status::Cancelled,
while a failure frees the donors it never reached and may leave the failing donor's entries
behind.
*× 2Splits this index into the empty receivers, the inverse of [DenseVerbs::absorb]: the entries are partitioned along the graph, each part lands in one receiver with its edges kept, and a closing refinement re-links every receiver, leaving this index empty and live.
fn split<'config>(
self,
config: impl Into<DenseSplitConfig<'config>>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseSplit<'config>>
where
Self::Completion: Accepts<DenseSplitConfig<'config>>,
*× 2Draws keys around the focus, or across the whole index, each hanging from a shown or earlier draw and placed in three dimensions beside the shown ones, writing up to wanted of them into arena as a Sample.
fn save<'config, Location: AsRef<Path>>(
self,
config: impl Into<DenseSaveConfig<'config, Location>>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseSave<'config, Location>>
where
Self::Completion: Accepts<DenseSaveConfig<'config, Location>>,
Demands a mutation-free window; concurrent readers keep searching throughout. A stopped save
leaves leftovers every reader rejects and the caller is left to remove.
save_*_buffer× 2Saves the index as one monolith into the arena its config carries.
fn save__buffer<'config, 'arena>(
self,
config: impl Into<DenseSaveBufferConfig<'config, 'arena>>,
) -> <<Self as DenseVerbs<'index>>::Completion as Completion>::Answer<'index, DenseSaveBuffer<'config, 'arena>>
where
Self::Completion: Accepts<DenseSaveBufferConfig<'config, 'arena>>,
*× 2Opens a walk over the current contents, pulled blocking on this receiver's executor; each shard's publication watermark is read as the walk enters it, so an entry once passed is never revisited.
Without starts the walk yields every entry as stored. With them it yields the start keys'
own entries first, in order, then the entries those link to breadth-first, each entry once.
search_*_stream× 2Opens a resumable nearest-neighbor stream for a one-row query, pulled blocking on this receiver's executor.
Best-first traversal is naturally pausable, so callers that post-filter aggressively pull
until satisfied instead of guessing "k" upfront. The stream keeps walk state rather than a
thread.
No residency implements the stream yet, so the engine answers Status::Unsupported.
StructBlockingThe completion of a blocking receiver, whose verbs return once their call is final and answer what it produced.
StructDeferredA submitted call, which owns its outcome, argument struct and inputs in one box the library reads until the call is final, and borrows its arena as a blocking call does.2
Deferred::status polls without blocking, and Deferred::wait blocks until the call is
final and hands back what it produced, while .await polls it from async code. Dropping an
unfinished one waits for it, then frees the box.
A panic a handler raised resurfaces from wait, .await, or a drop outside unwinding.
'index is the borrow of the On that submitted it.
Rust-only.
Implements
Drop, Debug, Send, IntoFuture
statusErr(Status::Deferred) until the call is final, then its final status, read without blocking.
One executor serves any number of index handles, so separate categories of indexes share one
machine's cores or devices instead of each claiming a private set. Its kind is fixed by the
constructor that produced it and must match the residency of every handle it drives.
Executor::default is the null executor: it owns nothing, and every call
handed it runs on the calling thread.
Calls on one executor start in submission order, so threads wanting to overlap take one apiece;
it is Send but not Sync.
Every constructor takes a deferred_limit sizing its table of deferred calls, where zero takes
the default; a deferred submission finding the table full fails with Status::Contended.
Arena::new_in places an arena where it computes: a host arena may outlive it, while a GPU
arena holds it until the arena drops.
Dropping an executor drains it, unless a GPU arena it made still holds it, which then does so.
Implements
Debug, Send, Default
for_ramSpawns host workers for RAM-resident indexes, interleaved across the compute domains local to memory_domains, which is the same list the indexes were initialized with.
Zero threads takes 7/8 of the cores those domains carry, since full occupancy starves
the ordered publication path.
A pinned cpu_class narrows the threads' waiters and atomics to that class's.
on_poolCreates a RAM executor over a borrowed ForkUnion pool, so several libraries in one process share one set of threads while each keeps its own scratch.
Zero threads derives from those devices' queue capabilities.
A pinned cpu_class narrows the threads' waiters and atomics to that class's.
for_vramOpens one submission stream on every named device, or borrows the caller's, all of them driven by the one host thread that calls the verbs - which is why this takes no thread count.
streams is empty, or holds one CUDA stream per device in devices order. Zero
scratch_limit lets scratch grow as geometry asks. A pinned gpu_class the devices do not
run is refused with Status::Unsupported.
Safety
Every stream must be a live CUDA stream on its device that outlives the executor.
drainWaits for every call submitted to this executor, then adopts mood whenever it next goes idle; a failure is the executor's own, such as a lost device, never one of a drained call.
The library has no wake-up, so an executor re-polls it while other tasks run, spinning only
when nothing else is ready; Deferred::wait blocks without spinning.