Rust · Serialization

Serialization

use usearch::DenseLoadRamBufferConfig;use usearch::DenseLoadRamConfig;use usearch::DenseLoadVramBufferConfig;use usearch::DenseLoadVramConfig;use usearch::DenseMetadata;use usearch::DenseSaveBufferConfig;use usearch::DenseSaveConfig;use usearch::DenseViewNvmeConfig;use usearch::Monolith; 3.0.0-rc.1 · 4c3e5023

The path extension picks the layout: a .safegraphs path is one monolith, and a .safegraphs.json path is a manifest cataloging many.

index.on(&executor).save("index.safegraphs")?;
let loaded = DenseIndex::load_ram("index.safegraphs")?;
let gpu = unsafe { Executor::for_vram(&[0], None, &[], 0)? };
let placed = DenseIndex::load_vram(DenseLoadVramConfig {
    executor: Some(&gpu),
    devices: &[0],
    ..DenseLoadVramConfig::from("index.safegraphs")
})?;
let viewed = DenseIndex::view_ram("index.safegraphs")?;          // read-only, served from the mapping
let metadata = DenseIndex::metadata("index.safegraphs")?;        // geometry without loading a shard

Checkpoints are residency-portable: one file materializes into RAM, uploads to a device, or opens in place on disk, decided by the constructor that reads it. The metadata carries only the four fields a checkpoint froze, so values_per_key and the placement stay yours to set in the load config. view_ram answers Status::Unsupported until RAM shards can serve a mapping in place. A stopped save leaves leftovers every reader rejects, which are yours to remove.

In-memory buffers hold single monoliths only, through save_buffer, load_ram_buffer or load_vram_buffer, and metadata_buffer.

let mut arena = Arena::new_in(&executor)?;
let monolith = index.on(&executor).save_buffer(&mut arena)?;
let restored = DenseIndex::load_ram_buffer(monolith.view()?)?;

API

StructDenseLoadRamBufferConfigA serialized monolith and how it is copied into a new index in host memory; see DenseIndex::load_ram_buffer.
dense_index.rs:839 ↗
pub struct DenseLoadRamBufferConfig<'config>

Fields

buffer&'config [u8]The monolith's bytes.
executorOption<&'config Executor>The executor the copy fans out over; None copies on the calling thread.
memory_domains&'config [i32]NUMA node IDs to place shards on, the monolith loading whole into the first; empty derives every node the process may use.
cpu_classOption<CPUClass>The class the index dispatches as; None dispatches as the machine.
values_per_keyusizeUpper bound on entries sharing one key, which no checkpoint records; zero is unbounded.
progressOption<&'config mut ProgressCallback<'config>>Reports the bytes placed and stops wherever it says to; a stopped load creates no index.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<&'config [u8]>, From<&'config Bytes>
StructDenseLoadRamConfigA checkpoint path and how it is copied into a new index in host memory; see DenseIndex::load_ram.
dense_index.rs:771 ↗
pub struct DenseLoadRamConfig<'config>

Fields

path&'config PathA .safegraphs monolith or a .safegraphs.json manifest.
executorOption<&'config Executor>The executor the read fans out over; None reads on the calling thread.
memory_domains&'config [i32]NUMA node IDs to place the parts on, part p loading whole into shard p modulo their count; empty derives every node the process may use.
cpu_classOption<CPUClass>The class the index dispatches as; None dispatches as the machine.
values_per_keyusizeUpper bound on entries sharing one key, which no checkpoint records; zero is unbounded.
progressOption<&'config mut ProgressCallback<'config>>Reports the shards placed and stops wherever it says to; a stopped load creates no index.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<&'config Location>
StructDenseLoadVramBufferConfigA serialized monolith and how it is uploaded into a new index in device memory; see DenseIndex::load_vram_buffer.
dense_index.rs:877 ↗
pub struct DenseLoadVramBufferConfig<'config>

Fields

buffer&'config [u8]The monolith's bytes, in host or device memory.
executorOption<&'config Executor>The executor the upload runs on; None uploads on the default stream of each device.
devices&'config [i32]CUDA device ordinals to place shards on, the monolith loading whole into the first; empty derives one free device.
gpu_classOption<GPUClass>The class the index dispatches as; None dispatches as the devices.
values_per_keyusizeUpper bound on entries sharing one key, which no checkpoint records; zero is unbounded.
progressOption<&'config mut ProgressCallback<'config>>Reports the bytes placed and stops wherever it says to; a stopped load creates no index.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<&'config [u8]>, From<&'config Bytes>
StructDenseLoadVramConfigA checkpoint path and how it is uploaded into a new index in device memory; see DenseIndex::load_vram.
dense_index.rs:805 ↗
pub struct DenseLoadVramConfig<'config>

Fields

path&'config PathA .safegraphs monolith or a .safegraphs.json manifest.
executorOption<&'config Executor>The executor the upload runs on; None uploads on the default stream of each device.
devices&'config [i32]CUDA device ordinals to place the parts on, part p loading whole into device p modulo their count; empty derives one free device.
gpu_classOption<GPUClass>The class the index dispatches as; None dispatches as the devices.
values_per_keyusizeUpper bound on entries sharing one key, which no checkpoint records; zero is unbounded.
progressOption<&'config mut ProgressCallback<'config>>Reports the shards placed and stops wherever it says to; a stopped load creates no index.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<&'config Location>
StructDenseMetadataThe geometry and metric a checkpoint froze, readable without loading it.
dense_index.rs:654 ↗
pub struct DenseMetadata

values_per_key and the placement are the reader's to choose when loading, so a checkpoint fixes neither.

Fields

dimensionsusizeScalars per stored vector.
metricDenseMetricThe metric kind the checkpoint scores with.
quantizationScalarThe scalar type its vectors are stored and quantized into.
connectivityusizeConnections per node the graph was built with.

Implements

Clone, Copy, StructuralPartialEq, PartialEq, Eq, Debug, Hash
StructDenseSaveBufferConfigThe arena a monolith is written into and the handler counting its bytes; see DenseVerbs::save_buffer.
dense_index.rs:1195 ↗
pub struct DenseSaveBufferConfig<'config, 'arena>

Fields

arena&'arena mut ArenaThe arena the monolith lands in, which the result borrows.
progressOption<&'config mut ProgressCallback<'config>>Reports the bytes written and stops wherever it says to; a stopped save leaves no monolith.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<&'arena mut Arena>
StructDenseSaveConfigThe path a checkpoint is written to and the handler counting its shards; see DenseVerbs::save.
dense_index.rs:1174 ↗
pub struct DenseSaveConfig<'config, Location>

Fields

pathLocationA .safegraphs path for one monolith, a .safegraphs.json path for a manifest.
progressOption<&'config mut ProgressCallback<'config>>Reports the shards written and stops wherever it says to.
progress_stepsusizeReports the call makes at most, the last at completion; zero derives to ten.

Implements

From<Location>
StructDenseViewNvmeConfigA checkpoint path and the drives serving it in place; see DenseIndex::view_nvme.
dense_index.rs:914 ↗
pub struct DenseViewNvmeConfig<'config>

Fields

path&'config PathA .safegraphs monolith or a .safegraphs.json manifest.
executorOption<&'config Executor>The NVMe executor whose rings read the checkpoint's headers; None reads on the calling thread.
paths&'config [&'config Path]Devices or directories to serve the parts from; empty serves each where its manifest points.
cpu_classOption<CPUClass>The class the index dispatches as; None dispatches as the machine.

Implements

From<&'config Location>
StructMonolithThe bytes DenseVerbs::save_buffer exported into the arena it borrows.1
dense_index.rs:1321 ↗
pub struct Monolith<'arena>

Rust-only.

viewThe monolith's bytes, refused with Status::Device over device memory.
dense_index.rs:1328 ↗
pub fn view(&self) -> Result<&'arena [u8]>