Go · Serialization

Serialization

import "github.com/unum-cloud/USearch/golang" 3.0.0-rc.1 · 4c3e5023

Checkpoints ride the SafeGraphs interchange format, and the path extension picks the layout. A .safegraphs path is one self-describing monolith; a .safegraphs.json path is a manifest cataloging many, with every shard path resolved relative to it. In-memory buffers hold single monoliths only.

if err := index.On(executor).Save("index.safegraphs", nil); err != nil {
		log.Fatal(err)
}

restored, err := usearch.DenseLoadRam("index.safegraphs", &usearch.DenseLoadRamConfig{Executor: executor})
	if err != nil {
		log.Fatal(err)
}
defer restored.Close()

monolith, err := index.SaveBuffer(arena, nil) // sized by the call itself, see Arenas
serialized, err := monolith.Bytes()
copied, err := usearch.DenseLoadRamBuffer(serialized, nil)

Loading creates the index, freezing its geometry from the checkpoint’s own headers. DenseMetadataRead reads that geometry without opening a shard:

metadata, err := usearch.DenseMetadataRead("index.safegraphs", nil) // a nil config reads on the calling goroutine
fmt.Println(metadata.Dimensions, metadata.Metric, metadata.Quantization)

The result carries only the four fields a checkpoint froze, so ValuesPerKey and the placement stay yours to set in the load config. Serving a checkpoint in place, rather than copying it, yields a read-only handle whose mutating verbs fail with StatusReadOnly, through a memory mapping with DenseViewRam, or from drives with DenseViewNvme:

mapped, err := usearch.DenseViewRam("index.safegraphs")
served, err := usearch.DenseViewNvme("index.safegraphs", &usearch.DenseViewNvmeConfig{
	Executor: drives,
	Paths:    []string{"/dev/nvme0n1"},
})

API

Types

DenseLoadRamBufferConfigDenseLoadRamBufferConfig carries the optional settings of DenseLoadRamBuffer.
types.go:301 ↗
type DenseLoadRamBufferConfig struct {
    Executor *Executor
    MemoryDomains []int32
    CPUClass CPUClass
    ValuesPerKey uint
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

Executor*ExecutorExecutor is the one the copy fans out over; nil copies on the calling goroutine.
MemoryDomains[]int32MemoryDomains are the NUMA node IDs to place shards on, the monolith loading whole into the first; empty derives every node the process may use.
CPUClassCPUClassCPUClass is the class to dispatch as; the zero value derives the CPU's own.
ValuesPerKeyuintValuesPerKey bounds how many entries may share one key; zero is unbounded.
ProgressProgressCallbackProgress reports the bytes placed and stops wherever it says to.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseLoadRamConfigDenseLoadRamConfig carries the optional settings of DenseLoadRam.
types.go:265 ↗
type DenseLoadRamConfig struct {
    Executor *Executor
    MemoryDomains []int32
    CPUClass CPUClass
    ValuesPerKey uint
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

Executor*ExecutorExecutor is the one the read fans out over; nil reads on the calling goroutine.
MemoryDomains[]int32MemoryDomains are the NUMA node IDs to place the parts on, part p loading whole into shard p modulo their count; empty derives every node the process may use.
CPUClassCPUClassCPUClass is the class to dispatch as; the zero value derives the CPU's own.
ValuesPerKeyuintValuesPerKey bounds how many entries may share one key, which no checkpoint records; zero is unbounded.
ProgressProgressCallbackProgress reports the shards placed and stops wherever it says to; a stopped load creates no index.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseLoadVramBufferConfigDenseLoadVramBufferConfig carries the optional settings of DenseLoadVramBuffer.
types.go:318 ↗
type DenseLoadVramBufferConfig struct {
    Executor *Executor
    Devices []int32
    GPUClass GPUClass
    ValuesPerKey uint
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

Executor*ExecutorExecutor is the one the upload runs on; nil uploads on the default stream of each device.
Devices[]int32Devices are the CUDA ordinals to place shards on, the monolith loading whole onto the first; empty derives one free device.
GPUClassGPUClassGPUClass is the class to dispatch as; the zero value derives the devices' own.
ValuesPerKeyuintValuesPerKey bounds how many entries may share one key; zero is unbounded.
ProgressProgressCallbackProgress reports the bytes placed and stops wherever it says to.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseLoadVramConfigDenseLoadVramConfig carries the optional settings of DenseLoadVram.
types.go:284 ↗
type DenseLoadVramConfig struct {
    Executor *Executor
    Devices []int32
    GPUClass GPUClass
    ValuesPerKey uint
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

Executor*ExecutorExecutor is the one the upload runs on; nil uploads on the default stream of each device.
Devices[]int32Devices are the CUDA ordinals to place the parts on, part p loading whole onto device p modulo their count; empty derives one free device.
GPUClassGPUClassGPUClass is the class to dispatch as; the zero value derives the devices' own.
ValuesPerKeyuintValuesPerKey bounds how many entries may share one key; zero is unbounded.
ProgressProgressCallbackProgress reports the shards placed and stops wherever it says to.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseMetadataReadBufferConfigDenseMetadataReadBufferConfig carries the optional settings of DenseMetadataReadBuffer.
types.go:353 ↗
type DenseMetadataReadBufferConfig struct {
    Executor *Executor
}

Fields

Executor*ExecutorExecutor is the one the read runs on; nil reads on the calling goroutine.
DenseMetadataReadConfigDenseMetadataReadConfig carries the optional settings of DenseMetadataRead.
types.go:347 ↗
type DenseMetadataReadConfig struct {
    Executor *Executor
}

Fields

Executor*ExecutorExecutor is the one the read runs on; nil reads on the calling goroutine.
DenseSaveBufferConfigDenseSaveBufferConfig carries the optional settings of SaveBuffer.
types.go:513 ↗
type DenseSaveBufferConfig struct {
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

ProgressProgressCallbackProgress reports the bytes written and stops wherever it says to; a stopped save leaves no monolith.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseSaveConfigDenseSaveConfig carries the optional settings of Save.
types.go:505 ↗
type DenseSaveConfig struct {
    Progress ProgressCallback
    ProgressSteps uint
}

Fields

ProgressProgressCallbackProgress reports the shards written and stops wherever it says to.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
DenseViewNvmeConfigDenseViewNvmeConfig carries the optional settings of DenseViewNvme.
types.go:335 ↗
type DenseViewNvmeConfig struct {
    Executor *Executor
    Paths []string
    CPUClass CPUClass
}

Fields

Executor*ExecutorExecutor is the NVMe executor whose rings read the checkpoint's headers; nil reads on the calling goroutine.
Paths[]stringPaths are the devices or directories to serve the parts from; empty serves each where its manifest points.
CPUClassCPUClassCPUClass is the class to dispatch as; the zero value derives the CPU's own.
StructDenseMetadataDenseMetadata is the geometry and metric a checkpoint froze - everything a reader learns without opening it, and nothing a reader must choose for itself.2
dense_index.go:133 ↗
type DenseMetadata struct {
    Dimensions uint
    Metric DenseMetric
    Quantization Scalar
    Connectivity uint
}

ValuesPerKey is absent because it bounds a live handle's mutable state rather than anything the serialized form carries, and the placement is absent because a checkpoint records where it was written, never where a reader must keep it. Both stay yours to set in the load config.

Fields

DimensionsuintDimensions is the scalars per stored vector.
MetricDenseMetricMetric is the distance kind the checkpoint scores with.
QuantizationScalarQuantization is the scalar type its vectors are stored and quantized into.
ConnectivityuintConnectivity is the connections per node the graph was built with.
DenseMetadataReadDenseMetadataRead reads a checkpoint's geometry and metric without loading it - from the manifest's summary when the path names one, so no shard is opened.
dense_index.go:924 ↗
func DenseMetadataRead(
    path string,
    config *DenseMetadataReadConfig,
) (DenseMetadata, error)
DenseMetadataReadBufferDenseMetadataReadBuffer reads geometry and metric out of a serialized monolith buffer.
dense_index.go:945 ↗
func DenseMetadataReadBuffer(
    buffer []byte,
    config *DenseMetadataReadBufferConfig,
) (DenseMetadata, error)
StructMonolithMonolith is the serialized index SaveBuffer wrote into its arena.1
dense_index.go:601 ↗
type Monolith struct {
    // contains filtered or unexported fields
}
BytesBytes borrows the monolith from its arena.
dense_index.go:607 ↗
func (monolith Monolith) Bytes() ([]byte, error)