Go · Retrieval and Streaming

Retrieval and Streaming

import "github.com/unum-cloud/USearch/golang" 3.0.0-rc.1 · 4c3e5023

Get exports every entry stored under one key, converted to the scalar kind you name, in one call however many there turn out to be:

vectors, _ := index.Get(usearch.KeyFromUint64(42), usearch.ScalarF32, arena)
// vectors.Rows() rows of vectors.Columns(), vectors.Stride() bytes apart
values, _ := usearch.DenseValues[float32](vectors)

A cursor walks the contents, each pull writing up to wanted keys, and the keys each links to for NextLinks, into an arena; the vectors themselves come from Get. NextKeys pulls the keys alone, and a batch of none means the walk is exhausted. Cursor() yields every entry as stored, and Cursor(starts...) the starts’ own entries first, then the entries those link to breadth-first, each once. Cursors answer on RAM and NVMe handles, while VRAM entries are read back by key or by search.

cursor, _ := index.On(executor).Cursor()
defer cursor.Close()
for {
	batch, err := cursor.NextLinks(1024, arena)
	if err != nil || batch.Written() == 0 {
		break
	}
	keys, _ := batch.Keys()
	first, _ := batch.Links(0)
	consume(keys, first)
}

A search stream resumes one query’s walk, so a caller that post-filters aggressively pulls until satisfied instead of guessing the wanted count upfront. Each Next writes up to wanted next-nearest matches into an arena as the one query of a search, and a query with none means the walk is drained. Ordering beyond the expansion horizon is best-effort nearest rather than globally exact. No residency streams matches yet, so SearchStream answers StatusUnsupported today.

stream, err := index.On(executor).SearchStream(usearch.RowsOf(query, dimensions), nil)
	if err != nil {
		log.Fatal(err)
}
defer stream.Close()
for accepted := 0; accepted < 10; {
	matches, err := stream.Next(64, arena)
	if err != nil {
		log.Fatal(err)
}
	found, _, _ := matches.Query(0)
	if len(found) == 0 {
		break // the walk is drained
}
	for _, key := range found {
		if admits(key) {
			accepted++
}
}
}

§ Sampling

Sample draws keys around Focus, or across the whole index, each hanging from a shown or earlier draw and placed in three dimensions beside the Shown keys already on screen, so a view grows one connected sample as it zooms. Equal seeds, shown keys and focus draw equally, and a larger wanted draws a superset.

draws, err := index.On(executor).Sample(256, arena, &usearch.DenseSampleConfig{
	Shown: shown, ShownCoordinates: places, Focus: near, Seed: 42,
})
keys, _ := draws.Keys()
entryOrdinals, _ := draws.EntryOrdinals()
parent, distance, _ := draws.Parents().Query(0) // empty for a root
points, _ := usearch.DenseValues[float32](draws.Coordinates())

API

Types

DenseSampleConfigDenseSampleConfig carries the optional settings of Sample.
types.go:484 ↗
type DenseSampleConfig struct {
    Shown []Key
    ShownCoordinates [][3]float32
    Focus []Key
    Seed uint64
    ExpansionFactor uint
    Progress ProgressCallback
    ProgressSteps uint
    Stats *DenseTraversalStats
}

A zero value draws across the whole index with nothing shown.

Fields

Shown[]KeyShown holds the keys already shown: never drawn again, and the paths reach toward them.
ShownCoordinates[][3]float32ShownCoordinates places each Shown key, one point apiece.
Focus[]KeyFocus holds the keys to look around; empty looks around the whole index.
Seeduint64Seed makes the draw repeatable: equal seeds, shown keys and focus over an unchanged index draw equally.
ExpansionFactoruintExpansionFactor is the breadth of the candidates of the walks behind the draws; zero derives.
ProgressProgressCallbackProgress reports the draws and stops wherever it says to.
ProgressStepsuintProgressSteps bounds the reports Progress hears, the last at completion; zero derives to ten.
Stats*DenseTraversalStatsStats is where the traversal telemetry lands, overwritten by the call.
DenseSearchStreamInitConfigDenseSearchStreamInitConfig carries the optional settings of SearchStream.
types.go:522 ↗
type DenseSearchStreamInitConfig struct {
    ExpansionFactor uint
}

Fields

ExpansionFactoruintExpansionFactor is the breadth of the candidates the walk explores; zero derives it from the geometry.
StructDenseBatchDenseBatch is one cursor pull written into its arena: the keys, and beside them the keys each links to when NextLinks asked for them; the vectors themselves come from Get.3
dense_index.go:616 ↗
type DenseBatch struct {
    // contains filtered or unexported fields
}
KeysKeys borrows the entries' keys from the arena.
dense_index.go:627 ↗
func (batch DenseBatch) Keys() ([]Key, error)
WrittenWritten reports the entries the pull wrote; zero means the walk is exhausted.
dense_index.go:624 ↗
func (batch DenseBatch) Written() uint
StructDenseCursorDenseCursor is a walk over the index contents, pulled blocking on the executor of the receiver that opened it.3
dense_index.go:1817 ↗
type DenseCursor struct {
    // contains filtered or unexported fields
}

Each shard's publication watermark is read as the walk enters that shard, so an entry once passed is never revisited.

CloseClose releases the cursor.
dense_index.go:1894 ↗
func (cursor *DenseCursor) Close() error

It is idempotent.

NextKeysNextKeys copies up to wanted next keys into arena; a batch of none means the walk is exhausted.
dense_index.go:1845 ↗
func (cursor *DenseCursor) NextKeys(
    wanted uint,
    arena *Arena,
) (DenseBatch, error)
StructDenseDrawsDenseDraws is what Sample wrote into its arena, one entry per draw: its key, which of that key's entries it is, the shown or earlier draw it hangs from, and where it is placed.5
dense_index.go:662 ↗
type DenseDraws struct {
    // contains filtered or unexported fields
}
CoordinatesCoordinates views where each draw is placed, one row of three float32 apiece.
dense_index.go:692 ↗
func (draws DenseDraws) Coordinates() DenseVectors
EntryOrdinalsEntryOrdinals borrows which of its key's entries each draw is, in the order Get writes them.
dense_index.go:679 ↗
func (draws DenseDraws) EntryOrdinals() ([]uint, error)
KeysKeys borrows the drawn keys from the arena.
dense_index.go:671 ↗
func (draws DenseDraws) Keys() ([]Key, error)
ParentsParents views each draw's parent and its distance as one column per draw, a root holding none.
dense_index.go:687 ↗
func (draws DenseDraws) Parents() MatchesView
WrittenWritten reports the draws; fewer than wanted only once the region is exhausted.
dense_index.go:668 ↗
func (draws DenseDraws) Written() uint
StructDenseSearchStreamDenseSearchStream is a resumable nearest-neighbor walk over one query, pulled blocking on the executor of the receiver that opened it.2
dense_index.go:1906 ↗
type DenseSearchStream struct {
    // contains filtered or unexported fields
}

Ordering beyond the expansion horizon is best-effort nearest, not globally exact.

CloseClose releases the stream and the walk state it keeps.
dense_index.go:1970 ↗
func (stream *DenseSearchStream) Close() error

It is idempotent.

NextNext pulls up to wanted next-nearest matches into arena, as the one query of a search; a query with none means the walk is drained.
dense_index.go:1940 ↗
func (stream *DenseSearchStream) Next(
    wanted uint,
    arena *Arena,
) (MatchesView, error)
StructDenseVectorsDenseVectors is the rows a call wrote packed into its arena: Get's entries sharing a key, or Attend's attended rows.4
dense_index.go:534 ↗
type DenseVectors struct {
    // contains filtered or unexported fields
}
ColumnsColumns reports the values in each row, as the library wrote them.
dense_index.go:547 ↗
func (vectors DenseVectors) Columns() uint
RowsRows reports how many rows the call wrote; zero when Get found the key absent.
dense_index.go:541 ↗
func (vectors DenseVectors) Rows() uint
ScalarScalar reports the scalar type the rows were written as.
dense_index.go:550 ↗
func (vectors DenseVectors) Scalar() Scalar
StrideStride reports the byte pitch between consecutive rows.
dense_index.go:544 ↗
func (vectors DenseVectors) Stride() uint