# Reads and Writes

> UStore for Rust

### Binary Values

Beneath the collections, every verb takes a batch, names its collections by identifier, and hands back what its call produced.

```rust
use ustore::{Arena, CollectionID, RAMOptions, Database, Durability, Entry, Key, Matching, Modality};

let database = Database::init_ram(&RAMOptions::default())?;
let mut arena = Arena::new()?;

let [ada, grace, kent] = [1u64, 2, 3].map(Key::from);
let people = database.collection_create("people", Modality::Blobs)?;
let entries = [Entry::new(ada, b"Ada"), Entry::new(grace, b"Grace")];
database.head().write(people, &entries, Durability::Buffered)?;

let values = database.head().read(people, &[ada, grace, kent], &mut arena)?;
assert_eq!(values.get(0), Some(b"Ada".as_slice()));
assert_eq!(values.get(2), None);
```

An `Entry` pairs a key with borrowed bytes, laid out so a slice of them crosses as it stands: the keys as one strided run, and the values through an accessor over the same entries, so nothing is staged on the way in.
A write also takes owned keys beside a `stringtape::BytesTape<u32>` or `BytesTape<u64>` of their values, one more offset than keys into one run of bytes, which the library reads as it stands.
A read hands back a `Values`, which addresses the tape in the arena rather than copying out of it, and reports a key the database does not hold as `None`.
`Values::tape` hands the whole batch out as a `Tape` over that tape: one more offset than values, and the bytes between them, 32-bit while the bytes fit in 4 GiB, as Arrow lays out `binary`, and 64-bit beyond, as it lays out `large_binary`.
A missing value is empty there, and `Values::presences` is the bitmap of the keys found, as Arrow reads a validity buffer.

A write carries values and nothing else, so an empty slice writes an empty value, and `On::remove` removes keys with their values in a batch of its own.
A removal therefore never hides inside a batch of writes.

```rust
let entries = [Entry::new(ada, b"rewritten"), Entry::new(kent, b"")];
database.head().write(people, &entries, Durability::Buffered)?;
database.head().remove(people, &[grace], Durability::Buffered)?;

let first = database.head().scan(people, Key::from(0u64), 100, &mut arena)?.keys;
let measured = database.head().measure(people, ada..kent, &mut arena)?;
let largest = KeyRange { direction: ScanDirection::Descending, ..KeyRange::default() };
let last = database.head().scan(people, largest, 10, &mut arena)?.keys;
let headers = database.head().read_slices(people, &[ada, kent], ..64, &mut arena)?;
database.head().remove_range(people, ada..kent, Durability::Buffered)?;
```

`On::remove` removes whole documents the same way, while a field goes with a merge that carries a JSON `null` for it.
`scan` lists up to the count it wants from a start key, `sample` draws keys at random, and `measure` bounds the keys and the bytes a range of keys holds.
Scans, cursors and measures take a `KeyRange`: a start key, inclusive, an end key, exclusive, and a `ScanDirection`, where a key alone walks up from it and `start..end` walks up between them.
`read_slices` reads one byte range of every value, and `remove_range`, `docs_remove_range` and `graph_remove_vertices_range` remove every key of a range in one atomic call.
`docs_sample`, `pages_sample` and `graph_sample_vertices` draw keys from the other modalities, as `docs_measure`, `pages_measure` and `graph_measure_vertices` measure them.

A `Cursor` walks one collection page by page, each pull landing in the arena it is given, with the values of its keys when it asks for them:

```rust
let mut cursor = database.head().on(&executor).cursor(people, Key::from(0u64), Matching::default())?;
loop {
    let page = cursor.next_pairs(1024, &mut arena)?;
    for (index, key) in page.keys().iter().enumerate() {
        println!("{key}: {:?}", page.get(index));
    }
    if page.keys().len() < 1024 { break; }
}
```

Every pull resumes past the last key the pull before it handed out, so a key written behind the walk is never handed out and one written ahead of it is.
`next` pulls the keys alone, and `next_pairs` reads their values in the same call, where a key removed at the head between the listing and the read of one pull reads as an empty value.
A page shorter than the count it wanted exhausts the walk for good, so every pull after it hands out nothing, even once more keys are written, and a pull that fails leaves the cursor where it was.
A cursor opens from a blocking view, sees what that view sees pull by pull, and pulls on its executor; it holds the database open, and the transaction or snapshot it reads through, until it is dropped.

`Database::changes_cursor` opens a feed of the changes committed to one collection after a `CommitStamp`, oldest first, pinning that stamp as a snapshot does:

```rust
let mut feed = database.changes_cursor(people, stamp)?;
let changes = feed.next_pairs(1024, &mut arena)?;
for (index, (key, kind)) in changes.keys().iter().zip(changes.kinds()).enumerate() {
    println!("{key} {kind} at {:?}: {:?}", changes.stamps()[index], changes.value(index));
}
```

Every pull resumes past the last change the one before it handed out, and an empty one reached the newest commit.
A `ChangeKind` is a write, a removal or a partial change, and a range removal comes as the removal of every key it removed.
A feed whose history is already gone is refused with `Status::WriteConflict`, and LevelDB, which keeps no stamps, refuses it with `Status::Unsupported`.

`use ustore::ArrowArray;`

`use ustore::ArrowArrayStream;`

`use ustore::ArrowExport;`

`use ustore::ArrowSchema;`

`use ustore::IngestOrder;`

`use ustore::BlobsScanConfig;`

`use ustore::Entry;`

`use ustore::Values;`

`use ustore::Blobs;`

`use ustore::Durability;`

`use ustore::Tape;`

`use ustore::Listing;`

`use ustore::Mapped;`

`use ustore::StringsSampleConfig;`

`use ustore::StringsScanConfig;`

`use ustore::TextRange;`

`use ustore::arrow::ArrowDeviceArray;`

`use ustore::blobs::Write;`

`use ustore::blobs::Remove;`

`use ustore::blobs::RemoveRange;`

`use ustore::blobs::Read;`

`use ustore::db::Rows;`

`use ustore::strings::StringsWrite;`

`use ustore::strings::StringsRemove;`

`use ustore::strings::StringsRead;`

`use ustore::strings::StringsScan;`

`use ustore::strings::StringsSample;`

`use ustore::strings::StringsMeasure;`

## ArrowArray

The buffers of an Arrow array, as `struct ArrowArray` lays them out.

```rust
pub struct ArrowArray
```

- `length` `i64`: The rows.
- `null_count` `i64`: The nulls, or -1 when not yet counted.
- `offset` `i64`: The first row, into every buffer.
- `n_buffers` `i64`: The number of buffers.
- `n_children` `i64`: The number of children.
- `buffers` `*mut *const c_void`: The buffers, validity first.
- `children` `*mut *mut ArrowArray`: The children.
- `dictionary` `*mut ArrowArray`: The dictionary, or null.
- `release` `Option<unsafe extern "C" fn(*mut ArrowArray)>`: Frees the array, null once released or moved out.
- `private_data` `*mut c_void`: The producer's own state.

Implements: `Debug`, `Drop`

## empty

A released array, for a producer to write into.

```rust
pub fn empty() -> Self
```

## ArrowArrayStream

A stream of Arrow record batches of one schema, as `struct ArrowArrayStream` lays it out.

```rust
pub struct ArrowArrayStream
```

- `get_schema` `Option<unsafe extern "C" fn(*mut ArrowArrayStream, *mut ArrowSchema) -> c_int>`: Writes the schema every batch shares, answering 0 or an `errno` code.
- `get_next` `Option<unsafe extern "C" fn(*mut ArrowArrayStream, *mut ArrowArray) -> c_int>`: Writes the next batch, released once the stream ends, answering 0 or an `errno` code.
- `get_last_error` `Option<unsafe extern "C" fn(*mut ArrowArrayStream) -> *const c_char>`: The message of the last failed callback, or null.
- `release` `Option<unsafe extern "C" fn(*mut ArrowArrayStream)>`: Frees the stream, which every batch it answered outlives; null once released or moved out.
- `private_data` `*mut c_void`: The producer's own state.

Implements: `Debug`, `Drop`

## empty

A released stream, for a producer to write into.

```rust
pub fn empty() -> Self
```

## schema

The schema every batch shares.

```rust
pub fn schema(&mut self) -> Result<ArrowSchema>
```

## next_batch

The next batch, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) once the stream ends.

```rust
pub fn next_batch(&mut self) -> Result<Option<ArrowArray>>
```

## last_error

The message of the last failed callback, which lives until the next call on the stream.

```rust
pub fn last_error(&mut self) -> Option<&CStr>
```

## ArrowExport

One column exported as Arrow in place, borrowing the arena it sits in until dropped.

```rust
pub struct ArrowExport<'arena>
```

Implements: `Debug`

## schema

The column's type.

```rust
pub fn schema(&self) -> &ArrowSchema
```

## array

The column's buffers and the device holding them.

```rust
pub fn array(&self) -> &ArrowDeviceArray
```

## into_parts

Hands the column over past the arena's borrow, for a consumer that takes the structures by value.

```rust
pub unsafe fn into_parts(self) -> (ArrowSchema, ArrowDeviceArray)
```

Safety: The arena must outlive the release of both, which no borrow checks any more.

## ArrowSchema

The type of an Arrow array: its format, name, metadata and children, as `struct ArrowSchema` lays them out.

```rust
pub struct ArrowSchema
```

- `format` `*const c_char`: The format string, such as `+s` for a struct or `w:16` for 16-byte keys.
- `name` `*const c_char`: The field name, or null.
- `metadata` `*const c_char`: The binary-encoded key-value metadata, or null.
- `flags` `i64`: The `ARROW_FLAG_*` bits.
- `n_children` `i64`: The number of children.
- `children` `*mut *mut ArrowSchema`: The children.
- `dictionary` `*mut ArrowSchema`: The dictionary type, or null.
- `release` `Option<unsafe extern "C" fn(*mut ArrowSchema)>`: Frees the schema, null once released or moved out.
- `private_data` `*mut c_void`: The producer's own state.

Implements: `Debug`, `Drop`

## empty

A released schema, for a producer to write into.

```rust
pub fn empty() -> Self
```

## IngestOrder

In what order an ingest's rows arrive, as `ustore_ingest_order_t` spells it.

```rust
pub enum IngestOrder
```

- `Sorted` = `0`: Keys strictly ascend across every batch, checked as each one is written into runs of its own.
- `Unsorted` = `1`: Keys arrive in any order and may repeat, the last row of a key winning; every batch is held until the end.

Implements: `Clone`, `Copy`, `StructuralPartialEq`, `PartialEq`, `Eq`, `Debug`, `Default`, `Hash`, `Display`, `FromStr`

## BlobsScanConfig

The keys a scan keeps, what admits them and how it reports; see [`On::scan`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#scan).

```rust
pub struct BlobsScanConfig<'config>
```

- `wanted` `usize`: Keys kept, at most.
- `matching` `Matching<'config>`: Admits only the keys it approves; every key by default.
- `mode` `ScanMode`: In what order the keys come.
- `progress` `Option<&'config mut ProgressCallback<'config>>`: Reports the scans finished and stops wherever it says to.

Implements: `From<usize>`

## Entry

One key and the bytes stored under it, laid out so a slice of them crosses as it stands: the keys as one strided run, and the values through an accessor over the same entries.

```rust
pub struct Entry<'value>
```

Implements: `Clone`, `Copy`, `Debug`

## new

Pairs a key with the bytes to store under it.

```rust
pub fn new(key: Key, value: &'value [u8]) -> Self
```

## key

The key this entry stores under.

```rust
pub fn key(&self) -> Key
```

## value

The bytes this entry stores.

```rust
pub fn value(&self) -> &'value [u8]
```

## Values

The values one read exported, addressed by the position of their key in the batch.

```rust
pub struct Values<'arena>
```

The presences, the offsets and the bytes all live in the arena, so the run borrows it, and the next call through it
cannot overwrite them while the run lives.

Implements: `Clone`, `Copy`, `Debug`, `IntoIterator`

## len

How many keys the read covered.

```rust
pub fn len(&self) -> usize
```

## is_empty

Whether the read covered no keys at all.

```rust
pub fn is_empty(&self) -> bool
```

## get

The value of the key at `index`, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) for a key the database does not hold.

```rust
pub fn get(&self, index: usize) -> Option<&'arena [u8]>
```

## iter

Every value in the order its key was asked for.

```rust
pub fn iter(&self) -> Rows<Values<'arena>, Option<&'arena [u8]>>
```

## presences

Which keys the read found, one bit per key as Arrow reads a validity buffer.

```rust
pub fn presences(&self) -> Bitmap<'arena>
```

## tape

Every value on the one tape the read exported, without a copy.

```rust
pub fn tape(&self) -> Tape<'arena>
```

A missing value is empty on the tape, and [`Values::presences`](#presences) tells it apart from an empty one.

## Blobs

The binary values of one collection.

```rust
pub struct Blobs<'collection, 'view, Parent = Database>
```

Every verb consumes the view; [`Collection::blobs`](https://www.unum.cloud/docs/ustore/rust/collections.md#blobs) hands out the next one.

Implements: `Debug`

## get

The value under [`key`](https://www.unum.cloud/docs/ustore/rust/documents.md#key), copied out of a one-off arena, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) for a key the collection does not hold.

```rust
pub fn get(self, key: Key) -> Result<Option<Vec<u8>>>
```

## read

The values under [`keys`](https://www.unum.cloud/docs/ustore/rust/scans.md#keys) in order, read into `arena`, which [`Values::tape`](https://www.unum.cloud/docs/ustore/rust/scans.md#tape) hands out as one tape.

```rust
pub fn read<'arena>(self, keys: &[Key], arena: &'arena mut Arena) -> Result<Values<'arena>>
```

## contains

Whether the collection holds [`key`](https://www.unum.cloud/docs/ustore/rust/documents.md#key), exporting no value into the one-off arena it reads through.

```rust
pub fn contains(self, key: Key) -> Result<bool>
```

## set

Stores [`value`](https://www.unum.cloud/docs/ustore/rust/scans.md#value) under [`key`](https://www.unum.cloud/docs/ustore/rust/documents.md#key), creating the collection to hold blobs when it is missing.

```rust
pub fn set(self, key: Key, value: &[u8]) -> Result<()>
```

## remove

Removes [`key`](https://www.unum.cloud/docs/ustore/rust/documents.md#key), whether or not the collection holds it.

```rust
pub fn remove(self, key: Key) -> Result<()>
```

## Durability

Whether a write reaches the drive before the call returns, as `ustore_durability_t` spells it.

```rust
pub enum Durability
```

A transaction flushes at its commit, so [`Flushed`](#Flushed) on a transaction's write is refused.

- `Buffered` = `0`: The engine decides when the bytes reach the drive.
- `Flushed` = `1`: The call returns once the bytes are on the drive, at a cost that varies by engine.

Implements: `Clone`, `Copy`, `StructuralPartialEq`, `PartialEq`, `Eq`, `Debug`, `Default`, `Hash`, `Display`, `FromStr`

## Tape

The strings a call wrote as one tape: one more offset than strings, and the bytes between them, as Arrow lays out `binary` while the bytes fit in 4 GiB and `large_binary` beyond.

```rust
pub enum Tape<'arena>
```

- `U32` = `(BytesTapeViewU32&lt;'arena&gt;)`: A tape of 32-bit offsets, as Arrow lays out `binary`.
- `U64` = `(BytesTapeViewU64&lt;'arena&gt;)`: A tape of 64-bit offsets, as Arrow lays out `large_binary`.

Implements: `Debug`

## len

How many strings the tape holds.

```rust
pub fn len(&self) -> usize
```

## is_empty

Whether the tape holds no strings.

```rust
pub fn is_empty(&self) -> bool
```

## *

The string at `index`, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) past the end.

```rust
pub fn get(&self, index: usize) -> Option<&[u8]>
```

## Listing

The strings one scan or sample listed, as byte runs inside the arena, and the BM25 scores of a scan whose patterns carried weights.

```rust
pub struct Listing<'arena>
```

Implements: `Clone`, `Copy`, `Debug`

## len

How many strings the call listed.

```rust
pub fn len(&self) -> usize
```

## is_empty

Whether the call listed nothing.

```rust
pub fn is_empty(&self) -> bool
```

## iter

Every string in the order the call listed it.

```rust
pub fn iter(&self) -> Rows<Listing<'arena>, &'arena [u8]>
```

## scores

One score per string when the patterns carried weights.

```rust
pub fn scores(&self) -> Option<&'arena [f32]>
```

## Mapped

The keys one read found, addressed by the position of their string in the batch.

```rust
pub struct Mapped<'arena>
```

Implements: `Clone`, `Copy`, `Debug`

## len

How many strings the read covered.

```rust
pub fn len(&self) -> usize
```

## is_empty

Whether the read covered no strings at all.

```rust
pub fn is_empty(&self) -> bool
```

## get

The key of the string at `index`, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) for a string the database does not hold.

```rust
pub fn get(&self, index: usize) -> Option<Key>
```

## iter

Every key in the order its string was asked for.

```rust
pub fn iter(&self) -> Rows<Mapped<'arena>, Option<Key>>
```

## presences

Which strings the read found, one bit per string as Arrow reads a validity buffer.

```rust
pub fn presences(&self) -> Bitmap<'arena>
```

## StringsSampleConfig

The strings a sample draws; see [`On::strings_sample`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_sample).

```rust
pub struct StringsSampleConfig
```

- `wanted` `usize`: Strings drawn, at most.

Implements: `Clone`, `Copy`, `Debug`, `From<usize>`

## StringsScanConfig

The strings a scan keeps, what admits them and how it reports; see [`On::strings_scan`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_scan).

```rust
pub struct StringsScanConfig<'config>
```

- `wanted` `usize`: Strings kept, at most.
- `matching` `Matching<'config>`: Admits only the strings it approves; every string by default.
- `mode` `ScanMode`: In what order the strings come.
- `progress` `Option<&'config mut ProgressCallback<'config>>`: Reports the scans finished and stops wherever it says to.

Implements: `From<usize>`

## TextRange

Which strings of one collection a walk covers: those starting with `prefix`, from `start`, inclusive, up to `end`, exclusive, in the order of their bytes or the reverse.

```rust
pub struct TextRange<'text>
```

A string converts into the ascending range of the strings starting with it.

- `prefix` `Option<&'text str>`: What every string starts with, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) for every string.
- `start` `Option<&'text str>`: The string the walk starts from, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) for the first one in its direction.
- `end` `Option<&'text str>`: The string the walk stops before, or [`None`](https://www.unum.cloud/docs/ustore/rust/transactions.md#None) to walk to the last one in its direction.
- `direction` `ScanDirection`: Which way the walk goes from `start`.

Implements: `Clone`, `Copy`, `StructuralPartialEq`, `PartialEq`, `Eq`, `Debug`, `Default`, `Hash`, `From<&'text str>`

## ArrowDeviceArray

An Arrow array and the device holding its buffers, as `struct ArrowDeviceArray` lays them out.

```rust
pub struct ArrowDeviceArray
```

The buffers of an array on a device are addresses there, which only the raw pointers of
[`ArrowArray`](#ArrowArray) carry, so no safe host slice ever reads them.

- `array` `ArrowArray`: The array.
- `device_id` `i64`: The vendor's ordinal of the device, or -1 on the host.
- `device_type` `i32`: The `ARROW_DEVICE_*` type of the device.
- `sync_event` `*mut c_void`: What to wait on before reading the buffers, or null, as an exporter here waited for its
device.
- `reserved` `[i64; 3]`: Zeroed.

Implements: `Debug`

## CPU

`ARROW_DEVICE_CPU`, the type of memory the host reads in place.

```rust
pub const CPU: i32 = 1;
```

## empty

A released array, for an exporter to write into.

```rust
pub fn empty() -> Self
```

## Write

An upsert of a batch of values; see [`On::write`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#write).

```rust
pub struct Write<'collections, Entries>
```

Implements: `Debug`

## Remove

A removal of a batch of keys; see [`On::remove`](https://www.unum.cloud/docs/ustore/rust/documents.md#remove).

```rust
pub struct Remove<'collections, Keys>
```

Implements: `Debug`

## RemoveRange

A removal of one range of keys; see [`On::remove_range`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#remove_range).

```rust
pub struct RemoveRange
```

Implements: `Debug`

## Read

A read of a batch of keys; see [`On::read`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#read).

```rust
pub struct Read<'arena, 'collections, Keys>
```

Implements: `Debug`

## Rows

The rows of a batch result in order, from either end, each read through the result's own [`get`](https://www.unum.cloud/docs/ustore/rust/documents.md#get).

```rust
pub struct Rows<Results, Row>
```

Implements: `Clone`, `Debug`, `Iterator`, `DoubleEndedIterator`, `ExactSizeIterator`, `FusedIterator`

## StringsWrite

An upsert of a batch of strings; see [`On::strings_write`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_write).

```rust
pub struct StringsWrite<'collections, Strs, Keys>
```

Implements: `Debug`

## StringsRemove

A removal of a batch of strings; see [`On::strings_remove`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_remove).

```rust
pub struct StringsRemove<'collections, Strs>
```

Implements: `Debug`

## StringsRead

A read of the keys of a batch of strings; see [`On::strings_read`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_read).

```rust
pub struct StringsRead<'arena, 'collections, Strs>
```

Implements: `Debug`

## StringsScan

A walk over the strings of one collection; see [`On::strings_scan`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_scan).

```rust
pub struct StringsScan<'arena, 'text, 'matching>
```

Implements: `Debug`

## StringsSample

A random draw of strings from one collection; see [`On::strings_sample`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_sample).

```rust
pub struct StringsSample<'arena>
```

Implements: `Debug`

## StringsMeasure

An estimate of what a range of strings holds; see [`On::strings_measure`](https://www.unum.cloud/docs/ustore/rust/batches-and-arenas.md#strings_measure).

```rust
pub struct StringsMeasure<'arena, 'text>
```

Implements: `Debug`