# Collections

> UStore for Python

### Keys

A key is a `uuid.UUID`, its text, sixteen raw bytes, or a non-negative integer read as `uuid.UUID(int=n)`, so integer keys order as integers do.
Keys, vertices and collection IDs come back as `uuid.UUID`, and a key batch passes without a copy as a contiguous array of 16-byte records, such as NumPy's `V16` or an (n, 16) `uint8` array, which is also how `np.asarray` reads a `Keys` or `Neighborhoods` tape.
A buffer of 64-bit integers converts into keys one by one, and any other sequence converts element by element.

### Collections, Modes & Stamps

Every C enum a caller meets is an `enum.IntEnum` of `ustore`: `Status`, `Mood`, `CPUClass`, `Watch`, `Durability`, `DropMode`, `ScanMode`, `ScanDirection`, `Modality`, `Isolation`, `Transport`, `Engine`, `Role`, `FieldType` and `Modification`.
The module builds each one at import through the C `_name` functions, valued as the enumerators and named by their canonical spellings in upper case, dashes as underscores, so "strict-serializable" is `Isolation.STRICT_SERIALIZABLE` and "read-only" is `Status.READ_ONLY`.
A verb takes any `int` that is an enumerator, refusing a bool, which would read as zero or one, and text.
`Error.status` and `Deferred.status` are `Status` members, `measure()` answers a `Measurement` struct sequence, `database.capabilities` a `Capabilities` one of `transactions` and `snapshots`, and `Columns.types` holds `FieldType` members.

`describe()` answers a `Description` struct sequence of the `capabilities`, the `engine`, the `isolation`, and the `keys`, `memory_bytes` and `disk_bytes` the engine estimates, zero where it keeps no estimate.
Its `pin_refused_bytes` and `crossing_requests` report where the NVMe engine went unpinned: the arena bytes the locked-memory cap refused to pin, and the requests that crossed a gibibyte of a pinned arena.
`clear(mode)` empties every collection at once, as `collection_drop` empties one, and `compact()` reclaims the space overwritten and removed data takes, one collection at a time.
A snapshot's `save(path)` writes it out as a new database of the same engine, holding exactly the collections and names the snapshot records:

```python
database.describe().engine                              # ustore.Engine.ROCKSDB
database.clear(ustore.DropMode.PAIRS)                   # every pair goes, the collections and their names stay
database.compact(progress=report)
with database.snapshot() as snapshot:
    snapshot.save(directory)                            # RocksDB and LevelDB create it, RAM writes into it
```

Every other field of the C ABI's argument structs that a caller chooses, such as `collection`, `durability` or `role`, is a keyword, while a verb whose only option is a mode, such as `transaction(watch)`, `commit(durability)`, `drain(mood)` or `collection_drop(collection, mode)`, takes it by position too.
No collection exists until `collection_create(name, modality=...)` makes it to hold one modality, and names starting with `_` are reserved for the system collections, which no listing shows.
Every verb names its collection as a `uuid.UUID` identifier or a name resolved on the calling thread, and `database["movies"]` answers a `Collection` whose `blobs`, `docs`, `graph` and `table` views create it on their first write.

```python
database.set(key, value, collection="movies", durability=ustore.Durability.FLUSHED)
database.collection_drop("movies", ustore.DropMode.PAIRS)
```

Stamps are plain `int`s of two kinds.
`stage()` answers the read stamp, the stamp a transaction reads at and its commit validates against.
`commit()` answers the commit stamp, the commit's position in commit order, which every snapshot whose read stamp is at or above it sees.
`Snapshot.read_stamp` is the stamp a snapshot reads at, so `before.read_stamp < commit_stamp <= after.read_stamp` for snapshots taken around a commit.
Staging reserves no position in commit order, so a read stamp is never greater than the commit stamp that follows it.

`changes_cursor(since, *, collection)` opens a feed of the changes committed to one collection after a commit stamp, oldest first, pinning that stamp as a snapshot does:

```python
with database.changes_cursor(commit_stamp, collection="movies") as feed:
    while (changes := feed.next(1024, arena, values=True)).keys:  # an empty pull reached the newest commit
        for key, stamp, kind in zip(changes.keys, changes.stamps, changes.kinds):
            consume(key, stamp, kind)  # a ChangeKind: WRITE, REMOVAL or CHANGE
```

A `Changes` tuple carries the keys as a `Page`, their `Values` when asked, and tuples of commit stamps and `ChangeKind` members, where a range removal comes as the removal of every key it removed.
A feed whose history is already gone raises `WriteConflictError`, and LevelDB, which keeps no stamps, raises `UnsupportedError`.

`import ustore`

`from ustore import Collection, Capabilities, Description`

## DropMode

What dropping a collection removes, valued as `ustore_drop_mode_t`.

```python
class DropMode(IntEnum)
```

- `COLLECTION` = `0`
- `PAIRS` = `1`
- `VALUES` = `2`

## Modality

What a collection holds, valued as `ustore_modality_t`.

```python
class Modality(IntEnum)
```

- `BLOBS` = `1`
- `DOCS` = `2`
- `GRAPH` = `3`
- `STRINGS` = `4`
- `PAGES` = `5`

## CollectionLike

```python
type CollectionLike = UUID | str
```

## KeyLike

```python
type KeyLike = UUID | int | str | bytes
```

## KeysLike

Keys as a sequence, or packed as (keys, 16) bytes or 64-bit integers in a buffer or a DLPack tensor, which may live on a CUDA or ROCm device when it holds the bytes.

```python
type KeysLike = Sequence[KeyLike] | Buffer | _DLPack
```

## Collection

A collection a receiver addresses by name, resolved on first use and created by its first write.

```python
class Collection
```

The mapping views and [`sample`](https://www.unum.cloud/docs/ustore/python/snapshots.md#sample) copy what they read out of a one-off arena each. Through a deferred view,
`blobs[key]`, `docs[key]`, `docs_json[key]`, [`scan`](https://www.unum.cloud/docs/ustore/python/snapshots.md#scan), [`sample_keys`](#sample_keys) and the reads of [`graph`](#graph) that take one call
answer a [`Deferred`](https://www.unum.cloud/docs/ustore/python/batches-and-arenas.md#Deferred) of what they answer here.

## sample

Up to `wanted` keys drawn at random onto copies of their values, through one-off arenas.

```python
def sample(wanted: int, /) -> dict[UUID, bytes | None]
```

## sample_keys

Up to `wanted` keys drawn at random.

```python
def sample_keys(wanted: int, arena: Arena, /) -> Keys
```

## scan

Up to `wanted` keys from `start` upwards.

```python
def scan(start: KeyLike, wanted: int, arena: Arena, /) -> Keys
```

## __arrow_c_stream__

The whole collection as one Arrow stream through the receiver's view, keys as `arrow.uuid` and values as binary, so `pyarrow.table(db['posters'])` reads it in place.

```python
def __arrow_c_stream__(
    requested_schema: object | None = None,
) -> object
```

## name

```python
name: str
```

## id

```python
id: UUID
```

## modality

```python
modality: Modality
```

## blobs

```python
blobs: BlobsView
```

## docs

```python
docs: DocsView
```

## docs_json

```python
docs_json: DocsJSONView
```

## graph

```python
graph: DiGraph
```

## table

```python
table: Table
```

## Capabilities

What an engine supports, as [`Database.capabilities`](#capabilities) reports it.

```python
class Capabilities(structseq[bool], tuple[bool, bool])
```

## transactions

Whether the engine groups operations into transactions.

```python
transactions: bool
```

## snapshots

Whether the engine pins point-in-time states.

```python
snapshots: bool
```

## Description

What an open database reports about itself, as `describe()` answers it, every size the engine's estimate.

```python
class Description(structseq[Any], tuple[Capabilities, Engine, Isolation, int, int, int])
```

## capabilities

What the engine supports.

```python
capabilities: Capabilities
```

## engine

The engine serving the database.

```python
engine: Engine
```

## isolation

How transactions see each other: what the RAM engine was opened with, and [`SNAPSHOT`](https://www.unum.cloud/docs/ustore/python/transactions.md#SNAPSHOT) elsewhere.

```python
isolation: Isolation
```

## keys

Keys across every collection.

```python
keys: int
```

## memory_bytes

Bytes of memory the engine holds for the data, or zero where it keeps no estimate.

```python
memory_bytes: int
```

## disk_bytes

Bytes the files of the database take on disk.

```python
disk_bytes: int
```