# Collections

> UStore for C

A database holds no collection until one is created, and every call names the collections it addresses, refusing a `collections` view with `NULL` data with `ustore_status_invalid_argument_k`.
A collection is created with an optional name and the `ustore_modality_t` it holds, and addressed afterwards by the identifier the call answers, a UUIDv7 its create mints once and never reuses:

```c
char const name[] = "movies";
ustore_collection_id_t movies = {0};
ustore_collection_create(&(ustore_collection_create_t){
    .database = database,
    .outcome = &outcome,
    .name = {(uint8_t const*)name, sizeof(name) - 1},
    .modality = ustore_modality_docs_k,
    .collection = &movies,
});

ustore_collection_id_t values = {0};
ustore_collection_create(&(ustore_collection_create_t){
    .database = database,
    .outcome = &outcome,
    .modality = ustore_modality_blobs_k,
    .collection = &values,
});
```

The examples below write binary values into `values` and documents into `movies`.

Each engine records names in the `_names` system collection and every collection's modality and name in `_collections`, which no identifier reaches and no listing shows, so a name starting with `_` is refused, and a name resolves in two phases: once through `ustore_collection_find`, which answers the identifier and the modality of every name of a batch at the head or at a snapshot, and by identifier from then on, as no other call carries one.
Collections are created and dropped only at the head, so their names are never read inside a transaction, and resolving one never joins the conflicts of the work that follows it.
A create records the name in the same step as the collection, refusing a name another collection carries even under a concurrent creator.
A name no collection carries resolves to the nil UUID with its presence bit cleared, and `ustore_collection_drop` with `ustore_drop_collection_k` forgets a collection's name before the collection goes.
`ustore_collection_list` exports the identifier of every collection and, where asked, its modality and its name, as a tape where a collection created without one has an empty name.
A listing describes the database rather than a transaction, so it reads the head or a snapshot.

### Describing, Clearing and Compacting

`ustore_database_describe` reports the engine of an open database, what it supports and how much it holds, while `ustore_database_clear` and `ustore_database_compact` work on every collection at once:

```c
ustore_database_description_t description = {0};
ustore_database_describe(&(ustore_database_describe_t){
    .database = database,
    .outcome = &outcome,
    .description = &description,
});
printf("%s holds %llu keys in %llu bytes on disk\n", ustore_engine_name(description.engine),
       (unsigned long long)description.keys, (unsigned long long)description.disk_bytes);

ustore_database_clear(&(ustore_database_clear_t){.database = database, .outcome = &outcome, .mode = ustore_drop_pairs_k});
ustore_database_compact(&(ustore_database_compact_t){.database = database, .outcome = &outcome});
```

A description's `capabilities` say whether the engine supports transactions, collections and snapshots, `engine` names the engine, and `isolation` says how its transactions see each other: as the `ram` engine was opened, and `ustore_isolation_snapshot_k` elsewhere.
`keys`, `memory_bytes` and `disk_bytes` are the engine's estimates of the keys across every collection and of the bytes the data takes, and zero where it keeps none, as LevelDB keeps no count of its keys.
`pin_refused_bytes` and `crossing_requests` report where the NVMe engine went unpinned: the arena bytes the locked-memory cap refused to pin, and the requests that crossed a gibibyte of a pinned arena.
`ustore_database_clear` empties every collection as its `mode` names:

- `ustore_drop_pairs_k` removes every pair, keeping the collections and their names;
- `ustore_drop_collection_k` drops every collection but the main one with its name, and empties the main one;
- `ustore_drop_values_k` empties every value, keeping the keys.

The `ram` and RocksDB engines refuse to drop the collections while a snapshot is open, with `ustore_status_invalid_argument_k`, since the snapshot belongs to its holder.
`ustore_database_compact` reclaims the space overwritten and removed data still takes, one collection at a time: RocksDB and LevelDB flush and compact their trees, and the `ram` engine frees the versions no reader needs and rewrites its image.

`#include <ustore/types.h>`

`#include <ustore/db.h>`

## ustore_arena_description_t

Where an arena's outputs live.

```c
struct ustore_arena_description_t
```

- `residency` `ustore_residency_t`: RAM for host pages, VRAM for one device's memory.
- `device` `int32_t`: The UStore ordinal of the device as [`ustore_gpu_devices`](https://www.unum.cloud/docs/ustore/c/batches-and-arenas.md#ustore_gpu_devices) lists it, or -1 for host pages.

## ustore_uuid_t

Sixteen bytes naming one thing everywhere, ordered as their bytes and compared with `memcmp`.

```c
struct ustore_uuid_t
```

An integer spells as its 128-bit big-endian bytes, as Python's `uuid.UUID(int=...)` does, so identifiers made from integers order as the integers.

- `bytes` `uint8_t`

## ustore_collection_id_t

The identifier of a collection: a UUIDv7 its create mints once and never reuses, so it stays the same across restarts and never names two collections.

```c
struct ustore_collection_id_t
```

The nil UUID names none.

- `uuid` `ustore_uuid_t`

## ustore_key_t

The identifier of a value within one collection.

```c
struct ustore_key_t
```

- `uuid` `ustore_uuid_t`

## ustore_keys_view_t

A strided run of keys, one per task, where a zero stride repeats the first.

```c
struct ustore_keys_view_t
```

- `data` `ustore_key_t const *`: First key of the run.
- `stride` `ustore_size_t`: Bytes between consecutive keys, where zero repeats the first one.

## ustore_collections_view_t

A strided run of collections, one per task, whose `NULL` data a call refuses with [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k), as no collection was named.

```c
struct ustore_collections_view_t
```

- `data` `ustore_collection_id_t const *`: First collection of the run.
- `stride` `ustore_size_t`: Bytes between consecutive collections, where zero repeats the first one.

## ustore_drop_mode_t

What dropping a collection removes.

```c
enum ustore_drop_mode_t
```

- `ustore_drop_collection_k` = `0`: The collection together with its name and every pair it holds.
- `ustore_drop_pairs_k` = `1`: Every key and value, keeping the collection.
- `ustore_drop_values_k` = `2`: Every value, keeping the keys, which a strings collection refuses, as its values are keys.

Spellings: ```
char const * ustore_drop_mode_name(ustore_drop_mode_t drop_mode);
```

```
ustore_drop_mode_t ustore_drop_mode_from_name(ustore_bytes_view_t name);
```

## ustore_capabilities_t

Capabilities an engine reports.

```c
enum ustore_capabilities_t
```

- `ustore_capabilities_transactions_k` = `1`: The engine supports transactions.
- `ustore_capabilities_snapshots_k` = `4`: The engine supports snapshots.

## ustore_database_description_t

What an open database reports about itself in one call, where every size is the engine's estimate, and zero where the engine keeps none.

```c
struct ustore_database_description_t
```

- `capabilities` `ustore_capabilities_t`: The capabilities of the engine.
- `engine` `ustore_engine_t`: The engine serving the database.
- `isolation` `ustore_isolation_t`: How transactions see each other: what the RAM engine was opened with, and [`ustore_isolation_snapshot_k`](https://www.unum.cloud/docs/ustore/c/transactions.md#ustore_isolation_snapshot_k) elsewhere.
- `keys` `ustore_size_t`: Keys across every collection.
- `memory_bytes` `ustore_size_t`: Bytes of memory the engine holds for the data.
- `disk_bytes` `ustore_size_t`: Bytes the files of the database take on disk.

## ustore_modality_t

What a collection holds, recorded beside its name so a view of another kind is refused.

```c
enum ustore_modality_t
```

Zero names no modality.

- `ustore_modality_blobs_k` = `1`: Binary values of `blobs.h`.
- `ustore_modality_docs_k` = `2`: JSON documents of `docs.h`.
- `ustore_modality_graph_k` = `3`: Vertices and their edges of `graph.h`.
- `ustore_modality_strings_k` = `4`: Values keyed by strings of `strings.h`.
- `ustore_modality_pages_k` = `5`: Ordered documents of `pages.h`.

Spellings: ```
char const * ustore_modality_name(ustore_modality_t modality);
```

```
ustore_modality_t ustore_modality_from_name(ustore_bytes_view_t name);
```

## ustore_database_describe

Reports the engine of an open database, what it supports and how much it holds.

```c
void ustore_database_describe(ustore_database_describe_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `description`: Output for the description, filled whole; zeroed on failure.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `description` output is `NULL`, on the RAM and NVMe engines.

## ustore_database_clear

Empties every collection at once, as [`ustore_collection_drop`](#ustore_collection_drop) empties one.

```c
void ustore_database_clear(ustore_database_clear_t *arguments);
```

The RAM and RocksDB engines refuse to drop the collections while a snapshot is open, since the snapshot belongs to its holder.

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `mode`: What goes: every pair, keeping the collections and their names; every collection with its name; or every value, keeping the keys where the engine can.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `mode` is unknown, or a snapshot is open while [`ustore_drop_collection_k`](#ustore_drop_collection_k) drops the collections on the RAM and RocksDB engines.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A listed collection was dropped meanwhile, on the NVMe engine.
- [`ustore_status_write_conflict_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_write_conflict_k) Concurrent commits kept conflicting, on the RAM and NVMe engines.

## ustore_database_compact

Reclaims the space overwritten and removed data still takes: RocksDB and LevelDB flush and compact their trees, and the RAM engine frees the versions no reader needs and rewrites its image.

```c
void ustore_database_compact(ustore_database_compact_t *arguments);
```

The work runs one collection at a time, and its progress counts the bytes of the collections compacted where the engine measures them, as RocksDB does, and one unit per collection elsewhere. A cancelled compaction stops between collections, keeping what it compacted.

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `progress`: Optional callback reporting the compaction's advance.
- `progress_state`: Opaque pointer passed to `progress`.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.

## ustore_database_flush

Makes every commit that returned before this call durable, as if each had asked for [`ustore_durability_flushed_k`](https://www.unum.cloud/docs/ustore/c/reads-and-writes.md#ustore_durability_flushed_k).

```c
void ustore_database_flush(ustore_database_flush_t *arguments);
```

LevelDB syncs only its current log, so a commit in a log it already rotated stays unsynced until its compaction lands.

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.

## ustore_collection_list

Lists the collections of the database.

```c
void ustore_collection_list(ustore_collection_list_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `snapshot`: Optional snapshot whose collections are listed, where `NULL` lists the head.
- `collections`: Optional output for the identifiers of the collections, inside `arena`.
- `modalities`: Optional output for what every collection holds, inside `arena`.
- `names`: Optional output for the names as a tape inside `arena`, in the order of the identifiers, where a collection without a name has an empty one.
- `written`: Output for the collections written; zeroed on failure.
- `arena`: Rewound by this call, then holding every output it writes; `NULL` is refused.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena` or the `written` output is `NULL`, or the `snapshot` belongs to another engine.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) The arena lives where another executor computes.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The arena serves another call, the executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A listed collection was dropped meanwhile, on the NVMe engine.

## ustore_collection_create

Creates an empty collection, named or not.

```c
void ustore_collection_create(ustore_collection_create_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `name`: Optional name no other collection carries and not starting with `_`, where a zeroed view creates a collection without one.
- `modality`: What the collection holds, which fixes how it is keyed and so which calls reach it.
- `collection`: Output for the identifier of the collection, zeroed on failure.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.
- [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `modality` is unknown, the `name` starts with `_` or is empty with non-null data, or the `collection` output is `NULL`.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device, or a documents, graph or pages collection is asked of an NVMe database whose values share one size.
- [`ustore_status_already_exists_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_already_exists_k) Another collection carries the `name`.
- [`ustore_status_write_conflict_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_write_conflict_k) Concurrent commits kept conflicting, on the RAM and NVMe engines.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The name outgrows the NVMe schema's record, or every shared channel is taken.

## ustore_collection_find

Reads the collections a batch of names address.

```c
void ustore_collection_find(ustore_collection_find_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `snapshot`: Optional snapshot whose names are read, where `NULL` reads the head.
- `count`: Number of tasks, one per name.
- `names`: Names to resolve, one per task.
- `presences`: Optional output for a bitset of `count` bits marking the names a collection carries, addressed as Apache Arrow does, inside `arena`.
- `collections`: Output for the identifiers of the collections the names address, inside `arena`.
- `modalities`: Optional output for what the collections hold, inside `arena`.
- `arena`: Rewound by this call, then holding every output it writes; `NULL` is refused.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena` or the `collections` output is `NULL`, the `names` come in an unknown layout, or the `snapshot` belongs to another engine.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) The arena lives where another executor computes.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The arena serves another call, the executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.

## ustore_collection_drop

Removes or clears a collection.

```c
void ustore_collection_drop(ustore_collection_drop_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `collection`: Identifier of an existing collection.
- `mode`: Whether values, pairs or the whole collection go.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `mode` is unknown, [`ustore_drop_values_k`](#ustore_drop_values_k) empties a strings collection, or a snapshot is open while [`ustore_drop_collection_k`](#ustore_drop_collection_k) drops the collection on the RAM and RocksDB engines.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) No collection carries the identifier, a system one included.
- [`ustore_status_write_conflict_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_write_conflict_k) Concurrent commits kept conflicting, on the RAM and NVMe engines.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) Every shared channel is taken on the NVMe engine.

## ustore_key_unknown_k

The key that stands for no key, every bit of it set.

```c
ustore_key_t const ustore_key_unknown_k;
```