C · Collections

Collections

#include <ustore/types.h>#include <ustore/db.h> 1.0.0-rc.1 · d6d40961

A database holds no collection until one is created, and every call names the collections it addresses, refusing a collections view with NULL data with ustore_status_invalid_argument_k. A collection is created with an optional name and the ustore_modality_t it holds, and addressed afterwards by the identifier the call answers, a UUIDv7 its create mints once and never reuses:

char const name[] = "movies";
ustore_collection_id_t movies = {0};
ustore_collection_create(&(ustore_collection_create_t){
    .database = database,
    .outcome = &outcome,
    .name = {(uint8_t const*)name, sizeof(name) - 1},
    .modality = ustore_modality_docs_k,
    .collection = &movies,
});

ustore_collection_id_t values = {0};
ustore_collection_create(&(ustore_collection_create_t){
    .database = database,
    .outcome = &outcome,
    .modality = ustore_modality_blobs_k,
    .collection = &values,
});

The examples below write binary values into values and documents into movies.

Each engine records names in the _names system collection and every collection’s modality and name in _collections, which no identifier reaches and no listing shows, so a name starting with _ is refused, and a name resolves in two phases: once through ustore_collection_find, which answers the identifier and the modality of every name of a batch at the head or at a snapshot, and by identifier from then on, as no other call carries one. Collections are created and dropped only at the head, so their names are never read inside a transaction, and resolving one never joins the conflicts of the work that follows it. A create records the name in the same step as the collection, refusing a name another collection carries even under a concurrent creator. A name no collection carries resolves to the nil UUID with its presence bit cleared, and ustore_collection_drop with ustore_drop_collection_k forgets a collection’s name before the collection goes. ustore_collection_list exports the identifier of every collection and, where asked, its modality and its name, as a tape where a collection created without one has an empty name. A listing describes the database rather than a transaction, so it reads the head or a snapshot.

§ Describing, Clearing and Compacting

ustore_database_describe reports the engine of an open database, what it supports and how much it holds, while ustore_database_clear and ustore_database_compact work on every collection at once:

ustore_database_description_t description = {0};
ustore_database_describe(&(ustore_database_describe_t){
    .database = database,
    .outcome = &outcome,
    .description = &description,
});
printf("%s holds %llu keys in %llu bytes on disk\n", ustore_engine_name(description.engine),
       (unsigned long long)description.keys, (unsigned long long)description.disk_bytes);

ustore_database_clear(&(ustore_database_clear_t){.database = database, .outcome = &outcome, .mode = ustore_drop_pairs_k});
ustore_database_compact(&(ustore_database_compact_t){.database = database, .outcome = &outcome});

A description’s capabilities say whether the engine supports transactions, collections and snapshots, engine names the engine, and isolation says how its transactions see each other: as the ram engine was opened, and ustore_isolation_snapshot_k elsewhere. keys, memory_bytes and disk_bytes are the engine’s estimates of the keys across every collection and of the bytes the data takes, and zero where it keeps none, as LevelDB keeps no count of its keys. pin_refused_bytes and crossing_requests report where the NVMe engine went unpinned: the arena bytes the locked-memory cap refused to pin, and the requests that crossed a gibibyte of a pinned arena. ustore_database_clear empties every collection as its mode names:

  • ustore_drop_pairs_k removes every pair, keeping the collections and their names;
  • ustore_drop_collection_k drops every collection but the main one with its name, and empties the main one;
  • ustore_drop_values_k empties every value, keeping the keys.

The ram and RocksDB engines refuse to drop the collections while a snapshot is open, with ustore_status_invalid_argument_k, since the snapshot belongs to its holder. ustore_database_compact reclaims the space overwritten and removed data still takes, one collection at a time: RocksDB and LevelDB flush and compact their trees, and the ram engine frees the versions no reader needs and rewrites its image.

API

Types

ustore_arena_description_tWhere an arena's outputs live.
types.h:1079 ↗
struct ustore_arena_description_t

Fields

residencyustore_residency_tRAM for host pages, VRAM for one device's memory.
deviceint32_tThe UStore ordinal of the device as ustore_gpu_devices lists it, or -1 for host pages.
ustore_uuid_tSixteen bytes naming one thing everywhere, ordered as their bytes and compared with memcmp.
db.h:53 ↗
struct ustore_uuid_t

An integer spells as its 128-bit big-endian bytes, as Python's uuid.UUID(int=...) does, so identifiers made from integers order as the integers.

Fields

bytesuint8_t
ustore_collection_id_tThe identifier of a collection: a UUIDv7 its create mints once and never reuses, so it stays the same across restarts and never names two collections.
db.h:59 ↗
struct ustore_collection_id_t

The nil UUID names none.

Fields

ustore_key_tThe identifier of a value within one collection.
db.h:64 ↗
struct ustore_key_t

Fields

ustore_keys_view_tA strided run of keys, one per task, where a zero stride repeats the first.
db.h:82 ↗
struct ustore_keys_view_t

Fields

dataustore_key_t const *First key of the run.
strideustore_size_tBytes between consecutive keys, where zero repeats the first one.
ustore_collections_view_tA strided run of collections, one per task, whose NULL data a call refuses with ustore_status_invalid_argument_k, as no collection was named.
db.h:94 ↗
struct ustore_collections_view_t

Fields

dataustore_collection_id_t const *First collection of the run.
strideustore_size_tBytes between consecutive collections, where zero repeats the first one.
ustore_drop_mode_tWhat dropping a collection removes.
db.h:213 ↗
enum ustore_drop_mode_t

Values

ustore_drop_collection_k0The collection together with its name and every pair it holds.
ustore_drop_pairs_k1Every key and value, keeping the collection.
ustore_drop_values_k2Every value, keeping the keys, which a strings collection refuses, as its values are keys.

Spellings

char const * ustore_drop_mode_name(ustore_drop_mode_t drop_mode);
ustore_drop_mode_t ustore_drop_mode_from_name(ustore_bytes_view_t name);
ustore_capabilities_tCapabilities an engine reports.
db.h:228 ↗
enum ustore_capabilities_t

Values

ustore_capabilities_transactions_k1The engine supports transactions.
ustore_capabilities_snapshots_k4The engine supports snapshots.
ustore_database_description_tWhat an open database reports about itself in one call, where every size is the engine's estimate, and zero where the engine keeps none.
db.h:589 ↗
struct ustore_database_description_t

Fields

capabilitiesustore_capabilities_tThe capabilities of the engine.
engineustore_engine_tThe engine serving the database.
isolationustore_isolation_tHow transactions see each other: what the RAM engine was opened with, and ustore_isolation_snapshot_k elsewhere.
keysustore_size_tKeys across every collection.
memory_bytesustore_size_tBytes of memory the engine holds for the data.
disk_bytesustore_size_tBytes the files of the database take on disk.
ustore_modality_tWhat a collection holds, recorded beside its name so a view of another kind is refused.
db.h:879 ↗
enum ustore_modality_t

Zero names no modality.

Values

ustore_modality_blobs_k1Binary values of blobs.h.
ustore_modality_docs_k2JSON documents of docs.h.
ustore_modality_graph_k3Vertices and their edges of graph.h.
ustore_modality_strings_k4Values keyed by strings of strings.h.
ustore_modality_pages_k5Ordered documents of pages.h.

Spellings

char const * ustore_modality_name(ustore_modality_t modality);
ustore_modality_t ustore_modality_from_name(ustore_bytes_view_t name);

Functions

ustore_database_describeReports the engine of an open database, what it supports and how much it holds.
db.h:644 ↗
void ustore_database_describe(ustore_database_describe_t *arguments);

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
descriptionustore_database_description_t *Output for the description, filled whole; zeroed on failure.

Statuses

ustore_database_clearEmpties every collection at once, as ustore_collection_drop empties one.
db.h:687 ↗
void ustore_database_clear(ustore_database_clear_t *arguments);

The RAM and RocksDB engines refuse to drop the collections while a snapshot is open, since the snapshot belongs to its holder.

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
modeustore_drop_mode_tWhat goes: every pair, keeping the collections and their names; every collection with its name; or every value, keeping the keys where the engine can.

Statuses

ustore_database_compactReclaims the space overwritten and removed data still takes: RocksDB and LevelDB flush and compact their trees, and the RAM engine frees the versions no reader needs and rewrites its image.
db.h:728 ↗
void ustore_database_compact(ustore_database_compact_t *arguments);

The work runs one collection at a time, and its progress counts the bytes of the collections compacted where the engine measures them, as RocksDB does, and one unit per collection elsewhere. A cancelled compaction stops between collections, keeping what it compacted.

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
progressustore_progress_callback_tOptional callback reporting the compaction's advance.
progress_statevoid *Opaque pointer passed to progress.

Statuses

ustore_database_flushMakes every commit that returned before this call durable, as if each had asked for ustore_durability_flushed_k.
db.h:761 ↗
void ustore_database_flush(ustore_database_flush_t *arguments);

LevelDB syncs only its current log, so a commit in a log it already rotated stays unsynced until its compaction lands.

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.

Statuses

ustore_collection_listLists the collections of the database.
db.h:1005 ↗
void ustore_collection_list(ustore_collection_list_t *arguments);

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
snapshotustore_snapshot_tOptional snapshot whose collections are listed, where NULL lists the head.
collectionsustore_collection_id_t const **Optional output for the identifiers of the collections, inside arena.
modalitiesustore_modality_t const **Optional output for what every collection holds, inside arena.
namesustore_strings_view_t *Optional output for the names as a tape inside arena, in the order of the identifiers, where a collection without a name has an empty one.
writtenustore_size_t *Output for the collections written; zeroed on failure.
arenaustore_arena_tRewound by this call, then holding every output it writes; NULL is refused.

Statuses

ustore_collection_createCreates an empty collection, named or not.
db.h:1064 ↗
void ustore_collection_create(ustore_collection_create_t *arguments);

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
nameustore_bytes_view_tOptional name no other collection carries and not starting with _, where a zeroed view creates a collection without one.
modalityustore_modality_tWhat the collection holds, which fixes how it is keyed and so which calls reach it.
collectionustore_collection_id_t *Output for the identifier of the collection, zeroed on failure.

Statuses

ustore_collection_findReads the collections a batch of names address.
db.h:1125 ↗
void ustore_collection_find(ustore_collection_find_t *arguments);

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
snapshotustore_snapshot_tOptional snapshot whose names are read, where NULL reads the head.
countustore_size_tNumber of tasks, one per name.
namesustore_strings_view_tNames to resolve, one per task.
presencesustore_octet_t const **Optional output for a bitset of count bits marking the names a collection carries, addressed as Apache Arrow does, inside arena.
collectionsustore_collection_id_t const **Output for the identifiers of the collections the names address, inside arena.
modalitiesustore_modality_t const **Optional output for what the collections hold, inside arena.
arenaustore_arena_tRewound by this call, then holding every output it writes; NULL is refused.

Statuses

ustore_collection_dropRemoves or clears a collection.
db.h:1169 ↗
void ustore_collection_drop(ustore_collection_drop_t *arguments);

Parameters

databaseustore_database_tAn open database.
executorustore_executor_tExecutor that runs the call, where NULL runs it on the calling thread.
outcomeustore_outcome_t *Output for the outcome, final once its status leaves ustore_status_deferred_k.
completionustore_completion_tWhether the call returns final, the default, or once submitted.
collectionustore_collection_id_tIdentifier of an existing collection.
modeustore_drop_mode_tWhether values, pairs or the whole collection go.

Statuses

Constants

ustore_key_unknown_kThe key that stands for no key, every bit of it set.
db.h:239 ↗
ustore_key_t const ustore_key_unknown_k;