# Documents

> UStore for C

A document collection stores JSON text, so reading a document back through `ustore_blobs_read` may not return the exact bytes written.
A field addresses a value inside a document, either as a top-level key such as `_id` or as an RFC 6901 JSON Pointer such as `/posts/0/text`, while an empty field or a zeroed `fields` view addresses the whole document.

- `ustore_docs_write` upserts, updates or inserts whole documents or fields, or applies RFC 6902 patches or RFC 7386 merge patches, as its `modification` selects.
- `ustore_docs_read` reads whole documents or fields, as JSON or as values printed into text.
- `ustore_docs_gist` lists the unique JSON Pointers to the leaves of the requested documents.
- `ustore_docs_gather` reads many fields of many documents into typed columns, with validity, conversion and collision bitmaps per column.
- `ustore_docs_remove` removes whole documents, and `ustore_docs_scan` lists their keys as `ustore_blobs_scan` lists the keys of blobs.
- `ustore_docs_sample` draws document keys at random as `ustore_blobs_sample` draws the keys of blobs.

```c
ustore_key_t key = {.uuid.bytes[15] = 1};
char const document[] = "{\"person\": \"Alice\", \"age\": 27}";
uint32_t const document_offsets[2] = {0, sizeof(document) - 1};
ustore_docs_write(&(ustore_docs_write_t){
    .database = database,
    .outcome = &outcome,
    .count = 1,
    .collections = {&movies, 0},
    .keys = {&key, sizeof(ustore_key_t)},
    .values = {.layout = ustore_strings_u32_tape_k, .data.u32_tape = {(uint8_t const*)document, document_offsets}},
    .type = ustore_docs_field_type_json_k,
    .modification = ustore_docs_modify_upsert_k,
});

char const field[] = "/person";
uint32_t const field_offsets[2] = {0, sizeof(field) - 1};
ustore_strings_view_t person = {0};
ustore_docs_read(&(ustore_docs_read_t){
    .database = database,
    .outcome = &outcome,
    .count = 1,
    .collections = {&movies, 0},
    .keys = {&key, sizeof(ustore_key_t)},
    .fields = {.layout = ustore_strings_u32_tape_k, .data.u32_tape = {(uint8_t const*)field, field_offsets}},
    .values = &person,
    .arena = arena,
    .outputs_type = ustore_docs_field_type_string_k,
});
```

A gather takes `fields_count` fields as one strings view and the type of every column in `outputs_types`, and answers one `ustore_docs_column_view_t` per field in the arena.
A column holds the validity, conversion and collision bitmaps of its rows beside either the scalars or a tape of the strings.

`#include <ustore/docs.h>`

`#include <ustore/arrow.h>`

## ustore_docs_field_type_t

Type of a document, or of a value extracted from one.

```c
enum ustore_docs_field_type_t
```

Every type but JSON text, strings and binary text is a fixed-width scalar.

- `ustore_docs_field_type_json_k` = `0`: JSON text.
- `ustore_docs_field_type_null_k` = `10`: The JSON null.
- `ustore_docs_field_type_bool_k` = `11`: A boolean.
- `ustore_docs_field_type_uuid_k` = `12`: A 128-bit UUID.
- `ustore_docs_field_type_f64_k` = `42`: A 64-bit float.
- `ustore_docs_field_type_f32_k` = `41`: A 32-bit float.
- `ustore_docs_field_type_f16_k` = `40`: A 16-bit float.
- `ustore_docs_field_type_i64_k` = `23`: A signed 64-bit integer.
- `ustore_docs_field_type_i32_k` = `22`: A signed 32-bit integer.
- `ustore_docs_field_type_i16_k` = `21`: A signed 16-bit integer.
- `ustore_docs_field_type_i8_k` = `20`: A signed 8-bit integer.
- `ustore_docs_field_type_u64_k` = `33`: An unsigned 64-bit integer.
- `ustore_docs_field_type_u32_k` = `32`: An unsigned 32-bit integer.
- `ustore_docs_field_type_u16_k` = `31`: An unsigned 16-bit integer.
- `ustore_docs_field_type_u8_k` = `30`: An unsigned 8-bit integer.
- `ustore_docs_field_type_binary_k` = `50`: The text of a value without a null terminator.
- `ustore_docs_field_type_string_k` = `51`: A value printed as text.

Spellings: ```
char const * ustore_docs_field_type_name(
    ustore_docs_field_type_t field_type);
```

```
ustore_docs_field_type_t ustore_docs_field_type_from_name(
    ustore_bytes_view_t name);
```

## ustore_docs_modification_t

Modification [`ustore_docs_write`](#ustore_docs_write) applies: an upsert, update or insert, an RFC 6902 patch, or an RFC 7386 merge.

```c
enum ustore_docs_modification_t
```

- `ustore_docs_modify_upsert_k` = `0`
- `ustore_docs_modify_update_k` = `1`
- `ustore_docs_modify_insert_k` = `2`
- `ustore_docs_modify_patch_k` = `3`
- `ustore_docs_modify_merge_k` = `4`

Spellings: ```
char const * ustore_docs_modification_name(
    ustore_docs_modification_t modification);
```

```
ustore_docs_modification_t ustore_docs_modification_from_name(
    ustore_bytes_view_t name);
```

## ustore_docs_column_view_t

One column of `ustore_docs_gather:` `count` rows of one field, as an Apache Arrow array of its type.

```c
struct ustore_docs_column_view_t
```

Validity, conversion and collision columns are bitmaps, least significant bit first. A value is valid when it was found and converted, converted when the conversion changed its semantics, such as a string parsed into an integer, and colliding when a value of an inconvertible type sits under the field.

- `validities` `ustore_octet_t const *`: Bits marking the rows whose value was found and converted.
- `conversions` `ustore_octet_t const *`: Bits marking the rows whose conversion changed the value's semantics.
- `collisions` `ustore_octet_t const *`: Bits marking the rows where a value of an inconvertible type sits under the field.
- `scalars` `ustore_byte_t const *`: The scalars, each as wide as the column's type and aligned to it, or `NULL` for a string column.
- `strings` `ustore_strings_view_t`: The `count` strings of a string column as a tape, or a zeroed view for a scalar column.

## ustore_docs_write

Writes whole documents or fields inside them.

```c
void ustore_docs_write(ustore_docs_write_t *arguments);
```

[`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) means the `collections` or `values` are missing, so are both the `keys` and the `id_field`, the fields or values come in an unknown layout, the `transaction` belongs to another engine or is staged, the `durability`, `watch` or `modification` is unknown, a flushed write names a `transaction`, [`ustore_watch_none_k`](https://www.unum.cloud/docs/ustore/c/transactions.md#ustore_watch_none_k) comes without one, a collection holds another modality, a patch or a modifier does not apply to its field, or a field or value is past 4 GiB.

- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device, or the `type` is not supported.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A collection does not exist or was dropped meanwhile, a field path does not resolve, or the document a field modification, an update or a patch needs is missing.
- [`ustore_status_already_exists_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_already_exists_k) An inserted document already exists.
- [`ustore_status_malformed_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_malformed_k) A document or a value does not parse, is empty or holds NaN or Infinity, an id field is neither an unsigned integer nor a UUID, or a document outgrows what its NVMe collection admits.
- [`ustore_status_write_conflict_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_write_conflict_k) Concurrent commits kept conflicting, or a staged RocksDB transaction holds a key.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The documents outgrow the NVMe transaction's buffer.

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `transaction`: Transaction the writes join, where `NULL` writes to the head.
- `count`: Number of documents or fields to write.
- `collections`: Collections of the documents.
- `keys`: Keys of the documents, where `NULL` data reads every key from `id_field`.
- `id_field`: Field holding the key of every document when `keys` has no data, such as "_id".
- `fields`: Fields to modify inside the documents, one per task, where a zeroed view modifies whole documents.
- `values`: Values to write, one per task.
- `type`: Format of the written values.
- `modification`: Modification applied to every document.
- `durability`: How durable the writes are once the call finishes at the head; a transaction's commit decides its own.
- `watch`: Whether the `transaction` records the documents it reads and writes for conflicts.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The `transaction` serves another call, the executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.

## ustore_docs_read

Reads whole documents or fields inside them.

```c
void ustore_docs_read(ustore_docs_read_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `view`: What the call sees, where `NULL` is the head.
- `count`: Number of documents or fields to read.
- `collections`: Collections of the documents.
- `keys`: Keys of the documents.
- `fields`: Fields to read inside the documents, one per task, where a zeroed view reads whole documents.
- `presences`: Optional output for a bitset of `count` bits marking found values, inside `arena`.
- `lengths`: Optional output for the length of every value, for reads that skip the bytes, inside `arena`.
- `values`: Optional output for the values as a tape inside `arena`, where a missing value reads as an empty one.
- `arena`: Rewound by this call, then holding every output it writes; `NULL` is refused.
- `outputs_type`: Format the values are converted into on export, either JSON or a string.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena`, `collections` or `keys` are missing, the fields come in an unknown layout, the `view` is none of the engine's, a watching read goes through a staged transaction, or a collection holds another modality.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) The arena lives where another executor computes, or the device refused a copy or a launch.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The arena or the transaction the `view` reads through serves another call, the executor's deferred table is full, a callback blocks on the executor running it, or an NVMe value keeps changing size across reads.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL` or an NVMe transfer was cancelled.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) Fields are read as neither JSON nor strings.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A collection does not exist or was dropped meanwhile.
- [`ustore_status_malformed_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_malformed_k) A document does not parse.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) Every shared NVMe channel is taken.

## ustore_docs_gist

Lists the unique paths to the leaves of the requested documents.

```c
void ustore_docs_gist(ustore_docs_gist_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `view`: What the call sees, where `NULL` is the head.
- `count`: Number of documents to inspect.
- `collections`: Collections of the documents.
- `keys`: Keys of the documents.
- `fields`: Output for the `written` unique JSON Pointers as a tape inside `arena`.
- `written`: Output for the unique paths written; zeroed on failure.
- `arena`: Rewound by this call, then holding every output it writes; `NULL` is refused.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena`, `collections` or `keys` are missing, the `view` was made by neither view constructor or belongs to another engine, a watching read goes through a staged transaction, or a collection holds another modality.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) The arena lives where another executor computes.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The arena or the transaction the `view` reads through serves another call, the executor's deferred table is full, a callback blocks on the executor running it, or an NVMe value keeps changing size across reads.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL` or an NVMe transfer was cancelled.
- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A collection does not exist or was dropped meanwhile.
- [`ustore_status_malformed_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_malformed_k) A document does not parse.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) A path is 512 bytes or longer, or every shared NVMe channel is taken.

## ustore_docs_gather

Reads, type-checks and converts `fields_count` fields of `count` documents into columns.

```c
void ustore_docs_gather(ustore_docs_gather_t *arguments);
```

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `view`: What the call sees, where `NULL` is the head.
- `count`: Number of documents, the rows of the output.
- `collections`: Collections of the documents.
- `keys`: Keys of the documents.
- `fields_count`: Number of fields, the columns of the output.
- `fields`: Fields to gather from every document, `fields_count` of them.
- `columns`: Output for `fields_count` columns of `count` rows, one per field, inside `arena`.
- `arena`: Rewound by this call, then holding every output it writes; `NULL` is refused.
- `outputs_types`: Types the columns are converted into, `fields_count` of them, one per field.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) The `arena`, `collections`, `keys` or `columns` are missing, the `fields` or their types are missing or unknown, the `view` is foreign, a watching read goes through a staged transaction, or a collection holds another modality.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) The arena lives where another executor computes, or the device refused a copy or a launch.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The arena or the transaction the `view` reads through serves another call, the executor's deferred table is full, a callback blocks on the executor running it, or an NVMe value keeps changing size across reads.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL` or an NVMe transfer was cancelled.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A collection does not exist or was dropped meanwhile.
- [`ustore_status_malformed_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_malformed_k) A document does not parse.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) Every shared NVMe channel is taken.

## ustore_docs_remove

Removes a batch of whole documents.

```c
void ustore_docs_remove(ustore_docs_remove_t *arguments);
```

[`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) means the `collections` or `keys` are missing, the `transaction` belongs to another engine or is staged, the `durability` or `watch` is unknown, a flushed remove names a `transaction`, [`ustore_watch_none_k`](https://www.unum.cloud/docs/ustore/c/transactions.md#ustore_watch_none_k) comes without one, or a collection holds another modality.

- [`ustore_status_unsupported_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_unsupported_k) The executor computes on a device.
- [`ustore_status_not_found_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_not_found_k) A collection does not exist or was dropped meanwhile.
- [`ustore_status_write_conflict_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_write_conflict_k) Concurrent commits kept conflicting, or a staged RocksDB transaction holds a key.
- [`ustore_status_malformed_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_malformed_k) A key falls outside what its NVMe collection admits.
- [`ustore_status_capacity_exhausted_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_capacity_exhausted_k) The removes outgrow the NVMe transaction's buffer.

- `database`: An open database.
- `executor`: Executor that runs the call, where `NULL` runs it on the calling thread.
- `outcome`: Output for the outcome, final once its status leaves [`ustore_status_deferred_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_deferred_k).
- `completion`: Whether the call returns final, the default, or once submitted.
- `transaction`: Transaction the removals join, where `NULL` removes at the head.
- `count`: Number of tasks, one per document.
- `collections`: Collections of the documents.
- `keys`: Keys of the documents to remove, or inclusive starts of ranges when `end_keys` is given.
- `end_keys`: Exclusive ends of the ranges to remove, or a zeroed view to remove `keys` alone.
- `durability`: How durable the removals are once the call finishes at the head; a transaction's commit decides its own.
- `watch`: Whether the `transaction` records the removed keys for conflicts.

Statuses: - [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) The `transaction` serves another call, the executor's deferred table is full, or a callback blocks on the executor running it.
- [`ustore_status_uninitialized_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_uninitialized_k) The `database` is `NULL`.
- [`ustore_status_read_only_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_read_only_k) The database was opened read-only.

## ustore_docs_column_view_to_arrow

Exports one column of `count` rows a [`ustore_docs_gather`](#ustore_docs_gather) of `type` wrote into `arena` as an Arrow column whose validity buffer is the column's validities, in place where Arrow lays the column out alike.

```c
ustore_status_t ustore_docs_column_view_to_arrow(
    ustore_docs_column_view_t const *column,
    ustore_size_t count,
    ustore_docs_field_type_t type,
    ustore_arena_t arena,
    struct ArrowSchema *schema,
    struct ArrowDeviceArray *array);
```

Scalars export as the Arrow type of their width, UUIDs as `fixed_size_binary(16)` tagged as `arrow.uuid`, and binary strings as `binary` or `large_binary` by the width of their offsets, all in place. Booleans are packed into bits, and strings copied without the null byte ending every row into `utf8` or `large_utf8`, behind the call's outputs in `arena` under the claim a call takes, on the arena's device for a device arena, through an executor the export spawns there, which also reads a `column` the gather left on the device. The array lives until `arena's` next call or free as [`ustore_keys_view_to_arrow`](https://www.unum.cloud/docs/ustore/c/reads-and-writes.md#ustore_keys_view_to_arrow) does, and leaves its null count unknown.

Statuses: - [`ustore_status_invalid_argument_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_invalid_argument_k) An output or the `column` is missing, `type` is no column type, or the column names memory outside `arena`.
- [`ustore_status_contended_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_contended_k) A call fills `arena`, or another export carves it.
- [`ustore_status_device_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_device_k) No executor spawns on the arena's device, or it has no kernels.
- [`ustore_status_out_of_memory_k`](https://www.unum.cloud/docs/ustore/c/errors.md#ustore_status_out_of_memory_k) An allocation failed.