# UStore | One Store for Transactions and Analytics

> UStore, Unum’s embedded database for blobs, documents, graphs, strings and pages: one transaction across every modality, scans and random sampling over the same copy, on local NVMe and GPUs

UStore · Open source · Storage engine · Unum

What if transactions and analytics shared One Store?

Transactions, scans and random sampling over one copy, across every modality.

One embedded database, **one C ABI**, four engines. From your laptop to the largest GPU servers.

[Open Source ](https://github.com/unum-cloud/UStore)[Use with your agent](#for-agents)

Get started

## A database in your process.

Write and read a batch into memory you own.

[Explore the Python API](https://www.unum.cloud/docs/ustore/python.md)

Python

```python
import ustore

with ustore.Database.open_ram() as db:
    arena = ustore.Arena()
    db.write([1, 2], [b"first", b"second"])
    values = db.read([1, 2], arena)
    print(bytes(values[0]))
```

[01 · HTAP](#htap)

## HTAP was a hardware problem.

> The hardware caught up. The architecture stayed split.

Separating storage from compute was the defining idea of the last decade of systems design, and it earned it: capacity and compute scale on their own, object storage is durable and cheap, and a scan does not care about latency. It runs out where reads depend on each other. A transaction validates what it read, a graph query follows edges, a training loop samples at random, and every hop pays a network round trip, so systems split into an OLTP database, an OLAP warehouse and a change-data-capture pipeline that always trails the writes.

The hardware that forced the split is gone. One server holds two dozen PCIe 5.0 drives and GPUs on the same PCIe fabric, and UStore puts transactions, scans and random sampling over one copy on that box.

### Fig. 1.1: Four copies, each a little behind the last, or one

The usual split of OLTP and OLAP systems against one UStore copy. The usual split: OLTP database (rows, one primary), then change capture (tails the log), then queue (a second copy), then warehouse (columns, a third), then training lake (files, a fourth). Four copies, each a little behind the last. UStore: one transaction (blobs, docs, graphs), then one copy (on local NVMe), then scan · sample (Arrow, zero-copy). One copy, readable the moment it commits

Each hop in the usual split is another system to run, another copy to pay for, and another place where the analytics disagree with the transactions they came from.

### Fig. 1.2: In one box the drive is the latency, not the network

Disaggregated: compute and an object store sit on either side of a network, and every query and compaction crosses it twice, in milliseconds.

UStore: one box holds CPU cores, eight GPUs and eight NVMe drives, joined by PCIe lanes through io_uring, SPDK and GPUDirect Storage, in microseconds.

Compaction on object storage turns into a remote copy, and a GPU that could merge sorted runs waits on bytes crossing the network twice. In one box, one thread reads 500K random 4 KiB blocks a second through Linux AIO against 14K through pread, and four stream 60 GB/s against 21.

[02 · vs RocksDB](#rocksdb)

## Your LSM tree has a GPU now.

> Large merges are bulk, regular work: a natural fit for GPUs.

The LSM tree is the right structure for write-heavy storage, and RocksDB is its embedded default; UStore runs it as an engine behind the same ABI. Its compaction is where the time goes: the CPU merges and rewrites every level through the page cache, while a GPU can read the same runs straight from NVMe and merge them in parallel.

A merge of sorted runs is bulk, regular work. UStore folds large merges on GPUs, from sorted adjacency runs to JSON patches, with GPUDirect Storage reading the runs from the drives into GPU memory, `io_uring` and SPDK bypassing the page cache, and SIMD kits keeping point lookups on the CPU. The engine links into your process, with no server or RAID layer.

### Fig. 2.1: One thread of UStore compacts 2 to 3 times faster than RocksDB

The same level-0 runs merged on one thread: through io_uring on the raw device, or ranked on the GPU with GPUDirect Storage reading the drive, all on one PCIe 4 SSD.

Compaction throughput, one thread each

RocksDB 0.266 GB/s

UStore over io_uring 0.648 GB/s

UStore merging on the GPU 0.786 GB/s

Measured on Core i9-13950HX · RTX 4060 Laptop · Crucial T710 · medians of three or more runs  · Sources [UStore’s compaction benchmark](https://github.com/unum-cloud/UStore/blob/main/bench/README.md#compaction)

### Fig. 2.2: A C ABI, UUID keys and staged commits the LSM defaults lack

| Capability | UStore | RocksDB | LevelDB |
| - | - | - | - |
| Interface What other languages link against, and fixed-width keys that compare as machine words. | | | |
| Calling convention | Yes: C99 ABI, one struct per call | Partly: C++ classes, thin C shim | Partly: C++ classes and std::string |
| Keys | Yes: 128-bit UUIDs | No: variable-length strings | No: variable-length strings |
| Data model | | | |
| Modalities | Yes: blobs, documents, graphs, strings, pages | Partly: bytes, wide columns | No: bytes |
| Transactions Validate now, publish later, as another engine catches up. | | | |
| Across collections | Yes: all five modalities together | Yes: across column families | No |
| Two-phase stage | Yes: ustore_transaction_stage | Partly: pessimistic Prepare only | No |
| Hardware | | | |
| Kernel bypass | Yes: io_uring, SPDK | Partly: io_uring for MultiGet | No |
| GPU compaction | Yes: adjacency runs and JSON patches | No | No |
| SIMD point lookups | Yes: SmashTable kits, picked at open | No: Bloom filters, binary search | No: Bloom filters, binary search |
| Workloads | | | |
| Random sampling | Yes: ustore_sample | No | No |
| JSONPath and regex | Yes: on GPUs, into Arrow buffers | No | No |
| Assurance | | | |
| Verification | Yes: Spin models and GenMC | Partly: stress and crash tests | Partly: unit tests |

- full
- partial
- none

UStore even runs RocksDB as an engine behind its own ABI.

[03 · One commit](#acid)

## One commit, across every modality.

> A payment document, a follow edge and an avatar blob commit together or not at all.

Per-collection atomicity and column families are enough while a record is one row. A user is a document, a vertex, an avatar blob and an embedding, and today those live in four systems that agree only eventually.

Blobs, documents, graphs, strings and pages share one 16-byte key with [USearch embeddings](https://www.unum.cloud/engines/usearch.md#use-cases), and one consistency model spans all five on the RAM, NVMe and RocksDB engines. Transactions watch touched keys and conflict rather than block; staging validates writes and hides them until every engine, log or shard is ready, then commit publishes them together.

### Fig. 3.1: One 16‑byte key across five modalities, shared with USearch

Every entry is addressed by 16 bytes, a UUID or a big-endian integer, so a join across modalities or engines is key equality, and a node is searched with SIMD compares.

- #### Blobs

  Binary values, committed with the metadata describing them.

- #### Documents

  JSON addressed by pointer and changed by patch, with no query language.

- #### Graphs

  Directed multigraphs with identified edges, beside the documents they relate.

- #### Strings

  Text read as views into one arena, scanned in key order.

- #### Pages

  HTML, XML, spreadsheets and slides as ordered element trees.

Python

```python
import ustore

db = ustore.Database.open_nvme("/var/lib/ustore/")
alice, bob = 1, 2

txn = db.transaction()
txn["payments"].docs[alice] = {"amount": 10, "currency": "USD"}
txn["connections"].graph.add_edge(alice, bob)
txn["avatars"].blobs[alice] = png_bytes
txn.commit()

# one key reads every modality back
payment = db["payments"].docs[alice]
followed = list(db["connections"].graph.successors(alice))
```

Rust

```rust
use ustore::{jsontape, Database, Durability, Key, NVMEOptions};

let database = Database::open_nvme(&NVMEOptions::at("/var/lib/ustore/"))?;
let [alice, bob] = [1u64, 2].map(Key::from);
let payment = jsontape::parse(br#"{"amount":10,"currency":"USD"}"#)?;

let mut transaction = database.transaction()?;
transaction.collection("payments")?.docs().set(alice, &payment)?;
transaction.collection("connections")?.graph().add_edge(alice, bob)?;
transaction.collection("avatars")?.blobs().set(alice, &png_bytes)?;
transaction.commit(Durability::Buffered)?;
```

C++

```cpp
#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_nvme({.directory = bytes_of("/var/lib/ustore/")});
auto const alice = key_from_integer(1), bob = key_from_integer(2);

auto transaction = *db.transaction();
auto payments = *transaction.find<docs_collection_t>("payments");
auto connections = *transaction.find<graph_collection_t>("connections");
auto avatars = *transaction.find("avatars");

payments[alice].assign(R"({"amount": 10, "currency": "USD"})");
connections.write_edge(edge_t {alice, bob, key_from_integer(100)});
avatars[alice].assign(png_bytes);
transaction.commit();
```

### Fig. 3.2: Stage now, commit when everyone is ready

A commit applies every staged write or none of them.

1. `ustore_transaction_init(&(ustore_transaction_init_t){.transaction = &transaction, .database = db});` A transaction begins.
2. `ustore_docs_write(&(ustore_docs_write_t){.transaction = transaction, .count = count, .keys = keys});` The written keys are watched.
3. `ustore_transaction_stage(&(ustore_transaction_stage_t){.transaction = transaction, .read_stamp = &stamp});` Stamped and hidden; a pinned snapshot sees neither state.
4. `ustore_transaction_commit(&(ustore_transaction_commit_t){.transaction = transaction, .durability = ustore_durability_flushed_k});` Visible and flushed; every reader sees every write.
5. `ustore_transaction_rollback(&(ustore_transaction_rollback_t){.transaction = transaction});` On a conflict, reopens on the newest state.

Names, URLs and paths enter once, through a gateway collection that resolves them to UUIDs, so only the gateway pays for variable-length comparisons.

- Isolation

  - strict_serializable
  - snapshot

- Conflicts

  - write
  - read
  - phantom

- Durability

  - buffered
  - flushed at commit

- Verification

  - Spin
  - GenMC under RC11

[Concurrent retrieval and atomic batches in USearch](https://www.unum.cloud/engines/usearch.md#use-cases)

[04 · Analytics](#machine-learning)

## Shuffle a petabyte without loading it.

> Training rereads a dataset in random order. Its metadata may already exceed RAM.

Columnar warehouses are right for scans and aggregates. Training is neither: it rereads a dataset in random order, so a shuffle of a petabyte is a second copy, and the metadata alone may exceed RAM.

Analytics and training read the copy transactions write. Sample keys uniformly, resolve JSONPath and regular expressions over the drawn documents on GPUs, and gather the matches into typed columns that Arrow, NumPy and [UForm’s loader](https://www.unum.cloud/engines/uform.md#training) read without a copy.

### Fig. 4.1: An epoch is a sample, not a scan

1. 1 · Sample Random keys Draw keys uniformly from a collection bigger than RAM: one walk of its keys fills a reservoir, with no shuffle buffer. `ustore_blobs_sample`
2. 2 · Resolve Fields on GPUs JSONPath and regular expressions run on GPUs over every drawn document in one pass. `ustore_docs_read`
3. 3 · Gather Typed columns Matching fields land in typed columns with validity bitmaps, ready for Arrow and NumPy. `ustore_docs_gather`

Python

```python
import ustore
from ustore.table_view import Table
from ustore.graph_view import DiGraph

arena = ustore.Arena()
keys = db.sample(1024, arena, collection="pictures")
values = db.read(keys, ustore.Arena(), collection="pictures")
loss = model.train_step(values)

columns = db.docs_gather(
    product_keys, ["price", "rating"],
    [ustore.FieldType.F64, ustore.FieldType.F32], arena,
    collection="products",
)
frame = Table(db, "products").astype({"price": "float64"}).to_pandas(page=1 << 16)
graph = DiGraph(db, "follows")
```

Rust

```rust
let mut pictures = database.collection("pictures")?;
let mut arena = ustore::Arena::new()?;

while loss > threshold {
    let batch = pictures.sample(1024, &mut arena)?;
    loss = model.train_step(batch.keys(), batch.values().tape())?;
}
```

[05 · One ABI](#abi)

## A C ABI, not a query language.

> Arguments are named, every omitted one is zero, and the order stops mattering.

A query language is the right boundary between teams. Inside one process it formats, parses, plans and serializes on every call. UStore takes one argument struct per call and returns typed results into an arena you own.

Every modality is implemented once over the in-memory, NVMe, RocksDB and LevelDB engines. Pick one at open and keep the calls; [the docs cover each engine’s limits](https://www.unum.cloud/docs/ustore.md).

### Fig. 5.1: One C ABI, five modalities, any engine

UStore layers, top to bottom. Bindings: C, C++, Python, Rust, Pandas · NetworkX; One C ABI: argument structs, UUID keys, arenas, strided batches, stage / commit; Modalities: blobs, docs, graphs, strings, pages; Engines: ram, nvme, rocksdb, leveldb; Hardware paths: io_uring, SPDK, GPUDirect, SIMD kits, GPU compaction

- Ours
- Third-party

The enabled engines build into ustore_static and ustore_shared, exporting one C interface.

Python · NVMe

```python
import ustore

db = ustore.Database.open_nvme("/var/lib/ustore/", transport=ustore.Transport.LINUX_URING, gpu_devices=range(8))
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()
```

Each collection is an LSM tree whose point lookups run on SIMD kits and whose compactions merge sorted runs on the GPUs it is given.

Python · RAM

```python
import ustore

db = ustore.Database.open_ram("/var/lib/ustore/", isolation=ustore.Isolation.SNAPSHOT)
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()
```

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

Python · RocksDB

```python
import ustore

db = ustore.Database.open_rocksdb("/var/lib/ustore/")
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()
```

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.

Rust · RAM

```rust
use ustore::{jsontape, Database, Durability, Key, Isolation, RAMOptions};

let database = Database::open_ram("/var/lib/ustore/", &RAMOptions {
    isolation: Isolation::Snapshot,
    ..RAMOptions::default()
})?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;
```

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

Rust · NVMe

```rust
use ustore::{jsontape, Database, Durability, Key, NVMEOptions, Transport};

let database = Database::open_nvme(&NVMEOptions {
    transport: Transport::LinuxUring,
    gpu_devices: &[0, 1, 2, 3, 4, 5, 6, 7],
    ..NVMEOptions::at("/var/lib/ustore/")
})?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;
```

Each collection is an LSM tree whose point lookups run on SIMD kits and whose compactions merge sorted runs on the GPUs it is given.

Rust · RocksDB

```rust
use ustore::{jsontape, Database, Durability, Key, RocksDBOptions};

let database = Database::open_rocksdb(&RocksDBOptions::at("/var/lib/ustore/"))?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;
```

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.

C++ · RAM

```cpp
#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_ram({.directory = bytes_of("/var/lib/ustore/"), .isolation = ustore_isolation_snapshot_k});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);
```

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

C++ · NVMe

```cpp
#include <ustore/ustore.hpp>
using namespace unum::ustore;

int const gpus[8] = {0, 1, 2, 3, 4, 5, 6, 7};
auto db = *database_t::open_nvme({
    .directory = bytes_of("/var/lib/ustore/"),
    .transport = ustore_transport_linux_uring_k,
    .gpu_devices = gpus,
    .gpu_devices_count = 8,
});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);
```

Each collection is an LSM tree whose point lookups run on SIMD kits and whose compactions merge sorted runs on the GPUs it is given.

C++ · RocksDB

```cpp
#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_rocksdb({.directory = bytes_of("/var/lib/ustore/")});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);
```

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.

[06 · Questions](#faq)

## Before you move your data

### Keys & data model

#### How do range scans work over UUID keys?

Keys are ordered as their bytes. A scan returns up to a page of keys from a start key upwards, and the next page starts right after the last key returned, so there are no server-side cursors to leak or expire. Time-ordered UUIDv7 keys make those ranges meaningful, since keys written together sort together.

#### How big can values be?

The C ABI counts lengths in 64 bits, as Arrow's `large_binary` does, so neither a value nor a batch stops at 4 GiB. The NVMe engine is tuned for small objects: values up to 64 bytes are inlined into their key slots, a collection defaults to 4 KiB values, and the ceiling is 4 MiB per value. Media files belong in a blob collection on another engine, or behind a key that points at them.

#### What should I know about documents and graphs?

Documents are stored as JSON text, so a raw read returns the same document but not necessarily the bytes you wrote, and `docs_gather` pulls fields from many of them into typed columns, down to `f16` and UUIDs, with bitmaps for which values were present, converted or collided. Graphs are directed multigraphs of source, target and edge-id triplets, indexed from both ends: parallel edges need distinct ids, upserting an edge upserts its vertices, and removing a vertex removes its edges.

#### Why does every call take a batch?

Because round trips cost more than bytes. Every call takes a count of tasks and one array per argument, each with its own byte stride, so an array of your own structs passes without being copied apart, a zero stride repeats one element for every task, and a per-task collection lets one batch span many collections. A batch is atomic: if one task fails, none is applied.

### Transactions & durability

#### Which isolation levels are there, and what can conflict?

Two. `strict_serializable`, the default, validates every read and every scanned window at commit, while `snapshot` validates only the reads a transaction watches, trading write skew for cheaper commits. Transactions are optimistic and conflict rather than block: a commit fails on a write conflict, a read conflict, or a phantom when another commit inserted into a range this one scanned. The NVMe engine watches keys rather than ranges, so its scans report no phantoms.

#### What does stage do that commit doesn't?

Stage is the first phase alone. Both phases keep the transaction's contents whatever the outcome, so a failure can be logged or retried instead of rebuilt, and a commit stages first if you didn't. On NVMe, staging also flushes the write buffer when the changes would not fit it; on RocksDB, it locks and validates every watched read and buffered write, and prepares them into RocksDB's log.

#### What survives a crash?

What was flushed, and commits don't flush by default, since a database built on top usually keeps its own write-ahead log: ask with `ustore_option_write_flush_k` in C, `Durability::Flushed` in Rust or `commit(flush=True)` in Python. RAM keeps no log and rewrites one image on flush and close through a partial file and an atomic rename, so a crash loses what came after it. RocksDB logs every write and keeps a consistent prefix of the commits across every collection, and NVMe keeps its catalog in two checksummed slots, so a torn write fails its checksum and the mount reads the other slot.

### Engines & hardware

#### What do io_uring, SPDK and the GPUs actually do?

io_uring is driven through raw system calls, with no liburing, on Linux 6.12 and newer; where it isn't available, the engine falls back to Linux AIO, [measured in exhibit 1.2](#hyper-converged). SPDK reads the raw drive from user space once it is bound to vfio-pci, and the same ring tests also run it over NVMe/TCP. GPU compaction reads through cuFile, NVIDIA's GPUDirect Storage. GPUs take the large merges, and the host and GPU compactors share one comparator, so their outputs match field by field.

#### What exactly is model-checked?

The orders that are hardest to test. Spin models crash the NVMe engine at every step of its write, flush, record and catalog-sync order and assert that acknowledged data survives, and check its submission rings for completion overflow and ownership hand-offs, while GenMC explores every interleaving the C11 memory model allows for the gates readers pass through. The models of [SmashTable](https://www.unum.cloud/docs/smashtable.md), the transactional containers beneath the RAM engine, found a lost update and a torn write across partitions, both fixed before release.

### Adopting UStore

#### Can UStore replace RocksDB in my DBMS?

Yes, in two steps. First link UStore with its RocksDB engine: your data stays in RocksDB's own format while your code moves onto a C99 ABI with caller-owned arenas and no exceptions; then open the NVMe engine instead, with every other call unchanged. Map string keys onto 128-bit UUIDs once, at the edge, and let your coordinator call `ustore_transaction_stage` as the first phase of a two-phase commit on the RAM, NVMe and RocksDB engines, where RocksDB alone offers it only through pessimistic transactions that lock at `Prepare`.

#### Who owns the memory a read returns, and what can threads share?

You do, until the next call through the same arena. The next call through it discards what the previous one exported unless asked to keep it. Rust enforces that with borrows, Python results own their arena outright, and nothing throws, since running out of memory is a status code like any other. That is what lets UStore live inside a DBMS with its own allocator. A database handle is shared across threads from open until free, while a transaction and an arena belong to one thread at a time, and the Python wheels include free-threaded builds that release the GIL around every call.

#### Can UStore replace an analytical columnar store?

For most businesses, yes: fast enough transactions let 90% of them skip the analytical store, its sync pipeline and its servers, and analyze the freshest data. Still, if you only ever append, only read in full sequential scans, and focus on a few fields of every large document, a good columnar store will beat an excellent transactional one, UStore included.