Model::load_ram(path) and Model::load_vram(path) copy a safetensors checkpoint into host memory or onto GPUs, and load_ram_buffer and load_vram_buffer take the checkpoint’s bytes instead.
A bare path or buffer loads on the calling thread with Scalar::BF16 blocks, and its config reaches the executor, the placement and its class pin, Scalar::E4M3 quantization and a progress handler:
let model = Model::load_vram(ModelLoadVramConfig {
executor: Some(&executor),
devices: &[0],
quantization: Scalar::E4M3,
..ModelLoadVramConfig::from("model.safetensors")
})?;
let description = model.describe()?; // geometry, placement, classes and headsA description reports the cpu_class or gpu_class the model dispatched as, and the heads the checkpoint carries: a nonzero token_width for token vectors, and a nonzero choice_temperature_groups for choices. Model carries no type parameter: quantization is the runtime Scalar passed at load, as in C.
ModelA loaded encoder: its weights, their packed copies, and on a GPU the session its attention runs on, whose verbs come from ModelVerbs and EncodeVerbs.8pub struct Modeluse uform::*;
let model = Model::load_ram("model.safetensors")?;
let executor = Executor::for_ram(&[], None, 0, 0)?;
let mut arena = Arena::new_in(&executor)?;
let encoded = model.on(&executor).text_encode(&["a caption"], Outputs::EMBEDDINGS, Scalar::F32, &mut arena)?;
if let Some(embeddings) = encoded.embeddings {
println!("{} rows of {}", embeddings.count(), embeddings.columns());
}
Mirrors uform_model_t; dropping it calls uform_model_free.
onThe receiver whose verbs block on executor, after the calls submitted there before them.pub fn on<'model>(&'model self, executor: &'model Executor) -> On<'model, Model>Rust-only.
deferredThe receiver whose verbs submit their calls to executor, answering a Deferred from the same verbs given owned arguments.pub fn deferred<'model>(&'model self, executor: &'model Executor) -> On<'model, Model, Deferring>The library reads a deferred call's inputs after the verb returns, so lending borrowed texts does not compile:
use uform::*;
let model = Model::load_ram("model.safetensors")?;
let executor = Executor::for_ram(&[], None, 0, 0)?;
let mut arena = Arena::new_in(&executor)?;
let texts = vec![String::from("a caption")];
let call = model.deferred(&executor).text_encode(&texts, Outputs::EMBEDDINGS, Scalar::F32, &mut arena)?;
Owned texts move into the call, and the arena is borrowed as a blocking call borrows it, and so is the waited result:
use uform::*;
let model = Model::load_ram("model.safetensors")?;
let executor = Executor::for_ram(&[], None, 0, 0)?;
let mut arena = Arena::new_in(&executor)?;
let texts = vec![String::from("a caption")];
let call = model.deferred(&executor).text_encode(texts, Outputs::EMBEDDINGS, Scalar::F32, &mut arena)?;
let encoded = call.wait()?;
Rust-only.
load_ramCopies a checkpoint file into host memory and packs its products for the CPU; a bare path loads on the calling thread in BFloat16.pub fn load_ram<'config>(config: impl Into<ModelLoadRamConfig<'config>>) -> Result<Self>Calls uform_model_load_ram.
load_ram_bufferCopies a checkpoint held in memory into host memory and packs its products for the CPU.pub fn load_ram_buffer<'config>(
config: impl Into<ModelLoadRamBufferConfig<'config>>,
) -> Result<Self>Calls uform_model_load_ram_buffer.
load_vramCopies a checkpoint file onto the named GPUs and packs its products there.pub fn load_vram<'config>(config: impl Into<ModelLoadVramConfig<'config>>) -> Result<Self>Calls uform_model_load_vram.
load_vram_bufferCopies a checkpoint held in memory onto the named GPUs and packs its products there.pub fn load_vram_buffer<'config>(
config: impl Into<ModelLoadVramBufferConfig<'config>>,
) -> Result<Self>Calls uform_model_load_vram_buffer.
*× 2The model this receiver reaches and the executor its calls run on.*× 2Reports the model's geometry and placement in one call.Inherited from ModelVerbs
fn describe(
self,
) -> <<Self as ModelVerbs<'model>>::Completion as Completion>::Answer<'model, ModelDescribe>Calls uform_model_describe.
ModelDescribeA report of the model's geometry; see ModelVerbs::describe.ModelDescriptionWhat a loaded model reports about itself.pub struct ModelDescriptionMirrors uform_model_description_t.
dimensions | usize | Scalars per embedding row; Matryoshka prefixes of it are embeddings too. |
|---|---|---|
context | usize | Tokens one item may span before it is truncated: bytes of text, patches of an image, tubes of a clip, frames of a recording. |
quantization | Scalar | The scalar the transformer blocks run in. |
memory_usage | usize | Bytes the model holds; activations are the executor's. |
cpu_class | Option<CPUClass> | The class a RAM model's tower was packed for; None for a VRAM model. |
gpu_class | Option<GPUClass> | The class every replica of a VRAM model runs as; None for a RAM model. |
token_width | usize | Scalars per token vector; zero when the checkpoint carries no token head. |
choice_temperature_groups | usize | Question types the choice temperatures were calibrated over: zero without the byte decoder, one when uncalibrated. |
Clone, Copy, StructuralPartialEq, PartialEq, Eq, Debug, HashModelLoadRamBufferConfigA checkpoint held in memory and how it is copied into a new model in host memory; see Model::load_ram_buffer.pub struct ModelLoadRamBufferConfig<'config>Mirrors uform_model_load_ram_buffer_t.
buffer | &'config [u8] | The safetensors checkpoint itself, copied before the load completes. |
|---|---|---|
executor | Option<&'config Executor> | The executor the load runs on; None loads on the calling thread. |
memory_domains | &'config [i32] | NUMA node IDs to place the weights in; empty derives every node the process may allocate on. |
cpu_class | Option<CPUClass> | The class to pack and dispatch as, as for ModelLoadRamConfig::cpu_class. |
quantization | Scalar | The scalar the transformer blocks run in, as for ModelLoadRamConfig::quantization. |
progress | Option<&'config mut ProgressCallback<'config>> | Reports the tensors placed and stops wherever it says to; a stopped load creates no model. |
progress_steps | usize | Reports the call makes at most, the last at completion; zero derives to ten. |
From<&'config [u8]>, From<&'config Bytes>ModelLoadRamConfigA checkpoint path and how it is copied into a new model in host memory; see Model::load_ram.pub struct ModelLoadRamConfig<'config>Mirrors uform_model_load_ram_t.
path | &'config Path | A safetensors checkpoint, read in full and closed before the load completes. |
|---|---|---|
executor | Option<&'config Executor> | The executor the load runs on; None loads on the calling thread. |
memory_domains | &'config [i32] | NUMA node IDs to place the weights in; empty derives every node the process may allocate on. |
cpu_class | Option<CPUClass> | The class to pack and dispatch as, admitting every rung whose features it holds; None
derives the CPU's own. |
quantization | Scalar | The scalar the transformer blocks run in: Scalar::BF16 as the checkpoint stores it, or
Scalar::E4M3 quantized at load. |
progress | Option<&'config mut ProgressCallback<'config>> | Reports the tensors placed and stops wherever it says to; a stopped load creates no model. |
progress_steps | usize | Reports the call makes at most, the last at completion; zero derives to ten. |
From<&'config Location>ModelLoadVramBufferConfigA checkpoint held in memory and how it is copied onto devices; see Model::load_vram_buffer.pub struct ModelLoadVramBufferConfig<'config>Mirrors uform_model_load_vram_buffer_t.
buffer | &'config [u8] | The safetensors checkpoint itself, in host or device memory, copied before the load completes. |
|---|---|---|
executor | Option<&'config Executor> | The executor the upload runs on, one stream per device; None uploads on
the calling thread. |
devices | &'config [i32] | Positions in what gpu_devices lists, each holding a replica; empty
derives one free device. |
gpu_class | Option<GPUClass> | The class every device dispatches as, as for ModelLoadVramConfig::gpu_class. |
quantization | Scalar | The scalar the transformer blocks run in, as for ModelLoadRamConfig::quantization. |
progress | Option<&'config mut ProgressCallback<'config>> | Reports the tensors placed and stops wherever it says to; a stopped load creates no model. |
progress_steps | usize | Reports the call makes at most, the last at completion; zero derives to ten. |
From<&'config [u8]>, From<&'config Bytes>ModelLoadVramConfigA checkpoint path and how it is copied onto devices; see Model::load_vram.pub struct ModelLoadVramConfig<'config>Mirrors uform_model_load_vram_t.
path | &'config Path | A safetensors checkpoint, read in full and closed before the load completes. |
|---|---|---|
executor | Option<&'config Executor> | The executor the upload runs on, one stream per device; None uploads on
the calling thread. |
devices | &'config [i32] | Positions in what gpu_devices lists, each holding a replica; empty
derives one free device. |
gpu_class | Option<GPUClass> | The class every device dispatches as; None derives the devices' own, which must agree. |
quantization | Scalar | The scalar the transformer blocks run in, as for ModelLoadRamConfig::quantization. |
progress | Option<&'config mut ProgressCallback<'config>> | Reports the tensors placed and stops wherever it says to; a stopped load creates no model. |
progress_steps | usize | Reports the call makes at most, the last at completion; zero derives to ten. |
From<&'config Location>ModelVerbsEvery verb of the model header, written once for every receiver: &Model blocks on the calling thread, and the On that Model::on or Model::deferred makes runs on an executor.1pub trait ModelVerbs<'model>: CopyA blocking receiver answers what the call produced; a deferring one answers a
Deferred that borrows the model. EncodeVerbs
extends it with the forward passes.
Rust-only.
impl<'model> ModelVerbs<'model> for &'model Modelimpl<'model, Completion: Completion> ModelVerbs<'model> for On<'model, Model, Completion>CompletionBlocking or Deferring.type Completion: Completion;ResidencyWhere an executor computes, and with it where the arenas it makes keep their memory.pub enum ResidencyMirrors uform_residency_t.
Ram | 1 | Host pages, which the host reads directly. |
|---|---|---|
Vram | 2 | One GPU's memory, which only device code reads. |
Clone, Copy, StructuralPartialEq, PartialEq, Eq, Debug, Hash