enum uform_residency_tValues
uform_residency_unknown_k | 0 | No residency named. |
|---|---|---|
uform_residency_ram_k | 1 | Host pages, which the host reads directly. |
uform_residency_vram_k | 2 | One GPU's memory, which only device code reads. |
Residency is in the verb name, as in USearch and UStore, so a residency a build lacks has no verb to call:
- RAM:
uform_model_load_ramanduform_model_load_ram_bufferplace the weights in host memory, on the NUMA nodesuform_ram_options_tnames, and every build carries them. - NVIDIA:
uform_model_load_vramanduform_model_load_vram_bufferplace them on the CUDA devicesuform_vram_options_tnames, in builds withUFORM_CUDA_ARCH. - AMD: the same VRAM verbs place them on ROCm devices, in builds with
UFORM_ROCM_ARCH.
On NVIDIA, text and JPEG images never leave the device; container demux, PNG and WebP, audio, and every modality on AMD decode on host threads the executor spawns beside its streams. uform_capabilities_compiled and uform_capabilities_available report what the binary was built with and what the machine offers.
uform_model_describe reports the model’s dimensions, context, quantization, memory usage and the CPU or GPU class it dispatched to.
It also reports the cpu_class a RAM model was packed for or the gpu_class a VRAM model runs as, the token_width of its token head, and the choice_temperature_groups of its byte decoder, each zero when the checkpoint carries no such part.
API
Types
uform_residency_tWhere an arena's outputs live, as the executor that made it decided.
uform_model_tA loaded encoder: its weights, their packed copies, and on a GPU the session its attention runs on.
typedef struct uform_model_handle_t * uform_model_t;Shared by every executor of its residency; calls through different executors take turns on its session.
uform_model_description_tWhat a loaded model reports about itself.
struct uform_model_description_tFields
dimensions | size_t | Scalars per embedding row; Matryoshka prefixes of it are embeddings too. |
|---|---|---|
context | size_t | Tokens one item may span before it is truncated: bytes of text, patches of an image, tubes of a clip, frames of a recording. |
quantization | uform_scalar_t | The scalar the transformer blocks run in. |
memory_usage | size_t | Bytes the model holds; activations are the executor's. |
cpu_class | uform_cpu_class_t | The class a RAM model's tower was packed for; uform_cpu_class_unknown_k for a VRAM model. |
gpu_class | uform_gpu_class_t | The class every replica of a VRAM model runs as; uform_gpu_class_unknown_k for a RAM model, and for one on its vendor's generic kernels, whose devices have no class of their own. |
token_width | size_t | Scalars per token vector; zero when the checkpoint carries no token head. |
choice_temperature_groups | size_t | Question types the choice temperatures were calibrated over: zero without the byte decoder, one when uncalibrated. |
Functions
uform_model_load_ramCopies a checkpoint file into host memory and packs its products for the CPU.
void uform_model_load_ram(uform_model_load_ram_t *arguments);The checkpoint carries the model's geometry.
Parameters
modeluform_model_t * | Output for the new model, written once the call completes. | |
|---|---|---|
executoruform_executor_t | The executor the load runs on. | |
outcomeuform_outcome_t * | Output for the outcome, final once its status leaves uform_status_deferred_k. | |
completionuform_completion_t | Whether the call blocks or returns once queued; zero blocks. | |
pathuform_bytes_view_t | Path to a safetensors checkpoint, read in full and closed before the load completes. | |
ramuform_ram_options_t | The memory domains to place the weights in. | |
quantizationuform_scalar_t | The scalar the transformer blocks run in: uform_scalar_bf16_k as the checkpoint stores it, or uform_scalar_e4m3_k quantized at load, with BFloat16 entries, attention pool and head. | |
progressuform_progress_t | The optional handler, called with tensors placed. |
uform_model_load_ram_bufferCopies a checkpoint held in memory into host memory and packs its products for the CPU.
void uform_model_load_ram_buffer(
uform_model_load_ram_buffer_t *arguments);Parameters
modeluform_model_t * | Output for the new model, written once the call completes. | |
|---|---|---|
executoruform_executor_t | The executor the load runs on. | |
outcomeuform_outcome_t * | Output for the outcome, final once its status leaves uform_status_deferred_k. | |
completionuform_completion_t | Whether the call blocks or returns once queued; zero blocks. | |
bufferuform_bytes_view_t | The safetensors checkpoint itself, copied before the load completes; nothing of it is kept. | |
ramuform_ram_options_t | The memory domains to place the weights in. | |
quantizationuform_scalar_t | The scalar the transformer blocks run in, as for uform_model_load_ram. | |
progressuform_progress_t | The optional handler, called with tensors placed. |
uform_model_load_vramCopies a checkpoint file onto the named GPUs and packs its products there.
void uform_model_load_vram(uform_model_load_vram_t *arguments);Parameters
modeluform_model_t * | Output for the new model, written once the call completes. | |
|---|---|---|
executoruform_executor_t | The executor the upload runs on, one stream per device. | |
outcomeuform_outcome_t * | Output for the outcome, final once its status leaves uform_status_deferred_k. | |
completionuform_completion_t | Whether the call blocks or returns once queued; zero blocks. | |
pathuform_bytes_view_t | Path to a safetensors checkpoint, read in full and closed before the load completes. | |
vramuform_vram_options_t | The devices to place the weights on, replicated on each, since a model is small next to the activations it serves. | |
quantizationuform_scalar_t | The scalar the transformer blocks run in, as for uform_model_load_ram. | |
progressuform_progress_t | The optional handler, called with tensors placed. |
uform_model_load_vram_bufferCopies a checkpoint held in memory onto the named GPUs and packs its products there.
void uform_model_load_vram_buffer(
uform_model_load_vram_buffer_t *arguments);Parameters
modeluform_model_t * | Output for the new model, written once the call completes. | |
|---|---|---|
executoruform_executor_t | The executor the upload runs on, one stream per device. | |
outcomeuform_outcome_t * | Output for the outcome, final once its status leaves uform_status_deferred_k. | |
completionuform_completion_t | Whether the call blocks or returns once queued; zero blocks. | |
bufferuform_bytes_view_t | The safetensors checkpoint itself, in host or device memory, copied before the load completes. | |
vramuform_vram_options_t | The devices to place the weights on, replicated on each. | |
quantizationuform_scalar_t | The scalar the transformer blocks run in, as for uform_model_load_ram. | |
progressuform_progress_t | The optional handler, called with tensors placed. |
uform_model_describeReports a model's geometry and placement in one call.
void uform_model_describe(uform_model_describe_t *arguments);Parameters
modeluform_model_t | The model to be described. | |
|---|---|---|
executoruform_executor_t | The executor the report runs on. | |
outcomeuform_outcome_t * | Output for the outcome, final once its status leaves uform_status_deferred_k. | |
completionuform_completion_t | Whether the call blocks or returns once queued; zero blocks. | |
descriptionuform_model_description_t * | Output, filled whole. |