Swift · Training

Training

import UForm 4.0.0-rc.1 · 3ee5e534

Training is GPU-only, so every constructor takes a VRAM executor, and the trainer’s verbs block on it.

let mesh = try Mesh.bootstrap(executor: deviceStreams, options: MeshOptions(
    hostDomainIndex: node, hostDomainCount: nodes, directory: "/shared/rendezvous", job: "run-42"))
let trainer = try Trainer.initVram(
    executor: deviceStreams, depth: 12, hidden: 512,
    batch: BatchShape(maxRowsPerDevice: 64, maxSegmentsPerDevice: 256, maxColumnsPerRow: 1024,
                      maxBagsPerDevice: 128, image: ImageShape(patchSide: 16)),
    objective: .joint, schedule: Schedule(steps: 100_000), mesh: mesh, seed: 42)
let step = try trainer.step(
    modalities: [.text, .image, .text, .image], contents: items, bags: [0, 0, 1, 1],
    relations: Relations(attract: Edges(offsets: [0, 1, 2, 3, 4], ids: [1, 0, 3, 2])))
print(step.stats.step, step.stats.loss, step.stats.contrastiveLoss)
try trainer.save(path: "checkpoint.safetensors")

Modality, Objective, Negatives and MultimemTier spell themselves through the library’s own name tables. Contrastive and LateInteraction each take a PairLossForm, and a positive LateInteraction.weight trains the token head over the items whose sides name them a query, a document or both. A positive Choices.weight trains calibrated choices, which needs BatchShape.maxChoicesPerItem, over the legal bytes each step’s choices names, as textChoose takes them, and their choiceTargets. The step reports both terms as lateInteractionLoss and choicesLoss. A step over arrays of items gathers each into one tape, its one copy, while a step over (data, offsets) tapes lends their storage to the library in place. Modality, Objective, Negatives and MultimemTier read a spelling back with init?(name:), answering nil for a name the library’s table would quietly read as its default.

API

StructModalityWhat the bytes of one training item hold, carrying the C enumerator.6
Train.swift:15 ↗
struct Modality

Conforms to

Hashable, Sendable, CustomStringConvertible, Equatable
textUTF-8 text, a byte per token.
Train.swift:20 ↗
static let text: Modality
imageA JPEG, PNG or WebP file.
Train.swift:22 ↗
static let image: Modality
videoAn MP4 or WebM file.
Train.swift:24 ↗
static let video: Modality
audioA WAV, FLAC, MP3 or Opus file.
Train.swift:26 ↗
static let audio: Modality
descriptionThe modality as the library spells it.
Train.swift:29 ↗
var description: String { get }

Calls uform_modality_name.

initReads a spelling back, and nil for one no modality carries.
Train.swift:33 ↗
init?(name: String)

Calls uform_modality_from_name.

StructObjectiveWhich losses a step relaxes, carrying the C enumerator.5
Train.swift:42 ↗
struct Objective

Conforms to

Hashable, Sendable, CustomStringConvertible, Equatable
maskedMasked reconstruction alone, as the Base pretrains.
Train.swift:47 ↗
static let masked: Objective
contrastiveThe pairwise contrastive loss alone, as the Embedder trains.
Train.swift:49 ↗
static let contrastive: Objective
jointBoth, joined at the final-norm states.
Train.swift:51 ↗
static let joint: Objective
descriptionThe objective as the library spells it.
Train.swift:54 ↗
var description: String { get }

Calls uform_objective_name.

initReads a spelling back, and nil for one no objective carries.
Train.swift:58 ↗
init?(name: String)

Calls uform_objective_from_name.

StructNegativesWhich pairs the contrastive loss pushes apart, carrying the C enumerator.4
Train.swift:67 ↗
struct Negatives

Conforms to

Hashable, Sendable, CustomStringConvertible, Equatable
ambientRepel edges, and every pair no edge names, at the ambient repulsion weight.
Train.swift:72 ↗
static let ambient: Negatives
namedRepel edges alone.
Train.swift:74 ↗
static let named: Negatives
descriptionThe policy as the library spells it.
Train.swift:77 ↗
var description: String { get }

Calls uform_negatives_name.

initReads a spelling back, and nil for one no policy carries.
Train.swift:80 ↗
init?(name: String)

Calls uform_negatives_from_name.

StructPairLossFormHow a contrastive term turns a row of scores into forces, carrying the C enumerator.4
Train.swift:112 ↗
struct PairLossForm

Conforms to

Hashable, Sendable, Equatable
derivedThe term's own form: sigmoid for pooled embeddings, ranked for token vectors.
Train.swift:117 ↗
static let derived: PairLossForm
balancedThe sigmoid cross-entropy of every cell, its attract and repel halves normalized apart.
Train.swift:119 ↗
static let balanced: PairLossForm
rankedThe sigmoid cross-entropy of every positive's score over every negative's, where the bias cancels.
Train.swift:122 ↗
static let ranked: PairLossForm
sigmoidSigLIP's sigmoid cross-entropy of every cell, each repel weighed like an attract.
Train.swift:124 ↗
static let sigmoid: PairLossForm
StructLateInteractionSideWhich side of a late-interaction score an item takes, carrying the C enumerator.4
Train.swift:128 ↗
struct LateInteractionSide

Conforms to

Hashable, Sendable, Equatable
noneThe item takes no part in late interaction.
Train.swift:133 ↗
static let none: LateInteractionSide
queryThe item's tokens score the documents it attracts and repels.
Train.swift:135 ↗
static let query: LateInteractionSide
documentThe item's tokens are scored by the queries naming it.
Train.swift:137 ↗
static let document: LateInteractionSide
queryAndDocumentThe item plays both sides.
Train.swift:139 ↗
static let queryAndDocument: LateInteractionSide
StructImageShapeThe image slice of a batch envelope; a zero patchSide leaves images out of the run.3
Train.swift:258 ↗
struct ImageShape

Conforms to

Sendable
patchSideThe square patch side images tile into, in pixels.
Train.swift:260 ↗
var patchSide: Int
channelsChannels every pixel carries; zero derives four.
Train.swift:262 ↗
var channels: Int
initTakes every field by name; a zero derives as its field documents.
Train.swift:265 ↗
init(patchSide: Int = 0, channels: Int = 0)
StructVideoShapeThe video slice of a batch envelope; a zero patchSide leaves video out of the run.5
Train.swift:272 ↗
struct VideoShape

Conforms to

Sendable
patchSideThe square patch side one tube tiles into, in pixels.
Train.swift:274 ↗
var patchSide: Int
tubeFramesFrames one tube folds ahead of its patch; zero derives two.
Train.swift:276 ↗
var tubeFrames: Int
frameStrideSource frames between sampled ones; zero derives one.
Train.swift:278 ↗
var frameStride: Int
channelsChannels every pixel carries; zero derives three.
Train.swift:280 ↗
var channels: Int
initTakes every field by name; a zero derives as its field documents.
Train.swift:283 ↗
init(patchSide: Int = 0, tubeFrames: Int = 0, frameStride: Int = 0, channels: Int = 0)
StructAudioShapeThe audio slice of a batch envelope; a zero chunkSamples leaves audio out of the run.3
Train.swift:292 ↗
struct AudioShape

Conforms to

Sendable
chunkSamplesRaw samples per frame, the fan-in of the learned frontend.
Train.swift:294 ↗
var chunkSamples: Int
sampleRateThe mono rate every recording is resampled to; zero derives 16000.
Train.swift:296 ↗
var sampleRate: Int
initTakes every field by name; a zero derives as its field documents.
Train.swift:299 ↗
init(chunkSamples: Int = 0, sampleRate: Int = 0)
StructBatchShapeWhat one device's step carries at most, whatever the modality mix.9
Train.swift:307 ↗
struct BatchShape

Mirrors uform_batch_shape_t.

Conforms to

Sendable
maxRowsPerDeviceRows one device's step packs.
Train.swift:309 ↗
var maxRowsPerDevice: Int
maxSegmentsPerDeviceItems one device's step carries across all its rows.
Train.swift:311 ↗
var maxSegmentsPerDevice: Int
maxColumnsPerRowPositions one row spans, the sequence length.
Train.swift:313 ↗
var maxColumnsPerRow: Int
maxBagsPerDeviceBags one device's step carries; zero keeps every item its own routing unit.
Train.swift:315 ↗
var maxBagsPerDevice: Int
maxChoicesPerItemLegal bytes one item offers to choose from at most; zero makes no choices.
Train.swift:317 ↗
var maxChoicesPerItem: Int
imageThe image geometry.
Train.swift:319 ↗
var image: ImageShape
videoThe video geometry.
Train.swift:321 ↗
var video: VideoShape
audioThe audio geometry.
Train.swift:323 ↗
var audio: AudioShape
initTakes every field by name; a zero derives as its field documents.
Train.swift:326 ↗
init(maxRowsPerDevice: Int = 0, maxSegmentsPerDevice: Int = 0, maxColumnsPerRow: Int = 0, maxBagsPerDevice: Int = 0, maxChoicesPerItem: Int = 0, image: ImageShape = ImageShape(), video: VideoShape = VideoShape(), audio: AudioShape = AudioShape())
StructMaskingHow the masked objective corrupts a batch and weighs its heads.12
Train.swift:367 ↗
struct Masking

Mirrors uform_masking_t, every zero deriving the default the C field names.

Conforms to

Sendable
tokenMaskProbabilityFraction of text positions covered by spans; zero derives 0.15.
Train.swift:369 ↗
var tokenMaskProbability: Float
tokenMeanSpanMean geometric span length in bytes; zero derives 16.
Train.swift:371 ↗
var tokenMeanSpan: Float
replaceFractionOf covered positions, the share shown as MASK; zero derives 0.8.
Train.swift:373 ↗
var replaceFraction: Float
randomFractionThe share of covered positions given a random byte, the rest kept clean; zero derives 0.1.
Train.swift:375 ↗
var randomFraction: Float
imageMaskRatioFraction of image patches masked; zero derives 0.75.
Train.swift:377 ↗
var imageMaskRatio: Float
videoMaskRatioFraction of video tubes masked, tubed across time; zero derives 0.75.
Train.swift:379 ↗
var videoMaskRatio: Float
audioMaskRatioFraction of audio frames masked; zero derives 0.75.
Train.swift:381 ↗
var audioMaskRatio: Float
textWeightThe weight λ of the text head; zero derives 1.
Train.swift:383 ↗
var textWeight: Float
imageWeightThe weight λ of the image head; zero derives 1.
Train.swift:385 ↗
var imageWeight: Float
videoWeightThe weight λ of the video head; zero derives 1.
Train.swift:387 ↗
var videoWeight: Float
audioWeightThe weight λ of the audio head; zero derives 1.
Train.swift:389 ↗
var audioWeight: Float
initStarts every field at zero, so each derives its default.
Train.swift:392 ↗
init()
StructContrastiveHow the contrastive loss scores the pooled embeddings.9
Train.swift:413 ↗
struct Contrastive

Mirrors uform_contrastive_t, every zero deriving the default the C field names.

Conforms to

Sendable
matryoshkaDimensionsMatryoshka prefix widths, strictly ascending, each a multiple of 8, at most 8; empty scores the full width.
Train.swift:416 ↗
var matryoshkaDimensions: [Int]
temperatureGroupsLearnable inverse temperature and bias pairs, one per group of a step's items; zero derives one.
Train.swift:419 ↗
var temperatureGroups: Int
inverseTemperatureThe starting inverse temperature of every group; zero derives 10 under the sigmoid form, SigLIP's, else 1.
Train.swift:422 ↗
var inverseTemperature: Float
biasThe starting logit bias of every group; zero derives −10 under the sigmoid form, SigLIP's, else zero.
Train.swift:425 ↗
var bias: Float
negativesWhich pairs repel.
Train.swift:427 ↗
var negatives: Negatives
ambientRepulsionWeightThe weight of every pair no edge names under .ambient negatives; zero derives 1.
Train.swift:429 ↗
var ambientRepulsionWeight: Float
maxEdgesPerDeviceEdges one device's step names per edge set at most; zero derives it from the segments and the largest bag.
Train.swift:432 ↗
var maxEdgesPerDevice: Int
pairLossThe pair loss over each anchor's row; .derived takes sigmoid.
Train.swift:434 ↗
var pairLoss: PairLossForm
initStarts every field at zero or empty, so each derives its default.
Train.swift:437 ↗
init()
StructLateInteractionHow the late-interaction loss scores the per-token vectors of query and document items.10
Train.swift:443 ↗
struct LateInteraction

Mirrors uform_late_interaction_t, every zero deriving the default the C field names; a zero weight trains no token head.

Conforms to

Sendable
weightThe loss weight; zero trains no token head.
Train.swift:445 ↗
var weight: Float
widthThe token vectors' width, a multiple of 8; zero derives 128.
Train.swift:447 ↗
var width: Int
inverseTemperatureThe starting inverse temperature of every contrastive group; zero derives 1.
Train.swift:449 ↗
var inverseTemperature: Float
biasThe starting logit bias of every contrastive group.
Train.swift:451 ↗
var bias: Float
maxQuerySpanTokens one query segment contributes at most, its first past the prefix; zero derives 64.
Train.swift:453 ↗
var maxQuerySpan: Int
maxQueryTokensPerDeviceQuery token rows one device's step holds at most; zero derives maxQuerySpan for every segment.
Train.swift:456 ↗
var maxQueryTokensPerDevice: Int
maxDocumentsPerDeviceDocument segments one device's step holds at most; zero derives maxSegmentsPerDevice.
Train.swift:458 ↗
var maxDocumentsPerDevice: Int
maxDocumentTokensPerDeviceDocument token rows one device's step holds at most; zero derives the batch's tokens per device.
Train.swift:461 ↗
var maxDocumentTokensPerDevice: Int
pairLossThe pair loss over each query's row of hard MaxSim scores; .derived takes ranked.
Train.swift:463 ↗
var pairLoss: PairLossForm
initStarts every field at zero, so each derives its default and no token head trains.
Train.swift:466 ↗
init()
StructScheduleThe learning-rate schedule: a linear warmup to the peak, then a cosine decay to the floor.5
Train.swift:500 ↗
struct Schedule

Mirrors uform_schedule_t.

Conforms to

Sendable
learningRateThe peak learning rate; zero derives 5e-3.
Train.swift:502 ↗
var learningRate: Float
minLearningRateThe floor the cosine settles at; zero derives 1e-5.
Train.swift:504 ↗
var minLearningRate: Float
warmupStepsSteps from zero to the peak; zero derives a tenth of steps, at most 400.
Train.swift:506 ↗
var warmupSteps: Int
stepsSteps the run takes across resumes, which the cosine spans; zero is refused.
Train.swift:508 ↗
var steps: Int
initTakes every field by name; a zero derives as its field documents.
Train.swift:511 ↗
init(learningRate: Float = 0, minLearningRate: Float = 0, warmupSteps: Int = 0, steps: Int = 0)
StructEdgesOne edge set over a step's items, as compressed sparse rows: item i links to ids[offsets[i] ..< offsets[i + 1]].4
Train.swift:523 ↗
struct Edges

Mirrors uform_edges_view_t.

Conforms to

Sendable
offsetsThe step's count + 1 non-decreasing offsets into ids.
Train.swift:525 ↗
var offsets: [Int]
idsIndices into the step's items, ascending within each run.
Train.swift:527 ↗
var ids: [UInt32]
weightsThe weight of each edge, paired with ids; nil weighs every edge one.
Train.swift:529 ↗
var weights: [Float]?
initTakes the compressed rows by name; nil weights weighs every edge one.
Train.swift:532 ↗
init(offsets: [Int], ids: [UInt32], weights: [Float]? = nil)
StructRelationsThe contrastive targets of one step; a pair ignore names is skipped, one attract names pulls together, one repel names pushes apart, and every other pair follows the run's negatives.4
Train.swift:542 ↗
struct Relations

Mirrors uform_relations_view_t.

Conforms to

Sendable
attractPairs pulled together.
Train.swift:544 ↗
var attract: Edges?
repelPairs pushed apart.
Train.swift:546 ↗
var repel: Edges?
ignorePairs skipped, like known false negatives; its weights are not read.
Train.swift:548 ↗
var ignore: Edges?
initTakes the edge sets by name; a nil set names no pairs.
Train.swift:551 ↗
init(attract: Edges? = nil, repel: Edges? = nil, ignore: Edges? = nil)
StructTrainerStepStatsWhat a step reports, every loss its world mean.12
Train.swift:559 ↗
struct TrainerStepStats

Mirrors uform_trainer_step_stats_t.

Conforms to

Sendable
stepThe optimizer step the call applied, counted from one across resumes.
Train.swift:561 ↗
var step: Int
learningRateThe learning rate it applied.
Train.swift:563 ↗
var learningRate: Float
lossThe total loss the step minimized.
Train.swift:565 ↗
var loss: Float
contrastiveLossThe contrastive term; zero when the objective is masked alone.
Train.swift:567 ↗
var contrastiveLoss: Float
maskedTextLossThe text reconstruction term, before its weight; zero when absent.
Train.swift:569 ↗
var maskedTextLoss: Float
maskedImageLossThe image reconstruction term, before its weight; zero when absent.
Train.swift:571 ↗
var maskedImageLoss: Float
maskedVideoLossThe video reconstruction term, before its weight; zero when absent.
Train.swift:573 ↗
var maskedVideoLoss: Float
maskedAudioLossThe audio reconstruction term, before its weight; zero when absent.
Train.swift:575 ↗
var maskedAudioLoss: Float
lateInteractionLossThe late-interaction term, weighted; zero when absent.
Train.swift:577 ↗
var lateInteractionLoss: Float
choicesLossThe choices term over the items that made a choice, before its weight; zero when absent.
Train.swift:579 ↗
var choicesLoss: Float
truncatedItems cut at a row's width.
Train.swift:581 ↗
var truncated: Int
droppedSegments dropped whole because the batch had no room for them, since the last step that read its stats.
Train.swift:584 ↗
var dropped: Int
StructTrainerStepWhat one step wrote: its report, and the pooled embeddings it scored when given an arena.2
Train.swift:603 ↗
struct TrainerStep
statsWhat the step reports.
Train.swift:605 ↗
let stats: TrainerStepStats
embeddingsOne row per item, as the contrastive loss scored it before the update, or nil without an arena.
Train.swift:608 ↗
let embeddings: Rows?
ClassTrainerA training run: the tower replicas, gradients, optimizer moments, heads and temperatures on the devices of its executor, every verb blocking on that executor.3
Train.swift:615 ↗
final class Trainer

Conforms to

Sendable
initVramBuilds one replica per device of executor and starts the run from seed, or continues it from the resume checkpoint; every process of the run calls it with the same geometry, objective and seed.
Train.swift:635 ↗
static func initVram(
    executor: Executor,
    depth: Int,
    hidden: Int,
    batch: BatchShape,
    objective: Objective = .joint,
    masking: Masking = Masking(),
    contrastive: Contrastive = Contrastive(),
    lateInteraction: LateInteraction = LateInteraction(),
    choices: Choices = Choices(),
    schedule: Schedule,
    mesh: Mesh? = nil,
    resume: String = "",
    seed: UInt64 = 0,
    progress: Progress? = nil,
    progressSteps: Int = 0
) throws(UFormError) -> Trainer

Calls uform_trainer_init_vram.

stepApplies one optimizer step over this process's items and every peer's; every process of the run calls it once per step.
Train.swift:700 ↗
func step(
    modalities: [Modality],
    contents: [[UInt8]],
    bags: [UInt32]? = nil,
    temperatureGroups: [UInt32]? = nil,
    sides: [LateInteractionSide]? = nil,
    relations: Relations = Relations(),
    choices: [[UInt8]]? = nil,
    choiceTargets: [[Float]]? = nil,
    arena: Arena? = nil,
    outputsScalar: Scalar = .f32
) throws(UFormError) -> TrainerStep
func step(
    modalities: [Modality],
    contents: (data: [UInt8], offsets: [Int]),
    bags: [UInt32]? = nil,
    temperatureGroups: [UInt32]? = nil,
    sides: [LateInteractionSide]? = nil,
    relations: Relations = Relations(),
    choices: (data: [UInt8], offsets: [Int])? = nil,
    choiceTargets: (data: [Float], offsets: [Int])? = nil,
    arena: Arena? = nil,
    outputsScalar: Scalar = .f32
) throws(UFormError) -> TrainerStep

Applies one optimizer step over tapes already packed as data and its count + 1 Apache Arrow offsets, lent to the library in place, without a copy, until the step returns.

With an arena, the step also lands the pooled embeddings it scored there, as outputsScalar. sides places each item in the late-interaction loss, nil leaving every item out, choices names each item's legal bytes as textChoose takes them, and choiceTargets their target probabilities, an entry as long as the item's choices. Each list of items is gathered into one tape, the one copy this form makes before lending it as the packed form does. Calls uform_trainer_step.

saveWrites the weights and the step to a safetensors checkpoint, which Model.loadVram and its siblings serve and initVram resumes; only the process of index zero writes.
Train.swift:807 ↗
func save(path: String, progress: Progress? = nil, progressSteps: Int = 0) throws(UFormError)

Calls uform_trainer_save.