struct Modality Conforms to
Hashable, Sendable, CustomStringConvertible, EquatableTraining is GPU-only, so every constructor takes a VRAM executor, and the trainer’s verbs block on it.
let mesh = try Mesh.bootstrap(executor: deviceStreams, options: MeshOptions(
hostDomainIndex: node, hostDomainCount: nodes, directory: "/shared/rendezvous", job: "run-42"))
let trainer = try Trainer.initVram(
executor: deviceStreams, depth: 12, hidden: 512,
batch: BatchShape(maxRowsPerDevice: 64, maxSegmentsPerDevice: 256, maxColumnsPerRow: 1024,
maxBagsPerDevice: 128, image: ImageShape(patchSide: 16)),
objective: .joint, schedule: Schedule(steps: 100_000), mesh: mesh, seed: 42)
let step = try trainer.step(
modalities: [.text, .image, .text, .image], contents: items, bags: [0, 0, 1, 1],
relations: Relations(attract: Edges(offsets: [0, 1, 2, 3, 4], ids: [1, 0, 3, 2])))
print(step.stats.step, step.stats.loss, step.stats.contrastiveLoss)
try trainer.save(path: "checkpoint.safetensors")Modality, Objective, Negatives and MultimemTier spell themselves through the library’s own name tables. Contrastive and LateInteraction each take a PairLossForm, and a positive LateInteraction.weight trains the token head over the items whose sides name them a query, a document or both.
A positive Choices.weight trains calibrated choices, which needs BatchShape.maxChoicesPerItem, over the legal bytes each step’s choices names, as textChoose takes them, and their choiceTargets.
The step reports both terms as lateInteractionLoss and choicesLoss.
A step over arrays of items gathers each into one tape, its one copy, while a step over (data, offsets) tapes lends their storage to the library in place. Modality, Objective, Negatives and MultimemTier read a spelling back with init?(name:), answering nil for a name the library’s table would quietly read as its default.
ModalityWhat the bytes of one training item hold, carrying the C enumerator.6textUTF-8 text, a byte per token.static let text: Modality imageA JPEG, PNG or WebP file.static let image: Modality videoAn MP4 or WebM file.static let video: Modality audioA WAV, FLAC, MP3 or Opus file.static let audio: Modality descriptionThe modality as the library spells it.initReads a spelling back, and nil for one no modality carries.ObjectiveWhich losses a step relaxes, carrying the C enumerator.5struct Objective Hashable, Sendable, CustomStringConvertible, EquatablemaskedMasked reconstruction alone, as the Base pretrains.static let masked: Objective contrastiveThe pairwise contrastive loss alone, as the Embedder trains.static let contrastive: Objective jointBoth, joined at the final-norm states.static let joint: Objective descriptionThe objective as the library spells it.initReads a spelling back, and nil for one no objective carries.NegativesWhich pairs the contrastive loss pushes apart, carrying the C enumerator.4struct Negatives Hashable, Sendable, CustomStringConvertible, EquatableambientRepel edges, and every pair no edge names, at the ambient repulsion weight.static let ambient: Negatives namedRepel edges alone.static let named: Negatives descriptionThe policy as the library spells it.initReads a spelling back, and nil for one no policy carries.PairLossFormHow a contrastive term turns a row of scores into forces, carrying the C enumerator.4derivedThe term's own form: sigmoid for pooled embeddings, ranked for token vectors.static let derived: PairLossForm balancedThe sigmoid cross-entropy of every cell, its attract and repel halves normalized apart.static let balanced: PairLossForm rankedThe sigmoid cross-entropy of every positive's score over every negative's, where the bias cancels.static let ranked: PairLossForm sigmoidSigLIP's sigmoid cross-entropy of every cell, each repel weighed like an attract.static let sigmoid: PairLossForm LateInteractionSideWhich side of a late-interaction score an item takes, carrying the C enumerator.4noneThe item takes no part in late interaction.static let none: LateInteractionSide queryThe item's tokens score the documents it attracts and repels.static let query: LateInteractionSide documentThe item's tokens are scored by the queries naming it.static let document: LateInteractionSide queryAndDocumentThe item plays both sides.static let queryAndDocument: LateInteractionSide ImageShapeThe image slice of a batch envelope; a zero patchSide leaves images out of the run.3patchSideThe square patch side images tile into, in pixels.var patchSide: Int channelsChannels every pixel carries; zero derives four.var channels: Int initTakes every field by name; a zero derives as its field documents.init(patchSide: Int = 0, channels: Int = 0) VideoShapeThe video slice of a batch envelope; a zero patchSide leaves video out of the run.5patchSideThe square patch side one tube tiles into, in pixels.var patchSide: Int tubeFramesFrames one tube folds ahead of its patch; zero derives two.var tubeFrames: Int frameStrideSource frames between sampled ones; zero derives one.var frameStride: Int channelsChannels every pixel carries; zero derives three.var channels: Int initTakes every field by name; a zero derives as its field documents.init(patchSide: Int = 0, tubeFrames: Int = 0, frameStride: Int = 0, channels: Int = 0) AudioShapeThe audio slice of a batch envelope; a zero chunkSamples leaves audio out of the run.3chunkSamplesRaw samples per frame, the fan-in of the learned frontend.var chunkSamples: Int sampleRateThe mono rate every recording is resampled to; zero derives 16000.var sampleRate: Int initTakes every field by name; a zero derives as its field documents.init(chunkSamples: Int = 0, sampleRate: Int = 0) BatchShapeWhat one device's step carries at most, whatever the modality mix.9maxRowsPerDeviceRows one device's step packs.var maxRowsPerDevice: Int maxSegmentsPerDeviceItems one device's step carries across all its rows.var maxSegmentsPerDevice: Int maxColumnsPerRowPositions one row spans, the sequence length.var maxColumnsPerRow: Int maxBagsPerDeviceBags one device's step carries; zero keeps every item its own routing unit.var maxBagsPerDevice: Int maxChoicesPerItemLegal bytes one item offers to choose from at most; zero makes no choices.var maxChoicesPerItem: Int imageThe image geometry.var image: ImageShape videoThe video geometry.var video: VideoShape audioThe audio geometry.var audio: AudioShape initTakes every field by name; a zero derives as its field documents.init(maxRowsPerDevice: Int = 0, maxSegmentsPerDevice: Int = 0, maxColumnsPerRow: Int = 0, maxBagsPerDevice: Int = 0, maxChoicesPerItem: Int = 0, image: ImageShape = ImageShape(), video: VideoShape = VideoShape(), audio: AudioShape = AudioShape()) MaskingHow the masked objective corrupts a batch and weighs its heads.12struct Masking Mirrors uform_masking_t, every
zero deriving the default the C field names.
SendabletokenMaskProbabilityFraction of text positions covered by spans; zero derives 0.15.var tokenMaskProbability: Float tokenMeanSpanMean geometric span length in bytes; zero derives 16.var tokenMeanSpan: Float replaceFractionOf covered positions, the share shown as MASK; zero derives 0.8.var replaceFraction: Float randomFractionThe share of covered positions given a random byte, the rest kept clean; zero derives 0.1.var randomFraction: Float imageMaskRatioFraction of image patches masked; zero derives 0.75.var imageMaskRatio: Float videoMaskRatioFraction of video tubes masked, tubed across time; zero derives 0.75.var videoMaskRatio: Float audioMaskRatioFraction of audio frames masked; zero derives 0.75.var audioMaskRatio: Float textWeightThe weight λ of the text head; zero derives 1.var textWeight: Float imageWeightThe weight λ of the image head; zero derives 1.var imageWeight: Float videoWeightThe weight λ of the video head; zero derives 1.var videoWeight: Float audioWeightThe weight λ of the audio head; zero derives 1.var audioWeight: Float initStarts every field at zero, so each derives its default.init() ContrastiveHow the contrastive loss scores the pooled embeddings.9struct Contrastive Mirrors uform_contrastive_t, every zero
deriving the default the C field names.
SendablematryoshkaDimensionsMatryoshka prefix widths, strictly ascending, each a multiple of 8, at most 8; empty scores the full width.var matryoshkaDimensions: [Int] temperatureGroupsLearnable inverse temperature and bias pairs, one per group of a step's items; zero derives one.var temperatureGroups: Int inverseTemperatureThe starting inverse temperature of every group; zero derives 10 under the sigmoid form, SigLIP's, else 1.var inverseTemperature: Float biasThe starting logit bias of every group; zero derives −10 under the sigmoid form, SigLIP's, else zero.var bias: Float negativesWhich pairs repel.var negatives: Negatives ambientRepulsionWeightThe weight of every pair no edge names under .ambient negatives; zero derives 1.var ambientRepulsionWeight: Float maxEdgesPerDeviceEdges one device's step names per edge set at most; zero derives it from the segments and the largest bag.var maxEdgesPerDevice: Int pairLossThe pair loss over each anchor's row; .derived takes sigmoid.var pairLoss: PairLossForm initStarts every field at zero or empty, so each derives its default.init() LateInteractionHow the late-interaction loss scores the per-token vectors of query and document items.10struct LateInteraction Mirrors
uform_late_interaction_t, every zero deriving the default the C field names; a zero weight
trains no token head.
SendableweightThe loss weight; zero trains no token head.var weight: Float widthThe token vectors' width, a multiple of 8; zero derives 128.var width: Int inverseTemperatureThe starting inverse temperature of every contrastive group; zero derives 1.var inverseTemperature: Float biasThe starting logit bias of every contrastive group.var bias: Float maxQuerySpanTokens one query segment contributes at most, its first past the prefix; zero derives 64.var maxQuerySpan: Int maxQueryTokensPerDeviceQuery token rows one device's step holds at most; zero derives maxQuerySpan for every segment.var maxQueryTokensPerDevice: Int maxDocumentsPerDeviceDocument segments one device's step holds at most; zero derives maxSegmentsPerDevice.var maxDocumentsPerDevice: Int maxDocumentTokensPerDeviceDocument token rows one device's step holds at most; zero derives the batch's tokens per device.var maxDocumentTokensPerDevice: Int pairLossThe pair loss over each query's row of hard MaxSim scores; .derived takes ranked.var pairLoss: PairLossForm initStarts every field at zero, so each derives its default and no token head trains.init() ScheduleThe learning-rate schedule: a linear warmup to the peak, then a cosine decay to the floor.5learningRateThe peak learning rate; zero derives 5e-3.var learningRate: Float minLearningRateThe floor the cosine settles at; zero derives 1e-5.var minLearningRate: Float warmupStepsSteps from zero to the peak; zero derives a tenth of steps, at most 400.var warmupSteps: Int stepsSteps the run takes across resumes, which the cosine spans; zero is refused.var steps: Int initTakes every field by name; a zero derives as its field documents.init(learningRate: Float = 0, minLearningRate: Float = 0, warmupSteps: Int = 0, steps: Int = 0) EdgesOne edge set over a step's items, as compressed sparse rows: item i links to ids[offsets[i] ..< offsets[i + 1]].4offsetsThe step's count + 1 non-decreasing offsets into ids.var offsets: [Int] idsIndices into the step's items, ascending within each run.var ids: [UInt32] weightsThe weight of each edge, paired with ids; nil weighs every edge one.var weights: [Float]? initTakes the compressed rows by name; nil weights weighs every edge one.init(offsets: [Int], ids: [UInt32], weights: [Float]? = nil) RelationsThe contrastive targets of one step; a pair ignore names is skipped, one attract names pulls together, one repel names pushes apart, and every other pair follows the run's negatives.4attractPairs pulled together.var attract: Edges? repelPairs pushed apart.var repel: Edges? ignorePairs skipped, like known false negatives; its weights are not read.var ignore: Edges? initTakes the edge sets by name; a nil set names no pairs.init(attract: Edges? = nil, repel: Edges? = nil, ignore: Edges? = nil) TrainerStepStatsWhat a step reports, every loss its world mean.12stepThe optimizer step the call applied, counted from one across resumes.var step: Int learningRateThe learning rate it applied.var learningRate: Float lossThe total loss the step minimized.var loss: Float contrastiveLossThe contrastive term; zero when the objective is masked alone.var contrastiveLoss: Float maskedTextLossThe text reconstruction term, before its weight; zero when absent.var maskedTextLoss: Float maskedImageLossThe image reconstruction term, before its weight; zero when absent.var maskedImageLoss: Float maskedVideoLossThe video reconstruction term, before its weight; zero when absent.var maskedVideoLoss: Float maskedAudioLossThe audio reconstruction term, before its weight; zero when absent.var maskedAudioLoss: Float lateInteractionLossThe late-interaction term, weighted; zero when absent.var lateInteractionLoss: Float choicesLossThe choices term over the items that made a choice, before its weight; zero when absent.var choicesLoss: Float truncatedItems cut at a row's width.var truncated: Int droppedSegments dropped whole because the batch had no room for them, since the last step that read its stats.var dropped: Int TrainerStepWhat one step wrote: its report, and the pooled embeddings it scored when given an arena.2struct TrainerStep statsWhat the step reports.let stats: TrainerStepStats embeddingsOne row per item, as the contrastive loss scored it before the update, or nil without an arena.let embeddings: Rows? TrainerA training run: the tower replicas, gradients, optimizer moments, heads and temperatures on the devices of its executor, every verb blocking on that executor.3initVramBuilds one replica per device of executor and starts the run from seed, or continues it from the resume checkpoint; every process of the run calls it with the same geometry, objective and seed.static func initVram(
executor: Executor,
depth: Int,
hidden: Int,
batch: BatchShape,
objective: Objective = .joint,
masking: Masking = Masking(),
contrastive: Contrastive = Contrastive(),
lateInteraction: LateInteraction = LateInteraction(),
choices: Choices = Choices(),
schedule: Schedule,
mesh: Mesh? = nil,
resume: String = "",
seed: UInt64 = 0,
progress: Progress? = nil,
progressSteps: Int = 0
) throws(UFormError) -> Trainer Calls uform_trainer_init_vram.
stepApplies one optimizer step over this process's items and every peer's; every process of the run calls it once per step.func step(
modalities: [Modality],
contents: [[UInt8]],
bags: [UInt32]? = nil,
temperatureGroups: [UInt32]? = nil,
sides: [LateInteractionSide]? = nil,
relations: Relations = Relations(),
choices: [[UInt8]]? = nil,
choiceTargets: [[Float]]? = nil,
arena: Arena? = nil,
outputsScalar: Scalar = .f32
) throws(UFormError) -> TrainerStep func step(
modalities: [Modality],
contents: (data: [UInt8], offsets: [Int]),
bags: [UInt32]? = nil,
temperatureGroups: [UInt32]? = nil,
sides: [LateInteractionSide]? = nil,
relations: Relations = Relations(),
choices: (data: [UInt8], offsets: [Int])? = nil,
choiceTargets: (data: [Float], offsets: [Int])? = nil,
arena: Arena? = nil,
outputsScalar: Scalar = .f32
) throws(UFormError) -> TrainerStep Applies one optimizer step over tapes already packed as data and its count + 1 Apache Arrow offsets, lent to the library in place, without a copy, until the step returns.
With an arena, the step also lands the pooled embeddings it
scored there, as outputsScalar. sides places each item in the late-interaction loss, nil
leaving every item out, choices names each item's legal bytes as textChoose takes them,
and choiceTargets their target probabilities, an entry as long as the item's choices. Each
list of items is gathered into one tape, the one copy this form makes before lending it as
the packed form does. Calls uform_trainer_step.
saveWrites the weights and the step to a safetensors checkpoint, which Model.loadVram and its siblings serve and initVram resumes; only the process of index zero writes.func save(path: String, progress: Progress? = nil, progressSteps: Int = 0) throws(UFormError) Calls uform_trainer_save.