# Training > UForm for Swift Training is GPU-only, so every constructor takes a VRAM executor, and the trainer's verbs block on it. ```swift let mesh = try Mesh.bootstrap(executor: deviceStreams, options: MeshOptions( hostDomainIndex: node, hostDomainCount: nodes, directory: "/shared/rendezvous", job: "run-42")) let trainer = try Trainer.initVram( executor: deviceStreams, depth: 12, hidden: 512, batch: BatchShape(maxRowsPerDevice: 64, maxSegmentsPerDevice: 256, maxColumnsPerRow: 1024, maxBagsPerDevice: 128, image: ImageShape(patchSide: 16)), objective: .joint, schedule: Schedule(steps: 100_000), mesh: mesh, seed: 42) let step = try trainer.step( modalities: [.text, .image, .text, .image], contents: items, bags: [0, 0, 1, 1], relations: Relations(attract: Edges(offsets: [0, 1, 2, 3, 4], ids: [1, 0, 3, 2]))) print(step.stats.step, step.stats.loss, step.stats.contrastiveLoss) try trainer.save(path: "checkpoint.safetensors") ``` `Modality`, `Objective`, `Negatives` and `MultimemTier` spell themselves through the library's own name tables. `Contrastive` and `LateInteraction` each take a `PairLossForm`, and a positive `LateInteraction.weight` trains the token head over the items whose `sides` name them a query, a document or both. A positive `Choices.weight` trains calibrated choices, which needs `BatchShape.maxChoicesPerItem`, over the legal bytes each step's `choices` names, as `textChoose` takes them, and their `choiceTargets`. The step reports both terms as `lateInteractionLoss` and `choicesLoss`. A step over arrays of items gathers each into one tape, its one copy, while a step over `(data, offsets)` tapes lends their storage to the library in place. `Modality`, `Objective`, `Negatives` and `MultimemTier` read a spelling back with `init?(name:)`, answering nil for a name the library's table would quietly read as its default. `import UForm` ## Modality What the bytes of one training item hold, carrying the C enumerator. ```swift struct Modality ``` Conforms to: `Hashable`, `Sendable`, `CustomStringConvertible`, `Equatable` ## text UTF-8 text, a byte per token. ```swift static let text: Modality ``` ## image A JPEG, PNG or WebP file. ```swift static let image: Modality ``` ## video An MP4 or WebM file. ```swift static let video: Modality ``` ## audio A WAV, FLAC, MP3 or Opus file. ```swift static let audio: Modality ``` ## description The modality as the library spells it. ```swift var description: String { get } ``` Calls `uform_modality_name`. ## init Reads a spelling back, and nil for one no modality carries. ```swift init?(name: String) ``` Calls `uform_modality_from_name`. ## Objective Which losses a step relaxes, carrying the C enumerator. ```swift struct Objective ``` Conforms to: `Hashable`, `Sendable`, `CustomStringConvertible`, `Equatable` ## masked Masked reconstruction alone, as the Base pretrains. ```swift static let masked: Objective ``` ## contrastive The pairwise contrastive loss alone, as the Embedder trains. ```swift static let contrastive: Objective ``` ## joint Both, joined at the final-norm states. ```swift static let joint: Objective ``` ## description The objective as the library spells it. ```swift var description: String { get } ``` Calls `uform_objective_name`. ## init Reads a spelling back, and nil for one no objective carries. ```swift init?(name: String) ``` Calls `uform_objective_from_name`. ## Negatives Which pairs the contrastive loss pushes apart, carrying the C enumerator. ```swift struct Negatives ``` Conforms to: `Hashable`, `Sendable`, `CustomStringConvertible`, `Equatable` ## ambient Repel edges, and every pair no edge names, at the ambient repulsion weight. ```swift static let ambient: Negatives ``` ## named Repel edges alone. ```swift static let named: Negatives ``` ## description The policy as the library spells it. ```swift var description: String { get } ``` Calls `uform_negatives_name`. ## init Reads a spelling back, and nil for one no policy carries. ```swift init?(name: String) ``` Calls `uform_negatives_from_name`. ## PairLossForm How a contrastive term turns a row of scores into forces, carrying the C enumerator. ```swift struct PairLossForm ``` Conforms to: `Hashable`, `Sendable`, `Equatable` ## derived The term's own form: sigmoid for pooled embeddings, ranked for token vectors. ```swift static let derived: PairLossForm ``` ## balanced The sigmoid cross-entropy of every cell, its attract and repel halves normalized apart. ```swift static let balanced: PairLossForm ``` ## ranked The sigmoid cross-entropy of every positive's score over every negative's, where the bias cancels. ```swift static let ranked: PairLossForm ``` ## sigmoid SigLIP's sigmoid cross-entropy of every cell, each repel weighed like an attract. ```swift static let sigmoid: PairLossForm ``` ## LateInteractionSide Which side of a late-interaction score an item takes, carrying the C enumerator. ```swift struct LateInteractionSide ``` Conforms to: `Hashable`, `Sendable`, `Equatable` ## none The item takes no part in late interaction. ```swift static let none: LateInteractionSide ``` ## query The item's tokens score the documents it attracts and repels. ```swift static let query: LateInteractionSide ``` ## document The item's tokens are scored by the queries naming it. ```swift static let document: LateInteractionSide ``` ## queryAndDocument The item plays both sides. ```swift static let queryAndDocument: LateInteractionSide ``` ## ImageShape The image slice of a batch envelope; a zero [`patchSide`](#patchSide) leaves images out of the run. ```swift struct ImageShape ``` Conforms to: `Sendable` ## patchSide The square patch side images tile into, in pixels. ```swift var patchSide: Int ``` ## channels Channels every pixel carries; zero derives four. ```swift var channels: Int ``` ## init Takes every field by name; a zero derives as its field documents. ```swift init(patchSide: Int = 0, channels: Int = 0) ``` ## VideoShape The video slice of a batch envelope; a zero [`patchSide`](#patchSide) leaves video out of the run. ```swift struct VideoShape ``` Conforms to: `Sendable` ## patchSide The square patch side one tube tiles into, in pixels. ```swift var patchSide: Int ``` ## tubeFrames Frames one tube folds ahead of its patch; zero derives two. ```swift var tubeFrames: Int ``` ## frameStride Source frames between sampled ones; zero derives one. ```swift var frameStride: Int ``` ## channels Channels every pixel carries; zero derives three. ```swift var channels: Int ``` ## init Takes every field by name; a zero derives as its field documents. ```swift init(patchSide: Int = 0, tubeFrames: Int = 0, frameStride: Int = 0, channels: Int = 0) ``` ## AudioShape The audio slice of a batch envelope; a zero [`chunkSamples`](#chunkSamples) leaves audio out of the run. ```swift struct AudioShape ``` Conforms to: `Sendable` ## chunkSamples Raw samples per frame, the fan-in of the learned frontend. ```swift var chunkSamples: Int ``` ## sampleRate The mono rate every recording is resampled to; zero derives 16000. ```swift var sampleRate: Int ``` ## init Takes every field by name; a zero derives as its field documents. ```swift init(chunkSamples: Int = 0, sampleRate: Int = 0) ``` ## BatchShape What one device's step carries at most, whatever the modality mix. ```swift struct BatchShape ``` Mirrors `uform_batch_shape_t`. Conforms to: `Sendable` ## maxRowsPerDevice Rows one device's step packs. ```swift var maxRowsPerDevice: Int ``` ## maxSegmentsPerDevice Items one device's step carries across all its rows. ```swift var maxSegmentsPerDevice: Int ``` ## maxColumnsPerRow Positions one row spans, the sequence length. ```swift var maxColumnsPerRow: Int ``` ## maxBagsPerDevice Bags one device's step carries; zero keeps every item its own routing unit. ```swift var maxBagsPerDevice: Int ``` ## maxChoicesPerItem Legal bytes one item offers to choose from at most; zero makes no choices. ```swift var maxChoicesPerItem: Int ``` ## image The image geometry. ```swift var image: ImageShape ``` ## video The video geometry. ```swift var video: VideoShape ``` ## audio The audio geometry. ```swift var audio: AudioShape ``` ## init Takes every field by name; a zero derives as its field documents. ```swift init(maxRowsPerDevice: Int = 0, maxSegmentsPerDevice: Int = 0, maxColumnsPerRow: Int = 0, maxBagsPerDevice: Int = 0, maxChoicesPerItem: Int = 0, image: ImageShape = ImageShape(), video: VideoShape = VideoShape(), audio: AudioShape = AudioShape()) ``` ## Masking How the masked objective corrupts a batch and weighs its heads. ```swift struct Masking ``` Mirrors `uform_masking_t`, every zero deriving the default the C field names. Conforms to: `Sendable` ## tokenMaskProbability Fraction of text positions covered by spans; zero derives 0.15. ```swift var tokenMaskProbability: Float ``` ## tokenMeanSpan Mean geometric span length in bytes; zero derives 16. ```swift var tokenMeanSpan: Float ``` ## replaceFraction Of covered positions, the share shown as MASK; zero derives 0.8. ```swift var replaceFraction: Float ``` ## randomFraction The share of covered positions given a random byte, the rest kept clean; zero derives 0.1. ```swift var randomFraction: Float ``` ## imageMaskRatio Fraction of image patches masked; zero derives 0.75. ```swift var imageMaskRatio: Float ``` ## videoMaskRatio Fraction of video tubes masked, tubed across time; zero derives 0.75. ```swift var videoMaskRatio: Float ``` ## audioMaskRatio Fraction of audio frames masked; zero derives 0.75. ```swift var audioMaskRatio: Float ``` ## textWeight The weight λ of the text head; zero derives 1. ```swift var textWeight: Float ``` ## imageWeight The weight λ of the image head; zero derives 1. ```swift var imageWeight: Float ``` ## videoWeight The weight λ of the video head; zero derives 1. ```swift var videoWeight: Float ``` ## audioWeight The weight λ of the audio head; zero derives 1. ```swift var audioWeight: Float ``` ## init Starts every field at zero, so each derives its default. ```swift init() ``` ## Contrastive How the contrastive loss scores the pooled embeddings. ```swift struct Contrastive ``` Mirrors `uform_contrastive_t`, every zero deriving the default the C field names. Conforms to: `Sendable` ## matryoshkaDimensions Matryoshka prefix widths, strictly ascending, each a multiple of 8, at most 8; empty scores the full width. ```swift var matryoshkaDimensions: [Int] ``` ## temperatureGroups Learnable inverse temperature and bias pairs, one per group of a step's items; zero derives one. ```swift var temperatureGroups: Int ``` ## inverseTemperature The starting inverse temperature of every group; zero derives 10 under the sigmoid form, SigLIP's, else 1. ```swift var inverseTemperature: Float ``` ## bias The starting logit bias of every group; zero derives −10 under the sigmoid form, SigLIP's, else zero. ```swift var bias: Float ``` ## negatives Which pairs repel. ```swift var negatives: Negatives ``` ## ambientRepulsionWeight The weight of every pair no edge names under `.ambient` negatives; zero derives 1. ```swift var ambientRepulsionWeight: Float ``` ## maxEdgesPerDevice Edges one device's step names per edge set at most; zero derives it from the segments and the largest bag. ```swift var maxEdgesPerDevice: Int ``` ## pairLoss The pair loss over each anchor's row; `.derived` takes sigmoid. ```swift var pairLoss: PairLossForm ``` ## init Starts every field at zero or empty, so each derives its default. ```swift init() ``` ## LateInteraction How the late-interaction loss scores the per-token vectors of query and document items. ```swift struct LateInteraction ``` Mirrors `uform_late_interaction_t`, every zero deriving the default the C field names; a zero [`weight`](#weight) trains no token head. Conforms to: `Sendable` ## weight The loss weight; zero trains no token head. ```swift var weight: Float ``` ## width The token vectors' width, a multiple of 8; zero derives 128. ```swift var width: Int ``` ## inverseTemperature The starting inverse temperature of every contrastive group; zero derives 1. ```swift var inverseTemperature: Float ``` ## bias The starting logit bias of every contrastive group. ```swift var bias: Float ``` ## maxQuerySpan Tokens one query segment contributes at most, its first past the prefix; zero derives 64. ```swift var maxQuerySpan: Int ``` ## maxQueryTokensPerDevice Query token rows one device's step holds at most; zero derives [`maxQuerySpan`](#maxQuerySpan) for every segment. ```swift var maxQueryTokensPerDevice: Int ``` ## maxDocumentsPerDevice Document segments one device's step holds at most; zero derives [`maxSegmentsPerDevice`](#maxSegmentsPerDevice). ```swift var maxDocumentsPerDevice: Int ``` ## maxDocumentTokensPerDevice Document token rows one device's step holds at most; zero derives the batch's tokens per device. ```swift var maxDocumentTokensPerDevice: Int ``` ## pairLoss The pair loss over each query's row of hard MaxSim scores; `.derived` takes ranked. ```swift var pairLoss: PairLossForm ``` ## init Starts every field at zero, so each derives its default and no token head trains. ```swift init() ``` ## Schedule The learning-rate schedule: a linear warmup to the peak, then a cosine decay to the floor. ```swift struct Schedule ``` Mirrors `uform_schedule_t`. Conforms to: `Sendable` ## learningRate The peak learning rate; zero derives 5e-3. ```swift var learningRate: Float ``` ## minLearningRate The floor the cosine settles at; zero derives 1e-5. ```swift var minLearningRate: Float ``` ## warmupSteps Steps from zero to the peak; zero derives a tenth of [`steps`](#steps), at most 400. ```swift var warmupSteps: Int ``` ## steps Steps the run takes across resumes, which the cosine spans; zero is refused. ```swift var steps: Int ``` ## init Takes every field by name; a zero derives as its field documents. ```swift init(learningRate: Float = 0, minLearningRate: Float = 0, warmupSteps: Int = 0, steps: Int = 0) ``` ## Edges One edge set over a step's items, as compressed sparse rows: item `i` links to `ids[offsets[i] ..< offsets[i + 1]]`. ```swift struct Edges ``` Mirrors `uform_edges_view_t`. Conforms to: `Sendable` ## offsets The step's count + 1 non-decreasing offsets into [`ids`](#ids). ```swift var offsets: [Int] ``` ## ids Indices into the step's items, ascending within each run. ```swift var ids: [UInt32] ``` ## weights The weight of each edge, paired with [`ids`](#ids); nil weighs every edge one. ```swift var weights: [Float]? ``` ## init Takes the compressed rows by name; nil [`weights`](#weights) weighs every edge one. ```swift init(offsets: [Int], ids: [UInt32], weights: [Float]? = nil) ``` ## Relations The contrastive targets of one step; a pair [`ignore`](#ignore) names is skipped, one [`attract`](#attract) names pulls together, one [`repel`](#repel) names pushes apart, and every other pair follows the run's negatives. ```swift struct Relations ``` Mirrors `uform_relations_view_t`. Conforms to: `Sendable` ## attract Pairs pulled together. ```swift var attract: Edges? ``` ## repel Pairs pushed apart. ```swift var repel: Edges? ``` ## ignore Pairs skipped, like known false negatives; its [`weights`](#weights) are not read. ```swift var ignore: Edges? ``` ## init Takes the edge sets by name; a nil set names no pairs. ```swift init(attract: Edges? = nil, repel: Edges? = nil, ignore: Edges? = nil) ``` ## TrainerStepStats What a step reports, every loss its world mean. ```swift struct TrainerStepStats ``` Mirrors `uform_trainer_step_stats_t`. Conforms to: `Sendable` ## step The optimizer step the call applied, counted from one across resumes. ```swift var step: Int ``` ## learningRate The learning rate it applied. ```swift var learningRate: Float ``` ## loss The total loss the step minimized. ```swift var loss: Float ``` ## contrastiveLoss The contrastive term; zero when the objective is masked alone. ```swift var contrastiveLoss: Float ``` ## maskedTextLoss The text reconstruction term, before its weight; zero when absent. ```swift var maskedTextLoss: Float ``` ## maskedImageLoss The image reconstruction term, before its weight; zero when absent. ```swift var maskedImageLoss: Float ``` ## maskedVideoLoss The video reconstruction term, before its weight; zero when absent. ```swift var maskedVideoLoss: Float ``` ## maskedAudioLoss The audio reconstruction term, before its weight; zero when absent. ```swift var maskedAudioLoss: Float ``` ## lateInteractionLoss The late-interaction term, weighted; zero when absent. ```swift var lateInteractionLoss: Float ``` ## choicesLoss The choices term over the items that made a choice, before its weight; zero when absent. ```swift var choicesLoss: Float ``` ## truncated Items cut at a row's width. ```swift var truncated: Int ``` ## dropped Segments dropped whole because the batch had no room for them, since the last step that read its stats. ```swift var dropped: Int ``` ## TrainerStep What one step wrote: its report, and the pooled embeddings it scored when given an arena. ```swift struct TrainerStep ``` ## stats What the step reports. ```swift let stats: TrainerStepStats ``` ## embeddings One row per item, as the contrastive loss scored it before the update, or nil without an arena. ```swift let embeddings: Rows? ``` ## Trainer A training run: the tower replicas, gradients, optimizer moments, heads and temperatures on the devices of its executor, every verb blocking on that executor. ```swift final class Trainer ``` Conforms to: `Sendable` ## initVram Builds one replica per device of `executor` and starts the run from `seed`, or continues it from the `resume` checkpoint; every process of the run calls it with the same geometry, objective and seed. ```swift static func initVram( executor: Executor, depth: Int, hidden: Int, batch: BatchShape, objective: Objective = .joint, masking: Masking = Masking(), contrastive: Contrastive = Contrastive(), lateInteraction: LateInteraction = LateInteraction(), choices: Choices = Choices(), schedule: Schedule, mesh: Mesh? = nil, resume: String = "", seed: UInt64 = 0, progress: Progress? = nil, progressSteps: Int = 0 ) throws(UFormError) -> Trainer ``` Calls `uform_trainer_init_vram`. ## step Applies one optimizer step over this process's items and every peer's; every process of the run calls it once per step. ```swift func step( modalities: [Modality], contents: [[UInt8]], bags: [UInt32]? = nil, temperatureGroups: [UInt32]? = nil, sides: [LateInteractionSide]? = nil, relations: Relations = Relations(), choices: [[UInt8]]? = nil, choiceTargets: [[Float]]? = nil, arena: Arena? = nil, outputsScalar: Scalar = .f32 ) throws(UFormError) -> TrainerStep ``` ```swift func step( modalities: [Modality], contents: (data: [UInt8], offsets: [Int]), bags: [UInt32]? = nil, temperatureGroups: [UInt32]? = nil, sides: [LateInteractionSide]? = nil, relations: Relations = Relations(), choices: (data: [UInt8], offsets: [Int])? = nil, choiceTargets: (data: [Float], offsets: [Int])? = nil, arena: Arena? = nil, outputsScalar: Scalar = .f32 ) throws(UFormError) -> TrainerStep ``` Applies one optimizer step over tapes already packed as `data` and its `count + 1` Apache Arrow [`offsets`](#offsets), lent to the library in place, without a copy, until the step returns. With an `arena`, the step also lands the pooled embeddings it scored there, as [`outputsScalar`](https://www.unum.cloud/docs/uform/swift/encoding.md#outputsScalar). `sides` places each item in the late-interaction loss, nil leaving every item out, `choices` names each item's legal bytes as [`textChoose`](https://www.unum.cloud/docs/uform/swift/models.md#textChoose) takes them, and `choiceTargets` their target probabilities, an entry as long as the item's choices. Each list of items is gathered into one tape, the one copy this form makes before lending it as the packed form does. Calls `uform_trainer_step`. ## save Writes the weights and the step to a safetensors checkpoint, which [`Model.loadVram`](https://www.unum.cloud/docs/uform/swift/models.md#loadVram) and its siblings serve and [`initVram`](#initVram) resumes; only the process of index zero writes. ```swift func save(path: String, progress: Progress? = nil, progressSteps: Int = 0) throws(UFormError) ``` Calls `uform_trainer_save`.