Runtime Dispatch

§ Capabilities and Devices

Linking the library is the default, and the way to ship one binary across many CPU generations. Every query writes its answer through a pointer and returns an nk_status_t. CPU capabilities are reported along two independent axes, plus the mask to dispatch with:

• nk_cpu_capabilities_detected is what this CPU can execute, from CPUID or the OS, or where the OS cannot be asked, from what the compiler flags guarantee.
• nk_cpu_capabilities_compiled is what this binary contains, from the ISA probes at build time.
• nk_cpu_capabilities_enabled is both axes at once, the mask to pass every CPU call, and always holds nk_cap_serial_k.

Ask for enabled unless you specifically mean one of the raw axes. The two axes are independent, and conflating them fails quietly rather than loudly: a binary whose ISA probes failed still reports this machine’s full detected mask while containing no SIMD kernels at all. To dispatch over fewer capabilities, pass a narrower mask to the call itself; there is no process-wide setting to change.

nk_cpu_configure_thread prepares the calling thread for the capabilities it is given, and only those. Most capabilities need nothing: AMX on Linux costs one arch_prctl syscall, which grants tile state to the whole process, and fused BF16 dot products on Arm cost one FPCR write per thread.

nk_capability_t enabled = nk_cap_serial_k;
nk_cpu_capabilities_enabled(&enabled);
nk_cpu_configure_thread(enabled);
if (enabled & nk_cap_sapphireamx_k) { /* AMX both detected and compiled in */ }
nk_dot_bf16_best(a, b, n, &dot, enabled & ~nk_cap_sapphireamx_k, NULL); // dispatch without AMX

GPUs are more ISAs with a mask of their own, one device at a time. Each vendor has its own queries, and a device index is that runtime’s own ordinal: the one cudaSetDevice or hipSetDevice takes, or the position in Metal’s device list. nk_cuda_count_devices counts the devices, and nk_cuda_capabilities_detected, nk_cuda_capabilities_compiled, and nk_cuda_capabilities_enabled report one device’s capabilities the way their CPU twins do; nk_rocm_* and nk_metal_* are the same for the other vendors. Each vendor’s baseline, nk_cap_cuda_k, nk_cap_rocm_k, or nk_cap_metal_k, plays the role of nk_cap_serial_k. A GPU capability, like nk_dots_packed_bf16_ampere, takes its CPU twin’s arguments, queues on the trailing stream — a cudaStream_t, a hipStream_t, or an id<MTLCommandQueue>, with null for the default one — and returns without waiting. Only the capability queries take an ordinal; the stream names the device for everything else. nk_memory_allocate_unified_best and nk_memory_free_unified_best hand out and return memory the host and that device both address, and nk_stream_synchronize_best waits for the stream. Batched kernels that may need scratch, like the packed and symmetric families and attention, take a workspace, workspace_bytes pair before the capabilities and never allocate. Their *_workspace_size_* twin reports the bytes from the shapes alone, and a zero-byte workspace runs the scratch-free path where the capability has one.

nk_size_t devices = 0, workspace_bytes = 0;
nk_capability_t gpu = 0;
if (nk_cuda_count_devices(&devices) == nk_success_k && nk_cuda_capabilities_enabled(0, &gpu) == nk_success_k) {
    nk_dots_packed_workspace_size_f64_best(rows, columns, depth, gpu, &workspace_bytes);
    void *workspace = NULL;
    nk_memory_allocate_unified_best(workspace_bytes, &workspace, gpu, cuda_stream);
    nk_dots_packed_f64_best(a, b_packed, c, rows, columns, depth, a_stride, c_stride, workspace, workspace_bytes, gpu,
                            cuda_stream);
    nk_stream_synchronize_best(gpu, cuda_stream);
}

nk_capabilities_name spells any such mask as the names bindings accept, like “serial,neon,neonhalf”, into a buffer of NUMKONG_CAPABILITIES_NAME_CAPACITY bytes.

For exact register-level details, see capabilities.h. Capability kernels, like nk_dot_f32_haswell, stay callable directly if you want to pin a path for testing or benchmarking.