The forward and reverse substring scans mirror libc’s memchr/memrchr and memmem, but always take explicit lengths and report a pointer into the haystack, or NULL when there is no match.
Like every dispatch point, each takes the mask to pick a capability from and a stream, null on the CPU, and returns sz_success_k, or sz_missing_kernel_k when no capability in the mask has the kernel:
sz_status_t sz_find_byte_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, //
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);
sz_status_t sz_rfind_byte_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, //
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);
sz_status_t sz_find_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length,
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);
sz_status_t sz_rfind_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length,
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);Byte-set scans replace strspn/strcspn and take a prebuilt sz_byteset_t:
sz_status_t sz_find_byteset_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_byteset_t const *set, //
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);
sz_status_t sz_rfind_byteset_best(sz_cptr_t haystack, sz_size_t haystack_length, sz_byteset_t const *set, //
sz_cptr_t *match, sz_capability_t capabilities, sz_stream_t stream);Four header-only shortcuts build the set for you from a needle string, optionally inverting it:
sz_cptr_t sz_find_byte_from(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length);
sz_cptr_t sz_find_byte_not_from(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length);
sz_cptr_t sz_rfind_byte_from(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length);
sz_cptr_t sz_rfind_byte_not_from(sz_cptr_t haystack, sz_size_t haystack_length, sz_cptr_t needle, sz_size_t needle_length);Example: find a substring, then count occurrences by re-scanning the tail.
#include <stringzilla/stringzilla.h>
sz_size_t count_occurrences(sz_cptr_t text, sz_size_t length, sz_cptr_t needle, sz_size_t needle_length,
sz_capability_t capabilities) {
sz_size_t total = 0;
sz_cptr_t cursor = text, end = text + length, match = NULL;
while (sz_find_best(cursor, (sz_size_t)(end - cursor), needle, needle_length, &match, capabilities, NULL) ==
sz_success_k && match) {
++total;
cursor = match + needle_length; // disjoint matches; use `+1` for overlapping
}
return total;
}
int main(void) {
sz_capability_t capabilities = sz_cap_serial_k;
sz_capabilities_enabled_cpu(&capabilities);
sz_byteset_t whitespace;
sz_byteset_init(&whitespace);
char const *spaces = " \t\n\r\v\f";
for (char const *p = spaces; *p; ++p) sz_byteset_add(&whitespace, *p);
char const *line = " hello world";
sz_cptr_t first_space = NULL;
sz_find_byteset_best(line, 13, &whitespace, &first_space, capabilities, NULL);
(void)first_space;
return 0;
}sz_match_tOne reported match, locating it by haystack, by needle, and by byte span.struct sz_match_thaystack_index | sz_size_t | Which haystack of the collection this match was found in. |
|---|---|---|
needle_index | sz_size_t | Which needle of the vocabulary matched: a substring, a pattern or a path. |
byte_offset | sz_size_t | Where the match starts inside that haystack, in its own bytes. |
byte_length | sz_size_t | Haystack bytes the match spans, which folding can make differ from the needle's length. |
sz_subdocuments_format_tThe document grammar a subdocuments engine reads, and the path language that addresses it.enum sz_subdocuments_format_tsz_subdocuments_json_k | 0 | RFC 8259 documents, addressed by RFC 9535 JSONPath. |
|---|---|---|
sz_subdocuments_xml_k | 1 | XML 1.0 documents, addressed by XPath 1.0 along forward axes. |
sz_subdocuments_options_tHow a subdocuments engine reads what RFC 8259 leaves open, combined by bitwise or.enum sz_subdocuments_options_tsz_subdocuments_options_none_k | 0 | Every member a name selector names is entered. |
|---|---|---|
sz_subdocuments_duplicate_names_first_k | 1 | A name selector enters only the first member of an object with that name, so each field a path names is at most one node. |
sz_substrings_overlap_policy_tHow matches that share bytes resolve: reported in full, or thinned to a leftmost run.enum sz_substrings_overlap_policy_tsz_substrings_overlapping_k | 0 | Every match of every needle, including ones that share bytes and ones nested in others. |
|---|---|---|
sz_substrings_leftmost_longest_k | 1 | Matches sharing no bytes: earliest start, then longest span, then lower needle index. |
sz_substrings_leftmost_first_k | 2 | Matches sharing no bytes: earliest start, then lower needle index, whatever the lengths. |
sz_kernel_utf8_count_tSignature of the kernels behind sz_utf8_count_best.typedef sz_status_t(*) sz_kernel_utf8_count_t(sz_cptr_t, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_utf8_count_strings_tSignature of the kernels behind sz_utf8_count_strings_best.typedef sz_status_t(*) sz_kernel_utf8_count_strings_t(sz_strings_view_t const *, sz_size_t const *, sz_size_t, sz_size_t *, sz_size_t *, sz_stream_t);sz_kernel_find_byte_tSignature of the sz_find_byte_best and sz_rfind_byte_best kernels.typedef sz_status_t(*) sz_kernel_find_byte_t(sz_cptr_t, sz_size_t, sz_cptr_t, sz_cptr_t *, sz_stream_t);sz_kernel_find_tSignature of the sz_find_best and sz_rfind_best kernels.typedef sz_status_t(*) sz_kernel_find_t(sz_cptr_t, sz_size_t, sz_cptr_t, sz_size_t, sz_cptr_t *, sz_stream_t);sz_kernel_find_byteset_tSignature of the sz_find_byteset_best and sz_rfind_byteset_best kernels.typedef sz_status_t(*) sz_kernel_find_byteset_t(sz_cptr_t, sz_size_t, sz_byteset_t const *, sz_cptr_t *, sz_stream_t);sz_kernel_find_strings_tSignature of the sz_find_strings_best kernels.typedef sz_status_t(*) sz_kernel_find_strings_t(sz_strings_view_t const *, sz_size_t const *, sz_size_t, sz_cptr_t, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_substrings_engine_init_tSignature of every sz_substrings_engine_init kernel.typedef sz_status_t(*) sz_kernel_substrings_engine_init_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_case_sensitivity_t, sz_substrings_overlap_policy_t, sz_size_t, sz_size_t, sz_size_t, sz_allocator_t *, sz_stream_t);sz_kernel_substrings_contains_tSignature of every sz_substrings_contains kernel.typedef sz_status_t(*) sz_kernel_substrings_contains_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_u1x8_t *, sz_stream_t);sz_kernel_substrings_counts_tSignature of every sz_substrings_counts kernel.typedef sz_status_t(*) sz_kernel_substrings_counts_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_size_t *, sz_size_t, sz_stream_t);sz_kernel_substrings_find_tSignature of every sz_substrings_find kernel.typedef sz_status_t(*) sz_kernel_substrings_find_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_match_t *, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_substrings_replace_tSignature of every sz_substrings_replace kernel.typedef sz_status_t(*) sz_kernel_substrings_replace_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_strings_view_t const *, sz_ptr_t, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_substrings_bm25_scores_tSignature of every sz_substrings_bm25_scores kernel.typedef sz_status_t(*) sz_kernel_substrings_bm25_scores_t(struct sz_substrings_engine_t *, sz_strings_view_t const *, sz_f32_t const *, struct sz_substrings_bm25_t const *, sz_f32_t const *, sz_f32_t *, sz_size_t, sz_stream_t);sz_kernel_regex_engine_init_tSignature of every sz_regex_engine_init kernel.typedef sz_status_t(*) sz_kernel_regex_engine_init_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_case_sensitivity_t, sz_size_t *, sz_allocator_t *, sz_stream_t);sz_kernel_regex_contains_tSignature of every sz_regex_contains kernel.typedef sz_status_t(*) sz_kernel_regex_contains_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_u1x8_t *, sz_stream_t);sz_kernel_regex_counts_tSignature of every sz_regex_counts kernel.typedef sz_status_t(*) sz_kernel_regex_counts_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_size_t *, sz_size_t, sz_stream_t);sz_kernel_regex_find_tSignature of every sz_regex_find kernel.typedef sz_status_t(*) sz_kernel_regex_find_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_match_t *, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_regex_replace_tSignature of every sz_regex_replace kernel.typedef sz_status_t(*) sz_kernel_regex_replace_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_strings_view_t const *, sz_ptr_t, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_regex_extract_tSignature of every sz_regex_extract kernel.typedef sz_status_t(*) sz_kernel_regex_extract_t(struct sz_regex_engine_t *, sz_strings_view_t const *, sz_size_t, sz_match_t *, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_subdocuments_engine_init_tSignature of every sz_subdocuments_engine_init kernel.typedef sz_status_t(*) sz_kernel_subdocuments_engine_init_t(struct sz_subdocuments_engine_t *, sz_strings_view_t const *, sz_subdocuments_format_t, sz_subdocuments_options_t, sz_size_t *, sz_allocator_t *, sz_stream_t);sz_kernel_subdocuments_contains_tSignature of every sz_subdocuments_contains kernel.typedef sz_status_t(*) sz_kernel_subdocuments_contains_t(struct sz_subdocuments_engine_t *, sz_strings_view_t const *, sz_u1x8_t *, sz_size_t *, sz_stream_t);sz_kernel_subdocuments_counts_tSignature of every sz_subdocuments_counts kernel.typedef sz_status_t(*) sz_kernel_subdocuments_counts_t(struct sz_subdocuments_engine_t *, sz_strings_view_t const *, sz_size_t *, sz_size_t, sz_size_t *, sz_stream_t);sz_kernel_subdocuments_find_tSignature of every sz_subdocuments_find kernel.typedef sz_status_t(*) sz_kernel_subdocuments_find_t(struct sz_subdocuments_engine_t *, sz_strings_view_t const *, sz_match_t *, sz_size_t, sz_size_t *, sz_size_t *, sz_stream_t);sz_subdocuments_slice_tThe array positions one selector picks, RFC 9535's [begin:end:step], an index n being [n:n+1:1], or [-1::1] for the last element; negative bounds count from the array's end.struct sz_subdocuments_slice_tbegin | sz_i64_t | The first position, or 2⁵³ past the end that a negative step walks from when omitted. |
|---|---|---|
end | sz_i64_t | The position the slice stops before, or 2⁵³ past the end its step walks to when omitted. |
step | sz_i64_t | Distance between selected positions, negative walking backward; zero selects none. |
sz_find_byte_bestLocates the first byte equal to the one at needle in haystack, like memchr.sz_status_t sz_find_byte_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_cptr_t needle,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_find_byteserialwestmerehaswellskylakeneonsvervvv128v128relaxedloongsonasxpowervsxhaystack | in | Haystack - the string to search in. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
needle | in | Needle - single-byte substring to find. |
match | out | Address of the first match, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_rfind_byte_bestLocates the last byte equal to the one at needle in haystack, like memrchr.sz_status_t sz_rfind_byte_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_cptr_t needle,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_rfind_byteserialwestmerehaswellskylakeneonsvervvv128v128relaxedloongsonasxpowervsxThere is no AArch64 reference implementation to link here.
haystack | in | Haystack - the string to search in. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
needle | in | Needle - single-byte substring to find. |
match | out | Address of the last match, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
x86_64 implementation in glibc: https://github.com/lattera/glibc/blob/master/sysdeps/x86_64/memrchr.S
sz_missing_kernel_k No capability in capabilities has it.sz_find_bestLocates the first occurrence of needle in haystack, like memmem in LibC, or like strstr for strings of known length.sz_status_t sz_find_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_cptr_t needle,
sz_size_t needle_length,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_findserialwestmerehaswellskylakeneonsvervvv128v128relaxedloongsonasxpowervsxhaystack | in | Haystack - the string to search in. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
needle | in | Needle - substring to find. |
needle_length | in | Number of bytes in the needle. |
match | out | Address of the first match, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_rfind_bestLocates the last matching substring.sz_status_t sz_rfind_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_cptr_t needle,
sz_size_t needle_length,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_rfindserialwestmerehaswellskylakeneonsvervvv128v128relaxedloongsonasxpowervsxhaystack | in | Haystack - the string to search in. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
needle | in | Needle - substring to find. |
needle_length | in | Number of bytes in the needle. |
match | out | Address of the last match, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_find_byteset_bestFinds the first character present from the set, present in haystack.sz_status_t sz_find_byteset_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_byteset_t const *set,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_find_bytesetserialhaswellicelakeneonsve2rvvv128v128relaxedloongsonasxpowervsxEquivalent to strspn(haystack, accepted) and strcspn(haystack, rejected) in LibC. May have identical implementation and performance to sz_rfind_byteset_best. Useful for parsing, when we want to skip a set of characters, such as:
haystack | in | Haystack - the string to scan. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
set | in | Set of relevant characters. |
match | out | Address of the first matching character from set, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_rfind_byteset_bestFinds the last character present from the set, present in haystack.sz_status_t sz_rfind_byteset_best(
sz_cptr_t haystack,
sz_size_t haystack_length,
sz_byteset_t const *set,
sz_cptr_t *match,
sz_capability_t capabilities,
sz_stream_t stream);sz_rfind_bytesetserialhaswellicelakeneonsve2rvvv128v128relaxedloongsonasxpowervsxEquivalent to strspn(haystack, accepted) and strcspn(haystack, rejected) in LibC. May have identical implementation and performance to sz_find_byteset_best. Useful for parsing, when we want to skip a set of characters, such as:
haystack | in | Haystack - the string to scan. |
|---|---|---|
haystack_length | in | Number of bytes in the haystack. |
set | in | Set of relevant characters. |
match | out | Address of the last matching character from set, or NULL if not found. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_find_strings_bestFinds needle in every one of strings[order[i]], or of the first count strings, like a LIKE 'needle' filter over a column.sz_status_t sz_find_strings_best(
sz_strings_view_t const *strings,
sz_size_t const *order,
sz_size_t count,
sz_cptr_t needle,
sz_size_t needle_length,
sz_size_t *positions,
sz_capability_t capabilities,
sz_stream_t stream);sz_find_stringsserialhaswellskylakeneonA tape read in its own order is searched as one run of bytes, 64 starts at a time, so a string costs no call of its own, and a match crossing into the next string counts for neither. Other layouts and orders search string by string.
strings | in | The strings to search, in any layout. |
|---|---|---|
order | in | Indices into strings, possibly repeated, or null for the first count. |
count | in | Number of strings to search. |
needle | in | The substring to find. |
needle_length | in | Number of bytes in the needle, zero matching at every start. |
positions | out | The offset of each first match within its string, or STRINGZILLA_SIZE_MAX where a string holds none, count of them. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_find_find_kernelFinds the search kernel of kind, from the best of capabilities.sz_status_t sz_find_find_kernel(
sz_kernel_kind_t kind,
sz_capability_t capabilities,
sz_kernel_punned_t *kernel,
sz_capability_t *capability);kind | in | The verb whose kernel to find. |
|---|---|---|
capabilities | in | The mask to pick the best capability from. |
kernel | out | The kernel, or null when none of capabilities has it. |
capability | out | The capability the kernel belongs to, or zero. |
sz_missing_kernel_ksz_missing_library_k Header-only.sz_substrings_engine_initCompiles needles for the best capability of capabilities that has an init kernel.sz_status_t sz_substrings_engine_init(
sz_substrings_engine_t *engine,
sz_strings_view_t const *needles,
sz_case_sensitivity_t case_sensitivity,
sz_substrings_overlap_policy_t overlap_policy,
sz_size_t hot_states,
sz_size_t matches_budget,
sz_size_t haystacks_budget,
sz_capability_t capabilities,
sz_allocator_t *allocator,
sz_stream_t stream);sz_substrings_engine_initserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalengine | out | Left untouched unless the call succeeds. |
|---|---|---|
needles | in | The vocabulary; an empty needle is refused rather than skipped, as it would match at every position and dropping it would shift every later needle's reported index. Read on the host. |
case_sensitivity | in | Whether both sides are folded before they meet, or compared as is. |
overlap_policy | in | The cover every round runs, as the arena is sized for that one alone. |
hot_states | in | States to keep in the dense hot rows, or STRINGZILLA_SUBSTRINGS_HOT_STATES_AUTO to fill a fixed byte budget, which holds more states the fewer classes the vocabulary spells. |
matches_budget | in | Matches one round may emit, read by a device tier only; a host tier walks straight into the caller's output and ignores it. Zero asks for a tier-chosen default. |
haystacks_budget | in | Haystacks one round may carry, read by a device tier only, zero asking for its default. The arena of counts, find and replace is sized for it here, so they allocate nothing; sz_substrings_bm25_scores grows it for its tables on its first call only. A round carrying more is refused with sz_unexpected_dimensions_k. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu or sz_capabilities_enabled_cuda report; its group picks the CPU or a GPU vendor. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_best of capabilities. Stored by value, so sz_substrings_engine_free needs none and cannot get the wrong one; on a device it must be host-writable, as the host writes the automaton there. |
stream | in | Null on the CPU. On a GPU, the stream to queue on, also naming the device: a cudaStream_t, a hipStream_t, or an id<MTLCommandQueue>; null for the default. |
May join stream; no compute verb ever does.
sz_missing_kernel_k No capability of the mask has an init.sz_overflow_risk_k The vocabulary exceeds a 32-bit state id.sz_invalid_utf8_k A malformed needle under sz_uncased_k.sz_unexpected_dimensions_k The vocabulary or one of its needles was empty.sz_device_memory_mismatch_k On a device, a block or a stream it cannot use.sz_*_engine_free× 3Returns both of the engine's blocks to the allocator that built them, once the work queued on stream is done with them, emptying engine.void sz_substrings_engine_free(
sz_substrings_engine_t *engine,
sz_stream_t stream);| Symbol | engine |
|---|---|
sz_substrings_engine_free | sz_substrings_engine_t |
sz_regex_engine_free | sz_regex_engine_t |
sz_subdocuments_engine_free | sz_subdocuments_engine_t |
sz_substrings_containsMarks which haystacks hold at least one match of any needle, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_substrings_contains(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_u1x8_t *contained,
sz_stream_t stream);sz_substrings_containsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmengine | in | A compiled vocabulary; a match under any overlap policy is a match. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
contained | out | One bit per haystack, eight to a byte, the lowest bit first: set when that haystack matches, with the bits past the last haystack written as zero. |
stream | in | As sz_substrings_counts takes it. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an argument or a stream that it has no way to use.sz_substrings_countsCounts the matches of every needle in every haystack, one count per haystack.sz_status_t sz_substrings_counts(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t *counts,
sz_size_t counts_stride,
sz_stream_t stream);sz_substrings_countsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalEvery verb runs the kernel of the capability engine was compiled for. A device one enqueues on stream and returns; every round of one engine runs out of its one arena and writes its one report, so the caller orders them - one stream, or events between two - and reads the report only after joining the round that wrote it.
engine | in | A compiled vocabulary, whose overlap_policy decides whether matches sharing bytes are all counted or thinned to a leftmost run. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. Any layout on the CPU, and on a device a tape sz_strings_to_tape_best wrote into a unified allocator, as every verb takes its haystacks and replacements. |
counts | out | The per-haystack counts, haystack h at counts[h * stride]. |
counts_stride | in | Entries from one haystack's count to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | Null on the CPU. On a GPU, the stream to queue on, which also names the device the round runs on; null for the default. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an argument or a stream it cannot use.sz_substrings_findReports every match of every needle in every haystack.sz_status_t sz_substrings_find(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_stream_t stream);sz_substrings_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalA capacity too small is not an error: the report's matches_emitted names the true total and shortfall names what did not fit, so a sizing call and a filling call need no walk between.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches, ascending by haystack; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds. |
matches_offsets | out | One boundary per haystack into matches plus one, the last being the total; filled whether or not the matches fit, which is what sizes the next call. |
stream | in | As sz_substrings_counts takes it. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an unaddressable argument.sz_substrings_replaceRewrites every haystack, substituting one replacement per needle, into one target.sz_status_t sz_substrings_replace(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_strings_view_t const *replacements,
sz_ptr_t target,
sz_size_t target_capacity,
sz_size_t *offsets,
sz_stream_t stream);sz_substrings_replaceserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalA target too small is not an error: the report's target_length names the bytes the rewrite needs and shortfall names what did not fit, leaving the contents of target unspecified.
engine | in | A vocabulary compiled under a leftmost policy; sz_substrings_overlapping_k is refused, since a substitution over matches that share bytes is not a function. |
|---|---|---|
haystacks | in | The texts to rewrite; their bytes are read in place and never copied. |
replacements | in | One replacement per needle, indexed by needle; an empty one deletes. |
target | out | Room for target_capacity bytes, or STRINGZILLA_NULL with zero capacity to size only. |
target_capacity | in | Bytes target holds. |
offsets | out | One rewritten boundary per haystack plus one, the last being the total; filled whether or not the target held the result, which is what sizes the next call. |
stream | in | As sz_substrings_counts takes it. |
sz_unexpected_dimensions_k replacements lacks one entry per needle.sz_status_unknown_k The policy leaves no cover.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k An argument no device kernel can address.sz_substrings_bm25_scoresScores every haystack against the vocabulary as one BM25 query, one score per haystack.sz_status_t sz_substrings_bm25_scores(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_f32_t const *document_lengths,
sz_substrings_bm25_t const *parameters,
sz_f32_t const *needle_weights,
sz_f32_t *scores,
sz_size_t scores_stride,
sz_stream_t stream);sz_substrings_bm25_scoresserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalThe vocabulary is the query: needle_weights holds each needle's IDF or boost. Term frequencies are raw overlapping counts, since a leftmost cover would suppress genuine occurrences of a needle nested in another, so the engine's own policy does not apply here. A CPU backend sums in ascending needle order and a CUDA backend in fixed-point integers, so each is bit-stable across its own runs, and the two agree.
The arguments are invalid when needle_weights is STRINGZILLA_NULL, scores_stride is zero, or length_normalization is positive while average_document_length is not.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The documents to score; their bytes are read in place and never copied. |
document_lengths | in | One length per haystack to normalize by, in the unit of average_document_length, or STRINGZILLA_NULL to use each haystack's byte length. |
parameters | in | BM25's continuous parameters. |
needle_weights | in | One weight per needle of the engine. |
scores | out | The per-haystack scores, haystack h at scores[h * stride]. |
scores_stride | in | Entries from one haystack's score to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | As sz_substrings_counts takes it. |
sz_unexpected_dimensions_k Invalid arguments, as above.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k An argument or grown arena a device cannot address.sz_substrings_engine_init_serialCompiles needles for the serial kernels.sz_status_t sz_substrings_engine_init_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *needles,
sz_case_sensitivity_t case_sensitivity,
sz_substrings_overlap_policy_t overlap_policy,
sz_size_t hot_states,
sz_size_t matches_budget,
sz_size_t haystacks_budget,
sz_allocator_t *allocator,
sz_stream_t stream);sz_substrings_engine_initserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalengine | out | Left untouched unless the call succeeds. |
|---|---|---|
needles | in | The vocabulary; an empty needle is refused rather than skipped, as it would match at every position and dropping it would shift every later needle's reported index. Read on the host. |
case_sensitivity | in | Whether both sides are folded before they meet, or compared as is. |
overlap_policy | in | The cover every round runs, as the arena is sized for that one alone. |
hot_states | in | States to keep in the dense hot rows, or STRINGZILLA_SUBSTRINGS_HOT_STATES_AUTO to fill a fixed byte budget, which holds more states the fewer classes the vocabulary spells. |
matches_budget | in | Matches one round may emit, read by a device tier only; the serial walk goes straight into the caller's output and ignores it. |
haystacks_budget | in | Haystacks one round may carry, read by a device tier only; the serial walk ignores it. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_serial. Stored by value, so sz_substrings_engine_free needs none and cannot get the wrong one. |
stream | in | Null on the CPU. |
sz_overflow_risk_k The vocabulary exceeds a 32-bit state id.sz_invalid_utf8_k A malformed needle under sz_uncased_k.sz_unexpected_dimensions_k The vocabulary or one of its needles was empty.sz_substrings_contains_serialMarks which haystacks hold at least one match of any needle, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_substrings_contains_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_u1x8_t *contained,
sz_stream_t stream);sz_substrings_containsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmengine | in | A compiled vocabulary; a match under any overlap policy is a match. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
contained | out | One bit per haystack, eight to a byte, the lowest bit first: set when that haystack matches, with the bits past the last haystack written as zero. |
stream | in | As sz_substrings_counts takes it. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an argument or a stream that it has no way to use.sz_substrings_counts_serialCounts the matches of every needle in every haystack, one count per haystack.sz_status_t sz_substrings_counts_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t *counts,
sz_size_t counts_stride,
sz_stream_t stream);sz_substrings_countsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalEvery verb runs the kernel of the capability engine was compiled for. A device one enqueues on stream and returns; every round of one engine runs out of its one arena and writes its one report, so the caller orders them - one stream, or events between two - and reads the report only after joining the round that wrote it.
engine | in | A compiled vocabulary, whose overlap_policy decides whether matches sharing bytes are all counted or thinned to a leftmost run. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. Any layout on the CPU, and on a device a tape sz_strings_to_tape_best wrote into a unified allocator, as every verb takes its haystacks and replacements. |
counts | out | The per-haystack counts, haystack h at counts[h * stride]. |
counts_stride | in | Entries from one haystack's count to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | Null on the CPU. On a GPU, the stream to queue on, which also names the device the round runs on; null for the default. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an argument or a stream it cannot use.sz_substrings_find_serialReports every match of every needle in every haystack.sz_status_t sz_substrings_find_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_stream_t stream);sz_substrings_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalA capacity too small is not an error: the report's matches_emitted names the true total and shortfall names what did not fit, so a sizing call and a filling call need no walk between.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches, ascending by haystack; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds. |
matches_offsets | out | One boundary per haystack into matches plus one, the last being the total; filled whether or not the matches fit, which is what sizes the next call. |
stream | in | As sz_substrings_counts takes it. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A device got an unaddressable argument.sz_substrings_replace_serialRewrites every haystack, substituting one replacement per needle, into one target.sz_status_t sz_substrings_replace_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_strings_view_t const *replacements,
sz_ptr_t target,
sz_size_t target_capacity,
sz_size_t *offsets,
sz_stream_t stream);sz_substrings_replaceserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalA target too small is not an error: the report's target_length names the bytes the rewrite needs and shortfall names what did not fit, leaving the contents of target unspecified.
engine | in | A vocabulary compiled under a leftmost policy; sz_substrings_overlapping_k is refused, since a substitution over matches that share bytes is not a function. |
|---|---|---|
haystacks | in | The texts to rewrite; their bytes are read in place and never copied. |
replacements | in | One replacement per needle, indexed by needle; an empty one deletes. |
target | out | Room for target_capacity bytes, or STRINGZILLA_NULL with zero capacity to size only. |
target_capacity | in | Bytes target holds. |
offsets | out | One rewritten boundary per haystack plus one, the last being the total; filled whether or not the target held the result, which is what sizes the next call. |
stream | in | As sz_substrings_counts takes it. |
sz_unexpected_dimensions_k replacements lacks one entry per needle.sz_status_unknown_k The policy leaves no cover.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k An argument no device kernel can address.sz_substrings_bm25_scores_serialScores every haystack against the vocabulary as one BM25 query, one score per haystack.sz_status_t sz_substrings_bm25_scores_serial(
sz_substrings_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_f32_t const *document_lengths,
sz_substrings_bm25_t const *parameters,
sz_f32_t const *needle_weights,
sz_f32_t *scores,
sz_size_t scores_stride,
sz_stream_t stream);sz_substrings_bm25_scoresserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalThe vocabulary is the query: needle_weights holds each needle's IDF or boost. Term frequencies are raw overlapping counts, since a leftmost cover would suppress genuine occurrences of a needle nested in another, so the engine's own policy does not apply here. A CPU backend sums in ascending needle order and a CUDA backend in fixed-point integers, so each is bit-stable across its own runs, and the two agree.
The arguments are invalid when needle_weights is STRINGZILLA_NULL, scores_stride is zero, or length_normalization is positive while average_document_length is not.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The documents to score; their bytes are read in place and never copied. |
document_lengths | in | One length per haystack to normalize by, in the unit of average_document_length, or STRINGZILLA_NULL to use each haystack's byte length. |
parameters | in | BM25's continuous parameters. |
needle_weights | in | One weight per needle of the engine. |
scores | out | The per-haystack scores, haystack h at scores[h * stride]. |
scores_stride | in | Entries from one haystack's score to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | As sz_substrings_counts takes it. |
sz_unexpected_dimensions_k Invalid arguments, as above.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k An argument or grown arena a device cannot address.sz_*_find_kernel× 3Finds the substrings kernel of kind, from the best of capabilities.sz_status_t sz_substrings_find_kernel(
sz_kernel_kind_t kind,
sz_capability_t capabilities,
sz_kernel_punned_t *kernel,
sz_capability_t *capability);| Symbol | Backends |
|---|---|
sz_substrings_find_kernel | 13 |
sz_regex_find_kernel | 7 |
sz_subdocuments_find_kernel | 12 |
sz_substrings_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudahopperrocmmetalsz_regex_findserialhaswellicelakeneoncudarocmmetalsz_subdocuments_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalkind | in | The verb whose kernel to find. |
|---|---|---|
capabilities | in | The mask to pick the best capability from. |
kernel | out | The kernel, or null when none of capabilities has it. |
capability | out | The capability the kernel belongs to, or zero. |
sz_missing_kernel_ksz_missing_library_k Header-only.sz_regex_engine_initCompiles patterns for the best capability of capabilities that has an init kernel.sz_status_t sz_regex_engine_init(
sz_regex_engine_t *engine,
sz_strings_view_t const *patterns,
sz_case_sensitivity_t case_sensitivity,
sz_size_t *malformed_index,
sz_capability_t capabilities,
sz_allocator_t *allocator,
sz_stream_t stream);sz_regex_engine_initserialhaswellicelakeneoncudarocmmetalengine | out | Left untouched unless the call succeeds. |
|---|---|---|
patterns | in | The vocabulary, read on the host; a pattern's index is the needle_index its matches report, and the lower index wins between patterns matching at one position. |
case_sensitivity | in | Whether patterns match bytes as written, or both sides are folded first, as the section on case folding describes. |
malformed_index | out | Where the index of the first refused pattern is written, only when the call returns sz_invalid_syntax_k; may be STRINGZILLA_NULL. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_best of capabilities. Stored by value, so sz_regex_engine_free needs none and cannot get the wrong one. Scratch grows to the widest call seen, so a warm engine allocates nothing. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_invalid_syntax_k An empty vocabulary or a pattern outside the syntax above.sz_missing_kernel_k No capability of the mask has an init.sz_overflow_risk_k An NFA exceeds a 32-bit state id.sz_device_memory_mismatch_k A stream or allocator its capability cannot use.sz_regex_containsMarks which haystacks hold at least one match of any pattern, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_regex_contains(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_u1x8_t *contained,
sz_stream_t stream);sz_regex_containsserialhaswellicelakeneoncudarocmDevice rounds walk every byte at once, as the counts' do, and keep only which haystacks matched.
engine | in | A compiled vocabulary, whose round scratch grows to the longest haystack. |
|---|---|---|
haystacks | in | The texts to search, in any layout; their bytes are read in place. |
contained | out | One bit per haystack, eight to a byte, the lowest bit first: set when that haystack matches, with the bits past the last haystack written as zero. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_countsCounts the vocabulary's leftmost-first matches in every haystack, one count per haystack.sz_status_t sz_regex_counts(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t *counts,
sz_size_t counts_stride,
sz_stream_t stream);sz_regex_countsserialhaswellicelakeneoncudarocmmetalengine | in | A compiled vocabulary, whose round scratch grows to the longest haystack. |
|---|---|---|
haystacks | in | The texts to search, in any layout; their bytes are read in place. Malformed UTF-8 is searched as is: no class or literal matches a malformed byte. |
counts | out | The per-haystack counts, haystack h at counts[h * stride]. |
counts_stride | in | Entries from one haystack's count to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_findReports every leftmost-first match of the vocabulary in every haystack.sz_status_t sz_regex_find(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_stream_t stream);sz_regex_findserialhaswellicelakeneoncudarocmmetalWithin a haystack, matches share no bytes and come in ascending order of start; an empty match is never reported where the previous match ended, as Rust's find_iter does, nor before a UTF-8 continuation byte, so none splits a codepoint.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches, ascending by haystack, then by start; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds; only the first that many are written. |
matches_offsets | out | One boundary per haystack into the full list of matches plus one, the last being the total; true whether or not the matches fit, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_replaceRewrites every haystack, substituting each match with its pattern's replacement, into one target.sz_status_t sz_regex_replace(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_strings_view_t const *replacements,
sz_ptr_t target,
sz_size_t target_capacity,
sz_size_t *offsets,
sz_stream_t stream);sz_regex_replaceserialhaswellicelakeneoncudarocmmetalA replacement is copied as written, except that $ stands for one dollar and $N, ${N} and ${name} for the text of that group of the match's pattern, $0 being the whole match; a group that took no part in the match or does not exist is empty, and a dollar starting no reference stands for itself, so $1a is group one followed by a. Groups are read only for the matches whose replacement names them, by a PikeVM over each such match's own span.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to rewrite; their bytes are read in place and never copied. |
replacements | in | One replacement per pattern, indexed by pattern; an empty one deletes. |
target | out | Room for target_capacity bytes, or STRINGZILLA_NULL with zero capacity to size only. |
target_capacity | in | Bytes target holds; only the first that many are written. |
offsets | out | One rewritten boundary per haystack plus one, the last being the total; true whether or not the target held the result, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k replacements lacks one entry per pattern.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_extractLocates the groups of each haystack's first match of pattern_index, as Pandas str.extract reads them.sz_status_t sz_regex_extract(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t pattern_index,
sz_match_t *groups,
sz_size_t groups_capacity,
sz_size_t *groups_offsets,
sz_stream_t stream);sz_regex_extractserialhaswellicelakeneoncudarocmmetalA haystack with such a match gets one row for the whole match, needle_index zero, then one row per group that took part, needle_index naming the group; a haystack without one gets no rows, so a group that took no part is told apart from an empty one. The first match of a pattern is the first the vocabulary reports with that needle_index.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
pattern_index | in | The pattern whose groups to locate. |
groups | out | Room for groups_capacity rows, or STRINGZILLA_NULL with zero capacity to size only. |
groups_capacity | in | Rows groups holds; only the first that many are written. |
groups_offsets | out | One boundary per haystack plus one, the last being the total; true whether or not groups held them, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k pattern_index is past the vocabulary.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_engine_init_serialCompiles patterns for the serial kernels.sz_status_t sz_regex_engine_init_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *patterns,
sz_case_sensitivity_t case_sensitivity,
sz_size_t *malformed_index,
sz_allocator_t *allocator,
sz_stream_t stream);sz_regex_engine_initserialhaswellicelakeneoncudarocmmetalengine | out | Left untouched unless the call succeeds. |
|---|---|---|
patterns | in | The vocabulary, read on the host; a pattern's index is the needle_index its matches report, and the lower index wins between patterns matching at one position. |
case_sensitivity | in | Whether patterns match bytes as written, or both sides are folded first, as the section on case folding describes. |
malformed_index | out | Where the index of the first refused pattern is written, only when the call returns sz_invalid_syntax_k; may be STRINGZILLA_NULL. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_serial. Stored by value, so sz_regex_engine_free needs none and cannot get the wrong one. Scratch grows to the widest call seen, so a warm engine allocates nothing. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k An empty vocabulary or a pattern outside the syntax above.sz_overflow_risk_k An NFA exceeds a 32-bit state id.sz_regex_contains_serialMarks which haystacks hold at least one match of any pattern, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_regex_contains_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_u1x8_t *contained,
sz_stream_t stream);sz_regex_containsserialhaswellicelakeneoncudarocmDevice rounds walk every byte at once, as the counts' do, and keep only which haystacks matched.
engine | in | A compiled vocabulary, whose round scratch grows to the longest haystack. |
|---|---|---|
haystacks | in | The texts to search, in any layout; their bytes are read in place. |
contained | out | One bit per haystack, eight to a byte, the lowest bit first: set when that haystack matches, with the bits past the last haystack written as zero. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_counts_serialCounts the vocabulary's leftmost-first matches in every haystack, one count per haystack.sz_status_t sz_regex_counts_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t *counts,
sz_size_t counts_stride,
sz_stream_t stream);sz_regex_countsserialhaswellicelakeneoncudarocmmetalengine | in | A compiled vocabulary, whose round scratch grows to the longest haystack. |
|---|---|---|
haystacks | in | The texts to search, in any layout; their bytes are read in place. Malformed UTF-8 is searched as is: no class or literal matches a malformed byte. |
counts | out | The per-haystack counts, haystack h at counts[h * stride]. |
counts_stride | in | Entries from one haystack's count to the next, at least one, so a strided call writes one column of a [haystacks, vocabularies] feature matrix. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_find_serialReports every leftmost-first match of the vocabulary in every haystack.sz_status_t sz_regex_find_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_stream_t stream);sz_regex_findserialhaswellicelakeneoncudarocmmetalWithin a haystack, matches share no bytes and come in ascending order of start; an empty match is never reported where the previous match ended, as Rust's find_iter does, nor before a UTF-8 continuation byte, so none splits a codepoint.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches, ascending by haystack, then by start; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds; only the first that many are written. |
matches_offsets | out | One boundary per haystack into the full list of matches plus one, the last being the total; true whether or not the matches fit, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_replace_serialRewrites every haystack, substituting each match with its pattern's replacement, into one target.sz_status_t sz_regex_replace_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_strings_view_t const *replacements,
sz_ptr_t target,
sz_size_t target_capacity,
sz_size_t *offsets,
sz_stream_t stream);sz_regex_replaceserialhaswellicelakeneoncudarocmmetalA replacement is copied as written, except that $ stands for one dollar and $N, ${N} and ${name} for the text of that group of the match's pattern, $0 being the whole match; a group that took no part in the match or does not exist is empty, and a dollar starting no reference stands for itself, so $1a is group one followed by a. Groups are read only for the matches whose replacement names them, by a PikeVM over each such match's own span.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to rewrite; their bytes are read in place and never copied. |
replacements | in | One replacement per pattern, indexed by pattern; an empty one deletes. |
target | out | Room for target_capacity bytes, or STRINGZILLA_NULL with zero capacity to size only. |
target_capacity | in | Bytes target holds; only the first that many are written. |
offsets | out | One rewritten boundary per haystack plus one, the last being the total; true whether or not the target held the result, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k replacements lacks one entry per pattern.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_regex_extract_serialLocates the groups of each haystack's first match of pattern_index, as Pandas str.extract reads them.sz_status_t sz_regex_extract_serial(
sz_regex_engine_t *engine,
sz_strings_view_t const *haystacks,
sz_size_t pattern_index,
sz_match_t *groups,
sz_size_t groups_capacity,
sz_size_t *groups_offsets,
sz_stream_t stream);sz_regex_extractserialhaswellicelakeneoncudarocmmetalA haystack with such a match gets one row for the whole match, needle_index zero, then one row per group that took part, needle_index naming the group; a haystack without one gets no rows, so a group that took no part is told apart from an empty one. The first match of a pattern is the first the vocabulary reports with that needle_index.
engine | in | A compiled vocabulary. |
|---|---|---|
haystacks | in | The texts to search; their bytes are read in place and never copied. |
pattern_index | in | The pattern whose groups to locate. |
groups | out | Room for groups_capacity rows, or STRINGZILLA_NULL with zero capacity to size only. |
groups_capacity | in | Rows groups holds; only the first that many are written. |
groups_offsets | out | One boundary per haystack plus one, the last being the total; true whether or not groups held them, which sizes the next call. |
stream | in | Null on the CPU, or the device stream the call is enqueued on. |
sz_unexpected_dimensions_k pattern_index is past the vocabulary.sz_missing_kernel_k An empty engine.sz_device_memory_mismatch_k A stream the engine's capability cannot use.sz_subdocuments_engine_initCompiles paths for the best capability of capabilities that has an init kernel.sz_status_t sz_subdocuments_engine_init(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *paths,
sz_subdocuments_format_t format,
sz_subdocuments_options_t options,
sz_size_t *malformed_index,
sz_capability_t capabilities,
sz_allocator_t *allocator,
sz_stream_t stream);sz_subdocuments_engine_initserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalJSON paths follow RFC 9535: the root $, child segments .name and .*, and bracketed unions of names 'name' or "name", wildcards *, indices n, slices begin:end:step and filters ?expression, the same after a descendant .., with blank space where the RFC allows it. Negative indices and bounds count from the array's end, and a negative step walks backward, though matches still come in document order. Filters hold comparisons, existence tests of relative queries, &&, ||, ! and parentheses, and the functions length(), count(), value(), match() and search(), whose I-Regexp patterns compile into one regex engine owned by this one. Each path compiles into one step per segment, numbered path after path, each filter's queries taking steps of their own. A path may also be a bare filter ?expression, which tests the document's root as $[?expression] tests an array's elements.
XML paths follow XPath 1.0 along forward axes: absolute paths of / and // steps, each a name test, *, @name, @* or text() with an optional position [n], which counts the siblings passing the same test. Name tests compare QNames as the document spells them, prefix included, and namespace declarations are no attributes. No blank space is allowed.
What no walk evaluates is an absolute query inside a filter, a comparison with no literal, length() or count() on either side, a pattern other than a string literal or one the regex family refuses, and in XML a reverse axis or a predicate other than a position.
engine | out | Left untouched unless the call succeeds. |
|---|---|---|
paths | in | The vocabulary, one path per string, read on the host. |
format | in | The grammar of the documents every verb of this engine reads. |
options | in | How JSON walks read what RFC 8259 leaves open, or sz_subdocuments_options_none_k for the default. |
malformed_index | out | The first path to break the grammar or leave the subset, on failure. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_best of capabilities. Stored by value, so sz_subdocuments_engine_free needs none and cannot get the wrong one. Scratch grows to the widest call seen, so a warm engine allocates nothing. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k Anything outside the grammar, and what no walk evaluates, as above.sz_missing_library_k match() or search() in a header-only build, whose regex dispatch links no library.sz_missing_kernel_k No capability of the mask has an init, a format no kernel reads yet, or on a GPU non-zero options or a bare filter, which malformed_index then names.sz_unexpected_dimensions_k An empty vocabulary.sz_overflow_risk_k Past 32-bit step or name offsets.sz_subdocuments_containsMarks which documents hold at least one node some path selects, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_subdocuments_contains(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_u1x8_t *contained,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_containsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmA walk validates nothing past the match it stops at, so a document malformed only after its first match is marked rather than refused. Device walks run whole, as the counts' do, so on a device that document fails the call.
engine | in | Compiled paths, serving one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
contained | out | One bit per document, eight to a byte, the lowest bit first: set when that document matches, with the bits past the last document written as zero. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k A malformed document, as sz_subdocuments_counts defines it, leaving contained unspecified.sz_missing_kernel_k An empty engine.sz_subdocuments_countsCounts the nodes of every document that some path selects, one count per document.sz_status_t sz_subdocuments_counts(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_size_t *counts,
sz_size_t counts_stride,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_countsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalA node several paths select counts once per path, as sz_subdocuments_find reports it.
A document is malformed with unbalanced brackets or tags, an unterminated string, attribute value, comment or CDATA section, a / outside the strings of JSON, broken structure where the walk reads it, or an XML DOCTYPE with an internal subset, and leaves the counts unspecified. A container or element no live step can enter is skipped by counting its nesting, so nothing else it holds is validated.
engine | in | Compiled paths; the walk grows the engine's depth stack, so one engine serves one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
counts | out | The per-document counts, document d at counts[d * stride]. |
counts_stride | in | Entries from one document's count to the next, at least one. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_invalid_syntax_k A malformed document.sz_subdocuments_findReports every node of every document that some path selects.sz_status_t sz_subdocuments_find(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalMatches come by document, then by start offset, then by path, so a container comes before the nodes inside it, and a node is reported once per path that selects it, however many of the path's selectors or descendant routes reach it. Each match spans the node's exact bytes: in JSON a string with its quotes, a number, true, false or null without the blank space around it, or an object or array through its closing bracket; in XML an element through its end tag, an attribute's value without its quotes, or a text node, whose CDATA sections stay as written.
engine | in | Compiled paths, serving one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds. |
matches_offsets | out | One boundary per document into the matches plus one, the last being the total; filled whether or not the matches fit, which is what sizes the next call. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k A malformed document, as sz_subdocuments_counts defines it, leaving the outputs unspecified.sz_missing_kernel_k An empty engine.sz_subdocuments_engine_init_serialCompiles paths for the serial kernels, with the grammar sz_subdocuments_engine_init lists.sz_status_t sz_subdocuments_engine_init_serial(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *paths,
sz_subdocuments_format_t format,
sz_subdocuments_options_t options,
sz_size_t *malformed_index,
sz_allocator_t *allocator,
sz_stream_t stream);sz_subdocuments_engine_initserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalengine | out | Left untouched unless the call succeeds. |
|---|---|---|
paths | in | The vocabulary, one path per string, read on the host. |
format | in | The grammar of the documents every verb of this engine reads. |
options | in | How JSON walks read what RFC 8259 leaves open, or sz_subdocuments_options_none_k for the default. |
malformed_index | out | The first path to break the grammar or leave the subset, on failure. |
allocator | in | Where both blocks come from, or STRINGZILLA_NULL for sz_allocator_init_unified_serial. Stored by value, so sz_subdocuments_engine_free needs none and cannot get the wrong one. Scratch grows to the widest call seen, so a warm engine allocates nothing. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k Anything outside the grammar or what no walk evaluates.sz_missing_library_k match() or search() in a header-only build.sz_missing_kernel_k A format no kernel reads yet.sz_unexpected_dimensions_k An empty vocabulary.sz_overflow_risk_k Past 32-bit step or name offsets.sz_subdocuments_contains_serialMarks which documents hold at least one node some path selects, stopping each walk at its first match, the lightest verb of the engine.sz_status_t sz_subdocuments_contains_serial(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_u1x8_t *contained,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_containsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmA walk validates nothing past the match it stops at, so a document malformed only after its first match is marked rather than refused. Device walks run whole, as the counts' do, so on a device that document fails the call.
engine | in | Compiled paths, serving one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
contained | out | One bit per document, eight to a byte, the lowest bit first: set when that document matches, with the bits past the last document written as zero. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k A malformed document, as sz_subdocuments_counts defines it, leaving contained unspecified.sz_missing_kernel_k An empty engine.sz_subdocuments_counts_serialCounts the nodes of every document that some path selects, one count per document.sz_status_t sz_subdocuments_counts_serial(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_size_t *counts,
sz_size_t counts_stride,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_countsserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalA node several paths select counts once per path, as sz_subdocuments_find reports it.
A document is malformed with unbalanced brackets or tags, an unterminated string, attribute value, comment or CDATA section, a / outside the strings of JSON, broken structure where the walk reads it, or an XML DOCTYPE with an internal subset, and leaves the counts unspecified. A container or element no live step can enter is skipped by counting its nesting, so nothing else it holds is validated.
engine | in | Compiled paths; the walk grows the engine's depth stack, so one engine serves one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
counts | out | The per-document counts, document d at counts[d * stride]. |
counts_stride | in | Entries from one document's count to the next, at least one. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_unexpected_dimensions_k counts_stride is zero.sz_missing_kernel_k An empty engine.sz_invalid_syntax_k A malformed document.sz_subdocuments_find_serialReports every node of every document that some path selects.sz_status_t sz_subdocuments_find_serial(
sz_subdocuments_engine_t *engine,
sz_strings_view_t const *documents,
sz_match_t *matches,
sz_size_t matches_capacity,
sz_size_t *matches_offsets,
sz_size_t *malformed_index,
sz_stream_t stream);sz_subdocuments_findserialhaswellicelakeneonsve2rvvv128loongsonasxpowervsxcudarocmmetalMatches come by document, then by start offset, then by path, so a container comes before the nodes inside it, and a node is reported once per path that selects it, however many of the path's selectors or descendant routes reach it. Each match spans the node's exact bytes: in JSON a string with its quotes, a number, true, false or null without the blank space around it, or an object or array through its closing bracket; in XML an element through its end tag, an attribute's value without its quotes, or a text node, whose CDATA sections stay as written.
engine | in | Compiled paths, serving one call at a time. |
|---|---|---|
documents | in | One document per string; their bytes are read in place and never copied. |
matches | out | Room for matches_capacity matches; STRINGZILLA_NULL together with a zero capacity makes the call a pure size query. |
matches_capacity | in | Entries matches holds. |
matches_offsets | out | One boundary per document into the matches plus one, the last being the total; filled whether or not the matches fit, which is what sizes the next call. |
malformed_index | out | The first malformed document, on failure. |
stream | in | Null on the CPU. |
sz_invalid_syntax_k A malformed document, as sz_subdocuments_counts defines it, leaving the outputs unspecified.sz_missing_kernel_k An empty engine.sz_utf8_count_bestCount the number of UTF-8 characters in a string.sz_status_t sz_utf8_count_best(
sz_cptr_t text,
sz_size_t length,
sz_size_t *count,
sz_capability_t capabilities,
sz_stream_t stream);sz_utf8_countserialhaswellicelakeneonsve2rvvv128v128relaxedloongsonasxpowervsxThe logic is to count the number of "continuation bytes" matching the 10xxxxxx pattern, and then subtract that from the total byte length to get the number of "start bytes" - coinciding with the number of UTF-8 characters. Counting the characters of a string:
sz_capability_t capabilities;
sz_capabilities_enabled_cpu(&capabilities);
sz_size_t char_count;
sz_utf8_count_best(text, length, &char_count, capabilities, NULL);
printf("String has %zu characters\n", char_count);text | in | String to be scanned. |
|---|---|---|
length | in | Number of bytes in the string. |
count | out | Number of UTF-8 characters in the string. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_missing_kernel_k No capability in capabilities has it.sz_utf8_count_strings_bestChecks that every one of strings[order[i]], or of the first count strings, is well-formed UTF-8 and, unless runes is null, counts each one's runes.sz_status_t sz_utf8_count_strings_best(
sz_strings_view_t const *strings,
sz_size_t const *order,
sz_size_t count,
sz_size_t *runes,
sz_size_t *malformed_index,
sz_capability_t capabilities,
sz_stream_t stream);sz_utf8_count_stringsserialhaswellicelakeneonWell-formed is RFC 3629: no overlong forms, surrogates, codepoints past U+10FFFF, stray continuation bytes or truncated sequences, and no sequence crossing from one string into the next. A tape read in its own order is validated as one run of bytes, resolving each string at its end in one step. Other layouts and orders go string by string.
strings | in | The strings to check, in any layout. |
|---|---|---|
order | in | Indices into strings, possibly repeated, or null for the first count. |
count | in | Number of strings to check. |
runes | out | Null, or the runes of each string, count of them, unspecified from the first ill-formed string on. |
malformed_index | out | The first ill-formed string, or count when all are well-formed. |
capabilities | in | One device's capabilities, like sz_capabilities_enabled_cpu reports. |
stream | in | Null on the CPU, or the GPU stream of that device to queue on. |
sz_invalid_utf8_k A string is ill-formed.sz_missing_kernel_k No capability in capabilities has it.