# Unicode Segmentation > StringZilla for C Splitting Unicode text into the units a human actually perceives is deceptively hard, and getting it wrong corrupts data. A single emoji can be dozens of bytes and several codepoints yet one perceived character, CJK runs carry no spaces, and a period is not always the end of a sentence. StringZilla ships the real algorithms β€” UAX-29 graphemes, words, and sentences, plus UAX-14 line-break opportunities β€” as lazy ranges that borrow from the source and yield one `sz::string_view_t` per segment with no per-segment allocation. A string's length is really three different numbers, and only the grapheme count matches what a user sees: ```cpp #include #include namespace sz = ashvardanian::stringzilla; int main() { sz::string_view_t family = "πŸ‘¨β€πŸ‘©β€πŸ‘§β€πŸ‘¦"; // one emoji, one perceived character std::size_t codepoints = 0; for (sz_rune_t r : family.utf8_runes()) (void)r, ++codepoints; std::size_t graphemes = 0; for (sz::string_view_t g : family.utf8_graphemes()) (void)g, ++graphemes; assert(family.size() == 25); // 25 bytes on the wire assert(codepoints == 7); // four people joined by three zero-width joiners assert(graphemes == 1); // one grapheme cluster return 0; } ``` This is exactly why truncating by bytes or codepoints tears emoji apart β€” it can slice a skin-tone modifier off its base, or split a flag in half. Iterating graphemes lets you cut on a real boundary: ```cpp sz::string_view_t msg = "πŸ™πŸ½πŸ‘πŸΏβœ¨ thanks"; std::size_t cut = 0, seen = 0; for (sz::string_view_t g : msg.utf8_graphemes()) { if (seen++ == 3) break; cut += g.size(); } assert(msg.sub(0, cut) == "πŸ™πŸ½πŸ‘πŸΏβœ¨"); // the three emoji keep their skin-tone modifiers ``` Word and sentence boundaries are just as treacherous: contractions, CJK, decimals, and URLs all defeat naive splitting. UAX-29 keeps `can't` whole while separating CJK ideographs that no space divides, and refuses to break a sentence inside `$9.99`: ```cpp sz::string_view_t phrase = "can't stop εŒ—δΊ¬!"; assert(*phrase.utf8_wordbreaks().begin() == "can't"); // apostrophe is interior; εŒ— and δΊ¬ split apart sz::string_view_t prose = "She paid $9.99. Cheap."; std::size_t sentences = 0; for (sz::string_view_t s : prose.utf8_sentences()) (void)s, ++sentences; assert(sentences == 2); // the dot inside $9.99 is not a sentence break ``` The full family of ranges, each borrowing from the source and yielding `sz::string_view_t` segments, or `sz_rune_t` for runes. Naming follows one rule: the bare name (`newlines`/`whitespaces`/`delimiters`) yields the __separators__ the kernel finds, while `split_*` yields the content __between__ them: - `utf8_runes()` β€” every codepoint as a decoded `sz_rune_t` UTF-32 scalar, with U+FFFD for ill-formed bytes and a truncated tail. - `utf8_graphemes()` β€” UAX-29 grapheme clusters, the user-perceived characters. - `utf8_wordbreaks()` β€” all UAX-29 word segments (words and the separators between them). - `utf8_sentences()` β€” UAX-29 sentence boundaries. - `utf8_linebreaks()` β€” UAX-14 line-break opportunities, both hard breaks and soft wrap points. - `utf8_split_newlines()` / `utf8_newlines()` β€” content between the seven Unicode newlines (CRLF as one), or the newline runs themselves. - `utf8_split_whitespaces()` / `utf8_whitespaces()` β€” content between Unicode "White_Space" runs (tokens), or the whitespace runs. - `utf8_split_delimiters()` / `utf8_delimiters()` β€” content between any Unicode punctuation/symbol/separator, or the delimiter runs. On any `split_*` range, `.skip_empty()` drops empty segments and `.with_separators()` interleaves segments and separators losslessly (concatenation reconstructs the input). The `utf8_wordbreaks`, `utf8_graphemes`, `utf8_sentences`, and `utf8_linebreaks` ranges _tile_ the input β€” every byte belongs to exactly one segment, with no gaps and no empty slices. Because every range borrows and walks the buffer once, segmenting a multi-megabyte document into graphemes or words allocates nothing. In C, `sz_utf8_graphemes_best` and its three tiling siblings write one length per segment, each segment starting where the previous one ended, so after a full batch the caller resumes at the sum of its lengths. The newline, whitespace and delimiter finders write offsets and lengths of their matches, and after a full batch `bytes_consumed` ends exactly at the last match; otherwise it is the whole input. `sz_utf8_decode_best` treats the end of the text as the end of the input, decoding a truncated final sequence to one U+FFFD, so a caller feeding it chunks splits them at a rune boundary. `#include ` `#include ` `#include ` `#include ` `#include ` `#include ` `#include ` ## sz_kernel_utf8_seek_t Signature of the kernels behind [`sz_utf8_seek_best`](#sz_utf8_seek_best). ```c typedef sz_status_t(*) sz_kernel_utf8_seek_t(sz_cptr_t, sz_size_t, sz_size_t, sz_cptr_t *, sz_stream_t); ``` ## sz_kernel_utf8_decode_t Signature of the kernels behind [`sz_utf8_decode_best`](#sz_utf8_decode_best). ```c typedef sz_status_t(*) sz_kernel_utf8_decode_t(sz_cptr_t, sz_size_t, sz_rune_t *, sz_size_t, sz_size_t *, sz_size_t *, sz_stream_t); ``` ## sz_kernel_utf8_segmenter_t Signature of the tiling segmenters - graphemes, words, sentences, lines. ```c typedef sz_status_t(*) sz_kernel_utf8_segmenter_t(sz_cptr_t, sz_size_t, sz_size_t *, sz_size_t, sz_size_t *, sz_stream_t); ``` Emits one length per segment and their count; each segment starts where the previous one ended. ## sz_kernel_utf8_tokenizer_t Signature of the token kernels - newlines, whitespace, delimiters. ```c typedef sz_status_t(*) sz_kernel_utf8_tokenizer_t(sz_cptr_t, sz_size_t, sz_size_t *, sz_size_t *, sz_size_t, sz_size_t *, sz_size_t *, sz_stream_t); ``` Emits parallel (offset, length) arrays for each match, their count, and a resume `bytes_consumed`. ## sz_utf8_seek_best Skip forward to the Nth UTF-8 character. ```c sz_status_t sz_utf8_seek_best( sz_cptr_t text, sz_size_t length, sz_size_t n, sz_cptr_t *position, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_seek` backends: serial, haswell, icelake, neon, sve2, rvv, v128, v128relaxed, loongsonasx, powervsx Skipping to character 1000, e.g. for pagination: ``` char const *position; sz_utf8_seek_best(text, length, 1000, &position, capabilities, NULL); if (!position) { // String has fewer than 1000 characters } ``` Truncating to 280 characters, Twitter-style: ``` char const *end; sz_utf8_seek_best(text, length, 280, &end, capabilities, NULL); size_t truncated_bytes = end ? (end - text) : length; ``` - `text` (in): String to be scanned. - `length` (in): Number of bytes in the string. - `n` (in): Number of UTF-8 characters to skip, 0-indexed, so `n = 0` yields `text`. - `position` (out): The Nth character, or NULL if the string has fewer than `n` characters. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_decode_best Unpack a UTF-8 string into UTF-32 codepoints. ```c sz_status_t sz_utf8_decode_best( sz_cptr_t text, sz_size_t length, sz_rune_t *runes, sz_size_t runes_capacity, sz_size_t *runes_count, sz_size_t *bytes_consumed, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_decode` backends: serial, haswell, icelake, neon, sve2, rvv, v128, loongsonasx, powervsx This function is designed for streaming-like decoding with smart iterators built on top of it. The iterator would unpack a continuous slice of UTF-8 text into UTF-32 codepoints in chunks, yielding them upstream - only one at a time. This avoids allocating large buffers for the entire UTF-32 string, which can be 4x the size of the UTF-8 input. This functionality is similar to the `simdutf` library's UTF-8 to UTF-32 conversion routines, and leverages an assumption that the absolute majority of written text doesn't mix codepoints of every length in each register-sized chunk: - English text and source code is predominantly 1-byte ASCII characters. - Broader European languages with diacritics: mostly 2-byte characters, 1-byte punctuation. - Chinese & Japanese mostly use 3-byte characters with rare 1- or 3-byte punctuation. - Korean uses 3-byte characters with 1-byte spaces; word are 2-6 syllables or 6-16 bytes.It's a different story for emoji-heavy texts, which can mix 4-byte characters more frequently. Every backend shares one contract, the basis for every codepoint iterator built on it: - __Fill-or-drain__: emits runes until `runes_capacity` is reached or `text` is exhausted, looping internally - one call fills the buffer however many byte-widths the text mixes. - __Total__ __and__ __safe:__ never reads past `text + length`; substitutes one __U+FFFD__ per maximal ill-formed subpart (overlong, surrogate, out-of-range, or a stray byte) and resyncs. - __Valid__ __scalar__ __values:__ every emitted rune is a Unicode scalar value (incl. U+FFFD), so callers may convert without re-validation.End of text is end of input: a well-formed but truncated final sequence decodes to one U+FFFD over its maximal subpart and is consumed, so `"a\xE2\x82"` yields `{'a', U+FFFD}` over all 3 bytes. Only a full `runes` buffer stops early, so `bytes_consumed` falls short of `length` only when `runes_count` reaches `runes_capacity`, and then at a rune boundary. Callers decoding a stream in chunks split it at a rune boundary. - `text` (in): UTF-8 string to unpack. - `length` (in): Number of bytes in the string. - `runes` (out): Buffer for UTF-32 codepoints, recommended to be at least __64__ entries wide. - `runes_capacity` (in): Capacity of the `runes` buffer, in `[sz_rune_t](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_rune_t)` entries. - `runes_count` (out): Number of runes unpacked. - `bytes_consumed` (out): Bytes of `text` decoded, the offset the next call resumes at. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_*_find_kernel Finds the UTF-8 runes kernel of `kind`, from the best of `capabilities`. ```c sz_status_t sz_utf8_runes_find_kernel( sz_kernel_kind_t kind, sz_capability_t capabilities, sz_kernel_punned_t *kernel, sz_capability_t *capability); ``` - `sz_utf8_runes_find_kernel` - `sz_utf8_tokens_find_kernel` - `kind` (in): The verb whose kernel to find. - `capabilities` (in): The mask to pick the best capability from. - `kernel` (out): The kernel, or null when none of `capabilities` has it. - `capability` (out): The capability the kernel belongs to, or zero. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) - [`sz_missing_library_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_library_k) Header-only. ## sz_utf8_newlines_best Enumerates every UTF-8 newline delimiter in a string. ```c sz_status_t sz_utf8_newlines_best( sz_cptr_t text, sz_size_t length, sz_size_t *match_offsets, sz_size_t *match_lengths, sz_size_t matches_capacity, sz_size_t *matches_count, sz_size_t *bytes_consumed, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_newlines` backends: serial, haswell, icelake, neon, sve2, rvv, v128, loongsonasx, powervsx Enumerates every newline delimiter in a single sweep: writes the byte offset and byte length of each match into the parallel `match_offsets` and `match_lengths` arrays, a `"\r\n"` CRLF being one match of length 2, and their number into `matches_count`. A caller resuming from `text + *bytes_consumed` after a full output obtains the identical remainder. Here are all the UTF-8 newline characters we are looking for (7 characters + CRLF): - single-byte chars (4 total): - U+000A for `"\n"` (LINE FEED) - U+000B for `"\v"` (VERTICAL TAB / LINE TABULATION) - U+000C for `"\f"` (FORM FEED) - U+000D for `"\r"` (CARRIAGE RETURN) - double-byte chars (1 total): - U+0085 for `0xC285` (NEXT LINE) - triple-byte chars (2 total): - U+2028 for `0xE280A8` (LINE SEPARATOR) - U+2029 for `0xE280A9` (PARAGRAPH SEPARATOR) - double-character sequence: - U+000D U+000A for `"\r\n"` that should be treated as a single new line!U+001C, U+001D, U+001E (FILE/GROUP/RECORD SEPARATOR) are not included. These are data structure delimiters used in formats like USV (Unicode Separated Values), not line breaks. Use [`sz_find_byte_best`](https://www.unum.cloud/docs/stringzilla/c/searching-and-counting.md#sz_find_byte_best) if you need to find these separators. - `text` (in): String to be scanned. - `length` (in): Number of bytes in the string. - `match_offsets` (out): Delimiter start offsets, at least `matches_capacity` entries. - `match_lengths` (out): Delimiter byte lengths, at least `matches_capacity` entries. - `matches_capacity` (in): Capacity of the output arrays. - `matches_count` (out): Number of delimiters written to the output arrays. - `bytes_consumed` (out): Optional resume offset: once `matches_count` reaches `matches_capacity`, the end of the last match, or zero with no capacity; else `length`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_whitespaces_best Enumerates every UTF-8 whitespace delimiter in a string. ```c sz_status_t sz_utf8_whitespaces_best( sz_cptr_t text, sz_size_t length, sz_size_t *match_offsets, sz_size_t *match_lengths, sz_size_t matches_capacity, sz_size_t *matches_count, sz_size_t *bytes_consumed, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_whitespaces` backends: serial, haswell, icelake, neon, sve2, rvv, v128, loongsonasx, powervsx Enumerates every whitespace delimiter in a single sweep, with the same contract as `sz_utf8_newlines_best:` writes the byte offset and byte length of each match into the parallel `match_offsets` and `match_lengths` arrays, their number into `matches_count`, and sets `bytes_consumed` to the resume offset. Each whitespace codepoint is one match: CR and LF are independent length-1 matches, as there is no CRLF merging in the whitespace set. Implements the Unicode White_Space property, 25 characters in total. Per the Unicode standard, whitespace includes all newline characters plus horizontal spaces. Matches the behavior of ICU's `u_isspace()` and Python's `str.isspace()`. - single-byte chars (6 total): - U+0009 tab `"\t"` (CHARACTER TABULATION) - U+000A `"\n"` (LINE FEED - newline) - U+000B `"\v"` (LINE TABULATION - newline) - U+000C `"\f"` (FORM FEED - newline) - U+000D `"\r"` (CARRIAGE RETURN - newline) - U+0020 (SPACE) - double-byte chars (2 total): - U+0085 `0xC285` (NEXT LINE - newline) - U+00A0 `0xC2A0` (NO-BREAK SPACE) - triple-byte chars (17 total): - U+1680 `0xE19A80` (OGHAM SPACE MARK) - U+2000 `0xE28080` (EN QUAD) - U+2001 `0xE28081` (EM QUAD) - U+2002 `0xE28082` (EN SPACE) - U+2003 `0xE28083` (EM SPACE) - U+2004 `0xE28084` (THREE-PER-EM SPACE) - U+2005 `0xE28085` (FOUR-PER-EM SPACE) - U+2006 `0xE28086` (SIX-PER-EM SPACE) - U+2007 `0xE28087` (FIGURE SPACE) - U+2008 `0xE28088` (PUNCTUATION SPACE) - U+2009 `0xE28089` (THIN SPACE) - U+200A `0xE2808A` (HAIR SPACE) - U+2028 `0xE280A8` (LINE SEPARATOR - newline) - U+2029 `0xE280A9` (PARAGRAPH SEPARATOR - newline) - U+202F `0xE280AF` (NARROW NO-BREAK SPACE) - U+205F `0xE2819F` (MEDIUM MATHEMATICAL SPACE) - U+3000 `0xE38080` (IDEOGRAPHIC SPACE)The last one, the IDEOGRAPHIC SPACE (U+3000), is commonly used in East Asian typography, like Japanese formatted text or Chinese traditional poetry alignments. Some implementations treat more codepoints as whitespace, but these are not included: - U+001C, U+001D, U+001E, U+001F (FILE/GROUP/RECORD/UNIT SEPARATOR) are data structure delimiters for formats like USV (Unicode Separated Values). Only Java's `Character.isWhitespace()` includes them; Unicode, ICU, and Python do not. - U+200B, U+200C, U+200D (ZERO WIDTH SPACE/NON-JOINER/JOINER) are Format characters, not whitespace. They have no width and affect rendering rather than spacing. - `text` (in): String to be scanned. - `length` (in): Number of bytes in the string. - `match_offsets` (out): Delimiter start offsets, at least `matches_capacity` entries. - `match_lengths` (out): Delimiter byte lengths, at least `matches_capacity` entries. - `matches_capacity` (in): Capacity of the output arrays. - `matches_count` (out): Number of delimiters written to the output arrays. - `bytes_consumed` (out): Optional resume offset: once `matches_count` reaches `matches_capacity`, the end of the last match, or zero with no capacity; else `length`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_delimiters_best Enumerates every UTF-8 delimiter codepoint (punctuation, symbol, separator, whitespace). ```c sz_status_t sz_utf8_delimiters_best( sz_cptr_t text, sz_size_t length, sz_size_t *match_offsets, sz_size_t *match_lengths, sz_size_t matches_capacity, sz_size_t *matches_count, sz_size_t *bytes_consumed, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_delimiters` backends: serial, haswell, icelake, neon, sve2, rvv, v128, loongsonasx, powervsx The general superset of [`sz_utf8_newlines_best`](#sz_utf8_newlines_best) and `sz_utf8_whitespaces_best:` every codepoint whose Unicode general category is a punctuation (P*), symbol (S*), or separator (Z*) is a delimiter. Shares the contract of `sz_utf8_newlines_best:` writes the byte offset and byte length of each match into the parallel `match_offsets` and `match_lengths` arrays, their number into `matches_count`, and sets `bytes_consumed` to the resume offset. A byte that does not begin a well-formed codepoint is skipped and never reported. - `text` (in): String to be scanned. - `length` (in): Number of bytes in the string. - `match_offsets` (out): Delimiter start offsets, at least `matches_capacity` entries. - `match_lengths` (out): Delimiter byte lengths, at least `matches_capacity` entries. - `matches_capacity` (in): Capacity of the output arrays. - `matches_count` (out): Number of delimiters written to the output arrays. - `bytes_consumed` (out): Optional resume offset: once `matches_count` reaches `matches_capacity`, the end of the last match, or zero with no capacity; else `length`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_wordbreaks_best Segment UTF-8 text into UAX-29 words in a single pass. ```c sz_status_t sz_utf8_wordbreaks_best( sz_cptr_t text, sz_size_t length, sz_size_t *lengths, sz_size_t capacity, sz_size_t *count, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_wordbreaks` backends: serial, haswell, icelake, neon, sve2, rvv, v128, loongsonasx, powervsx Walks the input left-to-right and writes the byte length of each word into `lengths`. Words are the spans between consecutive TR29 boundaries and tile the input: each starts where the previous one ended, the first at `text`. A call covers all of `text` unless it fills `lengths`, in which case it covers the sum of the lengths it wrote, and the caller resumes from `text` advanced by that sum. - `text` (in): UTF-8 encoded text. - `length` (in): Byte length of `text`. - `lengths` (out): Word byte lengths, at least `capacity` entries. - `capacity` (in): Capacity of `lengths`, in entries. - `count` (out): Number of words written, at most `capacity`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Notes: No zero-length words are emitted; `length` == 0 yields no words. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_wordbreaks_find_kernel Finds the UTF-8 word break kernel of `kind`, from the best of `capabilities`. ```c sz_status_t sz_utf8_wordbreaks_find_kernel( sz_kernel_kind_t kind, sz_capability_t capabilities, sz_kernel_punned_t *kernel, sz_capability_t *capability); ``` - `kind` (in): The verb whose kernel to find. - `capabilities` (in): The mask to pick the best capability from. - `kernel` (out): The kernel, or null when none of `capabilities` has it. - `capability` (out): The capability the kernel belongs to, or zero. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) - [`sz_missing_library_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_library_k) Header-only. ## sz_utf8_graphemes_best Segment UTF-8 text into UAX-29 grapheme clusters in a single pass. ```c sz_status_t sz_utf8_graphemes_best( sz_cptr_t text, sz_size_t length, sz_size_t *lengths, sz_size_t capacity, sz_size_t *count, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_graphemes` backends: serial, haswell, icelake, neon, sve2 Walks the input left-to-right and writes the byte length of each grapheme cluster into `lengths`. Clusters are the spans between consecutive UAX-29 grapheme boundaries (GB1-GB999) and tile the input: each starts where the previous one ended, the first at `text`. A call covers all of `text` unless it fills `lengths`, in which case it covers the sum of the lengths it wrote, and the caller resumes from `text` advanced by that sum. - `text` (in): UTF-8 encoded text. - `length` (in): Byte length of `text`. - `lengths` (out): Cluster byte lengths, at least `capacity` entries. - `capacity` (in): Capacity of `lengths`, in entries. - `count` (out): Number of clusters written, at most `capacity`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Notes: No zero-length clusters are emitted; `length` == 0 yields no clusters. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_graphemes_find_kernel Finds the UTF-8 grapheme kernel of `kind`, from the best of `capabilities`. ```c sz_status_t sz_utf8_graphemes_find_kernel( sz_kernel_kind_t kind, sz_capability_t capabilities, sz_kernel_punned_t *kernel, sz_capability_t *capability); ``` - `kind` (in): The verb whose kernel to find. - `capabilities` (in): The mask to pick the best capability from. - `kernel` (out): The kernel, or null when none of `capabilities` has it. - `capability` (out): The capability the kernel belongs to, or zero. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) - [`sz_missing_library_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_library_k) Header-only. ## sz_utf8_sentences_best Segment UTF-8 text into UAX-29 sentences in a single pass. ```c sz_status_t sz_utf8_sentences_best( sz_cptr_t text, sz_size_t length, sz_size_t *lengths, sz_size_t capacity, sz_size_t *count, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_sentences` backends: serial, haswell, icelake, neon, sve2 Walks the input left-to-right and writes the byte length of each sentence into `lengths`. Sentences are the spans between consecutive UAX-29 sentence boundaries (SB1-SB998) and tile the input: each starts where the previous one ended, the first at `text`. A call covers all of `text` unless it fills `lengths`, in which case it covers the sum of the lengths it wrote, and the caller resumes from `text` advanced by that sum. - `text` (in): UTF-8 encoded text. - `length` (in): Byte length of `text`. - `lengths` (out): Sentence byte lengths, at least `capacity` entries. - `capacity` (in): Capacity of `lengths`, in entries. - `count` (out): Number of sentences written, at most `capacity`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Notes: No zero-length sentences are emitted; `length` == 0 yields no sentences. Notes: Sentence segmentation is forward-only. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_sentences_find_kernel Finds the UTF-8 sentence kernel of `kind`, from the best of `capabilities`. ```c sz_status_t sz_utf8_sentences_find_kernel( sz_kernel_kind_t kind, sz_capability_t capabilities, sz_kernel_punned_t *kernel, sz_capability_t *capability); ``` - `kind` (in): The verb whose kernel to find. - `capabilities` (in): The mask to pick the best capability from. - `kernel` (out): The kernel, or null when none of `capabilities` has it. - `capability` (out): The capability the kernel belongs to, or zero. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) - [`sz_missing_library_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_library_k) Header-only. ## sz_utf8_linebreaks_best Segment UTF-8 text into UAX-14 line-break opportunities in one pass. ```c sz_status_t sz_utf8_linebreaks_best( sz_cptr_t text, sz_size_t length, sz_size_t *lengths, sz_size_t capacity, sz_size_t *count, sz_capability_t capabilities, sz_stream_t stream); ``` - `sz_utf8_linebreaks` backends: serial, haswell, icelake, neon, sve2 Walks the input left-to-right and writes the byte length of each line segment into `lengths`. Segments are the spans between consecutive UAX-14 break opportunities and tile the input: each starts where the previous one ended, the first at `text`. A call covers all of `text` unless it fills `lengths`, in which case it covers the sum of the lengths it wrote, and the caller resumes from `text` advanced by that sum. This emits every wrap opportunity (both the mandatory LB4/LB5 hard breaks and the allowed soft-wrap points). To split only on the hard breaks (the `str.splitlines` behavior) use [`sz_utf8_newlines_best`](#sz_utf8_newlines_best) instead, which enumerates exactly the LB4/LB5 break positions. - `text` (in): UTF-8 encoded text. - `length` (in): Byte length of `text`. - `lengths` (out): Segment byte lengths, at least `capacity` entries. - `capacity` (in): Capacity of `lengths`, in entries. - `count` (out): Number of segments written, at most `capacity`. - `capabilities` (in): One device's capabilities, like `sz_capabilities_enabled_cpu` reports. - `stream` (in): Null on the CPU, or the GPU stream of that device to queue on. Notes: No zero-length segments are emitted; `length` == 0 yields no segments. Notes: Line segmentation is forward-only. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) No capability in `capabilities` has it. ## sz_utf8_linebreaks_find_kernel Finds the UTF-8 line break kernel of `kind`, from the best of `capabilities`. ```c sz_status_t sz_utf8_linebreaks_find_kernel( sz_kernel_kind_t kind, sz_capability_t capabilities, sz_kernel_punned_t *kernel, sz_capability_t *capability); ``` - `kind` (in): The verb whose kernel to find. - `capabilities` (in): The mask to pick the best capability from. - `kernel` (out): The kernel, or null when none of `capabilities` has it. - `capability` (out): The capability the kernel belongs to, or zero. Statuses: - [`sz_missing_kernel_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_kernel_k) - [`sz_missing_library_k`](https://www.unum.cloud/docs/stringzilla/c/types.md#sz_missing_library_k) Header-only.