# Unicode Segmentation > StringZilla for Rust A single emoji such as the flag `๐Ÿ‡บ๐Ÿ‡ธ` is 8 bytes and 2 codepoints, yet a reader sees one character โ€” one grapheme cluster. StringZilla draws every one of these boundaries: `sz_utf8_runes` (`Utf8View`) walks codepoints, while `sz_utf8_graphemes` (`Utf8Graphemes`) walks user-perceived clusters. The `StringZillableUnary` trait turns any `AsRef<[u8]>` into a broad family of lazy UTF-8 iterators and views, each yielding borrowed byte sub-slices or `char`s on demand: ```rust fn sz_utf8_runes(&self) -> Utf8View<'_>; // codepoints / random codepoint access fn sz_utf8_split_newlines(&self) -> Utf8SplitNewlines<'_>; // content BETWEEN hard newlines (CRLF = one) fn sz_utf8_newlines(&self) -> Utf8Newlines<'_>; // the newline runs themselves fn sz_utf8_split_whitespaces(&self) -> Utf8SplitWhitespaces<'_>; // content BETWEEN Unicode whitespace fn sz_utf8_whitespaces(&self) -> Utf8Whitespaces<'_>; // the whitespace runs themselves fn sz_utf8_split_delimiters(&self) -> Utf8SplitDelimiters<'_>; // content BETWEEN whitespace/punctuation fn sz_utf8_delimiters(&self) -> Utf8Delimiters<'_>; // the delimiter runs themselves fn sz_utf8_wordbreaks(&self) -> Utf8Wordbreaks<'_>; // all UAX-29 word segments (tiling) fn sz_utf8_graphemes(&self) -> Utf8Graphemes<'_>; // UAX-29 grapheme clusters fn sz_utf8_sentences(&self) -> Utf8Sentences<'_>; // UAX-29 sentences fn sz_utf8_linebreaks(&self) -> Utf8Linebreaks<'_>; // UAX-14 line-break opportunities ``` The naming follows one rule: the bare name (`newlines`/`whitespaces`/`delimiters`) yields the __separators__ the kernel finds, while `split_*` yields the content __between__ them. Chain `.with_separators()` on a `split_*` iterator to interleave both losslessly. Every member of this family is lazy and zero-copy. `Utf8View`, `Utf8Runes`, `Utf8SplitNewlines`, `Utf8SplitWhitespaces`, `Utf8Wordbreaks`, `Utf8Graphemes`, `Utf8Sentences`, and `Utf8Linebreaks`, together with the `sz_splits` / `sz_rsplits` iterators, all borrow from the source string and yield `&[u8]` or `&str` slices on demand without allocating. There is no backing vector and no per-element heap buffer. Contrast this with the standard library, where collecting into a `Vec` allocates the vector and a fresh heap buffer for every element, and even a `Vec<&str>` allocates the backing vector up front. Because these iterators borrow from the input, you can stream over millions of words or grapheme clusters of a large document with effectively zero per-element allocation, and the borrow lifetimes keep the yielded slices zero-copy. `Utf8View` offers O(1) construction, `len()` counting the codepoints on demand, `offset_of(n)` for the byte offset of the Nth codepoint, and `iter()` for batched `char` iteration: ```rust use stringzilla::StringZillableUnary; let view = "Hello๐ŸŒ".sz_utf8_runes(); assert_eq!(view.len(), 6); assert_eq!(view.offset_of(5), Some(5)); let chars: Vec = view.iter().collect(); assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '๐ŸŒ']); ``` The boundary iterators `wordbreaks`, `graphemes`, `sentences`, and `linebreaks` __tile__ the input โ€” every byte belongs to exactly one segment, so consecutive segments are contiguous and no empty slices appear: ```rust use stringzilla::StringZillableUnary; let words: Vec<&[u8]> = b"Hi, world".sz_utf8_wordbreaks().collect(); assert_eq!(words, vec![&b"Hi"[..], &b","[..], &b" "[..], &b"world"[..]]); let sentences: Vec<&[u8]> = b"Hi. Bye.".sz_utf8_sentences().collect(); assert_eq!(sentences, vec![&b"Hi. "[..], &b"Bye."[..]]); ``` `split_newlines` splits on the 8 Unicode hard newlines, treating CRLF as a single delimiter and excluding the newline from each segment, while `split_whitespaces` splits on the 25 Unicode whitespace characters and __keeps__ empty segments by default; chain `.skip_empty()` for `str::split_whitespace`-style behavior: ```rust use stringzilla::StringZillableUnary; let lines: Vec<&str> = "Hello\nWorld\r\nRust" .sz_utf8_split_newlines() .map(|l| std::str::from_utf8(l).unwrap()) .collect(); assert_eq!(lines, vec!["Hello", "World", "Rust"]); let tokens: Vec<&[u8]> = b" hi ".sz_utf8_split_whitespaces().skip_empty().collect(); assert_eq!(tokens, vec![&b"hi"[..]]); ``` Every iterator is also constructible directly, for example `Utf8Wordbreaks::new(text)`. Each exposes a `STEPS` const-generic batch-size knob via `with_steps`, written with a turbofish such as `Utf8Runes::<256>::with_steps(bytes)`, that trades buffer size for FFI-call amortization without changing the yielded results. The default batch size is the public constant `ITERATORS_DEFAULT_STEPS = 64`. ### Low Level Decoding `utf8_decode` unpacks UTF-8 bytes into a UTF-32 `runes` buffer, returning how many bytes were consumed and codepoints written. ```rust pub fn utf8_decode(text: &[u8], runes: &mut [u32]) -> (usize, usize); // (bytes_consumed, runes_count) ``` `utf8_decode` decodes UTF-8 into UTF-32 codepoints, filling the output buffer or draining the input per call. For inputs larger than the buffer, loop and resume at `bytes_consumed`. It is total: ill-formed bytes decode to U+FFFD, so every value written is a valid Unicode scalar. The end of `text` is the end of the input, so a truncated final sequence also decodes to one U+FFFD and is consumed; cut a streamed input into chunks at rune boundaries: ```rust use stringzilla as sz; let mut runes = [0u32; 16]; assert_eq!(sz::utf8_decode(b"a\xE2\x82", &mut runes), (3, 2)); assert_eq!(&runes[..2], &['a' as u32, 0xFFFD]); let (bytes, count) = sz::utf8_decode("Hello".as_bytes(), &mut runes); assert_eq!((bytes, count), (5, 5)); assert_eq!(runes[0], 'H' as u32); ``` `use stringzilla::StringZillableUnary;` `use stringzilla::Graphemes;` `use stringzilla::Linebreaks;` `use stringzilla::Utf8View;` `use stringzilla::Utf8Runes;` `use stringzilla::Sentences;` `use stringzilla::TokenizerKernel;` `use stringzilla::SegmenterKernel;` `use stringzilla::Newlines;` `use stringzilla::Whitespaces;` `use stringzilla::Delimiters;` `use stringzilla::Between;` `use stringzilla::Separators;` `use stringzilla::Both;` `use stringzilla::Utf8Segments;` `use stringzilla::Wordbreaks;` `use stringzilla::*;` ## Utf8Graphemes An iterator over UAX-29 grapheme clusters in UTF-8 text, in order. ```rust pub type Utf8Graphemes<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Graphemes, STEPS>; ``` Unlike whitespace splitting, the grapheme clusters tile the input: every byte belongs to exactly one grapheme cluster, so consecutive clusters are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules. Examples: ``` use stringzilla::Utf8Graphemes; let graphemes: Vec<&[u8]> = Utf8Graphemes::new(b"Hi!").collect(); assert_eq!(graphemes, vec![&b"H"[..], &b"i"[..], &b"!"[..]]); ``` ## Utf8Linebreaks An iterator over UAX-14 line break opportunities in UTF-8 text, in order. ```rust pub type Utf8Linebreaks<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Linebreaks, STEPS>; ``` Unlike whitespace splitting, the lines tile the input: every byte belongs to exactly one line, so consecutive lines are contiguous and no empty slices are produced. Follows the Unicode TR14 rules. Each yielded segment ends at a TR14 break opportunity, including soft breaks where a renderer _may_ wrap but is not required to. To split only on hard line breaks (the "splitlines" behaviour), use the newline API ([`StringZillableUnary::sz_utf8_split_newlines`](#sz_utf8_split_newlines)) instead. Examples: ``` use stringzilla::Utf8Linebreaks; let lines: Vec<&[u8]> = Utf8Linebreaks::new(b"Hi\nBye").collect(); assert_eq!(lines, vec![&b"Hi\n"[..], &b"Bye"[..]]); ``` ## Utf8Sentences An iterator over UAX-29 sentences in UTF-8 text, in order. ```rust pub type Utf8Sentences<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Sentences, STEPS>; ``` Unlike whitespace splitting, the sentences tile the input: every byte belongs to exactly one sentence, so consecutive sentences are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules. Examples: ``` use stringzilla::Utf8Sentences; let sentences: Vec<&[u8]> = Utf8Sentences::new(b"Hi. Bye.").collect(); assert_eq!(sentences, vec![&b"Hi. "[..], &b"Bye."[..]]); ``` ## Utf8SplitNewlines An iterator over substrings of UTF-8 text split by newline characters. ```rust pub type Utf8SplitNewlines<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Newlines, Between, KeepEmpty, STEPS>; ``` This iterator yields slices between newline characters. The newline characters themselves are not included in the yielded slices. Handles all 8 Unicode newline characters including CRLF as a single delimiter. Examples: ``` use stringzilla::{Utf8SplitNewlines}; let text = b"Hello\nWorld\r\nRust"; let lines: Vec<&[u8]> = Utf8SplitNewlines::new(text).collect(); assert_eq!(lines, vec![&b"Hello"[..], &b"World"[..], &b"Rust"[..]]); ``` ## Utf8Newlines An iterator over the newline runs themselves, the separators, in order. ```rust pub type Utf8Newlines<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Newlines, Separators, KeepEmpty, STEPS>; ``` ## Utf8SplitWhitespaces An iterator over segments of UTF-8 text split by whitespace characters. ```rust pub type Utf8SplitWhitespaces<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Whitespaces, Between, KeepEmpty, STEPS>; ``` Splits on all 25 Unicode "White_Space" characters; N whitespace delimiters yield N+1 segments. By default empty segments are __kept__, matching `str::split`, so runs of whitespace and leading/trailing whitespace produce empty slices. Call [`Self::skip_empty`](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#skip_empty) for the `str::split_whitespace`-style behavior that drops empties and yields only non-empty tokens. Examples: ``` use stringzilla::{Utf8SplitWhitespaces}; // Default KEEP policy: empties around the words are preserved. let text = b" hi "; let segments: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).collect(); assert_eq!(segments, vec![&b""[..], &b""[..], &b"hi"[..], &b""[..], &b""[..]]); // Opt in to dropping empties for token-style splitting. let tokens: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).skip_empty().collect(); assert_eq!(tokens, vec![&b"hi"[..]]); ``` ## Utf8Whitespaces An iterator over the whitespace runs themselves, the separators, in order. ```rust pub type Utf8Whitespaces<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Whitespaces, Separators, KeepEmpty, STEPS>; ``` ## Utf8SplitDelimiters An iterator over segments of UTF-8 text split by any Unicode delimiter codepoint. ```rust pub type Utf8SplitDelimiters<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Delimiters, Between, KeepEmpty, STEPS>; ``` Splits on every codepoint whose Unicode general category is punctuation (`P*`), symbol (`S*`), or separator/whitespace (`Z*`) โ€” the superset of [`Utf8SplitWhitespaces`](#Utf8SplitWhitespaces). N delimiters yield N+1 segments; empty segments are __kept__ by default (call [`Self::skip_empty`](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#skip_empty) to drop them for token-style splitting). Examples: ``` use stringzilla::{Utf8SplitDelimiters}; // "Hi, worldโ€”foo" splits on ',', ' ', and U+2014 EM DASH. let tokens: Vec<&[u8]> = Utf8SplitDelimiters::new("Hi, world\u{2014}foo".as_bytes()).skip_empty().collect(); assert_eq!(tokens, vec![&b"Hi"[..], &b"world"[..], &b"foo"[..]]); ``` ## Utf8Delimiters An iterator over the delimiter runs themselves, the separators, in order. ```rust pub type Utf8Delimiters<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Delimiters, Separators, KeepEmpty, STEPS>; ``` ## Utf8Wordbreaks An iterator over UAX-29 words in UTF-8 text, in order. ```rust pub type Utf8Wordbreaks<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Wordbreaks, STEPS>; ``` Unlike whitespace splitting, the words tile the input: every byte belongs to exactly one word, so consecutive words are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules. Examples: ``` use stringzilla::Utf8Wordbreaks; let words: Vec<&[u8]> = Utf8Wordbreaks::new(b"Hi, world").collect(); assert_eq!(words, vec![&b"Hi"[..], &b","[..], &b" "[..], &b"world"[..]]); ``` ## StringZillableUnary Trait for unary string operations that only operate on `self` without needle parameters. ```rust pub trait StringZillableUnary ``` These operations include hash computation and byte sum calculation. Examples: Basic usage on a byte slice: ``` use stringzilla::StringZillableUnary; let text = b"Hello"; assert_eq!(text.sz_bytesum(), 500); ``` Implementors: - `impl [StringZillableUnary](#StringZillableUnary) for Source where Source: AsRef<[u8]> + ?Sized` ## sz_bytesum Computes the bytesum value of unsigned bytes in a given string. ```rust fn sz_bytesum(&self) -> u64 ``` This function is useful for verifying data integrity and detecting changes in binary data, such as files or network packets. Examples: ``` use stringzilla::StringZillableUnary; let text = b"Hello"; assert_eq!(text.sz_bytesum(), 500); ``` ## sz_hash Computes a 64-bit AES-based hash value for a given string. ```rust fn sz_hash(&self) -> u64 ``` This function is designed to provide a high-quality hash value for use in hash tables, data structures, and cryptographic applications. Unlike the bytesum function, the hash function is order-sensitive. Examples: ``` use stringzilla::StringZillableUnary; let s1 = b"Hello"; let s2 = b"World"; assert_ne!(s1.sz_hash(), s2.sz_hash()); ``` ## sz_utf8_runes Returns a lazy UTF-8 character view with SIMD-accelerated operations. ```rust fn sz_utf8_runes(&self) -> Utf8View<'_> ``` The view provides: - `.len()` for character count, computed on demand - `.offset_of(n)` for random access to Nth character offset - `.iter()` for efficient batched iteration over characters Examples: ``` use stringzilla::StringZillableUnary; let text = "Hello๐ŸŒ"; let view = text.sz_utf8_runes(); // Lazy character count assert_eq!(view.len(), 6); // Random access (byte offset of Nth character) assert_eq!(view.offset_of(5), Some(5)); // ๐ŸŒ at byte 5 // Efficient batched iteration let chars: Vec<char> = view.iter().collect(); assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '๐ŸŒ']); ``` ## sz_utf8_split_newlines Returns an iterator over lines split by UTF-8 newline characters. ```rust fn sz_utf8_split_newlines(&self) -> Utf8SplitNewlines<'_> ``` The iterator yields slices between newlines. Handles all Unicode newline characters including CRLF as a single delimiter. Examples: ``` use stringzilla::StringZillableUnary; let text = "Hello\nWorld\r\nRust"; let lines: Vec<&str> = text.sz_utf8_split_newlines() .map(|line| std::str::from_utf8(line).unwrap()) .collect(); assert_eq!(lines, vec!["Hello", "World", "Rust"]); ``` ## sz_utf8_newlines Returns an iterator over the newline runs themselves, the separators. ```rust fn sz_utf8_newlines(&self) -> Utf8Newlines<'_> ``` ## sz_utf8_split_whitespaces Returns an iterator over segments split by UTF-8 whitespace characters. ```rust fn sz_utf8_split_whitespaces(&self) -> Utf8SplitWhitespaces<'_> ``` Handles all 25 Unicode "White_Space" characters; N delimiters yield N+1 segments. By default __empty segments are kept__, matching `str::split`, so runs of whitespace surface empty slices. Chain `.skip_empty()` on the returned iterator to recover the `str::split_whitespace` token behavior that drops empties. Examples: ``` use stringzilla::StringZillableUnary; // KEEP (default): the double space between "Hello" and "World" yields an empty segment. let text = "Hello World\tRust"; let segments: Vec<&str> = text.sz_utf8_split_whitespaces() .map(|segment| std::str::from_utf8(segment).unwrap()) .collect(); assert_eq!(segments, vec!["Hello", "", "World", "Rust"]); // skip_empty: drops the empties to yield only the non-empty tokens. let tokens: Vec<&str> = text.sz_utf8_split_whitespaces() .skip_empty() .map(|token| std::str::from_utf8(token).unwrap()) .collect(); assert_eq!(tokens, vec!["Hello", "World", "Rust"]); ``` ## sz_utf8_whitespaces Returns an iterator over the whitespace runs themselves, the separators. ```rust fn sz_utf8_whitespaces(&self) -> Utf8Whitespaces<'_> ``` ## sz_utf8_split_delimiters Returns an iterator splitting on any Unicode delimiter (punctuation/symbol/separator/whitespace). ```rust fn sz_utf8_split_delimiters(&self) -> Utf8SplitDelimiters<'_> ``` ## sz_utf8_delimiters Returns an iterator over the delimiter runs themselves, the separators. ```rust fn sz_utf8_delimiters(&self) -> Utf8Delimiters<'_> ``` ## sz_utf8_wordbreaks Returns an iterator over UAX-29 words, in order. ```rust fn sz_utf8_wordbreaks(&self) -> Utf8Wordbreaks<'_> ``` Words tile the input contiguously. ## sz_utf8_graphemes Returns an iterator over UAX-29 grapheme clusters, in order. ```rust fn sz_utf8_graphemes(&self) -> Utf8Graphemes<'_> ``` Clusters tile the input contiguously. ## sz_utf8_sentences Returns an iterator over UAX-29 sentences, in order. ```rust fn sz_utf8_sentences(&self) -> Utf8Sentences<'_> ``` Sentences tile the input contiguously. ## sz_utf8_linebreaks Returns an iterator over UAX-14 line-break opportunities per Unicode TR14, in order. ```rust fn sz_utf8_linebreaks(&self) -> Utf8Linebreaks<'_> ``` Linewrap segments tile the input contiguously, including soft break opportunities. For hard line splits only, use [`Self::sz_utf8_split_newlines`](#sz_utf8_split_newlines). ## Graphemes Kernel behind [`Utf8Graphemes`](#Utf8Graphemes) (`sz_utf8_graphemes_best`). ```rust pub struct Graphemes; ``` Implements: `[SegmenterKernel](#SegmenterKernel)` ## * Reports the lengths of up to `capacity` segments tiling the front of `text`, returning the count; each segment starts where the previous one ends. ```rust unsafe fn segment(t: *const c_void, n: usize, l: *mut usize, c: usize) -> usize ``` Safety: `lengths` must point to at least `capacity` writable `usize` slots, and `text` to `length` readable bytes. ## Linebreaks Kernel behind [`Utf8Linebreaks`](#Utf8Linebreaks) (`sz_utf8_linebreaks_best`). ```rust pub struct Linebreaks; ``` Implements: `[SegmenterKernel](#SegmenterKernel)` ## Utf8View Lazy UTF-8 character view with SIMD-accelerated operations. ```rust pub struct Utf8View<'a> ``` Provides O(1) construction, character counting on demand, and efficient random access. Examples: ``` use stringzilla as sz; let text = "Hello๐ŸŒ"; let view = sz::Utf8View::new(text.as_bytes()); // Character count, each call counting anew assert_eq!(view.len(), 6); // Random access to byte offset of Nth character assert_eq!(view.offset_of(5), Some(5)); // ๐ŸŒ at byte 5 // Iterate over characters let chars: Vec<char> = view.iter().collect(); assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '๐ŸŒ']); ``` ## new Creates a new UTF-8 view (O(1) - no scanning). ```rust pub const fn new(octets: &'a [u8]) -> Self ``` ## len Returns the number of UTF-8 characters, counting them on every call. ```rust pub fn len(&self) -> usize ``` ## is_empty Checks if the view is empty. ```rust pub const fn is_empty(&self) -> bool ``` ## offset_of Gets the byte offset of the Nth character (0-indexed, SIMD-accelerated). ```rust pub fn offset_of(&self, n: usize) -> Option ``` ## iter Returns an iterator over UTF-8 characters. ```rust pub fn iter(&self) -> Utf8Runes<'a> ``` ## Utf8Runes Iterator over UTF-8 characters using batched decoding. ```rust pub struct Utf8Runes<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> ``` Each refill decodes up to `STEPS` codepoints in a single `sz_utf8_decode_best` FFI call (the decoder fills the whole buffer regardless of script width), then yields them one at a time - far cheaper than decoding character-by-character. Ill-formed bytes decode to the replacement character U+FFFD, so iteration is total and never silently truncates. Typically created through [`Utf8View::iter()`](https://www.unum.cloud/docs/stringzilla/rust/runtime-dispatch.md#iter). Examples: ``` use stringzilla as sz; let text = "Hello๐ŸŒ"; let view = sz::Utf8View::new(text.as_bytes()); let chars: Vec<char> = view.iter().collect(); assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '๐ŸŒ']); ``` Implements: `Iterator` ## with_steps Constructs an iterator buffering up to `STEPS` codepoints per FFI call. ```rust pub fn with_steps(octets: &'a [u8]) -> Self ``` ## Sentences Kernel behind [`Utf8Sentences`](#Utf8Sentences) (`sz_utf8_sentences_best`). ```rust pub struct Sentences; ``` Implements: `[SegmenterKernel](#SegmenterKernel)` ## TokenizerKernel A zero-sized selector of a UTF-8 kernel reporting separator runs. ```rust pub trait TokenizerKernel ``` Each implementor binds one FFI tokenizer, so the shared [`Utf8Split`](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#Utf8Split) iterator monomorphizes to a direct, branch-free call rather than a function pointer. Implementors: - `impl [TokenizerKernel](#TokenizerKernel) for [Newlines](#Newlines)` - `impl [TokenizerKernel](#TokenizerKernel) for [Whitespaces](#Whitespaces)` - `impl [TokenizerKernel](#TokenizerKernel) for [Delimiters](#Delimiters)` ## * Reports up to `capacity` separator runs of `text` into `offsets` / `lengths`, returning the count and writing the number of consumed bytes to `consumed`: the end of the last run when the count reaches `capacity`, otherwise `length`. ```rust unsafe fn tokenize( text: *const c_void, length: usize, offsets: *mut usize, lengths: *mut usize, capacity: usize, consumed: *mut usize, ) -> usize ``` Safety: `offsets` and `lengths` must each point to at least `capacity` writable `usize` slots, and `text` to `length` readable bytes. ## SegmenterKernel A zero-sized selector of a UTF-8 kernel tiling text into segments. ```rust pub trait SegmenterKernel ``` Each implementor binds one FFI segmenter, so the shared [`Utf8Segments`](#Utf8Segments) iterator monomorphizes to a direct call. Implementors: - `impl [SegmenterKernel](#SegmenterKernel) for [Graphemes](#Graphemes)` - `impl [SegmenterKernel](#SegmenterKernel) for [Linebreaks](#Linebreaks)` - `impl [SegmenterKernel](#SegmenterKernel) for [Sentences](#Sentences)` - `impl [SegmenterKernel](#SegmenterKernel) for [Wordbreaks](#Wordbreaks)` ## Newlines Kernel behind [`Utf8SplitNewlines`](#Utf8SplitNewlines) (`sz_utf8_newlines_best`). ```rust pub struct Newlines; ``` Implements: `[TokenizerKernel](#TokenizerKernel)` ## Whitespaces Kernel behind [`Utf8SplitWhitespaces`](#Utf8SplitWhitespaces) (`sz_utf8_whitespaces_best`). ```rust pub struct Whitespaces; ``` Implements: `[TokenizerKernel](#TokenizerKernel)` ## Delimiters Kernel behind [`Utf8SplitDelimiters`](#Utf8SplitDelimiters) (`sz_utf8_delimiters_best`). ```rust pub struct Delimiters; ``` Implements: `[TokenizerKernel](#TokenizerKernel)` ## Between Yields the segments between separators, the default. ```rust pub struct Between; ``` Implements: `[SplitParts](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#SplitParts)` ## Separators Yields the separator runs themselves. ```rust pub struct Separators; ``` Implements: `[SplitParts](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#SplitParts)` ## Both Yields segments and separators interleaved, so concatenating them reproduces the input. ```rust pub struct Both; ``` Implements: `[SplitParts](https://www.unum.cloud/docs/stringzilla/rust/splitting-and-partitioning.md#SplitParts)` ## Utf8Segments An iterator over the segments a kernel tiles UTF-8 text into, in order. ```rust pub struct Utf8Segments<'a, Kernel: SegmenterKernel, const STEPS: usize = ITERATORS_DEFAULT_STEPS> ``` Implements: `Iterator` ## new Constructs an iterator with the default batch size ([`ITERATORS_DEFAULT_STEPS`](https://www.unum.cloud/docs/stringzilla/rust/types.md#ITERATORS_DEFAULT_STEPS)). ```rust pub fn new(text: &'a [u8]) -> Self ``` For an explicit batch size use [`Self::with_steps`](#with_steps) with a turbofish, e.g. `Utf8Wordbreaks::<1>::with_steps(text)`. ## with_steps Constructs an iterator buffering up to `STEPS` segments per FFI call. ```rust pub fn with_steps(text: &'a [u8]) -> Self ``` ## Wordbreaks Kernel behind [`Utf8Wordbreaks`](#Utf8Wordbreaks) (`sz_utf8_wordbreaks_best`). ```rust pub struct Wordbreaks; ``` Implements: `[SegmenterKernel](#SegmenterKernel)` ## utf8_decode Unpacks a UTF-8 byte sequence into UTF-32 codepoints. ```rust pub fn utf8_decode(text: &[u8], runes: &mut [u32]) -> (usize, usize) ``` This function decodes UTF-8 encoded text into individual Unicode codepoints, storing them in a u32 array. It fills the output buffer, or drains the input, in a single call, looping internally regardless of how many byte-widths the text mixes. Ill-formed bytes decode to the replacement character U+FFFD, one per maximal ill-formed subpart, so every written value is a valid Unicode scalar value. The end of `text` is the end of the input: a well-formed but truncated final sequence decodes to one U+FFFD and is consumed, so a caller decoding a stream in chunks must cut them at rune boundaries. #### Returns A tuple `(bytes_consumed, runes_count)` where: - `bytes_consumed` is the number of bytes processed from `text` - `runes_count` is the number of codepoints written to `runes` Arguments: - `text`: The UTF-8 encoded byte slice to decode. - `runes`: Output buffer to store decoded codepoints. Examples: Processing pure ASCII text (most common case, single chunk): ``` use stringzilla as sz; let text = "Hello World!"; let mut runes = [0u32; 16]; let (bytes, count) = sz::utf8_decode(text.as_bytes(), &mut runes); assert_eq!(count, 12); // All 12 ASCII characters assert_eq!(bytes, 12); // 12 bytes consumed assert_eq!(runes[0], 'H' as u32); assert_eq!(runes[11], '!' as u32); ``` Each call fills the output buffer or drains the input; call repeatedly (resuming at `bytes_consumed`) to process a string longer than the buffer: ``` use stringzilla as sz; let text = "Hiไธ–็•Œ"; // 2 ASCII + 2 CJK let bytes = text.as_bytes(); let mut runes = [0u32; 16]; let mut all_runes = Vec::new(); let mut offset = 0; while offset < bytes.len() { let (consumed, count) = sz::utf8_decode(&bytes[offset..], &mut runes); all_runes.extend_from_slice(&runes[..count]); offset += consumed; } assert_eq!(all_runes.len(), 4); // 2 ASCII + 2 CJK = 4 codepoints ```