A single emoji such as the flag πΊπΈ is 8 bytes and 2 codepoints, yet a reader sees one character β one grapheme cluster.
StringZilla draws every one of these boundaries: sz_utf8_runes (Utf8View) walks codepoints, while sz_utf8_graphemes (Utf8Graphemes) walks user-perceived clusters.
The StringZillableUnary trait turns any AsRef<[u8]> into a broad family of lazy UTF-8 iterators and views, each yielding borrowed byte sub-slices or chars on demand:
fn sz_utf8_runes(&self) -> Utf8View<'_>; // codepoints / random codepoint accessfn sz_utf8_split_newlines(&self) -> Utf8SplitNewlines<'_>; // content BETWEEN hard newlines (CRLF = one)fn sz_utf8_newlines(&self) -> Utf8Newlines<'_>; // the newline runs themselvesfn sz_utf8_split_whitespaces(&self) -> Utf8SplitWhitespaces<'_>; // content BETWEEN Unicode whitespacefn sz_utf8_whitespaces(&self) -> Utf8Whitespaces<'_>; // the whitespace runs themselvesfn sz_utf8_split_delimiters(&self) -> Utf8SplitDelimiters<'_>; // content BETWEEN whitespace/punctuationfn sz_utf8_delimiters(&self) -> Utf8Delimiters<'_>; // the delimiter runs themselvesfn sz_utf8_wordbreaks(&self) -> Utf8Wordbreaks<'_>; // all UAX-29 word segments (tiling)fn sz_utf8_graphemes(&self) -> Utf8Graphemes<'_>; // UAX-29 grapheme clustersfn sz_utf8_sentences(&self) -> Utf8Sentences<'_>; // UAX-29 sentencesfn sz_utf8_linebreaks(&self) -> Utf8Linebreaks<'_>; // UAX-14 line-break opportunities
The naming follows one rule: the bare name (newlines/whitespaces/delimiters) yields the separators the kernel finds, while split_* yields the content between them.
Chain .with_separators() on a split_* iterator to interleave both losslessly.
Every member of this family is lazy and zero-copy. Utf8View, Utf8Runes, Utf8SplitNewlines, Utf8SplitWhitespaces, Utf8Wordbreaks, Utf8Graphemes, Utf8Sentences, and Utf8Linebreaks, together with the sz_splits / sz_rsplits iterators, all borrow from the source string and yield &[u8] or &str slices on demand without allocating.
There is no backing vector and no per-element heap buffer.
Contrast this with the standard library, where collecting into a Vec<String> allocates the vector and a fresh heap buffer for every element, and even a Vec<&str> allocates the backing vector up front.
Because these iterators borrow from the input, you can stream over millions of words or grapheme clusters of a large document with effectively zero per-element allocation, and the borrow lifetimes keep the yielded slices zero-copy.
Utf8View offers O(1) construction, len() counting the codepoints on demand, offset_of(n) for the byte offset of the Nth codepoint, and iter() for batched char iteration:
The boundary iterators wordbreaks, graphemes, sentences, and linebreakstile the input β every byte belongs to exactly one segment, so consecutive segments are contiguous and no empty slices appear:
split_newlines splits on the 8 Unicode hard newlines, treating CRLF as a single delimiter and excluding the newline from each segment, while split_whitespaces splits on the 25 Unicode whitespace characters and keeps empty segments by default; chain .skip_empty() for str::split_whitespace-style behavior:
use stringzilla::StringZillableUnary;let lines: Vec<&str> = "Hello\nWorld\r\nRust" .sz_utf8_split_newlines() .map(|l| std::str::from_utf8(l).unwrap()) .collect();assert_eq!(lines, vec!["Hello", "World", "Rust"]);let tokens: Vec<&[u8]> = b" hi ".sz_utf8_split_whitespaces().skip_empty().collect();assert_eq!(tokens, vec![&b"hi"[..]]);
Every iterator is also constructible directly, for example Utf8Wordbreaks::new(text).
Each exposes a STEPS const-generic batch-size knob via with_steps, written with a turbofish such as Utf8Runes::<256>::with_steps(bytes), that trades buffer size for FFI-call amortization without changing the yielded results.
The default batch size is the public constant ITERATORS_DEFAULT_STEPS = 64.
utf8_decode decodes UTF-8 into UTF-32 codepoints, filling the output buffer or draining the input per call.
For inputs larger than the buffer, loop and resume at bytes_consumed.
It is total: ill-formed bytes decode to U+FFFD, so every value written is a valid Unicode scalar.
The end of text is the end of the input, so a truncated final sequence also decodes to one U+FFFD and is consumed; cut a streamed input into chunks at rune boundaries:
use stringzilla as sz;let mut runes = [0u32; 16];assert_eq!(sz::utf8_decode(b"a\xE2\x82", &mut runes), (3, 2));assert_eq!(&runes[..2], &['a' as u32, 0xFFFD]);let (bytes, count) = sz::utf8_decode("Hello".as_bytes(), &mut runes);assert_eq!((bytes, count), (5, 5));assert_eq!(runes[0], 'H' as u32);
API
Types
Utf8GraphemesAn iterator over UAX-29 grapheme clusters in UTF-8 text, in order.
Unlike whitespace splitting, the grapheme clusters tile the input: every byte belongs to exactly
one grapheme cluster, so consecutive clusters are contiguous and no empty slices are produced.
Follows the Unicode UAX-29 rules.
Examples
use stringzilla::Utf8Graphemes;let graphemes: Vec<&[u8]> = Utf8Graphemes::new(b"Hi!").collect();assert_eq!(graphemes, vec![&b"H"[..], &b"i"[..], &b"!"[..]]);
Utf8LinebreaksAn iterator over UAX-14 line break opportunities in UTF-8 text, in order.
Unlike whitespace splitting, the lines tile the input: every byte belongs to exactly one
line, so consecutive lines are contiguous and no empty slices are produced. Follows the
Unicode TR14 rules.
Each yielded segment ends at a TR14 break opportunity, including soft breaks where a renderer
may wrap but is not required to. To split only on hard line breaks (the "splitlines"
behaviour), use the newline API (StringZillableUnary::sz_utf8_split_newlines) instead.
Examples
use stringzilla::Utf8Linebreaks;let lines: Vec<&[u8]> = Utf8Linebreaks::new(b"Hi\nBye").collect();assert_eq!(lines, vec![&b"Hi\n"[..], &b"Bye"[..]]);
Utf8SentencesAn iterator over UAX-29 sentences in UTF-8 text, in order.
Unlike whitespace splitting, the sentences tile the input: every byte belongs to exactly one
sentence, so consecutive sentences are contiguous and no empty slices are produced. Follows the
Unicode UAX-29 rules.
Examples
use stringzilla::Utf8Sentences;let sentences: Vec<&[u8]> = Utf8Sentences::new(b"Hi. Bye.").collect();assert_eq!(sentences, vec![&b"Hi. "[..], &b"Bye."[..]]);
Utf8SplitNewlinesAn iterator over substrings of UTF-8 text split by newline characters.
This iterator yields slices between newline characters. The newline characters themselves
are not included in the yielded slices. Handles all 8 Unicode newline characters including
CRLF as a single delimiter.
Examples
use stringzilla::{Utf8SplitNewlines};let text = b"Hello\nWorld\r\nRust";let lines: Vec<&[u8]> = Utf8SplitNewlines::new(text).collect();assert_eq!(lines, vec![&b"Hello"[..], &b"World"[..], &b"Rust"[..]]);
Utf8NewlinesAn iterator over the newline runs themselves, the separators, in order.
Splits on all 25 Unicode "White_Space" characters; N whitespace delimiters yield N+1 segments.
By default empty segments are kept, matching str::split, so runs of whitespace and
leading/trailing whitespace produce empty slices. Call Self::skip_empty for the
str::split_whitespace-style behavior that drops empties and yields only non-empty tokens.
Examples
use stringzilla::{Utf8SplitWhitespaces};// Default KEEP policy: empties around the words are preserved.let text = b" hi ";let segments: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).collect();assert_eq!(segments, vec![&b""[..], &b""[..], &b"hi"[..], &b""[..], &b""[..]]);// Opt in to dropping empties for token-style splitting.let tokens: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).skip_empty().collect();assert_eq!(tokens, vec![&b"hi"[..]]);
Utf8WhitespacesAn iterator over the whitespace runs themselves, the separators, in order.
Splits on every codepoint whose Unicode general category is punctuation (P*), symbol (S*),
or separator/whitespace (Z*) β the superset of Utf8SplitWhitespaces. N delimiters yield
N+1 segments; empty segments are kept by default (call Self::skip_empty to drop them
for token-style splitting).
Examples
use stringzilla::{Utf8SplitDelimiters};// "Hi, worldβfoo" splits on ',', ' ', and U+2014 EM DASH.let tokens: Vec<&[u8]> = Utf8SplitDelimiters::new("Hi, world\u{2014}foo".as_bytes()).skip_empty().collect();assert_eq!(tokens, vec![&b"Hi"[..], &b"world"[..], &b"foo"[..]]);
Utf8DelimitersAn iterator over the delimiter runs themselves, the separators, in order.
Unlike whitespace splitting, the words tile the input: every byte belongs to exactly one
word, so consecutive words are contiguous and no empty slices are produced. Follows the
Unicode UAX-29 rules.
This function is designed to provide a high-quality hash value for use in
hash tables, data structures, and cryptographic applications.
Unlike the bytesum function, the hash function is order-sensitive.
Examples
use stringzilla::StringZillableUnary;let s1 = b"Hello";let s2 = b"World";assert_ne!(s1.sz_hash(), s2.sz_hash());
sz_utf8_runesReturns a lazy UTF-8 character view with SIMD-accelerated operations.
Handles all 25 Unicode "White_Space" characters; N delimiters yield N+1 segments. By default
empty segments are kept, matching str::split, so runs of whitespace surface empty
slices. Chain .skip_empty() on the returned iterator to recover the
str::split_whitespace token behavior that drops empties.
Examples
use stringzilla::StringZillableUnary;// KEEP (default): the double space between "Hello" and "World" yields an empty segment.let text = "Hello World\tRust";let segments: Vec<&str> = text.sz_utf8_split_whitespaces() .map(|segment| std::str::from_utf8(segment).unwrap()) .collect();assert_eq!(segments, vec!["Hello", "", "World", "Rust"]);// skip_empty: drops the empties to yield only the non-empty tokens.let tokens: Vec<&str> = text.sz_utf8_split_whitespaces() .skip_empty() .map(|token| std::str::from_utf8(token).unwrap()) .collect();assert_eq!(tokens, vec!["Hello", "World", "Rust"]);
sz_utf8_whitespacesReturns an iterator over the whitespace runs themselves, the separators.
Provides O(1) construction, character counting on demand, and efficient random access.
Examples
use stringzilla as sz;let text = "Helloπ";let view = sz::Utf8View::new(text.as_bytes());// Character count, each call counting anewassert_eq!(view.len(), 6);// Random access to byte offset of Nth characterassert_eq!(view.offset_of(5), Some(5)); // π at byte 5// Iterate over characterslet chars: Vec<char> = view.iter().collect();assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', 'π']);
Each refill decodes up to STEPS codepoints in a single sz_utf8_decode_best FFI call (the
decoder fills the whole buffer regardless of script width), then yields them one at a time -
far cheaper than decoding character-by-character. Ill-formed bytes decode to the replacement
character U+FFFD, so iteration is total and never silently truncates.
*Γ 4Reports up to capacity separator runs of text into offsets / lengths, returning the count and writing the number of consumed bytes to consumed: the end of the last run when the count reaches capacity, otherwise length.
This function decodes UTF-8 encoded text into individual Unicode codepoints, storing them in a
u32 array. It fills the output buffer, or drains the input, in a single call, looping internally
regardless of how many byte-widths the text mixes. Ill-formed bytes decode to the replacement
character U+FFFD, one per maximal ill-formed subpart, so every written value is a valid Unicode
scalar value. The end of text is the end of the input: a well-formed but truncated final
sequence decodes to one U+FFFD and is consumed, so a caller decoding a stream in chunks must cut
them at rune boundaries.
Returns
A tuple (bytes_consumed, runes_count) where:
bytes_consumed is the number of bytes processed from text
runes_count is the number of codepoints written to runes
Arguments
text: The UTF-8 encoded byte slice to decode.
runes: Output buffer to store decoded codepoints.
Examples
Processing pure ASCII text (most common case, single chunk):
use stringzilla as sz;let text = "Hello World!";let mut runes = [0u32; 16];let (bytes, count) = sz::utf8_decode(text.as_bytes(), &mut runes);assert_eq!(count, 12); // All 12 ASCII charactersassert_eq!(bytes, 12); // 12 bytes consumedassert_eq!(runes[0], 'H' as u32);assert_eq!(runes[11], '!' as u32);
Each call fills the output buffer or drains the input; call repeatedly (resuming at
bytes_consumed) to process a string longer than the buffer: