Rust Β· Unicode Segmentation

Unicode Segmentation

use stringzilla::StringZillableUnary;use stringzilla::Graphemes;use stringzilla::Linebreaks;use stringzilla::Utf8View;use stringzilla::Utf8Runes;use stringzilla::Sentences;use stringzilla::TokenizerKernel;use stringzilla::SegmenterKernel;use stringzilla::Newlines;use stringzilla::Whitespaces;use stringzilla::Delimiters;use stringzilla::Between;use stringzilla::Separators;use stringzilla::Both;use stringzilla::Utf8Segments;use stringzilla::Wordbreaks;use stringzilla::*; 6.0.0-rc.1 Β· 91a27bdd

A single emoji such as the flag πŸ‡ΊπŸ‡Έ is 8 bytes and 2 codepoints, yet a reader sees one character β€” one grapheme cluster. StringZilla draws every one of these boundaries: sz_utf8_runes (Utf8View) walks codepoints, while sz_utf8_graphemes (Utf8Graphemes) walks user-perceived clusters. The StringZillableUnary trait turns any AsRef<[u8]> into a broad family of lazy UTF-8 iterators and views, each yielding borrowed byte sub-slices or chars on demand:

fn sz_utf8_runes(&self) -> Utf8View<'_>;                         // codepoints / random codepoint access
fn sz_utf8_split_newlines(&self) -> Utf8SplitNewlines<'_>;       // content BETWEEN hard newlines (CRLF = one)
fn sz_utf8_newlines(&self) -> Utf8Newlines<'_>;                  // the newline runs themselves
fn sz_utf8_split_whitespaces(&self) -> Utf8SplitWhitespaces<'_>; // content BETWEEN Unicode whitespace
fn sz_utf8_whitespaces(&self) -> Utf8Whitespaces<'_>;            // the whitespace runs themselves
fn sz_utf8_split_delimiters(&self) -> Utf8SplitDelimiters<'_>;   // content BETWEEN whitespace/punctuation
fn sz_utf8_delimiters(&self) -> Utf8Delimiters<'_>;              // the delimiter runs themselves
fn sz_utf8_wordbreaks(&self) -> Utf8Wordbreaks<'_>;              // all UAX-29 word segments (tiling)
fn sz_utf8_graphemes(&self) -> Utf8Graphemes<'_>;                // UAX-29 grapheme clusters
fn sz_utf8_sentences(&self) -> Utf8Sentences<'_>;                // UAX-29 sentences
fn sz_utf8_linebreaks(&self) -> Utf8Linebreaks<'_>;              // UAX-14 line-break opportunities

The naming follows one rule: the bare name (newlines/whitespaces/delimiters) yields the separators the kernel finds, while split_* yields the content between them. Chain .with_separators() on a split_* iterator to interleave both losslessly.

Every member of this family is lazy and zero-copy. Utf8View, Utf8Runes, Utf8SplitNewlines, Utf8SplitWhitespaces, Utf8Wordbreaks, Utf8Graphemes, Utf8Sentences, and Utf8Linebreaks, together with the sz_splits / sz_rsplits iterators, all borrow from the source string and yield &[u8] or &str slices on demand without allocating. There is no backing vector and no per-element heap buffer. Contrast this with the standard library, where collecting into a Vec<String> allocates the vector and a fresh heap buffer for every element, and even a Vec<&str> allocates the backing vector up front. Because these iterators borrow from the input, you can stream over millions of words or grapheme clusters of a large document with effectively zero per-element allocation, and the borrow lifetimes keep the yielded slices zero-copy.

Utf8View offers O(1) construction, len() counting the codepoints on demand, offset_of(n) for the byte offset of the Nth codepoint, and iter() for batched char iteration:

use stringzilla::StringZillableUnary;

let view = "Hello🌍".sz_utf8_runes();
assert_eq!(view.len(), 6);
assert_eq!(view.offset_of(5), Some(5));
let chars: Vec<char> = view.iter().collect();
assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '🌍']);

The boundary iterators wordbreaks, graphemes, sentences, and linebreaks tile the input β€” every byte belongs to exactly one segment, so consecutive segments are contiguous and no empty slices appear:

use stringzilla::StringZillableUnary;

let words: Vec<&[u8]> = b"Hi, world".sz_utf8_wordbreaks().collect();
assert_eq!(words, vec![&b"Hi"[..], &b","[..], &b" "[..], &b"world"[..]]);

let sentences: Vec<&[u8]> = b"Hi. Bye.".sz_utf8_sentences().collect();
assert_eq!(sentences, vec![&b"Hi. "[..], &b"Bye."[..]]);

split_newlines splits on the 8 Unicode hard newlines, treating CRLF as a single delimiter and excluding the newline from each segment, while split_whitespaces splits on the 25 Unicode whitespace characters and keeps empty segments by default; chain .skip_empty() for str::split_whitespace-style behavior:

use stringzilla::StringZillableUnary;

let lines: Vec<&str> = "Hello\nWorld\r\nRust"
    .sz_utf8_split_newlines()
    .map(|l| std::str::from_utf8(l).unwrap())
    .collect();
assert_eq!(lines, vec!["Hello", "World", "Rust"]);

let tokens: Vec<&[u8]> = b"  hi  ".sz_utf8_split_whitespaces().skip_empty().collect();
assert_eq!(tokens, vec![&b"hi"[..]]);

Every iterator is also constructible directly, for example Utf8Wordbreaks::new(text). Each exposes a STEPS const-generic batch-size knob via with_steps, written with a turbofish such as Utf8Runes::<256>::with_steps(bytes), that trades buffer size for FFI-call amortization without changing the yielded results. The default batch size is the public constant ITERATORS_DEFAULT_STEPS = 64.

Β§ Low Level Decoding

utf8_decode unpacks UTF-8 bytes into a UTF-32 runes buffer, returning how many bytes were consumed and codepoints written.

pub fn utf8_decode(text: &[u8], runes: &mut [u32]) -> (usize, usize); // (bytes_consumed, runes_count)

utf8_decode decodes UTF-8 into UTF-32 codepoints, filling the output buffer or draining the input per call. For inputs larger than the buffer, loop and resume at bytes_consumed. It is total: ill-formed bytes decode to U+FFFD, so every value written is a valid Unicode scalar. The end of text is the end of the input, so a truncated final sequence also decodes to one U+FFFD and is consumed; cut a streamed input into chunks at rune boundaries:

use stringzilla as sz;

let mut runes = [0u32; 16];
assert_eq!(sz::utf8_decode(b"a\xE2\x82", &mut runes), (3, 2));
assert_eq!(&runes[..2], &['a' as u32, 0xFFFD]);
let (bytes, count) = sz::utf8_decode("Hello".as_bytes(), &mut runes);
assert_eq!((bytes, count), (5, 5));
assert_eq!(runes[0], 'H' as u32);

API

Types

Utf8GraphemesAn iterator over UAX-29 grapheme clusters in UTF-8 text, in order.
utf8_graphemes.rs:33Β β†—
pub type Utf8Graphemes<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Graphemes, STEPS>;

Unlike whitespace splitting, the grapheme clusters tile the input: every byte belongs to exactly one grapheme cluster, so consecutive clusters are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules.

Examples

use stringzilla::Utf8Graphemes;

let graphemes: Vec<&[u8]> = Utf8Graphemes::new(b"Hi!").collect();
assert_eq!(graphemes, vec![&b"H"[..], &b"i"[..], &b"!"[..]]);
Utf8LinebreaksAn iterator over UAX-14 line break opportunities in UTF-8 text, in order.
utf8_linebreaks.rs:37Β β†—
pub type Utf8Linebreaks<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Linebreaks, STEPS>;

Unlike whitespace splitting, the lines tile the input: every byte belongs to exactly one line, so consecutive lines are contiguous and no empty slices are produced. Follows the Unicode TR14 rules.

Each yielded segment ends at a TR14 break opportunity, including soft breaks where a renderer may wrap but is not required to. To split only on hard line breaks (the "splitlines" behaviour), use the newline API (StringZillableUnary::sz_utf8_split_newlines) instead.

Examples

use stringzilla::Utf8Linebreaks;

let lines: Vec<&[u8]> = Utf8Linebreaks::new(b"Hi\nBye").collect();
assert_eq!(lines, vec![&b"Hi\n"[..], &b"Bye"[..]]);
Utf8SentencesAn iterator over UAX-29 sentences in UTF-8 text, in order.
utf8_sentences.rs:33Β β†—
pub type Utf8Sentences<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Sentences, STEPS>;

Unlike whitespace splitting, the sentences tile the input: every byte belongs to exactly one sentence, so consecutive sentences are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules.

Examples

use stringzilla::Utf8Sentences;

let sentences: Vec<&[u8]> = Utf8Sentences::new(b"Hi. Bye.").collect();
assert_eq!(sentences, vec![&b"Hi. "[..], &b"Bye."[..]]);
Utf8SplitNewlinesAn iterator over substrings of UTF-8 text split by newline characters.
utf8_tokens.rs:309Β β†—
pub type Utf8SplitNewlines<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Newlines, Between, KeepEmpty, STEPS>;

This iterator yields slices between newline characters. The newline characters themselves are not included in the yielded slices. Handles all 8 Unicode newline characters including CRLF as a single delimiter.

Examples

use stringzilla::{Utf8SplitNewlines};

let text = b"Hello\nWorld\r\nRust";
let lines: Vec<&[u8]> = Utf8SplitNewlines::new(text).collect();
assert_eq!(lines, vec![&b"Hello"[..], &b"World"[..], &b"Rust"[..]]);
Utf8NewlinesAn iterator over the newline runs themselves, the separators, in order.
utf8_tokens.rs:313Β β†—
pub type Utf8Newlines<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Newlines, Separators, KeepEmpty, STEPS>;
Utf8SplitWhitespacesAn iterator over segments of UTF-8 text split by whitespace characters.
utf8_tokens.rs:337Β β†—
pub type Utf8SplitWhitespaces<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Whitespaces, Between, KeepEmpty, STEPS>;

Splits on all 25 Unicode "White_Space" characters; N whitespace delimiters yield N+1 segments. By default empty segments are kept, matching str::split, so runs of whitespace and leading/trailing whitespace produce empty slices. Call Self::skip_empty for the str::split_whitespace-style behavior that drops empties and yields only non-empty tokens.

Examples

use stringzilla::{Utf8SplitWhitespaces};

// Default KEEP policy: empties around the words are preserved.
let text = b"  hi  ";
let segments: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).collect();
assert_eq!(segments, vec![&b""[..], &b""[..], &b"hi"[..], &b""[..], &b""[..]]);

// Opt in to dropping empties for token-style splitting.
let tokens: Vec<&[u8]> = Utf8SplitWhitespaces::new(text).skip_empty().collect();
assert_eq!(tokens, vec![&b"hi"[..]]);
Utf8WhitespacesAn iterator over the whitespace runs themselves, the separators, in order.
utf8_tokens.rs:341Β β†—
pub type Utf8Whitespaces<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Whitespaces, Separators, KeepEmpty, STEPS>;
Utf8SplitDelimitersAn iterator over segments of UTF-8 text split by any Unicode delimiter codepoint.
utf8_tokens.rs:360Β β†—
pub type Utf8SplitDelimiters<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Delimiters, Between, KeepEmpty, STEPS>;

Splits on every codepoint whose Unicode general category is punctuation (P*), symbol (S*), or separator/whitespace (Z*) β€” the superset of Utf8SplitWhitespaces. N delimiters yield N+1 segments; empty segments are kept by default (call Self::skip_empty to drop them for token-style splitting).

Examples

use stringzilla::{Utf8SplitDelimiters};

// "Hi, worldβ€”foo" splits on ',', ' ', and U+2014 EM DASH.
let tokens: Vec<&[u8]> = Utf8SplitDelimiters::new("Hi, world\u{2014}foo".as_bytes()).skip_empty().collect();
assert_eq!(tokens, vec![&b"Hi"[..], &b"world"[..], &b"foo"[..]]);
Utf8DelimitersAn iterator over the delimiter runs themselves, the separators, in order.
utf8_tokens.rs:364Β β†—
pub type Utf8Delimiters<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Split<'a, Delimiters, Separators, KeepEmpty, STEPS>;
Utf8WordbreaksAn iterator over UAX-29 words in UTF-8 text, in order.
utf8_wordbreaks.rs:33Β β†—
pub type Utf8Wordbreaks<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS> = Utf8Segments<'a, Wordbreaks, STEPS>;

Unlike whitespace splitting, the words tile the input: every byte belongs to exactly one word, so consecutive words are contiguous and no empty slices are produced. Follows the Unicode UAX-29 rules.

Examples

use stringzilla::Utf8Wordbreaks;

let words: Vec<&[u8]> = Utf8Wordbreaks::new(b"Hi, world").collect();
assert_eq!(words, vec![&b"Hi"[..], &b","[..], &b" "[..], &b"world"[..]]);
TraitStringZillableUnaryTrait for unary string operations that only operate on self without needle parameters.13
stringzilla.rs:909Β β†—
pub trait StringZillableUnary

These operations include hash computation and byte sum calculation.

Examples

Basic usage on a byte slice:

use stringzilla::StringZillableUnary;

let text = b"Hello";
assert_eq!(text.sz_bytesum(), 500);

Implementors

sz_bytesumComputes the bytesum value of unsigned bytes in a given string.
stringzilla.rs:922Β β†—
fn sz_bytesum(&self) -> u64

This function is useful for verifying data integrity and detecting changes in binary data, such as files or network packets.

Examples

use stringzilla::StringZillableUnary;

let text = b"Hello";
assert_eq!(text.sz_bytesum(), 500);
sz_hashComputes a 64-bit AES-based hash value for a given string.
stringzilla.rs:938Β β†—
fn sz_hash(&self) -> u64

This function is designed to provide a high-quality hash value for use in hash tables, data structures, and cryptographic applications. Unlike the bytesum function, the hash function is order-sensitive.

Examples

use stringzilla::StringZillableUnary;

let s1 = b"Hello";
let s2 = b"World";
assert_ne!(s1.sz_hash(), s2.sz_hash());
sz_utf8_runesReturns a lazy UTF-8 character view with SIMD-accelerated operations.
stringzilla.rs:965Β β†—
fn sz_utf8_runes(&self) -> Utf8View<'_>

The view provides:

  • .len() for character count, computed on demand
  • .offset_of(n) for random access to Nth character offset
  • .iter() for efficient batched iteration over characters

Examples

use stringzilla::StringZillableUnary;

let text = "Hello🌍";
let view = text.sz_utf8_runes();

// Lazy character count
assert_eq!(view.len(), 6);

// Random access (byte offset of Nth character)
assert_eq!(view.offset_of(5), Some(5)); // 🌍 at byte 5

// Efficient batched iteration
let chars: Vec<char> = view.iter().collect();
assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '🌍']);
sz_utf8_split_newlinesReturns an iterator over lines split by UTF-8 newline characters.
stringzilla.rs:983Β β†—
fn sz_utf8_split_newlines(&self) -> Utf8SplitNewlines<'_>

The iterator yields slices between newlines. Handles all Unicode newline characters including CRLF as a single delimiter.

Examples

use stringzilla::StringZillableUnary;

let text = "Hello\nWorld\r\nRust";
let lines: Vec<&str> = text.sz_utf8_split_newlines()
    .map(|line| std::str::from_utf8(line).unwrap())
    .collect();
assert_eq!(lines, vec!["Hello", "World", "Rust"]);
sz_utf8_newlinesReturns an iterator over the newline runs themselves, the separators.
stringzilla.rs:986Β β†—
fn sz_utf8_newlines(&self) -> Utf8Newlines<'_>
sz_utf8_split_whitespacesReturns an iterator over segments split by UTF-8 whitespace characters.
stringzilla.rs:1014Β β†—
fn sz_utf8_split_whitespaces(&self) -> Utf8SplitWhitespaces<'_>

Handles all 25 Unicode "White_Space" characters; N delimiters yield N+1 segments. By default empty segments are kept, matching str::split, so runs of whitespace surface empty slices. Chain .skip_empty() on the returned iterator to recover the str::split_whitespace token behavior that drops empties.

Examples

use stringzilla::StringZillableUnary;

// KEEP (default): the double space between "Hello" and "World" yields an empty segment.
let text = "Hello  World\tRust";
let segments: Vec<&str> = text.sz_utf8_split_whitespaces()
    .map(|segment| std::str::from_utf8(segment).unwrap())
    .collect();
assert_eq!(segments, vec!["Hello", "", "World", "Rust"]);

// skip_empty: drops the empties to yield only the non-empty tokens.
let tokens: Vec<&str> = text.sz_utf8_split_whitespaces()
    .skip_empty()
    .map(|token| std::str::from_utf8(token).unwrap())
    .collect();
assert_eq!(tokens, vec!["Hello", "World", "Rust"]);
sz_utf8_whitespacesReturns an iterator over the whitespace runs themselves, the separators.
stringzilla.rs:1017Β β†—
fn sz_utf8_whitespaces(&self) -> Utf8Whitespaces<'_>
sz_utf8_split_delimitersReturns an iterator splitting on any Unicode delimiter (punctuation/symbol/separator/whitespace).
stringzilla.rs:1021Β β†—
fn sz_utf8_split_delimiters(&self) -> Utf8SplitDelimiters<'_>
sz_utf8_delimitersReturns an iterator over the delimiter runs themselves, the separators.
stringzilla.rs:1024Β β†—
fn sz_utf8_delimiters(&self) -> Utf8Delimiters<'_>
sz_utf8_wordbreaksReturns an iterator over UAX-29 words, in order.
stringzilla.rs:1027Β β†—
fn sz_utf8_wordbreaks(&self) -> Utf8Wordbreaks<'_>

Words tile the input contiguously.

sz_utf8_graphemesReturns an iterator over UAX-29 grapheme clusters, in order.
stringzilla.rs:1031Β β†—
fn sz_utf8_graphemes(&self) -> Utf8Graphemes<'_>

Clusters tile the input contiguously.

sz_utf8_sentencesReturns an iterator over UAX-29 sentences, in order.
stringzilla.rs:1034Β β†—
fn sz_utf8_sentences(&self) -> Utf8Sentences<'_>

Sentences tile the input contiguously.

sz_utf8_linebreaksReturns an iterator over UAX-14 line-break opportunities per Unicode TR14, in order.
stringzilla.rs:1039Β β†—
fn sz_utf8_linebreaks(&self) -> Utf8Linebreaks<'_>

Linewrap segments tile the input contiguously, including soft break opportunities. For hard line splits only, use Self::sz_utf8_split_newlines.

StructGraphemesKernel behind Utf8Graphemes (sz_utf8_graphemes_best).1
utf8_graphemes.rs:10Β β†—
pub struct Graphemes;

Implements

*Γ— 5Reports the lengths of up to capacity segments tiling the front of text, returning the count; each segment starts where the previous one ends.
utf8_graphemes.rs:12Β β†—

Inherited from SegmenterKernel

unsafe fn segment(t: *const c_void, n: usize, l: *mut usize, c: usize) -> usize

Safety

lengths must point to at least capacity writable usize slots, and text to length readable bytes.

StructLinebreaksKernel behind Utf8Linebreaks (sz_utf8_linebreaks_best).
utf8_linebreaks.rs:10Β β†—
pub struct Linebreaks;

Implements

StructUtf8ViewLazy UTF-8 character view with SIMD-accelerated operations.5
utf8_runes.rs:207Β β†—
pub struct Utf8View<'a>

Provides O(1) construction, character counting on demand, and efficient random access.

Examples

use stringzilla as sz;

let text = "Hello🌍";
let view = sz::Utf8View::new(text.as_bytes());

// Character count, each call counting anew
assert_eq!(view.len(), 6);

// Random access to byte offset of Nth character
assert_eq!(view.offset_of(5), Some(5)); // 🌍 at byte 5

// Iterate over characters
let chars: Vec<char> = view.iter().collect();
assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '🌍']);
newCreates a new UTF-8 view (O(1) - no scanning).
utf8_runes.rs:213Β β†—
pub const fn new(octets: &'a [u8]) -> Self
lenReturns the number of UTF-8 characters, counting them on every call.
utf8_runes.rs:218Β β†—
pub fn len(&self) -> usize
is_emptyChecks if the view is empty.
utf8_runes.rs:223Β β†—
pub const fn is_empty(&self) -> bool
offset_ofGets the byte offset of the Nth character (0-indexed, SIMD-accelerated).
utf8_runes.rs:228Β β†—
pub fn offset_of(&self, n: usize) -> Option<usize>
iterReturns an iterator over UTF-8 characters.
utf8_runes.rs:233Β β†—
pub fn iter(&self) -> Utf8Runes<'a>
StructUtf8RunesIterator over UTF-8 characters using batched decoding.1
utf8_runes.rs:257Β β†—
pub struct Utf8Runes<'a, const STEPS: usize = ITERATORS_DEFAULT_STEPS>

Each refill decodes up to STEPS codepoints in a single sz_utf8_decode_best FFI call (the decoder fills the whole buffer regardless of script width), then yields them one at a time - far cheaper than decoding character-by-character. Ill-formed bytes decode to the replacement character U+FFFD, so iteration is total and never silently truncates.

Typically created through Utf8View::iter().

Examples

use stringzilla as sz;

let text = "Hello🌍";
let view = sz::Utf8View::new(text.as_bytes());
let chars: Vec<char> = view.iter().collect();
assert_eq!(chars, vec!['H', 'e', 'l', 'l', 'o', '🌍']);

Implements

Iterator
with_stepsConstructs an iterator buffering up to STEPS codepoints per FFI call.
utf8_runes.rs:275Β β†—
pub fn with_steps(octets: &'a [u8]) -> Self
StructSentencesKernel behind Utf8Sentences (sz_utf8_sentences_best).
utf8_sentences.rs:10Β β†—
pub struct Sentences;

Implements

TraitTokenizerKernelA zero-sized selector of a UTF-8 kernel reporting separator runs.1
utf8_tokens.rs:17Β β†—
pub trait TokenizerKernel

Each implementor binds one FFI tokenizer, so the shared Utf8Split iterator monomorphizes to a direct, branch-free call rather than a function pointer.

Implementors

*Γ— 4Reports up to capacity separator runs of text into offsets / lengths, returning the count and writing the number of consumed bytes to consumed: the end of the last run when the count reaches capacity, otherwise length.
utf8_tokens.rs:25Β β†—
unsafe fn tokenize(
    text: *const c_void,
    length: usize,
    offsets: *mut usize,
    lengths: *mut usize,
    capacity: usize,
    consumed: *mut usize,
) -> usize

Safety

offsets and lengths must each point to at least capacity writable usize slots, and text to length readable bytes.

TraitSegmenterKernelA zero-sized selector of a UTF-8 kernel tiling text into segments.
utf8_tokens.rs:37Β β†—
pub trait SegmenterKernel

Each implementor binds one FFI segmenter, so the shared Utf8Segments iterator monomorphizes to a direct call.

Implementors

StructNewlinesKernel behind Utf8SplitNewlines (sz_utf8_newlines_best).
utf8_tokens.rs:48Β β†—
pub struct Newlines;

Implements

StructWhitespacesKernel behind Utf8SplitWhitespaces (sz_utf8_whitespaces_best).
utf8_tokens.rs:69Β β†—
pub struct Whitespaces;

Implements

StructDelimitersKernel behind Utf8SplitDelimiters (sz_utf8_delimiters_best).
utf8_tokens.rs:90Β β†—
pub struct Delimiters;

Implements

StructBetweenYields the segments between separators, the default.
utf8_tokens.rs:120Β β†—
pub struct Between;

Implements

StructSeparatorsYields the separator runs themselves.
utf8_tokens.rs:126Β β†—
pub struct Separators;

Implements

StructBothYields segments and separators interleaved, so concatenating them reproduces the input.
utf8_tokens.rs:132Β β†—
pub struct Both;

Implements

StructUtf8SegmentsAn iterator over the segments a kernel tiles UTF-8 text into, in order.2
utf8_tokens.rs:368Β β†—
pub struct Utf8Segments<'a, Kernel: SegmenterKernel, const STEPS: usize = ITERATORS_DEFAULT_STEPS>

Implements

Iterator
newConstructs an iterator with the default batch size (ITERATORS_DEFAULT_STEPS).
utf8_tokens.rs:380Β β†—
pub fn new(text: &'a [u8]) -> Self

For an explicit batch size use Self::with_steps with a turbofish, e.g. Utf8Wordbreaks::<1>::with_steps(text).

with_stepsConstructs an iterator buffering up to STEPS segments per FFI call.
utf8_tokens.rs:387Β β†—
pub fn with_steps(text: &'a [u8]) -> Self
StructWordbreaksKernel behind Utf8Wordbreaks (sz_utf8_wordbreaks_best).
utf8_wordbreaks.rs:10Β β†—
pub struct Wordbreaks;

Implements

Functions

utf8_decodeUnpacks a UTF-8 byte sequence into UTF-32 codepoints.
utf8_runes.rs:63Β β†—
pub fn utf8_decode(text: &[u8], runes: &mut [u32]) -> (usize, usize)

This function decodes UTF-8 encoded text into individual Unicode codepoints, storing them in a u32 array. It fills the output buffer, or drains the input, in a single call, looping internally regardless of how many byte-widths the text mixes. Ill-formed bytes decode to the replacement character U+FFFD, one per maximal ill-formed subpart, so every written value is a valid Unicode scalar value. The end of text is the end of the input: a well-formed but truncated final sequence decodes to one U+FFFD and is consumed, so a caller decoding a stream in chunks must cut them at rune boundaries.

Returns

A tuple (bytes_consumed, runes_count) where:

  • bytes_consumed is the number of bytes processed from text
  • runes_count is the number of codepoints written to runes

Arguments

  • text: The UTF-8 encoded byte slice to decode.
  • runes: Output buffer to store decoded codepoints.

Examples

Processing pure ASCII text (most common case, single chunk):

use stringzilla as sz;
let text = "Hello World!";
let mut runes = [0u32; 16];
let (bytes, count) = sz::utf8_decode(text.as_bytes(), &mut runes);
assert_eq!(count, 12);  // All 12 ASCII characters
assert_eq!(bytes, 12);  // 12 bytes consumed
assert_eq!(runes[0], 'H' as u32);
assert_eq!(runes[11], '!' as u32);

Each call fills the output buffer or drains the input; call repeatedly (resuming at bytes_consumed) to process a string longer than the buffer:

use stringzilla as sz;
let text = "HiδΈ–η•Œ";  // 2 ASCII + 2 CJK
let bytes = text.as_bytes();
let mut runes = [0u32; 16];
let mut all_runes = Vec::new();
let mut offset = 0;
while offset < bytes.len() {
    let (consumed, count) = sz::utf8_decode(&bytes[offset..], &mut runes);
    all_runes.extend_from_slice(&runes[..count]);
    offset += consumed;
}
assert_eq!(all_runes.len(), 4);  // 2 ASCII + 2 CJK = 4 codepoints