Rust · Splitting and Partitioning

Splitting and Partitioning

use stringzilla::FindSplits;use stringzilla::RFindSplits;use stringzilla::SplitKind;use stringzilla::EmptySegments;use stringzilla::KeepEmpty;use stringzilla::SkipEmpty;use stringzilla::SplitParts;use stringzilla::Utf8Split; 6.0.0-rc.1 · 91a27bdd

The crate exposes iterator-based match and split operations. The StringZillableBinary trait yields these iterators; the underlying FindMatches, RFindMatches, FindSplits, and RFindSplits structs can also be constructed directly from a Matcher.

Trait methods, each taking a borrowed needle &'a N:

fn sz_matches(&'a self, needle: &'a N) -> FindMatches<'a>;
fn sz_rmatches(&'a self, needle: &'a N) -> RFindMatches<'a>;
fn sz_splits(&'a self, needle: &'a N) -> FindSplits<'a>;
fn sz_rsplits(&'a self, needle: &'a N) -> RFindSplits<'a>;
fn sz_find_first_of(&'a self, needles: &'a N) -> FindMatches<'a>;
fn sz_find_last_of(&'a self, needles: &'a N) -> RFindMatches<'a>;
fn sz_find_first_not_of(&'a self, needles: &'a N) -> FindMatches<'a>;
fn sz_find_last_not_of(&'a self, needles: &'a N) -> RFindMatches<'a>;

Matching yields the matched sub-slices; splitting yields the segments between matches:

use stringzilla::StringZillableBinary;

let haystack = b"hello world hello universe";
let needle = b"hello";
let fwd: Vec<&[u8]> = haystack.sz_matches(needle).collect();
let rev: Vec<&[u8]> = haystack.sz_rmatches(needle).collect();
assert_eq!(fwd, vec![b"hello", b"hello"]);
assert_eq!(rev, vec![b"hello", b"hello"]);

let csv = b"a,b,c,d";
let splits: Vec<&[u8]> = csv.sz_splits(b",").collect();
let rsplits: Vec<&[u8]> = csv.sz_rsplits(b",").collect();
assert_eq!(splits, vec![b"a", b"b", b"c", b"d"]);
assert_eq!(rsplits, vec![b"d", b"c", b"b", b"a"]);

find_first_of / find_last_of match any byte present in the set; the not_of variants match any byte absent from the set:

use stringzilla::StringZillableBinary;

let haystack = b"hello world";
let any: Vec<&[u8]> = haystack.sz_find_first_of(b"or").collect();
assert_eq!(any, vec![b"o", b"o", b"r"]);

let other: Vec<&[u8]> = b"aabbbcccd".sz_find_first_not_of(b"ab").collect();
assert_eq!(other, vec![b"c", b"c", b"c", b"d"]);

By default the split iterators keep empty segments, mirroring str::split / str::rsplit; chain .skip_empty() to drop zero-length segments:

use stringzilla::StringZillableBinary;

let kept: Vec<&[u8]> = b"a,,b,".sz_splits(b",").collect();
assert_eq!(kept, vec![b"a", &b""[..], b"b", &b""[..]]);

let nonempty: Vec<&[u8]> = b"a,,b,".sz_splits(b",").skip_empty().collect();
assert_eq!(nonempty, vec![b"a", b"b"]);

For direct construction, a Matcher<'a> names what to look for, Matcher::Substring(needle) or Matcher::Bytes(set), and the iterator type picks the direction: FindMatches and FindSplits search front to back, RFindMatches and RFindSplits back to front. An inverted Byteset matches the bytes outside the set. Policies are compile-time markers, opted into with builder methods: .overlapping() (default NonOverlapping, like str::matches) and .skip_empty() (default KeepEmpty, like str::split):

use stringzilla::{Byteset, FindMatches, FindSplits, Matcher};

let non_overlapping: Vec<&[u8]> = FindMatches::new(b"aaaa", Matcher::Substring(b"aa")).collect();
assert_eq!(non_overlapping, vec![&b"aa"[..], &b"aa"[..]]);
let overlapping: Vec<&[u8]> = FindMatches::new(b"aaaa", Matcher::Substring(b"aa")).overlapping().collect();
assert_eq!(overlapping, vec![&b"aa"[..], &b"aa"[..], &b"aa"[..]]);

let separators = Matcher::Bytes(Byteset::from(b",;"));
let split: Vec<&[u8]> = FindSplits::new(b",a;;b,", separators).skip_empty().collect();
assert_eq!(split, vec![b"a", b"b"]);

An empty needle matches at every offset, n + 1 times in an n-byte haystack, like "abc".matches(""). An empty separator never splits, so the whole haystack comes back as one segment.

API

StructFindSplitsAn iterator over the segments of a byte slice between the matches of a pattern, front to back.2
find.rs:476 ↗
pub struct FindSplits<'a, Empty: EmptySegments = KeepEmpty>

By default empty segments are kept, mirroring str::split, so adjacent separators and a leading or trailing one yield empty slices. Call Self::skip_empty to drop them.

Empty needle and empty haystack

An empty needle never splits, yielding the whole haystack as one segment. An empty haystack yields one empty segment, mirroring "".split(",") == [""].

Examples

use stringzilla::{FindSplits, Matcher};

let splits: Vec<&[u8]> = FindSplits::new(b"a,b,c,d", Matcher::Substring(b",")).collect();
assert_eq!(splits, vec![b"a", b"b", b"c", b"d"]);

Implements

Iterator
newSplits haystack at every match of matcher, front to back, keeping empty pieces.
find.rs:484 ↗
pub fn new(haystack: &'a [u8], matcher: Matcher<'a>) -> Self
skip_*_empty× 2Drop zero-length segments, a compile-time policy returning the SkipEmpty variant.
find.rs:493 ↗
pub fn skip__empty(self) -> FindSplits<'a, SkipEmpty>
StructRFindSplitsAn iterator over the segments of a byte slice between the matches of a pattern, back to front.1
find.rs:541 ↗
pub struct RFindSplits<'a, Empty: EmptySegments = KeepEmpty>

By default empty segments are kept, mirroring str::rsplit. Call Self::skip_empty to drop them. An empty needle never splits, yielding the whole haystack as one segment.

Examples

use stringzilla::{Matcher, RFindSplits};

let splits: Vec<&[u8]> = RFindSplits::new(b"a,b,c,d", Matcher::Substring(b",")).collect();
assert_eq!(splits, vec![b"d", b"c", b"b", b"a"]);

Implements

Iterator
newSplits haystack at every match of matcher, back to front, keeping empty pieces.
find.rs:549 ↗
pub fn new(haystack: &'a [u8], matcher: Matcher<'a>) -> Self
EnumSplitKindHow Strings::split_in ends its parts.
strings.rs:381 ↗
pub enum SplitKind

Corresponds to sz_split_kind_t in the C API.

Values

Each0Every delimiter ends a part, so neighbouring delimiters leave empty parts, as slice::split does at one byte.
Runs1A run of delimiters ends one part and empty parts are skipped, as str::split_ascii_whitespace does at ASCII whitespace.

Implements

Debug, Clone, Copy, StructuralPartialEq, PartialEq, Eq
TraitEmptySegmentsCompile-time policy for whether a split keeps or drops empty, zero-length segments.1
types.rs:947 ↗
pub trait EmptySegments

A named marker type rather than a raw bool, so the choice is branchless and readable at call sites.

Implementors

SKIPWhether zero-length segments are skipped.
types.rs:949 ↗
const SKIP: bool;
StructKeepEmptyKeep empty segments, the default.
types.rs:952 ↗
pub struct KeepEmpty;

Implements

StructSkipEmptyDrop empty segments, via .skip_empty().
types.rs:957 ↗
pub struct SkipEmpty;

Implements

TraitSplitPartsWhich parts a Utf8Split yields, as a compile-time (FIRST, STRIDE) over the span boundaries: the segments between separators, the separators themselves, or both interleaved losslessly.2
utf8_tokens.rs:113 ↗
pub trait SplitParts

Implementors

FIRSTFirst boundary index to visit (0 for between/both, 1 for separators).
utf8_tokens.rs:115 ↗
const FIRST: usize;
STRIDEStep between visited boundaries (2 for between/separators, 1 for both).
utf8_tokens.rs:117 ↗
const STRIDE: usize;
StructUtf8SplitA range over UTF-8 text split on the separators a kernel reports, selecting which parts to yield.4
utf8_tokens.rs:145 ↗
pub struct Utf8Split<'a, Kernel: TokenizerKernel, Parts: SplitParts = Between, Empty: EmptySegments = KeepEmpty, const STEPS: usize = ITERATORS_DEFAULT_STEPS>

A batch's boundaries are 0, each separator's start and end, and the end of the text once the batch reaches it; part k spans boundaries k and k + 1, and Parts walks them as (FIRST, STRIDE), so all three modes share one formula. A batch holding STEPS separators stops at the end of its last one, where the next batch begins.

Implements

Iterator
newConstructs an iterator with the default batch size (ITERATORS_DEFAULT_STEPS).
utf8_tokens.rs:165 ↗
pub fn new(text: &'a [u8]) -> Self

For an explicit batch size use Self::with_steps with a turbofish.

with_stepsConstructs an iterator buffering up to STEPS separators per FFI call.
utf8_tokens.rs:174 ↗
pub fn with_steps(text: &'a [u8]) -> Self
skip_emptySkips zero-length spans, returning the SkipEmpty variant.
utf8_tokens.rs:238 ↗
pub fn skip_empty(self) -> Utf8Split<'a, Kernel, Parts, SkipEmpty, STEPS>

A compile-time policy rather than a runtime flag, so the keep-empties default stays branchless.

with_separatorsThe same split yielding segments and separators interleaved.
utf8_tokens.rs:256 ↗
pub fn with_separators(self) -> Utf8Split<'a, Kernel, Both, Empty, STEPS>

Lossless (concatenation reproduces the input) only when empties are kept; the Empty policy carries through the type, so .skip_empty() and .with_separators() compose in either order.