pub enum Matcher<'a> Values
Substring | (&'a [u8]) | An exact byte sequence. |
|---|---|---|
Bytes | (<a data-ref="Byteset">Byteset</a>) | Any one byte of the set; invert the set to match the bytes outside it. |
Implements
Debug, Clone, CopyAll search functions accept any AsRef<[u8]> and return byte offsets as Option<usize>.
They are available both as sz:: free functions and as StringZillableBinary trait methods.
Free functions:
pub fn find<H: AsRef<[u8]>, N: AsRef<[u8]>>(haystack: H, needle: N) -> Option<usize>;
pub fn rfind<H: AsRef<[u8]>, N: AsRef<[u8]>>(haystack: H, needle: N) -> Option<usize>;
pub fn find_byteset<H: AsRef<[u8]>>(haystack: H, needles: Byteset) -> Option<usize>;
pub fn rfind_byteset<H: AsRef<[u8]>>(haystack: H, needles: Byteset) -> Option<usize>;
pub fn find_byte_from<H, N>(haystack: H, needles: N) -> Option<usize>; // first byte in the set
pub fn rfind_byte_from<H, N>(haystack: H, needles: N) -> Option<usize>; // last byte in the set
pub fn find_byte_not_from<H, N>(haystack: H, needles: N) -> Option<usize>; // first byte not in the set
pub fn rfind_byte_not_from<H, N>(haystack: H, needles: N) -> Option<usize>; // last byte not in the setuse stringzilla as sz;
assert_eq!(sz::find("Hello, world!", "world"), Some(7));
assert_eq!(sz::rfind("Hello, world, world!", "world"), Some(14));
// First/last byte from a set of delimiters, or the first/last byte NOT in a set.
assert_eq!(sz::find_byte_from("Hello, world!", "aeiou"), Some(1)); // 'e'
assert_eq!(sz::rfind_byte_from("Hello, world!", "aeiou"), Some(8)); // 'o'
assert_eq!(sz::find_byte_not_from("Hello, world!", "aeiou"), Some(0)); // 'H'
assert_eq!(sz::rfind_byte_not_from("Hello, world!", "aeiou"), Some(12));// '!'The StringZillableBinary<'a, N> trait provides the same operations as methods on the haystack.
The blanket impl covers every T: AsRef<[u8]> + ?Sized, so they work on &str, String, &[u8], Cow<str>, and more:
use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_find("world".as_bytes()), Some(7));
assert_eq!(haystack.sz_rfind("world".as_bytes()), Some(7));
assert_eq!(haystack.sz_find_byte_from("aeiou".as_bytes()), Some(1));
assert_eq!(haystack.sz_rfind_byte_from("aeiou".as_bytes()), Some(8));
assert_eq!(haystack.sz_find_byte_not_from("aeiou".as_bytes()), Some(0));
assert_eq!(haystack.sz_rfind_byte_not_from("aeiou".as_bytes()), Some(12));Trait methods: sz_find, sz_rfind, sz_find_byte_from, sz_rfind_byte_from, sz_find_byte_not_from, sz_rfind_byte_not_from.
These two functions reason about codepoint boundaries without fully decoding the string.
pub fn count_utf8<T: AsRef<[u8]>>(text: T) -> usize;
pub fn find_nth_utf8<T: AsRef<[u8]>>(text: T, n: usize) -> Option<usize>;count_utf8 counts codepoint start bytes with SIMD; find_nth_utf8 returns the byte offset of the Nth, zero-indexed, codepoint without decoding the whole string:
use stringzilla as sz;
assert_eq!(sz::count_utf8("你好世界"), 4);
assert_eq!(sz::count_utf8("Hello🌍"), 6);
assert_eq!(sz::find_nth_utf8("Hello🌍", 5), Some(5)); // 🌍 starts at byte 5
assert_eq!(sz::find_nth_utf8("Hello", 5), None);SubstringsEngine compiles a whole needle set into one Aho-Corasick automaton and walks every haystack against all of them at once, so a dictionary of thousands of terms costs one pass rather than thousands.
Building it is the expensive half and the engine is reusable, so a long-lived one amortizes that across every later batch.
fn new<N: AsRef<[u8]>>(needles: &[N], case_sensitivity: CaseSensitivity,
overlap_policy: SubstringsOverlapPolicy, hot_states: usize, matches_budget: usize,
haystacks_budget: usize, stream: &Stream) -> Result<SubstringsEngine, Status>;
fn counts<'scope>(&'scope mut self, scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>, counts: &'scope mut [usize], counts_stride: usize) -> Result<(), Status>;
fn find<'scope>(&'scope mut self, scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>, matches: &'scope mut [Match],
matches_offsets: &'scope mut [usize]) -> Result<(), Status>;
fn replace<'scope>(&'scope mut self, scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>, replacements: StringsView<'scope>,
target: &'scope mut [u8], offsets: &'scope mut [usize]) -> Result<(), Status>;
fn bm25_scores<'scope>(&'scope mut self, scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>, document_lengths: Option<&'scope [f32]>, parameters: &'scope Bm25Params,
needle_weights: &'scope [f32], scores: &'scope mut [f32], scores_stride: usize) -> Result<(), Status>;
fn report(&self) -> SubstringsReport;hot_states sizes the automaton’s dense tier, or takes SUBSTRINGS_HOT_STATES_AUTO to fill a fixed byte budget instead, which holds more states the fewer byte classes the vocabulary spells. matches_budget bounds what one round may emit and haystacks_budget how many haystacks it may carry, and both are read by a device tier alone, so a host engine takes SUBSTRINGS_MATCHES_BUDGET_AUTO and zero and walks straight into the caller’s output.
CaseSensitivity::Cased matches bytes exactly and accepts arbitrary needles, while CaseSensitivity::Uncased folds both sides under full Unicode case folding and requires valid UTF-8.
Folding is not a byte-length-preserving operation, so a 1-byte needle can match a 3-byte span — the Kelvin sign U+212A folds to k — which is why every Match carries its own byte_length rather than borrowing the needle’s.
SubstringsOverlapPolicy decides what a walk reports, and it sizes the engine’s arena, so it is fixed at construction and one engine runs exactly one of the three:
Overlapping — every match of every needle, nested and overlapping ones included.LeftmostLongest — a cover taking the widest match at the earliest start.LeftmostFirst — a cover taking the lowest needle index at the earliest start.A capacity shortfall is not an error.
The sizing walk always runs, so report() names matches_emitted, the true total; matches_stored, what was written; target_length, what a rewrite needs; and shortfall, what did not fit.
That is what lets one call with an empty output size the next one, with no walk in between.
use stringzilla::{
Capabilities, CaseSensitivity, Stream, StringsView, SubstringsEngine, Match, SubstringsOverlapPolicy,
SUBSTRINGS_HOT_STATES_AUTO, SUBSTRINGS_MATCHES_BUDGET_AUTO,
};
let cpu = Stream::default(Capabilities::cpu_enabled());
let mut engine = SubstringsEngine::new(
&["cat", "catalog"],
CaseSensitivity::Cased,
SubstringsOverlapPolicy::Overlapping,
SUBSTRINGS_HOT_STATES_AUTO,
SUBSTRINGS_MATCHES_BUDGET_AUTO,
0,
&cpu,
)
.unwrap();
let documents = StringsView::from_slices(&["a catalog of cats", "nothing here"]);
let mut counts = [0usize; 2];
cpu.scope(|scope| engine.counts(scope, documents, &mut counts, 1)).unwrap();
assert_eq!(counts, [3, 0]); // "catalog", the "cat" inside it, and the "cat" of "cats"
// `find` fills one boundary per haystack plus a final total, whether or not the matches fit.
let mut matches = [Match::default(); 3];
let mut offsets = [0usize; 3];
cpu.scope(|scope| engine.find(scope, documents, &mut matches, &mut offsets)).unwrap();
assert_eq!(offsets, [0, 3, 3]);
assert_eq!(engine.report().matches_emitted, 3);
assert_eq!(engine.report().shortfall, 0);The same automaton scores documents with BM25 and rewrites them, each in a single walk. bm25_scores treats the dictionary itself as the query — needle_weights[i] is needle i’s IDF or boost — and writes one score per haystack, so a many-term query over a large corpus never materializes per-term frequency rows.
Term frequencies are raw overlapping counts, which is classic BM25, so the engine’s own policy does not apply here, and document_lengths falls back to each haystack’s byte length when None.
Bm25Params has no Default, because a corpus mean has no correct default value.
Its two constructors name the two configurations that exist: Bm25Params::normalized(mean) is the literature’s k1 = 1.2 and b = 0.75 against a corpus whose mean document length you know, and Bm25Params::unnormalized() switches length normalization off and leaves document_lengths unread.
replace takes one replacement per needle, inserted verbatim, an empty one deleting its match, and writes every rewritten haystack into one target buffer beside their boundaries.
Rewriting is defined only under a cover, so an engine built with SubstringsOverlapPolicy::Overlapping is refused there — an overlapping rewrite is not a function.
A target too small is not an error either: report().target_length names the bytes the rewrite needed and the buffer’s contents are then unspecified, so an empty target is how the next call is sized.
use stringzilla::{Bm25Params, Capabilities, CaseSensitivity, Stream, StringsView, SubstringsEngine,
SubstringsOverlapPolicy, SUBSTRINGS_HOT_STATES_AUTO, SUBSTRINGS_MATCHES_BUDGET_AUTO};
let cpu = Stream::default(Capabilities::cpu_enabled());
let mut engine = SubstringsEngine::new(
&["cat", "dog"],
CaseSensitivity::Cased,
SubstringsOverlapPolicy::LeftmostLongest,
SUBSTRINGS_HOT_STATES_AUTO,
SUBSTRINGS_MATCHES_BUDGET_AUTO,
0,
&cpu,
)
.unwrap();
let documents = StringsView::from_slices(&["cat and dog", "nothing here"]);
let mut scores = [0.0f32; 2];
let parameters = Bm25Params::normalized(10.0);
cpu.scope(|scope| engine.bm25_scores(scope, documents, None, ¶meters, &[1.0, 1.0], &mut scores, 1))
.unwrap();
assert_eq!(scores[1], 0.0);
// One replacement per needle: an empty target sizes the rewrite, a second scope performs it.
let replacements = StringsView::from_slices(&["feline", "canine"]);
let mut offsets = [0usize; 3];
cpu.scope(|scope| engine.replace(scope, documents, replacements, &mut [], &mut offsets)).unwrap();
let mut target = vec![0u8; engine.report().target_length];
cpu.scope(|scope| engine.replace(scope, documents, replacements, &mut target, &mut offsets)).unwrap();
assert_eq!(&target[offsets[0]..offsets[1]], b"feline and canine");MatcherWhat a match or split iterator looks for; the iterator type picks the direction.3pub enum Matcher<'a> Substring | (&'a [u8]) | An exact byte sequence. |
|---|---|---|
Bytes | (<a data-ref="Byteset">Byteset</a>) | Any one byte of the set; invert the set to match the bytes outside it. |
Debug, Clone, CopyfindOffset of the first match in haystack.pub fn find(&self, haystack: &[u8]) -> Option<usize> rfindOffset of the last match in haystack.pub fn rfind(&self, haystack: &[u8]) -> Option<usize> widthBytes every match spans: the needle's length, or one for a byteset.pub fn width(&self) -> usize FindMatchesAn iterator over the matches of a pattern in a byte slice, front to back.2pub struct FindMatches<'a, Overlap: Overlaps = NonOverlapping> An empty needle matches at every offset from 0 to haystack.len(), so an n-byte haystack
yields n + 1 empty matches, mirroring "abc".matches("").count() == 4.
use stringzilla::{FindMatches, Matcher};
let matches: Vec<&[u8]> = FindMatches::new(b"abababa", Matcher::Substring(b"aba")).collect();
assert_eq!(matches, vec![b"aba", b"aba"]);
IteratornewIterates the non-overlapping matches of matcher in haystack, front to back.pub fn new(haystack: &'a [u8], matcher: Matcher<'a>) -> Self *× 2Report overlapping matches too, a compile-time policy returning the Overlapping variant.pub fn overlapping(self) -> FindMatches<'a, Overlapping> RFindMatchesAn iterator over the matches of a pattern in a byte slice, back to front.1pub struct RFindMatches<'a, Overlap: Overlaps = NonOverlapping> An empty needle matches at every offset from haystack.len() down to 0, so an n-byte haystack
yields n + 1 empty matches.
use stringzilla::{Matcher, RFindMatches};
let matches: Vec<&[u8]> = RFindMatches::new(b"abababa", Matcher::Substring(b"aba")).collect();
assert_eq!(matches, vec![b"aba", b"aba"]);
IteratornewIterates the non-overlapping matches of matcher in haystack, back to front.pub fn new(haystack: &'a [u8], matcher: Matcher<'a>) -> Self StringZillableBinaryTrait for binary string operations that take a needle parameter.14pub trait StringZillableBinary<'a, Needle>
where
Needle: AsRef<[u8]> + 'a, These operations include searching, splitting, and pattern matching.
Basic usage on a string slice:
use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_find("world".as_bytes()), Some(7));
impl<'a, Source, Needle> StringZillableBinary<'a, Needle> for Source where Source: AsRef<[u8]> + ?Sized, Needle: AsRef<[u8]> + 'asz_findSearches for the first occurrence of needle in self.fn sz_find(&self, needle: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_find("world".as_bytes()), Some(7));
sz_rfindSearches for the last occurrence of needle in self.fn sz_rfind(&self, needle: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world, world!";
assert_eq!(haystack.sz_rfind("world".as_bytes()), Some(14));
sz_find_byte_fromFinds the index of the first character in self that is also present in needles.fn sz_find_byte_from(&self, needles: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_find_byte_from("aeiou".as_bytes()), Some(1));
sz_rfind_byte_fromFinds the index of the last character in self that is also present in needles.fn sz_rfind_byte_from(&self, needles: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_rfind_byte_from("aeiou".as_bytes()), Some(8));
sz_find_byte_not_fromFinds the index of the first character in self that is not present in needles.fn sz_find_byte_not_from(&self, needles: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_find_byte_not_from("aeiou".as_bytes()), Some(0));
sz_rfind_byte_not_fromFinds the index of the last character in self that is not present in needles.fn sz_rfind_byte_not_from(&self, needles: Needle) -> Option<usize> use stringzilla::StringZillableBinary;
let haystack = "Hello, world!";
assert_eq!(haystack.sz_rfind_byte_not_from("aeiou".as_bytes()), Some(12));
sz_matchesReturns an iterator over all non-overlapping matches of the given needle in self.fn sz_matches(&'a self, needle: &'a Needle) -> FindMatches<'a> needle: The byte slice to search for within self.use stringzilla::StringZillableBinary;
let haystack = b"abababa";
let needle = b"aba";
let matches: Vec<&[u8]> = haystack.sz_matches(needle).collect();
assert_eq!(matches, vec![b"aba", b"aba"]); // non-overlapping by default (like str::matches)
let overlapping: Vec<&[u8]> = haystack.sz_matches(needle).overlapping().collect();
assert_eq!(overlapping, vec![b"aba", b"aba", b"aba"]); // opt in with .overlapping()
sz_rmatchesReturns an iterator over all non-overlapping matches of the given needle in self, searching from the end.fn sz_rmatches(&'a self, needle: &'a Needle) -> RFindMatches<'a> needle: The byte slice to search for within self.use stringzilla::StringZillableBinary;
let haystack = b"abababa";
let needle = b"aba";
let matches: Vec<&[u8]> = haystack.sz_rmatches(needle).collect();
assert_eq!(matches, vec![b"aba", b"aba"]); // non-overlapping by default
let overlapping: Vec<&[u8]> = haystack.sz_rmatches(needle).overlapping().collect();
assert_eq!(overlapping, vec![b"aba", b"aba", b"aba"]); // opt in with .overlapping()
sz_splitsReturns an iterator over the substrings of self that are separated by the given needle.fn sz_splits(&'a self, needle: &'a Needle) -> FindSplits<'a> needle: The byte slice to split self by.use stringzilla::StringZillableBinary;
let haystack = b"a,b,c,d";
let needle = b",";
let splits: Vec<&[u8]> = haystack.sz_splits(needle).collect();
assert_eq!(splits, vec![b"a", b"b", b"c", b"d"]);
sz_rsplitsReturns an iterator over the substrings of self that are separated by the given needle, searching from the end.fn sz_rsplits(&'a self, needle: &'a Needle) -> RFindSplits<'a> needle: The byte slice to split self by.use stringzilla::StringZillableBinary;
let haystack = b"a,b,c,d";
let needle = b",";
let splits: Vec<&[u8]> = haystack.sz_rsplits(needle).collect();
assert_eq!(splits, vec![b"d", b"c", b"b", b"a"]);
sz_find_first_ofReturns an iterator over all non-overlapping matches of any of the bytes in needles within self.fn sz_find_first_of(&'a self, needles: &'a Needle) -> FindMatches<'a> needles: The set of bytes to search for within self.use stringzilla::StringZillableBinary;
let haystack = b"Hello, world!";
let needles = b"aeiou";
let matches: Vec<&[u8]> = haystack.sz_find_first_of(needles).collect();
assert_eq!(matches, vec![b"e", b"o", b"o"]);
sz_find_last_ofReturns an iterator over all non-overlapping matches of any of the bytes in needles within self, searching from the end.fn sz_find_last_of(&'a self, needles: &'a Needle) -> RFindMatches<'a> needles: The set of bytes to search for within self.use stringzilla::StringZillableBinary;
let haystack = b"Hello, world!";
let needles = b"aeiou";
let matches: Vec<&[u8]> = haystack.sz_find_last_of(needles).collect();
assert_eq!(matches, vec![b"o", b"o", b"e"]);
sz_find_first_not_ofReturns an iterator over all non-overlapping matches of any byte not in needles within self.fn sz_find_first_not_of(&'a self, needles: &'a Needle) -> FindMatches<'a> needles: The set of bytes that should not be matched within self.use stringzilla::StringZillableBinary;
let haystack = b"Hello, world!";
let needles = b"aeiou";
let matches: Vec<&[u8]> = haystack.sz_find_first_not_of(needles).collect();
assert_eq!(matches, vec![b"H", b"l", b"l", b",", b" ", b"w", b"r", b"l", b"d", b"!"]);
sz_find_last_not_ofReturns an iterator over all non-overlapping matches of any byte not in needles within self, searching from the end.fn sz_find_last_not_of(&'a self, needles: &'a Needle) -> RFindMatches<'a> needles: The set of bytes that should not be matched within self.use stringzilla::StringZillableBinary;
let haystack = b"Hello, world!";
let needles = b"aeiou";
let matches: Vec<&[u8]> = haystack.sz_find_last_not_of(needles).collect();
assert_eq!(matches, vec![b"!", b"d", b"l", b"r", b"w", b" ", b",", b"l", b"l", b"H"]);
RegexEngineA vocabulary of RE2-syntax patterns compiled into two automata, walked over as many batches of haystacks as a caller has.6pub struct RegexEngine Matches are leftmost-first across the union of patterns, ties going to the lower pattern index,
which is what Rust's regex, RE2 and Python's re report; a Match names its pattern as
needle_index. Every haystack streams through the automata once whatever the pattern count.
The fields mirror sz_regex_engine_t one for one and only patterns_count is read from Rust,
so the layout is load-bearing and the engine travels to C by pointer. Every verb grows the
engine's round scratch, so it takes &mut self.
use stringzilla::{Capabilities, CaseSensitivity, Match, RegexEngine, Stream, StringsView};
let cpu = Stream::default(Capabilities::cpu_enabled());
let mut engine = RegexEngine::new(&["[0-9]+", "[a-z]+"], CaseSensitivity::Cased, &cpu).unwrap();
let documents = StringsView::from_slices(&["abc 123 de", "!"]);
let (mut matches, mut offsets) = ([Match::default(); 4], [0usize; 3]);
cpu.scope(|scope| engine.find(scope, documents, &mut matches, &mut offsets)).unwrap();
assert_eq!(offsets, [0, 3, 3]);
assert_eq!((matches[1].needle_index, matches[1].byte_offset, matches[1].byte_length), (0, 4, 3));
DropnewCompiles patterns with the capabilities of stream, like Capabilities::cpu_enabled, on the device stream names.pub fn new<Pattern>(
patterns: &[Pattern],
case_sensitivity: CaseSensitivity,
stream: &Stream,
) -> Result<Self, StringsError>
where
Pattern: AsRef<[u8]>, A pattern outside the supported syntax, like a backreference, a capturing group inside a
lookaround or an inline (?i), is refused as StringsError::Malformed with its index;
case folding is the engine-wide case_sensitivity instead.
patterns_countPatterns the vocabulary holds, which bounds every reported needle_index.pub fn patterns_count(&self) -> usize countsQueues on the stream of scope the count of every haystack's leftmost-first matches, one count per haystack.pub fn counts<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
counts: &'scope mut [usize],
counts_stride: usize,
) -> Result<(), Status> counts receives haystack h at counts[h * counts_stride], the stride counting entries
rather than bytes and being at least one. A stream of another group than the engine's is
refused with Status::DeviceMemoryMismatch, here and by every verb below.
findQueues on the stream of scope a report of every leftmost-first match, ascending by haystack, then by start.pub fn find<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
matches: &'scope mut [Match],
matches_offsets: &'scope mut [usize],
) -> Result<(), Status> matches_offsets holds one boundary per haystack plus a final total, filled whether or not
the matches fit, which sizes the next call; an empty matches makes the call a size query.
replaceRewrites every haystack into one target buffer, substituting each match with its pattern's replacement, on the CPU only.pub fn replace<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
replacements: StringsView<'scope>,
target: &'scope mut [u8],
offsets: &'scope mut [usize],
) -> Result<(), Status> replacements is indexed by pattern, so it holds exactly RegexEngine::patterns_count
entries; a replacement names groups as $1, ${1} or ${name}, $0 being the whole match
and $ one dollar. offsets holds one boundary per haystack plus a final total and is
filled whether or not target held the result, which sizes the next call.
extractQueues on the stream of scope the groups of each haystack's first match of pattern_index, as Pandas str.extract reads them.pub fn extract<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
pattern_index: usize,
groups: &'scope mut [Match],
groups_offsets: &'scope mut [usize],
) -> Result<(), Status> A haystack with such a match gets a Match row for the whole match, needle_index zero,
then one per group that took part, needle_index naming the group, so a group left out
differs from an empty one; a haystack without one gets no rows. groups_offsets holds one
boundary per haystack plus a final total, filled whether or not the rows fit, which sizes
the next call; an empty groups makes the call a size query.
use stringzilla::{Capabilities, CaseSensitivity, Match, RegexEngine, Stream, StringsView};
let cpu = Stream::default(Capabilities::cpu_enabled());
let mut engine = RegexEngine::new(&["(\\d+)-(\\d+)"], CaseSensitivity::Cased, &cpu).unwrap();
let documents = StringsView::from_slices(&["a 12-34", "none"]);
let (mut groups, mut offsets) = ([Match::default(); 3], [0usize; 3]);
cpu.scope(|scope| engine.extract(scope, documents, 0, &mut groups, &mut offsets)).unwrap();
assert_eq!(offsets, [0, 3, 3]);
assert_eq!((groups[2].needle_index, groups[2].byte_offset, groups[2].byte_length), (2, 5, 2));
SubdocumentsFormatThe grammar of every document an engine reads, and the path language that addresses it.pub enum SubdocumentsFormat Corresponds to sz_subdocuments_format_t in the C API.
Json | 0 | RFC 8259 documents, addressed by RFC 9535 JSONPath. |
|---|---|---|
Xml | 1 | XML 1.0 documents, addressed by XPath 1.0 location paths along forward axes. |
Debug, Clone, Copy, StructuralPartialEq, PartialEq, EqSubdocumentsEngineA vocabulary of paths compiled into one automaton over keys, walked over as many batches of documents as a caller has.5pub struct SubdocumentsEngine Every document is walked once whatever the path count, and no tree is built: a container no path
can enter is skipped by counting its brackets. A Match spans the selected node's exact bytes
inside its document, in JSON a string with its quotes, a number or literal, or a container
through its closing bracket, and in XML an element, an attribute's value or a text node, which a
caller reads in place or hands to another parser.
The fields mirror sz_subdocuments_engine_t one for one and only paths_count and format are
read from Rust, so the layout is load-bearing and the engine travels to C by pointer. The regex
engine of match() and search() filters is embedded by value and freed by the C side, hence
ManuallyDrop. Every verb grows the engine's depth stack, so it takes &mut self.
use stringzilla::{Capabilities, Match, Stream, StringsView, SubdocumentsEngine, SubdocumentsFormat};
let cpu = Stream::default(Capabilities::cpu_enabled());
let paths = ["$.store.book[?@.price < 10].title"];
let mut engine = SubdocumentsEngine::new(&paths, SubdocumentsFormat::Json, &cpu).unwrap();
let document = r#"{"store": {"book": [{"title": "A", "price": 8}, {"title": "B", "price": 12}]}}"#;
let documents = [document];
let documents = StringsView::from_slices(&documents);
let (mut matches, mut offsets, mut malformed_index) = ([Match::default(); 4], [0usize; 2], 0);
cpu.scope(|scope| engine.find(scope, documents, &mut matches, &mut offsets, &mut malformed_index)).unwrap();
assert_eq!(offsets[1], 1);
let title = &matches[0];
assert_eq!(&document[title.byte_offset..title.byte_offset + title.byte_length], "\"A\"");
DropnewCompiles paths for format with the capabilities of stream, like Capabilities::cpu_enabled, on the device stream names.pub fn new<Path>(
paths: &[Path],
format: SubdocumentsFormat,
stream: &Stream,
) -> Result<Self, StringsError>
where
Path: AsRef<[u8]>, JSON paths follow RFC 9535, filters and their functions included, and XML paths follow
XPath 1.0 along forward axes. A path that breaks its grammar, or that no single walk can
evaluate, like an absolute query inside a filter, is refused as
StringsError::Malformed with its index.
paths_countPaths the vocabulary holds, which bounds every reported needle_index.pub fn paths_count(&self) -> usize formatThe grammar every document of every verb follows.pub fn format(&self) -> SubdocumentsFormat countsQueues on the stream of scope the count of every document's nodes that some path selects, once per path, one count per document.pub fn counts<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
documents: StringsView<'scope>,
counts: &'scope mut [usize],
counts_stride: usize,
malformed_index: &'scope mut usize,
) -> Result<(), Status> counts receives document d at counts[d * counts_stride], the stride counting entries
rather than bytes and being at least one. A malformed document fails the call, and
malformed_index then names the first one once the scope joins.
findQueues on the stream of scope a report of every node of every document that some path selects, by document, then by start offset, then by path, so a container precedes what it holds.pub fn find<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
documents: StringsView<'scope>,
matches: &'scope mut [Match],
matches_offsets: &'scope mut [usize],
malformed_index: &'scope mut usize,
) -> Result<(), Status> matches_offsets holds one boundary per document plus a final total, and is filled whether
or not the matches fit, which is what sizes the next call; an empty matches makes the call
a pure size query. A malformed document fails the call, and malformed_index then names the
first one once the scope joins.
SubstringsOverlapPolicyHow matches that share bytes resolve: reported in full, or thinned to a leftmost run.pub enum SubstringsOverlapPolicy The policy sizes the engine's arena, so it is fixed at construction rather than travelling per call, and one engine runs exactly one of the three.
Corresponds to sz_substrings_overlap_policy_t in the C API.
Overlapping | 0 | Every match of every needle, including ones that share bytes and ones nested in others. |
|---|---|---|
LeftmostLongest | 1 | Matches sharing no bytes: earliest start, then longest span, then lower needle index. |
LeftmostFirst | 2 | Matches sharing no bytes: earliest start, then lower needle index, however long the rival. |
Debug, Clone, Copy, StructuralPartialEq, PartialEq, EqMatchOne reported match, locating it by haystack, by needle, and by byte span.pub struct Match Under case folding a needle's own byte length is not the length of every match - needle k
matches both the 1-byte k and the 3-byte Kelvin sign U+212A - so the span travels per match.
Corresponds to sz_match_t in the C API.
haystack_index | usize | Which haystack of the sequence this match was found in. |
|---|---|---|
needle_index | usize | Which needle of the vocabulary matched. |
byte_offset | usize | Where the match starts inside that haystack, in its own bytes. |
byte_length | usize | Haystack bytes the match spans, which folding can make differ from the needle's own length. |
Debug, Clone, Copy, Default, StructuralPartialEq, PartialEq, EqSubstringsReportWhat a round found, which is how a capacity shortfall is reported rather than as an error.pub struct SubstringsReport The sizing walk always runs, so matches_emitted is the truth whatever the caller's
output could hold, and a nonzero shortfall is the one signal that an output is
incomplete rather than wrong.
Corresponds to sz_substrings_report_t in the C API.
matches_emitted | usize | Matches the sizing walk found, which is the truth whatever the output held. |
|---|---|---|
matches_stored | usize | Matches written out, which is matches_emitted clipped at the capacity. |
target_length | usize | Bytes a rewrite needs, which is the truth whatever the target held. |
shortfall | usize | Matches or bytes the round could not hold, zero when everything fit. |
Debug, Clone, Copy, Default, StructuralPartialEq, PartialEq, EqBm25ParamsBM25's continuous parameters.2pub struct Bm25Params There is no correct default for the corpus mean, so there is no Default: reach for
Bm25Params::normalized or Bm25Params::unnormalized, which name the two configurations
that exist. A positive length_normalization beside a non-positive average_document_length is
refused, since it would divide by a mean that is not there.
Corresponds to sz_substrings_bm25_t in the C API.
term_frequency_saturation | f32 | The literature's k1: how slowly repeated occurrences stop adding score; 1.2 is customary. |
|---|---|---|
length_normalization | f32 | The literature's b, in [0, 1]: 0 ignores document length, 1 normalizes it fully. |
average_document_length | f32 | The corpus-wide mean document length, in the unit of the lengths scored; read only when
length_normalization is positive. |
Debug, Clone, Copy, StructuralPartialEq, PartialEqnormalizedClassic BM25 with the literature's customary k1 = 1.2 and b = 0.75, normalized against a corpus whose mean document length is average_document_length, in the unit the per-document lengths use.pub const fn normalized(average_document_length: f32) -> Self unnormalizedBM25 with length normalization switched off, for a corpus whose mean length is unknown or whose documents are uniform enough not to need it; the per-document lengths then go unread.pub const fn unnormalized() -> Self SubstringsEngineA vocabulary compiled into one automaton, walked over as many batches of haystacks as a caller has.8pub struct SubstringsEngine Building the automaton is the expensive half and every verb below reuses it, so one long-lived engine amortizes that across every later batch: a dictionary of thousands of terms costs one pass over a haystack rather than thousands.
The fields mirror sz_substrings_engine_t one for one and only needles_count,
overlap_policy and report are read from Rust, so the layout is load-bearing and the engine
travels to C by pointer. Owning raw pointers makes it neither Send nor Sync, which is what
the C contract wants: a compute verb writes the engine's round arena, so it mutates, and every
verb below takes &mut self for that reason.
use stringzilla::{
Capabilities, CaseSensitivity, Stream, StringsView, SubstringsEngine, SubstringsOverlapPolicy,
SUBSTRINGS_HOT_STATES_AUTO, SUBSTRINGS_MATCHES_BUDGET_AUTO,
};
let cpu = Stream::default(Capabilities::cpu_enabled());
let mut engine = SubstringsEngine::new(
&["cat", "catalog"],
CaseSensitivity::Cased,
SubstringsOverlapPolicy::Overlapping,
SUBSTRINGS_HOT_STATES_AUTO,
SUBSTRINGS_MATCHES_BUDGET_AUTO,
0,
&cpu,
)
.unwrap();
let documents = StringsView::from_slices(&["a catalog of cats", "nothing here"]);
let mut counts = [0usize; 2];
cpu.scope(|scope| engine.counts(scope, documents, &mut counts, 1)).unwrap();
assert_eq!(counts, [3, 0]); // "catalog", the "cat" inside it, and the "cat" of "cats"
DropnewCompiles needles into an automaton the matching verbs read, with the capabilities of stream, like Capabilities::cpu_enabled or a GPU's Capabilities::cuda_enabled, on the device stream names, which it may join.pub fn new<Needle>(
needles: &[Needle],
case_sensitivity: CaseSensitivity,
overlap_policy: SubstringsOverlapPolicy,
hot_states: usize,
matches_budget: usize,
haystacks_budget: usize,
stream: &Stream,
) -> Result<Self, Status>
where
Needle: AsRef<[u8]>, An empty needle is refused rather than skipped, since it would match at every position
and dropping it would shift every later needle's reported index. hot_states is the
count kept in the dense hot rows, or SUBSTRINGS_HOT_STATES_AUTO to fill a fixed byte
budget instead.
The budgets are read by a device kernel alone, so a host engine takes
SUBSTRINGS_MATCHES_BUDGET_AUTO and zero and walks straight into the caller's output.
matches_budget is what a device round may emit before every later kernel retires, and
haystacks_budget how many haystacks it may carry; the round's arena is sized for both
here, zero asking for a default, so no verb allocates, a round carrying more haystacks is
refused, and matches past the budget surface as the report's shortfall. A GPU engine's
verbs take haystacks, replacements and every buffer, weights and lengths included, in
memory the device reaches, such as a Strings and a Vec in a UnifiedAllocator.
needles_countNeedles the vocabulary holds, which bounds every reported needle_index.pub fn needles_count(&self) -> usize overlap_policyThe cover every round runs, fixed when the arena was sized.pub fn overlap_policy(&self) -> SubstringsOverlapPolicy reportWhat the last round found, which is how a capacity shortfall surfaces instead of as an error.pub fn report(&self) -> SubstringsReport A verb borrows the engine until its scope joins, so the record a device wrote is complete.
countsQueues on the stream of scope the count of every needle's matches in every haystack, one count per haystack.pub fn counts<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
counts: &'scope mut [usize],
counts_stride: usize,
) -> Result<(), Status> counts receives haystack h at counts[h * counts_stride], the stride counting entries
rather than bytes and being at least one, so a strided call writes one column of a
[haystacks, vocabularies] feature matrix. A stream of another group than the engine's is
refused with Status::DeviceMemoryMismatch, here and by every verb below.
findQueues on the stream of scope a report of every match of every needle in every haystack, ascending by haystack.pub fn find<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
matches: &'scope mut [Match],
matches_offsets: &'scope mut [usize],
) -> Result<(), Status> matches_offsets holds one boundary per haystack plus a final total, and is filled whether
or not the matches fit, which is what sizes the next call. A capacity too small is not an
error: SubstringsEngine::report names the true total and what did not fit, so a sizing
call with an empty matches followed by one filling call needs no walk in between.
replaceQueues on the stream of scope a rewrite of every haystack into one target buffer, substituting one replacement per needle.pub fn replace<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
replacements: StringsView<'scope>,
target: &'scope mut [u8],
offsets: &'scope mut [usize],
) -> Result<(), Status> replacements is indexed by needle and an empty one deletes the match, so it holds exactly
SubstringsEngine::needles_count entries. offsets holds one boundary per haystack plus
a final total and is filled whether or not target held the result, which is what sizes the
next call; a target too small leaves its contents unspecified rather than failing, and
SubstringsEngine::report names the bytes the rewrite needed.
The engine's policy must be a cover, since a substitution over matches that share bytes is not a function.
bm25_scoresQueues on the stream of scope the score of every haystack against the whole vocabulary as one BM25 query, one score per haystack.pub fn bm25_scores<'scope>(
&'scope mut self,
scope: &'scope Scope<'scope, '_>,
haystacks: StringsView<'scope>,
document_lengths: Option<&'scope [f32]>,
parameters: &'scope Bm25Params,
needle_weights: &'scope [f32],
scores: &'scope mut [f32],
scores_stride: usize,
) -> Result<(), Status> The vocabulary is the query and needle_weights holds each needle's IDF or boost, so a
many-term query over a large corpus never materializes per-term frequency rows. Term
frequencies are raw overlapping counts, since a cover would suppress genuine occurrences of
a needle nested in another, so the engine's own policy does not apply here.
document_lengths is None to normalize by each haystack's byte length instead.
scores receives haystack h at scores[h * scores_stride], on the same convention as
SubstringsEngine::counts.
IndexSpanRepresents a byte span with offset and length.4pub struct IndexSpan Used for matches of UTF-8 characters, substrings, or any byte-level operations. Stores the byte offset from the start of the text and the length in bytes.
offset | usize | Byte offset from the start of the text |
|---|---|---|
length | usize | Length in bytes of the matched span |
use stringzilla::IndexSpan;
let text = "Hello\nWorld";
let span = IndexSpan::new(5, 1);
assert_eq!(span.offset, 5);
assert_eq!(span.length, 1);
let matched = span.extract(text.as_bytes());
assert_eq!(matched, Some(&b"\n"[..]));
Debug, Clone, Copy, StructuralPartialEq, PartialEq, Eq, HashnewCreates a new IndexSpan with the given offset and length.pub const fn new(offset: usize, length: usize) -> Self rangeReturns the range of bytes covered by this span, ending at Self::end.pub const fn range(&self) -> Range<usize> use stringzilla::IndexSpan;
let span = IndexSpan::new(5, 3);
assert_eq!(span.range(), 5..8);
extractExtracts the matched bytes from the source text, or None if the span runs past its end.pub fn extract<'a>(&self, text: &'a [u8]) -> Option<&'a [u8]> use stringzilla::IndexSpan;
let text = b"Hello World";
assert_eq!(IndexSpan::new(6, 5).extract(text), Some(&b"World"[..]));
assert_eq!(IndexSpan::new(6, 6).extract(text), None);
endReturns the end offset, offset + length, saturating at usize::MAX.pub const fn end(&self) -> usize use stringzilla::IndexSpan;
let span = IndexSpan::new(5, 3);
assert_eq!(span.end(), 8);
OverlapsCompile-time policy for whether overlapping matches are reported - a named marker, not a raw bool.1pub trait Overlaps impl Overlaps for NonOverlappingimpl Overlaps for OverlappingOVERLAPWhether overlapping matches are included.const OVERLAP: bool; NonOverlappingReport only non-overlapping matches, the default, like str::matches.OverlappingReport overlapping matches too, via .overlapping().findLocates the first matching substring within haystack that equals needle.pub fn find<Haystack, Needle>(haystack: Haystack, needle: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, This function is similar to the memmem() function in LibC, but, unlike strstr(),
it requires the length of both haystack and needle to be known beforehand.
An Option<usize> representing the starting index of the first occurrence of needle
within haystack if found, otherwise None.
The C core returns the start of haystack for an empty needle, like strstr, so
find(haystack, b"") is always Some(0), matching "abc".find("") == Some(0). This holds even
for an empty haystack.
haystack: The byte slice to search.needle: The byte slice to find within the haystack.rfindLocates the last matching substring within haystack that equals needle.pub fn rfind<Haystack, Needle>(haystack: Haystack, needle: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, This function is useful for finding the most recent or last occurrence of a pattern within a byte slice.
An Option<usize> representing the starting index of the last occurrence of needle
within haystack if found, otherwise None.
The C core returns the end of haystack for an empty needle, the reverse mirror of strstr, so
rfind(haystack, b"") is always Some(haystack.len()), matching "abc".rfind("") == Some(3).
This holds even for an empty haystack.
haystack: The byte slice to search.needle: The byte slice to find within the haystack.containsChecks whether needle occurs anywhere within haystack.pub fn contains<Haystack, Needle>(haystack: Haystack, needle: Needle) -> bool
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, true if needle occurs within haystack, false otherwise.
Mirrors str::contains: an empty needle is always present, so contains(haystack, b"") is
always true, matching "abc".contains("") == true, even for an empty haystack.
haystack: The byte slice to search.needle: The byte slice to look for within the haystack.find_bytesetFinds the index of the first character in haystack that is also present in needles.pub fn find_byteset<Haystack>(haystack: Haystack, needles: Byteset) -> Option<usize>
where
Haystack: AsRef<[u8]>, This function is particularly useful for parsing and tokenization tasks where a set of delimiter characters is used.
An Option<usize> representing the index of the first occurrence of any byte from
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes to search for within the haystack.rfind_bytesetFinds the index of the last character in haystack that is also present in needles.pub fn rfind_byteset<Haystack>(haystack: Haystack, needles: Byteset) -> Option<usize>
where
Haystack: AsRef<[u8]>, This can be used to find the last occurrence of any character from a specified set, useful in parsing scenarios such as finding the last delimiter in a string.
An Option<usize> representing the index of the last occurrence of any byte from
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes to search for within the haystack.find_byte_fromFinds the index of the first character in haystack that is also present in needles.pub fn find_byte_from<Haystack, Needle>(haystack: Haystack, needles: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, This function is particularly useful for parsing and tokenization tasks where a set of delimiter characters is used.
An Option<usize> representing the index of the first occurrence of any byte from
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes to search for within the haystack.rfind_byte_fromFinds the index of the last character in haystack that is also present in needles.pub fn rfind_byte_from<Haystack, Needle>(haystack: Haystack, needles: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, This can be used to find the last occurrence of any character from a specified set, useful in parsing scenarios such as finding the last delimiter in a string.
An Option<usize> representing the index of the last occurrence of any byte from
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes to search for within the haystack.find_byte_not_fromFinds the index of the first character in haystack that is not present in needles.pub fn find_byte_not_from<Haystack, Needle>(haystack: Haystack, needles: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, This function is useful for skipping over a known set of characters and finding the first character that does not belong to that set.
An Option<usize> representing the index of the first occurrence of any byte not in
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes that should not be matched within the haystack.rfind_byte_not_fromFinds the index of the last character in haystack that is not present in needles.pub fn rfind_byte_not_from<Haystack, Needle>(haystack: Haystack, needles: Needle) -> Option<usize>
where
Haystack: AsRef<[u8]>,
Needle: AsRef<[u8]>, Useful for text processing tasks such as trimming trailing characters that belong to a specified set.
An Option<usize> representing the index of the last occurrence of any byte not in
needles within haystack, if found, otherwise None.
haystack: The byte slice to search.needles: The set of bytes that should not be matched within the haystack.count_utf8Counts the number of UTF-8 characters in the text.pub fn count_utf8<Text>(text: Text) -> usize
where
Text: AsRef<[u8]>, This function efficiently counts UTF-8 characters by identifying character start bytes, the non-continuation bytes. Uses SIMD acceleration when available.
The number of UTF-8 characters, or codepoints, in the text.
text: The UTF-8 encoded byte slice to count characters in.use stringzilla as sz;
let text = "Hello";
assert_eq!(sz::count_utf8(text), 5);
let text_unicode = "Hello🌍";
assert_eq!(sz::count_utf8(text_unicode), 6);
let text_cjk = "你好世界";
assert_eq!(sz::count_utf8(text_cjk), 4);
find_nth_utf8Finds the byte offset of the Nth UTF-8 character (0-indexed).pub fn find_nth_utf8<Text>(text: Text, n: usize) -> Option<usize>
where
Text: AsRef<[u8]>, This function efficiently locates the Nth UTF-8 character without decoding the entire string. Uses SIMD acceleration when available.
An Option<usize> containing the byte offset of the Nth character.
Returns None if the string has fewer than N+1 characters.
text: The UTF-8 encoded byte slice to search.n: The 0-based index of the character to find.use stringzilla as sz;
let text = "Hello";
assert_eq!(sz::find_nth_utf8(text, 0), Some(0)); // 'H'
assert_eq!(sz::find_nth_utf8(text, 4), Some(4)); // 'o'
assert_eq!(sz::find_nth_utf8(text, 5), None);
let text_unicode = "Hello🌍";
assert_eq!(sz::find_nth_utf8(text_unicode, 5), Some(5)); // 🌍 starts at byte 5
assert_eq!(sz::find_nth_utf8(text_unicode, 6), None);
SUBSTRINGS_HOT_STATES_AUTOThe hot_states argument's "size the hot tier yourself" value, so zero stays a real all-cold request.pub const SUBSTRINGS_HOT_STATES_AUTO: usize = usize::MAX; SUBSTRINGS_MATCHES_BUDGET_AUTOThe matches_budget argument's "let the tier choose" value; a host tier reads no budget at all.pub const SUBSTRINGS_MATCHES_BUDGET_AUTO: usize = 0;