class RegexEngine Example:
engine = sz.RegexEngine(sz.Strings(['[0-9]+', '[a-z]+'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['abc 123 de']), out) out[0] 3
All search methods accept optional start and end indices that bound the search window; start defaults to 0 and end to the string length.
They are available both as String methods and as module-level functions.
find(substring, start=0, end=len) — index of the first occurrence, or -1 if absent.rfind(substring, start=0, end=len) — index of the last occurrence, or -1.index(substring, start=0, end=len) — like find, but raises ValueError if absent.rindex(substring, start=0, end=len) — like rfind, but raises ValueError if absent.contains(substring, start=0, end=len) — True/False membership test.startswith(prefix, start=0, end=len) — True if the windowed string starts with prefix.endswith(suffix, start=0, end=len) — True if the windowed string ends with suffix.count(substring, start=0, end=len, allowoverlap=False) — number of occurrences; pass allowoverlap=True to count overlapping matches.import stringzilla as sz
text = sz.String("the quick brown fox")
assert text.find("brown") == 10
assert text.rfind("o") == 17
assert text.index("quick") == 4
assert text.contains("quick")
assert text.startswith("the")
assert text.endswith("fox")
assert sz.String("banana").count("a") == 3
assert sz.String("aaaa").count("aa") == 2 # non-overlapping
assert sz.String("aaaa").count("aa", allowoverlap=True) == 3 # overlappingThe *_of / *_not_of family searches for any byte from a set, the byteset-accelerated counterpart of substring search. count_byteset counts how many bytes fall in the set.
find_first_of(chars, start=0, end=len) — index of the first byte that is in chars, or -1.find_last_of(chars, start=0, end=len) — index of the last byte that is in chars, or -1.find_first_not_of(chars, start=0, end=len) — index of the first byte not in chars, or -1.find_last_not_of(chars, start=0, end=len) — index of the last byte not in chars, or -1.count_byteset(chars, start=0, end=len) — number of bytes that are in chars.import stringzilla as sz
text = sz.String("hello world")
assert text.find_first_of("aeiou") == 1
assert text.find_last_of("aeiou") == 7
assert text.find_first_not_of("he") == 2
assert text.find_last_not_of("lo") == 10
assert text.count_byteset("lo") == 5RegexEngineCompile a vocabulary of RE2-syntax patterns once and stream many collections of haystacks through it, reporting leftmost-first matches across all of them, the lower index winning ties.5class RegexEngine Example:
engine = sz.RegexEngine(sz.Strings(['[0-9]+', '[a-z]+'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['abc 123 de']), out) out[0] 3
RegexEngineRegexEngine(
patterns: Strings,
case_sensitivity: str = 'cased',
*,
capabilities: Capability | None = None,
stream: int | None = None,
) *× 2counts(haystacks, out, *, stream=None) -> Nonedef counts(
haystacks: Strings,
out: _Buffer,
*,
stream: int | None = None,
) -> None Count the leftmost-first matches of the vocabulary in every haystack, into out.
Example:
engine = sz.RegexEngine(sz.Strings(['[0-9]+', '[a-z]+'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['abc 123 de']), out) out[0] 3
Count the matches of every needle in every haystack, into out.
Example:
engine = sz.SubstringsEngine(sz.Strings(['he', 'she'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['hershey']), out) out[0] 3
haystacksStrings | Texts to search, from Strings.copy on a GPU. | |
|---|---|---|
out_Buffer | Writable 1-D buffer of pointer-width unsigned integers, one per haystack. | |
streamint | None | A stream of the engine's device as an integer, or None. |
*× 2find(haystacks, matches, offsets, *, stream=None) -> Nonedef find(
haystacks: Strings,
matches: _Buffer | None,
offsets: _Buffer,
*,
stream: int | None = None,
) -> None Locate every leftmost-first match, writing one row per match and one boundary per haystack.
A capacity too small is not an error: offsets is filled either way, which sizes the next call.
Example:
engine = sz.RegexEngine(sz.Strings(['[0-9]+'])) matches = memoryview(bytearray(2 * 4 * 8)).cast('Q', (2, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['a 12 b 345']), matches, offsets) matches[1, 2], matches[1, 3] (7, 3)
Locate every match, writing one row per match and one boundary per haystack.
A capacity too small is not an error: offsets and report are filled either way, which is
what sizes the next call.
Example:
engine = sz.SubstringsEngine(sz.Strings(['he', 'she'])) matches = memoryview(bytearray(3 * 4 * 8)).cast('Q', (3, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['hershey']), matches, offsets) offsets[1] 3
haystacksStrings | Texts to search, from Strings.copy on a GPU. | |
|---|---|---|
matches_Buffer | None | Writable, fully contiguous 2-D buffer of pointer-width unsigned
integers, shaped (capacity, 4) - haystack index, pattern index, byte offset, byte length.
None makes the call a pure size query. | |
offsets_Buffer | Writable, contiguous 1-D buffer of pointer-width unsigned integers holding
len(haystacks) + 1 boundaries into matches, the last being the total. | |
streamint | None | A stream of the engine's device as an integer, or None. |
replacereplace(haystacks, replacements, target, offsets) -> Nonedef replace(
haystacks: Strings,
replacements: Strings,
target: _Buffer | None,
offsets: _Buffer,
) -> None Rewrite every haystack into one target buffer, one replacement per pattern.
A replacement names groups as $1, ${1} or ${name}, $0 being the whole match and $
one dollar. A target too small is not an error: offsets is filled either way. CPU only.
Example:
engine = sz.RegexEngine(sz.Strings(['(\w+)@(\w+)'])) target = memoryview(bytearray(16)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.replace(sz.Strings(['bob@host']), sz.Strings(['$2/$1']), target, offsets) bytes(target[offsets[0]:offsets[1]]) b'host/bob'
haystacksStrings | Texts to rewrite. | |
|---|---|---|
replacementsStrings | One replacement per pattern; an empty one deletes the match. | |
target_Buffer | None | Writable, contiguous byte buffer; None makes the call a size query. | |
offsets_Buffer | Writable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 rewritten boundaries, the last being the total. |
extractextract(haystacks, pattern_index, groups, offsets, *, stream=None) -> Nonedef extract(
haystacks: Strings,
pattern_index: int,
groups: _Buffer | None,
offsets: _Buffer,
*,
stream: int | None = None,
) -> None Locate the groups of each haystack's first match of a pattern, as Pandas str.extract reads them.
A haystack with such a match gets a row for the whole match, group index zero, then one row per
group that took part, so a group left out differs from an empty one; a haystack without one gets
no rows. A capacity too small is not an error: offsets is filled either way, which sizes the
next call. haystacks.matched(groups) reads the rows as strings.
Example:
engine = sz.RegexEngine(sz.Strings(['(\d+)-(\d+)'])) groups = memoryview(bytearray(3 * 4 * 8)).cast('Q', (3, 4)) offsets = memoryview(bytearray(3 * 8)).cast('Q') haystacks = sz.Strings(['a 12-34', 'none']) engine.extract(haystacks, 0, groups, offsets) list(map(bytes, haystacks.matched(groups))), offsets[2] ([b'12-34', b'12', b'34'], 3)
haystacksStrings | Texts to search, from Strings.copy on a GPU. | |
|---|---|---|
pattern_indexint | The pattern whose groups to locate. | |
groups_Buffer | None | Writable, fully contiguous 2-D buffer of pointer-width unsigned
integers, shaped (capacity, 4) - haystack index, group index, byte offset, byte length.
None makes the call a pure size query. | |
offsets_Buffer | Writable, contiguous 1-D buffer of pointer-width unsigned integers holding
len(haystacks) + 1 boundaries into groups, the last being the total. | |
streamint | None | A stream of the engine's device as an integer, or None. |
SubdocumentsEngineCompile JSONPath or XPath queries once and pull what they select out of many documents.3class SubdocumentsEngine Every document is walked once, whatever the path count, and no tree is built. A match spans the node's exact bytes: in JSON a string with its quotes, a number or literal, or a container through its closing bracket; in XML an element, an attribute's value, or a text node.
Raises: ValueError: Naming the first path that breaks its grammar or leaves the supported subset. Example:
engine = sz.SubdocumentsEngine(sz.Strings(['$.a[?@ > 1]'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['{"a": [1, 2, 3]}']), out) out[0] 2
SubdocumentsEngineSubdocumentsEngine(
paths: Strings,
format: str = 'json',
*,
capabilities: Capability | None = None,
stream: int | None = None,
) countscounts(documents, out, *, stream=None) -> Nonedef counts(
documents: Strings,
out: _Buffer,
*,
stream: int | None = None,
) -> None Count the nodes of every document that some path selects, once per path, into out.
Raises: ValueError: Naming the first malformed document. Example:
engine = sz.SubdocumentsEngine(sz.Strings(['$..name'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['{"name": "a", "b": {"name": "c"}}']), out) out[0] 2
documentsStrings | Documents to walk, from Strings.copy on a GPU. | |
|---|---|---|
out_Buffer | Writable 1-D buffer of pointer-width unsigned integers, one per document. | |
streamint | None | A stream of the engine's device as an integer, or None.
A device engine enqueues there and returns, so out must be device-reachable
and read only after the caller joins the stream; a host engine ignores it. |
findfind(documents, matches, offsets, *, stream=None) -> Nonedef find(
documents: Strings,
matches: _Buffer | None,
offsets: _Buffer,
*,
stream: int | None = None,
) -> None Locate every selected node, writing one row per match and one boundary per document.
Matches come by document, then start offset, then path, so a container precedes what it holds.
A capacity too small is not an error: offsets is filled either way, which is what sizes the next
call, and Strings.matched gathers the spans the rows locate.
Raises: ValueError: Naming the first malformed document. Example:
engine = sz.SubdocumentsEngine(sz.Strings(['/a/b/text()']), format='xml') matches = memoryview(bytearray(2 * 4 * 8)).cast('Q', (2, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['<a><b>x</b><b>y</b></a>']), matches, offsets) offsets[1] 2
documentsStrings | Documents to walk, from Strings.copy on a GPU. | |
|---|---|---|
matches_Buffer | None | Writable, fully contiguous 2-D buffer of pointer-width unsigned
integers, shaped (capacity, 4) - document index, path index, byte offset, byte length.
None makes the call a pure size query. | |
offsets_Buffer | Writable, contiguous 1-D buffer of pointer-width unsigned integers holding
len(documents) + 1 boundaries into matches, the last being the total. | |
streamint | None | A stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so both buffers must be device-reachable and read only after the caller joins the stream; a host engine ignores it. |
SubstringsEngineSubstringsEngine(needles, case_sensitivity='cased', overlap_policy='overlapping', hot_states=None, matches_budget=0, *, haystacks_budget=0, capabilities=None, stream=None)4class SubstringsEngine Compile a vocabulary once and stream many collections of haystacks through it.
One Aho-Corasick pass answers every needle at once, whatever the vocabulary size.
Example:
engine = sz.SubstringsEngine(sz.Strings(['he', 'she']), case_sensitivity='cased') out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['hershey']), out) out[0] 3
SubstringsEngineSubstringsEngine(
needles: Strings,
case_sensitivity: str = 'cased',
overlap_policy: str = 'overlapping',
hot_states: int | None = None,
matches_budget: int = 0,
*,
haystacks_budget: int = 0,
capabilities: Capability | None = None,
stream: int | None = None,
) replacereplace(haystacks, replacements, target, offsets, *, stream=None) -> Nonedef replace(
haystacks: Strings,
replacements: Strings,
target: _Buffer | None,
offsets: _Buffer,
*,
stream: int | None = None,
) -> None Rewrite every haystack into one target buffer, one replacement per needle.
Needs a leftmost policy, since a substitution over matches that share bytes is not a function.
A target too small is not an error: offsets and report are filled either way.
Example:
engine = sz.SubstringsEngine(sz.Strings(['he', 'she']), overlap_policy='leftmost-longest') target = memoryview(bytearray(16)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.replace(sz.Strings(['hershey']), sz.Strings(['HE', 'SHE']), target, offsets) bytes(target[offsets[0]:offsets[1]]) b'HErSHEy'
haystacksStrings | Texts to rewrite, from Strings.copy on a GPU. | |
|---|---|---|
replacementsStrings | One replacement per needle; an empty one deletes the match. On a GPU
engine, from Strings.copy too. | |
target_Buffer | None | Writable, contiguous byte buffer; None makes the call a size query. | |
offsets_Buffer | Writable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 rewritten boundaries, the last being the total. | |
streamint | None | A stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so both buffers must be device-reachable and read only after the caller joins the stream; a host engine ignores it. |
bm25_scoresbm25_scores(haystacks, needle_weights, out, *, document_lengths=None, term_frequency_saturation=1.2, length_normalization=0.75, average_document_length=0.0, stream=None) -> Nonedef bm25_scores(
haystacks: Strings,
needle_weights: _Buffer,
out: _Buffer,
*,
document_lengths: _Buffer | None = None,
term_frequency_saturation: float = 1.2,
length_normalization: float = 0.75,
average_document_length: float = 0.0,
stream: int | None = None,
) -> None Score every haystack against the vocabulary, which is the query, into out.
Term frequencies are raw overlapping counts, so the engine's own overlap policy does not apply.
Example:
engine = sz.SubstringsEngine(sz.Strings(['cat', 'dog'])) weights = memoryview(bytearray(2 * 4)).cast('f') weights[0], weights[1] = 1.0, 1.0 out = memoryview(bytearray(2 * 4)).cast('f') engine.bm25_scores(sz.Strings(['cat and dog', 'nothing here']), weights, out, ... average_document_length=10.0) out[1] 0.0
haystacksStrings | Documents to score, from Strings.copy on a GPU. | |
|---|---|---|
needle_weights_Buffer | One 32-bit float IDF or boost per needle. | |
out_Buffer | Writable 1-D buffer of 32-bit floats, one score per haystack. | |
document_lengths_Buffer | None | One 32-bit float length per haystack; byte lengths when None. | |
term_frequency_saturationfloat | The literature's k1. | |
length_normalizationfloat | The literature's b, in [0, 1]. | |
average_document_lengthfloat | Corpus-wide mean of the lengths above; required whenever length_normalization is positive. | |
streamint | None | A stream of the engine's device as an integer, or None.
A device engine enqueues there and returns, so out must be device-reachable
and read only after the caller joins the stream; a host engine ignores it. |
reportWhat the last round found, as a dict of four counts.report: dict[str, int] matches_emitted is the truth whatever the outputs could hold, matches_stored what was
written, target_length what a rewrite needs, and shortfall what did not fit. On a device
engine these are written by the device, so join the stream before reading them.
>>> engine = sz.SubstringsEngine(sz.Strings(['he', 'she']))
>>> out = memoryview(bytearray(8)).cast('Q')
>>> engine.counts(sz.Strings(['hershey']), out)
>>> engine.report['matches_emitted']
3
containsCheck if a string contains a substring.def contains(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> bool Returns: bool: True if the substring is found, False otherwise.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to search for. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').contains('ell')
True
findFind the first occurrence of a substring.def find(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The index of the first occurrence, or -1 if not found.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to find. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').find('l')
2
indexFind the first occurrence of a substring or raise an error if not found.def index(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The index of the first occurrence. Raises: ValueError: If the substring is not found.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to find. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').index('l')
2
rfindFind the last occurrence of a substring.def rfind(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The index of the last occurrence, or -1 if not found.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to find. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').rfind('l')
3
rindexFind the last occurrence of a substring or raise an error if not found.def rindex(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The index of the last occurrence. Raises: ValueError: If the substring is not found.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to find. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').rindex('l')
3
countCount the occurrences of a substring.def count(
text: _StringLike,
substring: _StringLike,
start: int = 0,
end: int | None = None,
allowoverlap: bool = False,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The number of occurrences of the substring.
text_StringLike | The string object. | |
|---|---|---|
substring_StringLike | The substring to count. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
allowoverlapbool | Count overlapping occurrences, defaulting to False. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('banana').count('a')
3
startswithCheck if a string starts with a given prefix.def startswith(
text: _StringLike,
prefix: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> bool Returns: bool: True if the string starts with the prefix, False otherwise.
text_StringLike | The string object. | |
|---|---|---|
prefix_StringLike | The prefix to check. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').startswith('he')
True
endswithCheck if a string ends with a given suffix.def endswith(
text: _StringLike,
suffix: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> bool Returns: bool: True if the string ends with the suffix, False otherwise.
text_StringLike | The string object. | |
|---|---|---|
suffix_StringLike | The suffix to check. | |
startint | The starting index, defaulting to 0. | |
endint | None | The ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').endswith('lo')
True
find_*_of× 2Find the index of the first occurrence of any character from another string.def find_first_of(
text: _StringLike,
chars: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int | Symbol |
|---|
find_first_of |
find_last_of |
Returns: int: Index of the first matching character, or -1 if none found.
Returns: int: Index of the last matching character, or -1 if none found.
text_StringLike | The string object. | |
|---|---|---|
chars_StringLike | A string containing characters to search for. | |
startint | Starting index, defaulting to 0. | |
endint | None | Ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').find_first_of('aeiou')
1
>>> sz.String('hello').find_last_of('aeiou')
4
find_*_not_of× 2Find the index of the first character not in another string.def find_first_not_of(
text: _StringLike,
chars: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int | Symbol |
|---|
find_first_not_of |
find_last_not_of |
Returns: int: Index of the first non-matching character, or -1 if all match.
Returns: int: Index of the last non-matching character, or -1 if all match.
text_StringLike | The string object. | |
|---|---|---|
chars_StringLike | A string containing characters to exclude. | |
startint | Starting index, defaulting to 0. | |
endint | None | Ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello').find_first_not_of('he')
2
>>> sz.String('hello').find_last_not_of('lo')
1
count_bytesetCount the occurrences of any character from a set of characters.def count_byteset(
text: _StringLike,
chars: _StringLike,
start: int = 0,
end: int | None = None,
*,
capabilities: Capability | None = None,
) -> int Returns: int: The number of occurrences of any character from the set.
text_StringLike | The string object. | |
|---|---|---|
chars_StringLike | A string containing characters to count. | |
startint | Starting index, defaulting to 0. | |
endint | None | Ending index, defaulting to the string length. | |
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.String('hello world').count_byteset('lo')
5
utf8_countCount the number of UTF-8 characters in a string.def utf8_count(
text: _StringLike,
*,
capabilities: Capability | None = None,
) -> int Unlike len() which returns bytes, this counts actual Unicode characters, handling multi-byte UTF-8 sequences correctly.
Returns: int: Number of UTF-8 characters in the string.
text_StringLike | The string object. | |
|---|---|---|
capabilitiesCapability | None | Capabilities to run, defaulting to the CPU's enabled ones. |
>>> sz.utf8_count('hello') # 5 ASCII chars = 5
5
>>> sz.utf8_count('\xc3\xa9') # 1 e-acute char = 1
1