Python · Searching and Counting

Searching and Counting

import stringzillafrom stringzilla import RegexEngine, SubdocumentsEngine, SubstringsEngine 6.0.0-rc.1 · 91a27bdd

All search methods accept optional start and end indices that bound the search window; start defaults to 0 and end to the string length. They are available both as String methods and as module-level functions.

  • find(substring, start=0, end=len) — index of the first occurrence, or -1 if absent.
  • rfind(substring, start=0, end=len) — index of the last occurrence, or -1.
  • index(substring, start=0, end=len) — like find, but raises ValueError if absent.
  • rindex(substring, start=0, end=len) — like rfind, but raises ValueError if absent.
  • contains(substring, start=0, end=len) — True/False membership test.
  • startswith(prefix, start=0, end=len) — True if the windowed string starts with prefix.
  • endswith(suffix, start=0, end=len) — True if the windowed string ends with suffix.
  • count(substring, start=0, end=len, allowoverlap=False) — number of occurrences; pass allowoverlap=True to count overlapping matches.
import stringzilla as sz

text = sz.String("the quick brown fox")
assert text.find("brown") == 10
assert text.rfind("o") == 17
assert text.index("quick") == 4
assert text.contains("quick")
assert text.startswith("the")
assert text.endswith("fox")
assert sz.String("banana").count("a") == 3
assert sz.String("aaaa").count("aa") == 2                     # non-overlapping
assert sz.String("aaaa").count("aa", allowoverlap=True) == 3  # overlapping

The *_of / *_not_of family searches for any byte from a set, the byteset-accelerated counterpart of substring search. count_byteset counts how many bytes fall in the set.

  • find_first_of(chars, start=0, end=len) — index of the first byte that is in chars, or -1.
  • find_last_of(chars, start=0, end=len) — index of the last byte that is in chars, or -1.
  • find_first_not_of(chars, start=0, end=len) — index of the first byte not in chars, or -1.
  • find_last_not_of(chars, start=0, end=len) — index of the last byte not in chars, or -1.
  • count_byteset(chars, start=0, end=len) — number of bytes that are in chars.
import stringzilla as sz

text = sz.String("hello world")
assert text.find_first_of("aeiou") == 1
assert text.find_last_of("aeiou") == 7
assert text.find_first_not_of("he") == 2
assert text.find_last_not_of("lo") == 10
assert text.count_byteset("lo") == 5

API

ClassRegexEngineCompile a vocabulary of RE2-syntax patterns once and stream many collections of haystacks through it, reporting leftmost-first matches across all of them, the lower index winning ties.5
__init__.pyi:2561 ↗
class RegexEngine

Example:

engine = sz.RegexEngine(sz.Strings(['[0-9]+', '[a-z]+'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['abc 123 de']), out) out[0] 3

RegexEngine
__init__.pyi:2584 ↗
RegexEngine(
    patterns: Strings,
    case_sensitivity: str = 'cased',
    *,
    capabilities: Capability | None = None,
    stream: int | None = None,
)
*× 2counts(haystacks, out, *, stream=None) -> None
__init__.pyi:2592 ↗
def counts(
    haystacks: Strings,
    out: _Buffer,
    *,
    stream: int | None = None,
) -> None

Count the leftmost-first matches of the vocabulary in every haystack, into out.

Example:

engine = sz.RegexEngine(sz.Strings(['[0-9]+', '[a-z]+'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['abc 123 de']), out) out[0] 3

Count the matches of every needle in every haystack, into out.

Example:

engine = sz.SubstringsEngine(sz.Strings(['he', 'she'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['hershey']), out) out[0] 3

Parameters

haystacksStringsTexts to search, from Strings.copy on a GPU.
out_BufferWritable 1-D buffer of pointer-width unsigned integers, one per haystack.
streamint | NoneA stream of the engine's device as an integer, or None.
*× 2find(haystacks, matches, offsets, *, stream=None) -> None
__init__.pyi:2608 ↗
def find(
    haystacks: Strings,
    matches: _Buffer | None,
    offsets: _Buffer,
    *,
    stream: int | None = None,
) -> None

Locate every leftmost-first match, writing one row per match and one boundary per haystack.

A capacity too small is not an error: offsets is filled either way, which sizes the next call.

Example:

engine = sz.RegexEngine(sz.Strings(['[0-9]+'])) matches = memoryview(bytearray(2 * 4 * 8)).cast('Q', (2, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['a 12 b 345']), matches, offsets) matches[1, 2], matches[1, 3] (7, 3)

Locate every match, writing one row per match and one boundary per haystack.

A capacity too small is not an error: offsets and report are filled either way, which is what sizes the next call.

Example:

engine = sz.SubstringsEngine(sz.Strings(['he', 'she'])) matches = memoryview(bytearray(3 * 4 * 8)).cast('Q', (3, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['hershey']), matches, offsets) offsets[1] 3

Parameters

haystacksStringsTexts to search, from Strings.copy on a GPU.
matches_Buffer | NoneWritable, fully contiguous 2-D buffer of pointer-width unsigned integers, shaped (capacity, 4) - haystack index, pattern index, byte offset, byte length. None makes the call a pure size query.
offsets_BufferWritable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 boundaries into matches, the last being the total.
streamint | NoneA stream of the engine's device as an integer, or None.
replacereplace(haystacks, replacements, target, offsets) -> None
__init__.pyi:2631 ↗
def replace(
    haystacks: Strings,
    replacements: Strings,
    target: _Buffer | None,
    offsets: _Buffer,
) -> None

Rewrite every haystack into one target buffer, one replacement per pattern.

A replacement names groups as $1, ${1} or ${name}, $0 being the whole match and $ one dollar. A target too small is not an error: offsets is filled either way. CPU only.

Example:

engine = sz.RegexEngine(sz.Strings(['(\w+)@(\w+)'])) target = memoryview(bytearray(16)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.replace(sz.Strings(['bob@host']), sz.Strings(['$2/$1']), target, offsets) bytes(target[offsets[0]:offsets[1]]) b'host/bob'

Parameters

haystacksStringsTexts to rewrite.
replacementsStringsOne replacement per pattern; an empty one deletes the match.
target_Buffer | NoneWritable, contiguous byte buffer; None makes the call a size query.
offsets_BufferWritable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 rewritten boundaries, the last being the total.
extractextract(haystacks, pattern_index, groups, offsets, *, stream=None) -> None
__init__.pyi:2653 ↗
def extract(
    haystacks: Strings,
    pattern_index: int,
    groups: _Buffer | None,
    offsets: _Buffer,
    *,
    stream: int | None = None,
) -> None

Locate the groups of each haystack's first match of a pattern, as Pandas str.extract reads them.

A haystack with such a match gets a row for the whole match, group index zero, then one row per group that took part, so a group left out differs from an empty one; a haystack without one gets no rows. A capacity too small is not an error: offsets is filled either way, which sizes the next call. haystacks.matched(groups) reads the rows as strings.

Example:

engine = sz.RegexEngine(sz.Strings(['(\d+)-(\d+)'])) groups = memoryview(bytearray(3 * 4 * 8)).cast('Q', (3, 4)) offsets = memoryview(bytearray(3 * 8)).cast('Q') haystacks = sz.Strings(['a 12-34', 'none']) engine.extract(haystacks, 0, groups, offsets) list(map(bytes, haystacks.matched(groups))), offsets[2] ([b'12-34', b'12', b'34'], 3)

Parameters

haystacksStringsTexts to search, from Strings.copy on a GPU.
pattern_indexintThe pattern whose groups to locate.
groups_Buffer | NoneWritable, fully contiguous 2-D buffer of pointer-width unsigned integers, shaped (capacity, 4) - haystack index, group index, byte offset, byte length. None makes the call a pure size query.
offsets_BufferWritable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 boundaries into groups, the last being the total.
streamint | NoneA stream of the engine's device as an integer, or None.
ClassSubdocumentsEngineCompile JSONPath or XPath queries once and pull what they select out of many documents.3
__init__.pyi:2684 ↗
class SubdocumentsEngine

Every document is walked once, whatever the path count, and no tree is built. A match spans the node's exact bytes: in JSON a string with its quotes, a number or literal, or a container through its closing bracket; in XML an element, an attribute's value, or a text node.

Raises: ValueError: Naming the first path that breaks its grammar or leaves the supported subset. Example:

engine = sz.SubdocumentsEngine(sz.Strings(['$.a[?@ > 1]'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['{"a": [1, 2, 3]}']), out) out[0] 2

SubdocumentsEngine
__init__.pyi:2712 ↗
SubdocumentsEngine(
    paths: Strings,
    format: str = 'json',
    *,
    capabilities: Capability | None = None,
    stream: int | None = None,
)
countscounts(documents, out, *, stream=None) -> None
__init__.pyi:2715 ↗
def counts(
    documents: Strings,
    out: _Buffer,
    *,
    stream: int | None = None,
) -> None

Count the nodes of every document that some path selects, once per path, into out.

Raises: ValueError: Naming the first malformed document. Example:

engine = sz.SubdocumentsEngine(sz.Strings(['$..name'])) out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['{"name": "a", "b": {"name": "c"}}']), out) out[0] 2

Parameters

documentsStringsDocuments to walk, from Strings.copy on a GPU.
out_BufferWritable 1-D buffer of pointer-width unsigned integers, one per document.
streamint | NoneA stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so out must be device-reachable and read only after the caller joins the stream; a host engine ignores it.
findfind(documents, matches, offsets, *, stream=None) -> None
__init__.pyi:2735 ↗
def find(
    documents: Strings,
    matches: _Buffer | None,
    offsets: _Buffer,
    *,
    stream: int | None = None,
) -> None

Locate every selected node, writing one row per match and one boundary per document.

Matches come by document, then start offset, then path, so a container precedes what it holds. A capacity too small is not an error: offsets is filled either way, which is what sizes the next call, and Strings.matched gathers the spans the rows locate.

Raises: ValueError: Naming the first malformed document. Example:

engine = sz.SubdocumentsEngine(sz.Strings(['/a/b/text()']), format='xml') matches = memoryview(bytearray(2 * 4 * 8)).cast('Q', (2, 4)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.find(sz.Strings(['<a><b>x</b><b>y</b></a>']), matches, offsets) offsets[1] 2

Parameters

documentsStringsDocuments to walk, from Strings.copy on a GPU.
matches_Buffer | NoneWritable, fully contiguous 2-D buffer of pointer-width unsigned integers, shaped (capacity, 4) - document index, path index, byte offset, byte length. None makes the call a pure size query.
offsets_BufferWritable, contiguous 1-D buffer of pointer-width unsigned integers holding len(documents) + 1 boundaries into matches, the last being the total.
streamint | NoneA stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so both buffers must be device-reachable and read only after the caller joins the stream; a host engine ignores it.
ClassSubstringsEngineSubstringsEngine(needles, case_sensitivity='cased', overlap_policy='overlapping', hot_states=None, matches_budget=0, *, haystacks_budget=0, capabilities=None, stream=None)4
__init__.pyi:2765 ↗
class SubstringsEngine

Compile a vocabulary once and stream many collections of haystacks through it.

One Aho-Corasick pass answers every needle at once, whatever the vocabulary size.

Example:

engine = sz.SubstringsEngine(sz.Strings(['he', 'she']), case_sensitivity='cased') out = memoryview(bytearray(8)).cast('Q') engine.counts(sz.Strings(['hershey']), out) out[0] 3

SubstringsEngine
__init__.pyi:2794 ↗
SubstringsEngine(
    needles: Strings,
    case_sensitivity: str = 'cased',
    overlap_policy: str = 'overlapping',
    hot_states: int | None = None,
    matches_budget: int = 0,
    *,
    haystacks_budget: int = 0,
    capabilities: Capability | None = None,
    stream: int | None = None,
)
replacereplace(haystacks, replacements, target, offsets, *, stream=None) -> None
__init__.pyi:2865 ↗
def replace(
    haystacks: Strings,
    replacements: Strings,
    target: _Buffer | None,
    offsets: _Buffer,
    *,
    stream: int | None = None,
) -> None

Rewrite every haystack into one target buffer, one replacement per needle.

Needs a leftmost policy, since a substitution over matches that share bytes is not a function. A target too small is not an error: offsets and report are filled either way.

Example:

engine = sz.SubstringsEngine(sz.Strings(['he', 'she']), overlap_policy='leftmost-longest') target = memoryview(bytearray(16)) offsets = memoryview(bytearray(2 * 8)).cast('Q') engine.replace(sz.Strings(['hershey']), sz.Strings(['HE', 'SHE']), target, offsets) bytes(target[offsets[0]:offsets[1]]) b'HErSHEy'

Parameters

haystacksStringsTexts to rewrite, from Strings.copy on a GPU.
replacementsStringsOne replacement per needle; an empty one deletes the match. On a GPU engine, from Strings.copy too.
target_Buffer | NoneWritable, contiguous byte buffer; None makes the call a size query.
offsets_BufferWritable, contiguous 1-D buffer of pointer-width unsigned integers holding len(haystacks) + 1 rewritten boundaries, the last being the total.
streamint | NoneA stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so both buffers must be device-reachable and read only after the caller joins the stream; a host engine ignores it.
bm25_scoresbm25_scores(haystacks, needle_weights, out, *, document_lengths=None, term_frequency_saturation=1.2, length_normalization=0.75, average_document_length=0.0, stream=None) -> None
__init__.pyi:2899 ↗
def bm25_scores(
    haystacks: Strings,
    needle_weights: _Buffer,
    out: _Buffer,
    *,
    document_lengths: _Buffer | None = None,
    term_frequency_saturation: float = 1.2,
    length_normalization: float = 0.75,
    average_document_length: float = 0.0,
    stream: int | None = None,
) -> None

Score every haystack against the vocabulary, which is the query, into out.

Term frequencies are raw overlapping counts, so the engine's own overlap policy does not apply.

Example:

engine = sz.SubstringsEngine(sz.Strings(['cat', 'dog'])) weights = memoryview(bytearray(2 * 4)).cast('f') weights[0], weights[1] = 1.0, 1.0 out = memoryview(bytearray(2 * 4)).cast('f') engine.bm25_scores(sz.Strings(['cat and dog', 'nothing here']), weights, out, ... average_document_length=10.0) out[1] 0.0

Parameters

haystacksStringsDocuments to score, from Strings.copy on a GPU.
needle_weights_BufferOne 32-bit float IDF or boost per needle.
out_BufferWritable 1-D buffer of 32-bit floats, one score per haystack.
document_lengths_Buffer | NoneOne 32-bit float length per haystack; byte lengths when None.
term_frequency_saturationfloatThe literature's k1.
length_normalizationfloatThe literature's b, in [0, 1].
average_document_lengthfloatCorpus-wide mean of the lengths above; required whenever length_normalization is positive.
streamint | NoneA stream of the engine's device as an integer, or None. A device engine enqueues there and returns, so out must be device-reachable and read only after the caller joins the stream; a host engine ignores it.
reportWhat the last round found, as a dict of four counts.
__init__.pyi:2807 ↗
report: dict[str, int]

matches_emitted is the truth whatever the outputs could hold, matches_stored what was written, target_length what a rewrite needs, and shortfall what did not fit. On a device engine these are written by the device, so join the stream before reading them.

Examples

>>> engine = sz.SubstringsEngine(sz.Strings(['he', 'she']))
>>> out = memoryview(bytearray(8)).cast('Q')
>>> engine.counts(sz.Strings(['hershey']), out)
>>> engine.report['matches_emitted']
3

Functions

containsCheck if a string contains a substring.
__init__.pyi:366 ↗
def contains(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> bool

Returns: bool: True if the substring is found, False otherwise.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to search for.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').contains('ell')
True
findFind the first occurrence of a substring.
__init__.pyi:390 ↗
def find(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The index of the first occurrence, or -1 if not found.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to find.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').find('l')
2
indexFind the first occurrence of a substring or raise an error if not found.
__init__.pyi:414 ↗
def index(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The index of the first occurrence. Raises: ValueError: If the substring is not found.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to find.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').index('l')
2
rfindFind the last occurrence of a substring.
__init__.pyi:440 ↗
def rfind(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The index of the last occurrence, or -1 if not found.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to find.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').rfind('l')
3
rindexFind the last occurrence of a substring or raise an error if not found.
__init__.pyi:464 ↗
def rindex(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The index of the last occurrence. Raises: ValueError: If the substring is not found.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to find.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').rindex('l')
3
countCount the occurrences of a substring.
__init__.pyi:524 ↗
def count(
    text: _StringLike,
    substring: _StringLike,
    start: int = 0,
    end: int | None = None,
    allowoverlap: bool = False,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The number of occurrences of the substring.

Parameters

text_StringLikeThe string object.
substring_StringLikeThe substring to count.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
allowoverlapboolCount overlapping occurrences, defaulting to False.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('banana').count('a')
3
startswithCheck if a string starts with a given prefix.
__init__.pyi:550 ↗
def startswith(
    text: _StringLike,
    prefix: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> bool

Returns: bool: True if the string starts with the prefix, False otherwise.

Parameters

text_StringLikeThe string object.
prefix_StringLikeThe prefix to check.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').startswith('he')
True
endswithCheck if a string ends with a given suffix.
__init__.pyi:574 ↗
def endswith(
    text: _StringLike,
    suffix: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> bool

Returns: bool: True if the string ends with the suffix, False otherwise.

Parameters

text_StringLikeThe string object.
suffix_StringLikeThe suffix to check.
startintThe starting index, defaulting to 0.
endint | NoneThe ending index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').endswith('lo')
True
find_*_of× 2Find the index of the first occurrence of any character from another string.
__init__.pyi:598 ↗
def find_first_of(
    text: _StringLike,
    chars: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int
Symbol
find_first_of
find_last_of

Returns: int: Index of the first matching character, or -1 if none found.

Returns: int: Index of the last matching character, or -1 if none found.

Parameters

text_StringLikeThe string object.
chars_StringLikeA string containing characters to search for.
startintStarting index, defaulting to 0.
endint | NoneEnding index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').find_first_of('aeiou')
1
>>> sz.String('hello').find_last_of('aeiou')
4
find_*_not_of× 2Find the index of the first character not in another string.
__init__.pyi:622 ↗
def find_first_not_of(
    text: _StringLike,
    chars: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int
Symbol
find_first_not_of
find_last_not_of

Returns: int: Index of the first non-matching character, or -1 if all match.

Returns: int: Index of the last non-matching character, or -1 if all match.

Parameters

text_StringLikeThe string object.
chars_StringLikeA string containing characters to exclude.
startintStarting index, defaulting to 0.
endint | NoneEnding index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello').find_first_not_of('he')
2
>>> sz.String('hello').find_last_not_of('lo')
1
count_bytesetCount the occurrences of any character from a set of characters.
__init__.pyi:694 ↗
def count_byteset(
    text: _StringLike,
    chars: _StringLike,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> int

Returns: int: The number of occurrences of any character from the set.

Parameters

text_StringLikeThe string object.
chars_StringLikeA string containing characters to count.
startintStarting index, defaulting to 0.
endint | NoneEnding index, defaulting to the string length.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.String('hello world').count_byteset('lo')
5
utf8_countCount the number of UTF-8 characters in a string.
__init__.pyi:1563 ↗
def utf8_count(
    text: _StringLike,
    *,
    capabilities: Capability | None = None,
) -> int

Unlike len() which returns bytes, this counts actual Unicode characters, handling multi-byte UTF-8 sequences correctly.

Returns: int: Number of UTF-8 characters in the string.

Parameters

text_StringLikeThe string object.
capabilitiesCapability | NoneCapabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> sz.utf8_count('hello')  # 5 ASCII chars = 5
5
>>> sz.utf8_count('\xc3\xa9')  # 1 e-acute char = 1
1