from stringzilla import Strings, String6.0.0-rc.1 · 91a27bdd
The stringzilla module exposes String, Strings, and File, the incremental hashers Hasher, Sha256 and Sha256s, the batch engines LevenshteinEngine, OverlapEngine and SubstringsEngine, plus a family of UTF-8 iterator types.
String(source) is an immutable, zero-copy view over a str, bytes, bytearray, another String, or a memory-mapped File.
It implements the read-only half of the str/bytes protocol — indexing, slicing, iteration, len, in, rich comparison, hashing, and the buffer protocol — without materializing new Python objects.
Slicing returns a new String view into the same backing buffer rather than copying bytes.
Two read-only properties expose its memory layout for zero-copy interop with PyArrow, ctypes, and similar:
String.address — integer memory address of the first byte.
String.nbytes — length of the view in bytes.
import stringzilla as sztext = sz.String("the quick brown fox")assert str(text[4:9]) == "quick" # a zero-copy viewassert str(text[10:15]) == "brown"assert "quick" in textassert len(text) == 19assert text.nbytes == 19assert isinstance(text.address, int)
Strings(sequence=()) is an immutable, indexable, sliceable sequence of byte strings, like a tuple of bytes, kept on one tape: the bytes back to back and an offsets array, the Apache Arrow string or large_string layout.
A pyarrow.Array of strings or binaries is borrowed in place, while a list, tuple, generator, or any other iterable of str and bytes-like objects is copied once.
The eager splits return one too, with the parts copied onto a tape of their own. strings.copy(capabilities, stream=None) copies the strings into one tape in the unified memory of capabilities, as Engines on a GPU describes; the host reads it as a normal read-only Strings.
Indexing yields String views into the tape, and slicing with a step of one borrows it, so neither copies any bytes.
Any other step, sorted, shuffled, and sample gather into a new tape, while argsort and intersect return indices and copy nothing.
One tape is also dramatically more memory-efficient than a Python list of str.
N strings cost two allocations, whereas a list of str holds N separately heap-allocated objects, each carrying a PyObject header, a hash cache, and per-object allocator overhead.
Splitting a large document into millions of tokens therefore stays compact and cache-friendly instead of fragmenting the heap.
The lazy split_iter, rsplit_iter, and *_byteset_iter forms go a step further and build no collection at all, yielding one String view at a time.
Five read-only properties expose the tape for Apache Arrow and StringTape interop:
Strings.tape_address — address of the tape’s first byte, which the offsets count from.
Strings.tape_nbytes — bytes from that address to the end of the last string.
Strings.offsets_address — address of the first offset.
Strings.offsets_nbytes — length of the offsets array in bytes, one offset more than there are strings.
Strings.offsets_are_large — True for the 64-bit offsets a tape past 4 GiB needs, False for 32-bit ones.
A borrowed slice shares tape_address with its source, while its offsets start past zero.
import stringzilla as sznames = sz.Strings(["banana", "apple", "cherry", "date"])assert len(names) == 4assert str(names[0]) == "banana"assert [str(x) for x in names[1:3]] == ["apple", "cherry"] # a borrowed sliceassert names[1:3].tape_address == names.tape_address
File(path, mode='r') memory-maps a path so a String can view data larger than RAM without reading it into memory.
The default mode is 'r', for read-only access.
import os, tempfile, pathlibimport stringzilla as szpath = os.path.join(tempfile.mkdtemp(), "log.txt")pathlib.Path(path).write_text("error: disk full\nok\nerror: timeout\n")mapped = sz.String(sz.File(path)) # scan in place, no copy into RAMassert mapped.count("error") == 2
API
ClassStringsAn immutable sequence of byte strings on one tape - their bytes back to back and an array of offsets, the Apache Arrow string or large_string layout - like a tuple of bytes that costs two allocations rather than one per element.20
Slicing with a step of one borrows the tape, while any other step, sorted, shuffled and
sample gather into a new one, and argsort returns the order without copying, as find,
hashes and utf8_count return one number per string in a single pass over the tape.
Examples
>>> names = sz.Strings(['banana', 'apple', 'cherry', 'date'])>>> [str(x) for x in names.sorted()]['apple', 'banana', 'cherry', 'date']>>> [str(x) for x in names.sorted(top=2)] # partial top-k is cheaper than a full sort['apple', 'banana']
Find the first occurrence of needle in every string, like str.find over each, counting bytes
as bytes.find does, in one pass over the whole tape.
Returns:
tuple[int, ...]: The byte offset of the first match in each string, or -1 where it holds none,
or out itself when an out buffer is given.
Example:
Hash every string under one seed, each equal to sz.hash(text, seed), like the built-in hash
over each but stable across runs and platforms, for group-by keys, joins and sharding.
Returns:
tuple[int, ...]: One unsigned 64-bit hash per string, or out itself when an out buffer is
given.
Example:
Count the codepoints of every string, like len(text.decode()) over each, validating every one
as UTF-8 in the same pass over the whole tape.
Returns:
tuple[int, ...]: The codepoints of each string, or out itself when an out buffer is given.
Raises:
ValueError: Naming the index of the first string holding ill-formed UTF-8.
Example:
Return a new Strings holding the bytes each match row spans, in row order, gathering what
a find of SubstringsEngine or SubdocumentsEngine located in this collection.
Returns:
Strings: A new collection of the matched spans on a tape of its own.
Raises:
IndexError: When a row reaches outside this collection.
Example:
Return the positions of strings present in both collections.
Each distinct shared value is matched exactly once, even if either side has duplicates.
Returns:
tuple[tuple[int, ...], tuple[int, ...]]: Parallel position tuples - result[0][i] in
this collection and result[1][i] in other point to equal strings.
Example:
Copy the strings into a tape in the unified memory of capabilities, which engines of those
capabilities read in place and the host reads as a normal read-only Strings.
Returns:
Strings: A tape owning its unified memory, freed with it.
Example:
tape = sz.Strings(['kitten', 'sitting']).copy(sz.cpu_capabilities_enabled())
[str(text) for text in tape]
['kitten', 'sitting']
Parameters
capabilitiesint
A producer's mask, like sz.cuda_capabilities_enabled(0).
streamint | None
A stream of that vendor as an integer, naming the device the tape is
made on, or None for the default.
Map every byte of every string through a look-up table, like bytes.translate over each.
Returns:
Strings: A new collection, each string as long as its source.
Raises:
ValueError: If the table is not 256 bytes long, or a dict names a character above U+00FF.
Example:
[str(text) for text in sz.Strings(['cab', 'bee']).translate({'a': 'A', 'b': 'B'})]
['cAB', 'Bee']
Parameters
tablestr | bytes | dict[str, str]
A 256-byte table, or a dict mapping characters below U+0100.
Escape every string for a text format, in one pass over the whole collection.
Returns:
Strings: A new collection of the escaped strings.
Example:
[str(text) for text in sz.Strings(['<b>', 'Tom & Jerry']).escape('html')]
['<b>', 'Tom & Jerry']
Reverse escape for every string, in one pass over the whole collection. The json kinds
decode what json.loads decodes in a string body, html what html.unescape decodes of the
five XML entities and numeric references, url what urllib.parse.unquote_to_bytes decodes,
and csv one field as csv.reader reads it.
Returns:
Strings: A new collection of the unescaped strings.
Raises:
ValueError: Naming the index of the first string holding a malformed escape.
Example:
[str(text) for text in sz.Strings(['a%20b', '100%25']).unescape('url')]
['a b', '100%']
Capabilities to run, defaulting to the CPU's enabled ones.
Kinds
json: A JSON string body, as json.dumps(text, ensure_ascii=False) writes it unquoted.
json_ascii: The same with every non-ASCII codepoint as \uXXXX, like json.dumps(text).
hex: Two lowercase digits per byte, as bytes.hex writes them.
html: &, <, >, " and ' as entities, like html.escape, with ' for '.
url: Percent-encoding, as urllib.parse.quote(text, safe='') writes it.
console: Controls, ill-formed UTF-8 and bidirectional overrides as \n, \xHH or \uXXXX.
csv: A field quoted only around a comma, a quote, CR or LF, as csv.writer writes it.
Bring every string into a Unicode normalization form, in one pass over the whole collection,
like unicodedata.normalize over each. 'NFKC_Casefold' is the Unicode toNFKC_Casefold, the
keys search engines match on: NFKC with full case folding and no default-ignorable codepoints.
Returns:
Strings: A new collection of the normalized strings.
Example:
[str(text) for text in sz.Strings(['\ufb01', 'Stra\u00dfe']).normalize('NFKC_Casefold')]
['fi', 'strasse']
Parameters
formstr
'NFC', 'NFD', 'NFKC', 'NFKD' or 'NFKC_Casefold', defaulting to 'NFC'.
Provides high-performance byte-string operations using modern CPU instructions.
Functions
find(), rfind(): Fast substring search with SIMD acceleration
count(): Count occurrences with optional overlap support
split(), rsplit(): String splitting with various separators
contains(): Fast membership testing
translate(): Byte-level translations with lookup tables
Examples
>>> text = sz.String('the quick brown fox')>>> text.find('brown') # SIMD-accelerated substring search10>>> str(text[10:15]) # slicing returns a zero-copy view, not a new buffer'brown'
def hash(
seed: int = 0,
/,
*,
capabilities: Capability | None = None,
) -> int
This function can be called as a method on a String object or as a standalone function.
Args:
text (String or str or bytes): The string to hash, positional-only when standalone.
seed (int, optional): The seed value for hashing. Defaults to 0. Can be positional or keyword.
capabilities (Capability, optional): Capabilities to run, defaulting to the CPU's enabled ones.
Returns:
int: The hash value as an unsigned 64-bit integer. This differs from Python's
built-in hash() which returns a Py_hash_t and may be platform-dependent.
Raises:
TypeError: If the argument is not string-like or incorrect number of arguments is provided.
Signature:
Equivalent to tuple(hash(text, s) for s in seeds), but normalizes the input into AES
blocks once and replays the cheap per-seed rounds - markedly faster for short strings under
many seeds (feature hashing, Count-Min sketches, Bloom/cuckoo filters, MinHash/LSH).
Returns:
tuple[int, ...] | None: Tuple of 64-bit hashes, or None when writing into out.
Parameters
seeds_Buffer
Contiguous 64-bit seeds, e.g. numpy.uint64 array or array('Q', ...).
Plain int lists are not accepted - wrap them in array('Q', seeds) or numpy.
out_Buffer | None
A contiguous buffer of at least len(seeds) elements.
If given, it is filled in place and None returned; otherwise a tuple of ints is returned.
This function can be called as a method on a String object or as a standalone function.
Args:
text (String or str or bytes): The input data to hash.
capabilities (Capability, optional): Capabilities to run, defaulting to the CPU's enabled ones.
Returns:
bytes: The 32-byte (256-bit) SHA256 digest.
Raises:
TypeError: If the argument is not string-like or incorrect number of arguments is provided.
Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF.
Unlike splitlines(), this returns an iterator for memory-efficient processing.
Returns:
iterator: An iterator yielding the lines as String objects.
Parameters
skip_emptybool
Skip empty lines, defaulting to False.
with_separatorsbool
Interleave the separators losslessly, defaulting to False.
Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF.
Yields each newline separator; utf8_split_newlines() yields the lines between them.
Returns:
iterator: An iterator yielding the newline separators as String objects.
Uses SIMD-accelerated detection of all 25 Unicode White_Space characters.
Splits on each whitespace codepoint; pass skip_empty=True for str.split()-style tokens.
Returns:
iterator: An iterator yielding the non-whitespace tokens as String objects.
Parameters
skip_emptybool
Skip empty segments, defaulting to False.
with_separatorsbool
Interleave the separators losslessly, defaulting to False.
Capabilities to run, defaulting to the CPU's enabled ones.
Recognized whitespace
ASCII: TAB, LF, VT, FF, CR, SPACE
Latin-1: NEXT LINE, NO-BREAK SPACE
General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc.
CJK: IDEOGRAPHIC SPACE (U+3000)
Examples
>>> sum(1 for _ in sz.String('foo bar baz').utf8_split_whitespaces())3
utf8_whitespacesCreate an iterator over Unicode whitespace separators, one per codepoint, CR+LF kept together.
Uses SIMD-accelerated detection of all 25 Unicode White_Space characters.
Yields each whitespace codepoint; utf8_split_whitespaces() yields the tokens between them.
Returns:
iterator: An iterator yielding the whitespace separators as String objects.
Capabilities to run, defaulting to the CPU's enabled ones.
Recognized whitespace
ASCII: TAB, LF, VT, FF, CR, SPACE
Latin-1: NEXT LINE, NO-BREAK SPACE
General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc.
CJK: IDEOGRAPHIC SPACE (U+3000)
Examples
>>> sum(1 for _ in sz.String('foo bar').utf8_whitespaces())1
utf8_split_delimitersCreate an iterator over the fields, the content between Unicode delimiters.
Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and
separator/whitespace (Z*) codepoint - the superset of utf8_split_whitespaces().
Splits on delimiters, one separator per codepoint, and yields the segments between them.
Returns:
iterator: An iterator yielding the non-delimiter segments as String objects.
Parameters
skip_emptybool
Skip empty segments, defaulting to False.
with_separatorsbool
Interleave the separators losslessly, defaulting to False.
Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and
separator/whitespace (Z*) codepoint. Yields each delimiter codepoint;
utf8_split_delimiters() yields the fields between them.
Returns:
iterator: An iterator yielding the delimiter separators as String objects.
Transforms the input into NFD, NFC, NFKD, or NFKC, mirroring unicodedata.normalize but
returning raw UTF-8 bytes, or into NFKC_Casefold, the Unicode toNFKC_Casefold: NFKC with full
case folding and no default-ignorable codepoints.
Parameters
formstr
One of 'NFC', 'NFD', 'NFKC', 'NFKD', 'NFKC_Casefold'.
validatebool
If True, validate UTF-8 before processing. Default: False.
Scans the UTF-8 string and returns the byte offset of the first codepoint
that breaks the given normalization form (first non-Yes Quick-Check result
or canonical-ordering violation). Returns None when the string is already
fully normalized.
Parameters
formstr
One of 'NFC', 'NFD', 'NFKC', 'NFKD', 'NFKC_Casefold'.
This function uses Unicode case folding for proper handling of
international text. The matched region length may differ from the
needle length due to case folding expansions, like 'ß' matching 'SS'.
An empty needle matches at every codepoint boundary, the end included.
Returns:
Union[None, str, bytes]: If inplace is False, a translated copy of the [start, end) slice is
returned, otherwise None.
Raises:
ValueError: If the table is not 256 bytes long, or a dict names a character above U+00FF.
UnicodeDecodeError: If a str input translates into malformed UTF-8.
TypeError: If the table is not a string or dictionary.
Parameters
tablestr | bytes | dict[str, str]
A 256-byte string, or a dict mapping characters below U+0100 to such.
inplacebool
If True, the string is modified in place, defaulting to False.
startint
The starting index for translation, defaulting to 0.
endint | None
The ending index for translation, defaulting to the string length.