Sorting and Sets

These operate on a Strings collection and return new collections or tuples of numbers, leaving the original unchanged.

§ Sorting

• sorted(reverse=False, uncased=False, top=None) — return a new, stably sorted Strings. reverse sorts descending, uncased orders by Unicode case-folding, and top keeps only the leading top elements (a partial top-k, cheaper than a full sort).
• argsort(reverse=False, uncased=False, top=None, out=None) — return the stable permutation of indices that sorts the collection, as a tuple of ints. out is an optional writable, C-contiguous 64-bit-unsigned buffer such as numpy.uintp or array('Q'), receiving the indices with zero allocation; when given, out itself is returned.

import stringzilla as sz

names = sz.Strings(["banana", "apple", "cherry"])
assert [str(x) for x in names.sorted()] == ["apple", "banana", "cherry"]
assert [str(x) for x in names.sorted(reverse=True)] == ["cherry", "banana", "apple"]
assert [str(x) for x in names.sorted(top=2)] == ["apple", "banana"]
assert names.argsort() == (1, 0, 2)

§ Intersection

• intersect(other, seed=0) — return the positions of strings present in both collections, as a pair of parallel index tuples: result[0][i] in this collection and result[1][i] in other point to equal strings. Each distinct shared value is matched exactly once, even if either side holds duplicates; seed reshuffles the underlying hash table to resist adversarial inputs.

import stringzilla as sz

ours, theirs = sz.Strings(["banana", "apple", "cherry"]).intersect(sz.Strings(["cherry", "orange", "banana"]))
assert sorted(ours) == [0, 2]                 # "banana" and "cherry" on our side
assert len(ours) == len(theirs) == 2

§ Searching and Hashing Every String

• find(needle, out=None) — return the byte offset of the first needle in every string, or -1 where a string holds none, like str.find over each; a str, bytes, or String needle is accepted. out is an optional writable, C-contiguous pointer-width signed buffer such as numpy.intp.
• hashes(seed=0, out=None) — return one unsigned 64-bit hash per string, each equal to sz.hash(text, seed), for group-by keys, joins, and sharding. out is an optional writable, C-contiguous 64-bit unsigned buffer such as numpy.uint64.
• utf8_count(out=None) — return the codepoints of every string, like len(text.decode()) over each, validating every string as UTF-8 on the way and raising ValueError that names the index of the first ill-formed one. out is an optional writable, C-contiguous pointer-width unsigned buffer such as numpy.uintp.

All three run once over the whole tape and, like argsort, return out itself when it is given.

import stringzilla as sz

names = sz.Strings(["hello", "world", "help"])
assert names.find("el") == (1, -1, 1)
assert names.hashes(seed=7) == tuple(sz.hash(name, 7) for name in ["hello", "world", "help"])
assert sz.Strings(["hello", "café", ""]).utf8_count() == (5, 4, 0)

§ Sampling and Shuffling

• sample(size, seed=None) — return a new Strings of size elements drawn at random with replacement; seed makes it reproducible.
• shuffled(seed=None) — return a new Strings with the elements randomly permuted; seed makes it reproducible.

import stringzilla as sz

pool = sz.Strings(["a", "b", "c", "d"])
assert len(pool.sample(2)) == 2
assert [str(x) for x in sorted(pool.shuffled(seed=42))] == ["a", "b", "c", "d"]

§ Random Byte Generation

These module-level functions generate random bytes and are not tied to Strings:

• sz.random(length, nonce=0, alphabet=None) — return a fresh random bytes of length; if alphabet is given, each byte b is mapped to alphabet[b % len(alphabet)].
• sz.fill_random(buffer, nonce=0, alphabet=None, start=0, end=len) — fill a writable, contiguous byte buffer such as a bytearray, memoryview, or String in place with pseudo-random bytes, optionally remapped to alphabet, optionally limited to the [start, end) slice. Returns None. Also available as the String.fill_random method.

import stringzilla as sz

assert len(sz.random(16)) == 16
sz.random(8, alphabet="ACGT")                 # e.g. b'GCTAACGT'

buf = bytearray(16)
sz.fill_random(buf)                           # mutates buf in place
assert len(buf) == 16

§ Escaping and Normalizing Collections

Every verb below is one native call over the whole collection and returns a new Strings owning one tape.

• translate(table): maps every byte through a 256-byte table or a dict, as bytes.translate does for each string.
• escape(kind='json'): escapes every string for a text format.
• unescape(kind='json'): reverses escape, raising ValueError that names the index of the first string holding a malformed escape.
• normalize(form='NFC'): brings every string into a Unicode form, as unicodedata.normalize does, taking the forms utf8_norm takes.

The escape kinds follow the standard library’s own writers and readers:

• json: a string body as json.dumps(text, ensure_ascii=False)[1:-1] writes it, decoded as json.loads decodes it.
• json_ascii: the same with every non-ASCII codepoint as \uXXXX, as json.dumps(text) writes it.
• hex: two lowercase digits per byte, as bytes.hex writes them and bytes.fromhex reads them.
• html: &, <, >, " and ' as entities, as html.escape writes them but with ' for ', decoding the five XML entities and numeric references as html.unescape does.
• url: every byte outside A-Z a-z 0-9 - . _ ~ percent-encoded, as urllib.parse.quote(text, safe="") writes it and urllib.parse.unquote_to_bytes reads it.
• console: controls, DEL, ill-formed UTF-8 and the bidirectional overrides as \n, \xHH or \uXXXX, so a logged string can neither hide nor reorder what a terminal shows.
• csv: a field quoted only around a comma, a quote, CR or LF, with every quote doubled, as csv.writer writes it and csv.reader reads it.

import stringzilla as sz

names = sz.Strings(["Ada", 'Grace "Amazing" Hopper', "Linus\nTorvalds"])
assert [str(x) for x in names.escape("csv")] == ["Ada", '"Grace ""Amazing"" Hopper"', '"Linus\nTorvalds"']
assert [str(x) for x in names.escape("url").unescape("url")] == [str(x) for x in names]
assert [str(x) for x in names.translate({"a": "A"})][0] == "AdA"
assert [str(x) for x in sz.Strings(["Straße", "file"]).normalize("NFKC_Casefold")] == ["strasse", "file"]

§ Dumping JSON and CSV

Writing a column of strings out as JSON or CSV usually costs a Python object per cell: json.dumps walks a list of str, csv.writer formats one row at a time, and pandas first builds a DataFrame of objects. sz.to_json and sz.to_csv instead escape each column once, in one native pass over the whole column, then join the framed rows with one more native call into a single bytes object of exactly the right size.

• Strings.to_json(ensure_ascii=True): one compact JSON array, byte for byte what json.dumps(strings, ensure_ascii=ensure_ascii, separators=(",", ":")) writes.
• sz.to_json(columns, ensure_ascii=True): an array of records, one object per row, byte for byte what json.dumps writes for those records, and what pandas.DataFrame(columns).to_json(orient="records") writes apart from pandas spelling / as \/.
• sz.to_csv(columns, lineterminator="\n"): a header line of the names, then a line per row, byte for byte what csv.writer and pandas.DataFrame(columns).to_csv(index=False) write.

columns maps every name to a Strings or to any sequence of str, all of the same length, and the names are escaped the same way as the cells.

import csv, io, json
import pandas as pd
import stringzilla as sz

names = ["Ada", 'Grace "Amazing" Hopper', "Linus\nTorvalds"]
cities = ["London", "New York, NY", "Helsinki"]
columns = {"name": sz.Strings(names), "city": cities}
frame = pd.DataFrame({"name": names, "city": cities})

records = [{"name": name, "city": city} for name, city in zip(names, cities)]
assert sz.to_json(columns) == json.dumps(records, separators=(",", ":")).encode()
assert sz.to_json(columns) == frame.to_json(orient="records").encode()

buffer = io.StringIO()
writer = csv.writer(buffer, lineterminator="\n")
writer.writerow(columns)
writer.writerows(zip(names, cities))
assert sz.to_csv(columns) == buffer.getvalue().encode()
assert sz.to_csv(columns) == frame.to_csv(index=False, lineterminator="\n").encode()

On an Apple M5 Pro with CPython 3.12 and pandas 3.0, the input is 200,000 English sentences from Leipzig 1M plus a second column holding the first word of each, 27.1 MB of UTF-8 in all. Times are the best of 5 runs, each producing the whole bytes object:

• A JSON array of the sentences with ensure_ascii=True: 39.4 ms for json.dumps and 12.0 ms for Strings.to_json, 3.3x faster.
• The same array with ensure_ascii=False: 75.8 ms and 10.9 ms, 7.0x faster.
• CSV of both columns: 174.5 ms for csv.writer and 188.5 ms for DataFrame.to_csv, against 20.8 ms for sz.to_csv over lists and 17.1 ms over Strings, 8.4x and 10.2x faster than csv.writer.
• JSON records of both columns: 94.1 ms for json.dumps with the dicts built inline and 68.2 ms for DataFrame.to_json, against 15.5 ms for sz.to_json over Strings, 6.1x and 4.4x faster.