# Sorting and Sets

> StringZilla for Python

These operate on a `Strings` collection and return new collections or tuples of numbers, leaving the original unchanged.

### Sorting

- `sorted(reverse=False, uncased=False, top=None)` — return a new, stably sorted `Strings`.
  `reverse` sorts descending, `uncased` orders by Unicode case-folding, and `top` keeps only the leading `top` elements (a partial top-k, cheaper than a full sort).
- `argsort(reverse=False, uncased=False, top=None, out=None)` — return the stable permutation of indices that sorts the collection, as a tuple of ints.
  `out` is an optional writable, C-contiguous 64-bit-unsigned buffer such as `numpy.uintp` or `array('Q')`, receiving the indices with zero allocation; when given, `out` itself is returned.

```python
import stringzilla as sz

names = sz.Strings(["banana", "apple", "cherry"])
assert [str(x) for x in names.sorted()] == ["apple", "banana", "cherry"]
assert [str(x) for x in names.sorted(reverse=True)] == ["cherry", "banana", "apple"]
assert [str(x) for x in names.sorted(top=2)] == ["apple", "banana"]
assert names.argsort() == (1, 0, 2)
```

### Intersection

- `intersect(other, seed=0)` — return the positions of strings present in both collections, as a pair of parallel index tuples: `result[0][i]` in this collection and `result[1][i]` in `other` point to equal strings.
  Each distinct shared value is matched exactly once, even if either side holds duplicates; `seed` reshuffles the underlying hash table to resist adversarial inputs.

```python
import stringzilla as sz

ours, theirs = sz.Strings(["banana", "apple", "cherry"]).intersect(sz.Strings(["cherry", "orange", "banana"]))
assert sorted(ours) == [0, 2]                 # "banana" and "cherry" on our side
assert len(ours) == len(theirs) == 2
```

### Searching and Hashing Every String

- `find(needle, out=None)` — return the byte offset of the first `needle` in every string, or `-1` where a string holds none, like `str.find` over each; a `str`, `bytes`, or `String` needle is accepted.
  `out` is an optional writable, C-contiguous pointer-width signed buffer such as `numpy.intp`.
- `hashes(seed=0, out=None)` — return one unsigned 64-bit hash per string, each equal to `sz.hash(text, seed)`, for group-by keys, joins, and sharding.
  `out` is an optional writable, C-contiguous 64-bit unsigned buffer such as `numpy.uint64`.
- `utf8_count(out=None)` — return the codepoints of every string, like `len(text.decode())` over each, validating every string as UTF-8 on the way and raising `ValueError` that names the index of the first ill-formed one.
  `out` is an optional writable, C-contiguous pointer-width unsigned buffer such as `numpy.uintp`.

All three run once over the whole tape and, like `argsort`, return `out` itself when it is given.

```python
import stringzilla as sz

names = sz.Strings(["hello", "world", "help"])
assert names.find("el") == (1, -1, 1)
assert names.hashes(seed=7) == tuple(sz.hash(name, 7) for name in ["hello", "world", "help"])
assert sz.Strings(["hello", "café", ""]).utf8_count() == (5, 4, 0)
```

### Sampling and Shuffling

- `sample(size, seed=None)` — return a new `Strings` of `size` elements drawn at random with replacement; `seed` makes it reproducible.
- `shuffled(seed=None)` — return a new `Strings` with the elements randomly permuted; `seed` makes it reproducible.

```python
import stringzilla as sz

pool = sz.Strings(["a", "b", "c", "d"])
assert len(pool.sample(2)) == 2
assert [str(x) for x in sorted(pool.shuffled(seed=42))] == ["a", "b", "c", "d"]
```

### Random Byte Generation

These module-level functions generate random bytes and are not tied to `Strings`:

- `sz.random(length, nonce=0, alphabet=None)` — return a fresh random `bytes` of `length`; if `alphabet` is given, each byte `b` is mapped to `alphabet[b % len(alphabet)]`.
- `sz.fill_random(buffer, nonce=0, alphabet=None, start=0, end=len)` — fill a writable, contiguous byte buffer such as a `bytearray`, `memoryview`, or `String` in place with pseudo-random bytes, optionally remapped to `alphabet`, optionally limited to the `[start, end)` slice.
  Returns `None`.
  Also available as the `String.fill_random` method.

```python
import stringzilla as sz

assert len(sz.random(16)) == 16
sz.random(8, alphabet="ACGT")                 # e.g. b'GCTAACGT'

buf = bytearray(16)
sz.fill_random(buf)                           # mutates buf in place
assert len(buf) == 16
```

### Escaping and Normalizing Collections

Every verb below is one native call over the whole collection and returns a new `Strings` owning one tape.

- `translate(table)`: maps every byte through a 256-byte table or a `dict`, as `bytes.translate` does for each string.
- `escape(kind='json')`: escapes every string for a text format.
- `unescape(kind='json')`: reverses `escape`, raising `ValueError` that names the index of the first string holding a malformed escape.
- `normalize(form='NFC')`: brings every string into a Unicode form, as `unicodedata.normalize` does, taking the forms `utf8_norm` takes.

The escape kinds follow the standard library's own writers and readers:

- `json`: a string body as `json.dumps(text, ensure_ascii=False)[1:-1]` writes it, decoded as `json.loads` decodes it.
- `json_ascii`: the same with every non-ASCII codepoint as `\uXXXX`, as `json.dumps(text)` writes it.
- `hex`: two lowercase digits per byte, as `bytes.hex` writes them and `bytes.fromhex` reads them.
- `html`: `&`, `<`, `>`, `"` and `'` as entities, as `html.escape` writes them but with `&#39;` for `'`, decoding the five XML entities and numeric references as `html.unescape` does.
- `url`: every byte outside `A-Z a-z 0-9 - . _ ~` percent-encoded, as `urllib.parse.quote(text, safe="")` writes it and `urllib.parse.unquote_to_bytes` reads it.
- `console`: controls, DEL, ill-formed UTF-8 and the bidirectional overrides as `\n`, `\xHH` or `\uXXXX`, so a logged string can neither hide nor reorder what a terminal shows.
- `csv`: a field quoted only around a comma, a quote, CR or LF, with every quote doubled, as `csv.writer` writes it and `csv.reader` reads it.

```python
import stringzilla as sz

names = sz.Strings(["Ada", 'Grace "Amazing" Hopper', "Linus\nTorvalds"])
assert [str(x) for x in names.escape("csv")] == ["Ada", '"Grace ""Amazing"" Hopper"', '"Linus\nTorvalds"']
assert [str(x) for x in names.escape("url").unescape("url")] == [str(x) for x in names]
assert [str(x) for x in names.translate({"a": "A"})][0] == "AdA"
assert [str(x) for x in sz.Strings(["Straße", "file"]).normalize("NFKC_Casefold")] == ["strasse", "file"]
```

### Dumping JSON and CSV

Writing a column of strings out as JSON or CSV usually costs a Python object per cell: `json.dumps` walks a `list` of `str`, `csv.writer` formats one row at a time, and pandas first builds a `DataFrame` of objects.
`sz.to_json` and `sz.to_csv` instead escape each column once, in one native pass over the whole column, then join the framed rows with one more native call into a single `bytes` object of exactly the right size.

- `Strings.to_json(ensure_ascii=True)`: one compact JSON array, byte for byte what `json.dumps(strings, ensure_ascii=ensure_ascii, separators=(",", ":"))` writes.
- `sz.to_json(columns, ensure_ascii=True)`: an array of records, one object per row, byte for byte what `json.dumps` writes for those records, and what `pandas.DataFrame(columns).to_json(orient="records")` writes apart from pandas spelling `/` as `\/`.
- `sz.to_csv(columns, lineterminator="\n")`: a header line of the names, then a line per row, byte for byte what `csv.writer` and `pandas.DataFrame(columns).to_csv(index=False)` write.

`columns` maps every name to a `Strings` or to any sequence of `str`, all of the same length, and the names are escaped the same way as the cells.

```python
import csv, io, json
import pandas as pd
import stringzilla as sz

names = ["Ada", 'Grace "Amazing" Hopper', "Linus\nTorvalds"]
cities = ["London", "New York, NY", "Helsinki"]
columns = {"name": sz.Strings(names), "city": cities}
frame = pd.DataFrame({"name": names, "city": cities})

records = [{"name": name, "city": city} for name, city in zip(names, cities)]
assert sz.to_json(columns) == json.dumps(records, separators=(",", ":")).encode()
assert sz.to_json(columns) == frame.to_json(orient="records").encode()

buffer = io.StringIO()
writer = csv.writer(buffer, lineterminator="\n")
writer.writerow(columns)
writer.writerows(zip(names, cities))
assert sz.to_csv(columns) == buffer.getvalue().encode()
assert sz.to_csv(columns) == frame.to_csv(index=False, lineterminator="\n").encode()
```

On an Apple M5 Pro with CPython 3.12 and pandas 3.0, the input is 200,000 English sentences from Leipzig 1M plus a second column holding the first word of each, 27.1 MB of UTF-8 in all.
Times are the best of 5 runs, each producing the whole `bytes` object:

- A JSON array of the sentences with `ensure_ascii=True`: 39.4 ms for `json.dumps` and 12.0 ms for `Strings.to_json`, 3.3x faster.
- The same array with `ensure_ascii=False`: 75.8 ms and 10.9 ms, 7.0x faster.
- CSV of both columns: 174.5 ms for `csv.writer` and 188.5 ms for `DataFrame.to_csv`, against 20.8 ms for `sz.to_csv` over `list`s and 17.1 ms over `Strings`, 8.4x and 10.2x faster than `csv.writer`.
- JSON records of both columns: 94.1 ms for `json.dumps` with the `dict`s built inline and 68.2 ms for `DataFrame.to_json`, against 15.5 ms for `sz.to_json` over `Strings`, 6.1x and 4.4x faster.

`import stringzilla`

## fill_random

Fill a string-like buffer in place with pseudo-random bytes.

```python
def fill_random(
    buffer: _Buffer,
    nonce: int = 0,
    alphabet: str | bytes | None = None,
    start: int = 0,
    end: int | None = None,
    *,
    capabilities: Capability | None = None,
) -> None
```

Returns:
None: Mutates the buffer slice in place.

- `buffer`: Writable, contiguous byte buffer (e.g., memoryview/bytearray).
- `nonce`: Seed ensuring reproducible output for the same inputs, defaulting to 0.
- `alphabet`: If provided, remaps random bytes to its characters.
- `start`: Starting index, defaulting to 0.
- `end`: Ending index, defaulting to len(buffer).
- `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones.

Examples: ```
>>> buf = bytearray(16)
>>> sz.fill_random(buf)
>>> len(buf)
16
```

## random

random(length, *, nonce=0, alphabet=None, capabilities=None) -> bytes

```python
def random(
    length: int,
    *,
    nonce: int = 0,
    alphabet: str | bytes | None = None,
    capabilities: Capability | None = None,
) -> bytes
```

Generate a new random byte string, optionally remapped to a given alphabet.
If alphabet is provided, each byte is mapped to alphabet[b % len(alphabet)].
Capabilities default to the CPU's enabled ones.

Examples: ```
>>> len(sz.random(16))
16
```

## to_json

to_json(columns, ensure_ascii=True, *, capabilities=None) -> bytes

```python
def to_json(
    columns: Mapping[str, Strings | Sequence[str]],
    ensure_ascii: bool = True,
    *,
    capabilities: Capability | None = None,
) -> bytes
```

Dump columns of strings as a compact JSON array of records, one object per row, the bytes of
`json.dumps([dict(zip(columns, row)) for row in zip(*columns.values())], separators=(',', ':'))`.
Each column is escaped in one pass, and the rows are written straight into the result.

Returns:
bytes: The UTF-8 JSON text.
Raises:
ValueError: If the columns differ in length.
Example:

sz.to_json({'name': sz.Strings(['Ada', 'Linus']), 'city': ['London', 'Helsinki']})
b'[{"name":"Ada","city":"London"},{"name":"Linus","city":"Helsinki"}]'

- `columns`: Columns of equal length, keyed by name.
- `ensure_ascii`: Write every non-ASCII codepoint as `\uXXXX`, defaulting to True.
- `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones.

## to_csv

to_csv(columns, lineterminator='\n', *, capabilities=None) -> bytes

```python
def to_csv(
    columns: Mapping[str, Strings | Sequence[str]],
    lineterminator: str = '\n',
    *,
    capabilities: Capability | None = None,
) -> bytes
```

Dump columns of strings as CSV with a header row of their names, the bytes `csv.writer` writes
with its default dialect, quoting a field only around a comma, a quote, CR or LF. Each column
is escaped in one pass, and the rows are written straight into the result.

Returns:
bytes: The UTF-8 CSV text.
Raises:
ValueError: If the columns differ in length, or the terminator is another string.
Example:

sz.to_csv({'name': ['Ada', 'Grace "Amazing" Hopper'], 'city': ['London', 'New York']})
b'name,city\nAda,London\n"Grace ""Amazing"" Hopper",New York\n'

- `columns`: Columns of equal length, keyed by name.
- `lineterminator`: '\n' or '\r\n' after every row, defaulting to '\n'.
- `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones.