# Types > StringZilla for Rust The `sz` module exposes a handful of public value types used throughout the API. - `Byteset` — a 256-bit set of bytes used by the byteset search and replace functions. - `IndexSpan` — a `{ offset, length }` byte span, returned by uncased match iterators. - `Hasher` — an incremental AES-based 64-bit hasher implementing `core::hash::Hasher`. - `Sha256` — an incremental SHA-256 hasher with a one-shot convenience constructor. - `BuildSzHasher` — a `std::hash::BuildHasher`, gated on feature `std`, for using `Hasher` with `HashMap`/`HashSet`. - `Utf8View`, `Utf8Runes`, `Utf8SplitNewlines`, `Utf8SplitWhitespaces`, `Utf8Wordbreaks`, `Utf8Graphemes`, `Utf8Sentences`, `Utf8Linebreaks` — lazy UTF-8 views and iterators. - `Utf8UncasedNeedle`, `Utf8UncasedMatches`, `NormalForm` — uncased search and Unicode normalization helpers. - `SemVer`, `Status` — version and error types. - `Capability`, `Capabilities` — one CPU or GPU capability, and a set of them printed as their names, with the producers that report a device's set by its ordinal. - `Stream`, `Scope` — a stream of one device, which every engine, conversion and allocator takes, and the span its engine verbs queue in before one join. - `StringsView`, `StringsIter` — a borrowed column, an Arrow tape or slices read in place, which every engine verb and conversion reads. - `Strings`, `UnifiedAllocator` — an owned column in any `Allocator`, and the memory the host and a GPU both address. - `EscapeKind`, `SplitKind`, `StringsError` — the formats a column escapes for, whether a split ends a part at every delimiter or every run of them, and the first malformed string unescaping or UTF-8 validation stopped at. - `ArgsortOptions` — knobs for sorting. `Status` is the error of every fallible call: one variant per failure code of C's `sz_status_t`, plus `Unrecognized` for a code this crate's header does not list. Success has no variant, being the `Ok` side of the `Result`, and `Status` implements `Display` and `core::error::Error`. A buffer of the wrong length is reported as `Status::UnexpectedDimensions` rather than asserted. `Byteset` is constructed from bytes and supports inversion, useful for "not from" semantics: ```rust use stringzilla::{self as sz, Byteset}; let vowels = Byteset::from("aeiou"); let mut punct = Byteset::new(); punct.add(b','); punct.add(b'.'); let everything = Byteset::new_ascii(); // all ASCII bytes set let not_vowels = vowels.inverted(); // leaves `vowels` unchanged let _ = (everything, not_vowels, punct); ``` `Byteset` API: `new()`, `new_ascii()`, `from_bytes(&[u8])` / `From>`, `add(u8)`, `invert(&mut self)`, `inverted(&self) -> Byteset`. `IndexSpan` API: `new(offset, length)`, `range() -> Range`, `extract<'a>(&self, &'a [u8]) -> Option<&'a [u8]>`, `end() -> usize`, plus public `offset` and `length` fields. `end` saturates rather than overflowing, and `extract` returns `None` for a span that runs past the text. ```rust use stringzilla::IndexSpan; let span = IndexSpan::new(6, 5); assert_eq!(span.range(), 6..11); assert_eq!(span.end(), 11); assert_eq!(span.extract(b"Hello World"), Some(&b"World"[..])); assert_eq!(span.extract(b"Hello"), None); ``` `use stringzilla::StringsError;` `use stringzilla::Strings;` `use stringzilla::StringsView;` `use stringzilla::StringsIter;` `use stringzilla::SemVer;` `use stringzilla::Status;` `use stringzilla::Byteset;` `use stringzilla::*;` ## StringsError Why [`StringsView::unescape_in`](#unescape_in) or [`StringsView::utf8_count`](#utf8_count) failed. ```rust pub enum StringsError ``` - `Malformed` = `(usize)`: The position of the first string holding a malformed sequence, like a lone surrogate in JSON, an unbalanced quote in a CSV field, or ill-formed UTF-8. - `Failed` = `(Status)`: Any other failure, like a block that could not grow. Implements: `Clone`, `Copy`, `Debug`, `StructuralPartialEq`, `PartialEq`, `Eq`, `Display`, `Error` ## Strings An owned column: one payload and its u32 or u64 offsets, both from `A`. ```rust pub struct Strings ``` It is C's `sz_strings_span_t`, the layout of Apache Arrow's `String` and `LargeString` arrays, with 32-bit offsets while the payload fits in 4 GiB and 64-bit beyond. Both blocks go back to `A` on drop, and every conversion joins its stream first, so a column is always readable. Examples: ``` use std::alloc::Global; use stringzilla::{Capabilities, EscapeKind, Stream, Strings, StringsView}; let cpu = Stream::default(Capabilities::cpu_enabled()); let names = Strings::copy_in(StringsView::from_slices(&["Ada", "Grace \"Amazing\" Hopper"]), Global, &cpu)?; let fields = names.view().escape_in(EscapeKind::Csv, Global, &cpu)?; assert_eq!(fields.view().get(1), Some(&br#""Grace ""Amazing"" Hopper""#[..])); ``` Implements: `Drop` ## copy_in Copies `strings` into a column from `allocator`, gathered on the device of `stream`, which on a GPU reads only tapes in memory it reaches. ```rust pub fn copy_in(strings: StringsView<'_>, allocator: A, stream: &Stream) -> Result ``` ## split_in Splits `text` at the bytes of `delimiters` into a column of the parts between them, every delimiter or every run of them ending a part as `kind` says. ```rust pub fn split_in( text: &[u8], delimiters: Byteset, kind: SplitKind, allocator: A, stream: &Stream, ) -> Result ``` Examples: ``` use std::alloc::Global; use stringzilla::{Byteset, Capabilities, SplitKind, Stream, Strings}; let cpu = Stream::default(Capabilities::cpu_enabled()); let fields = Strings::split_in(b"Ada,,Grace", Byteset::from(","), SplitKind::Each, Global, &cpu)?; assert!(fields.view().iter().eq([&b"Ada"[..], b"", b"Grace"])); let words = Strings::split_in(b" Ada Grace ", Byteset::from(" "), SplitKind::Runs, Global, &cpu)?; assert!(words.view().iter().eq([&b"Ada"[..], b"Grace"])); ``` ## split_lines_in Splits UTF-8 `text` into a column of its lines without their breaks, like Python's `str.splitlines()`, which takes CR LF as one break and leaves no empty line after the last. ```rust pub fn split_lines_in(text: &[u8], allocator: A, stream: &Stream) -> Result ``` ## view Borrows the column, to read it or convert it into another one. ```rust pub fn view(&self) -> StringsView<'_> ``` ## StringsView A borrowed column in any layout of `sz_strings_view_t`: an Arrow tape, or slices read through an accessor. ```rust pub struct StringsView<'a> ``` Engines and conversions read it where it lies. A GPU reads only tapes in memory it reaches, like a [`Strings`](#Strings) from a [`UnifiedAllocator`](https://www.unum.cloud/docs/stringzilla/rust/memory-operations.md#UnifiedAllocator), refusing any other view, accessors included, with [`Status::DeviceMemoryMismatch`](#DeviceMemoryMismatch). Examples: ``` use stringzilla::{Capabilities, LevenshteinEngine, LevenshteinSymbol, Status, Stream, StringsView}; // An Arrow `String` array sliced from its second element, so its offsets start past zero. let (data, offsets) = (b"catsittingkitten", [3u32, 10, 16]); let candidates = StringsView::from_u32_tape(data, &offsets)?; assert!(candidates.iter().eq([&b"sitting"[..], b"kitten"])); assert_eq!(StringsView::from_u32_tape(data, &[3, 17]).err(), Some(Status::UnexpectedDimensions)); let cpu = Stream::default(Capabilities::cpu_enabled()); let mut engine = LevenshteinEngine::new(&["kitten"], LevenshteinSymbol::Bytes, &cpu)?; let mut distances = [0usize; 2]; cpu.scope(|scope| engine.distances(scope, candidates, None, &mut distances, 2))?; assert_eq!(distances, [3, 0]); ``` Implements: `Clone`, `Copy` ## from_u32_tape Views an Arrow `String` tape: string `i` spans `data[offsets[i]..offsets[i + 1]]`. ```rust pub fn from_u32_tape(data: &'a [u8], offsets: &'a [u32]) -> Result ``` Offsets that are empty, fall, or run past `data` are refused with [`Status::UnexpectedDimensions`](#UnexpectedDimensions). ## from_u64_tape [`StringsView::from_u32_tape`](#from_u32_tape), for an Arrow `LargeString` tape. ```rust pub fn from_u64_tape(data: &'a [u8], offsets: &'a [u64]) -> Result ``` ## from_slices Views `texts` where they lie, read through an accessor rather than copied. ```rust pub fn from_slices>(texts: &'a [Text]) -> Self ``` ## len Strings the column holds. ```rust pub fn len(&self) -> usize ``` ## is_empty Whether the column holds no strings. ```rust pub fn is_empty(&self) -> bool ``` ## get The bytes of string `index`, or `None` past the last one, like [`slice::get`](#get). ```rust pub fn get(&self, index: usize) -> Option<&'a [u8]> ``` ## iter The strings in order, like [`slice::iter`](https://www.unum.cloud/docs/stringzilla/rust/runtime-dispatch.md#iter). ```rust pub fn iter(&self) -> StringsIter<'a> ``` ## escape_in Escapes every string for `kind` into a new column from `allocator`, on the device of `stream`. ```rust pub fn escape_in( &self, kind: EscapeKind, allocator: A, stream: &Stream, ) -> Result, Status> ``` ## unescape_in Reverses [`StringsView::escape_in`](#escape_in) for `kind`, stopping at the first string holding a malformed sequence, whose position [`StringsError::Malformed`](#Malformed) reports. ```rust pub fn unescape_in( &self, kind: EscapeKind, allocator: A, stream: &Stream, ) -> Result, StringsError> ``` ## lookup_in Maps every byte of every string through `table` into a new column, as [`lookup`](https://www.unum.cloud/docs/stringzilla/rust/trimming-and-translating.md#lookup) does for one buffer. ```rust pub fn lookup_in( &self, table: &[u8; 256], allocator: A, stream: &Stream, ) -> Result, Status> ``` ## normalize_in Brings every string into the Unicode `form` into a new column, as [`utf8_norm`](https://www.unum.cloud/docs/stringzilla/rust/case-folding-and-normalization.md#utf8_norm) does for one string, copying the strings already in it as they are. ```rust pub fn normalize_in( &self, form: NormalForm, allocator: A, stream: &Stream, ) -> Result, Status> ``` ## join_in Joins the strings into one buffer from `allocator`, with `between` between neighbours, like `slice::join`. ```rust pub fn join_in( &self, between: &[u8], allocator: A, stream: &Stream, ) -> Result, Status> ``` ## join_columns_in Writes row `r` of `columns` as `frames[0]`, `columns[0][r]`, `frames[1]` and so on through `frames[columns.len()]`, with `between` between rows, into one buffer from `allocator`, to serialize CSV lines or JSON records without a string per cell. ```rust pub fn join_columns_in( columns: &[StringsView<'_>], frames: &[&[u8]], between: &[u8], allocator: A, stream: &Stream, ) -> Result, Status> ``` Columns of different lengths, or `frames` not one longer than `columns`, are refused with [`Status::UnexpectedDimensions`](#UnexpectedDimensions). Examples: ``` use std::alloc::Global; use stringzilla::{Capabilities, Stream, StringsView}; let cpu = Stream::default(Capabilities::cpu_enabled()); let names = StringsView::from_slices(&["Ada", "Grace"]); let years = StringsView::from_u32_tape(b"18151906", &[0, 4, 8])?; let csv = StringsView::join_columns_in(&[names, years], &[b"", b",", b""], b"\n", Global, &cpu)?; assert_eq!(csv, b"Ada,1815\nGrace,1906"); ``` ## to_rows Writes every string into its own NUL-padded row of [`width`](https://www.unum.cloud/docs/stringzilla/rust/searching-and-counting.md#width) bytes, back to back in `rows`, truncating longer ones, and returns the longest length, to retry with wider rows. ```rust pub fn to_rows(&self, rows: &mut [u8], width: usize, stream: &Stream) -> Result ``` `rows` holds at least `len() * width` bytes, or [`Status::UnexpectedDimensions`](#UnexpectedDimensions) is returned, and on a GPU it lies in memory the device reaches, like a `Vec` in a [`UnifiedAllocator`](https://www.unum.cloud/docs/stringzilla/rust/memory-operations.md#UnifiedAllocator). ## find Finds `needle` in every string, as [`find`](https://www.unum.cloud/docs/stringzilla/rust/searching-and-counting.md#find) does in one, writing the offset of each first match into `positions`, or `usize::MAX` where a string holds none. ```rust pub fn find(&self, needle: &[u8], positions: &mut [usize], stream: &Stream) -> Result<(), Status> ``` `positions` holds exactly `len()` entries, or [`Status::UnexpectedDimensions`](#UnexpectedDimensions) is returned. A match crossing from one string of a tape into the next counts for neither. Examples: ``` use stringzilla::{Capabilities, Stream, StringsView}; let cpu = Stream::default(Capabilities::cpu_enabled()); let requests = StringsView::from_slices(&["GET /index", "POST /login", "GET /login"]); let mut positions = [0usize; 3]; requests.find(b"login", &mut positions, &cpu)?; assert_eq!(positions, [usize::MAX, 6, 5]); ``` ## hashes Hashes every string under `seed` into [`hashes`](#hashes), each exactly as [`hash_with_seed`](https://www.unum.cloud/docs/stringzilla/rust/hashing-and-checksums.md#hash_with_seed) would. ```rust pub fn hashes(&self, seed: u64, hashes: &mut [u64], stream: &Stream) -> Result<(), Status> ``` [`hashes`](#hashes) holds exactly `len()` entries, or [`Status::UnexpectedDimensions`](#UnexpectedDimensions) is returned. Examples: ``` use stringzilla::{self as sz, Capabilities, Stream, StringsView}; let cpu = Stream::default(Capabilities::cpu_enabled()); let keys = StringsView::from_u32_tape(b"adagrace", &[0, 3, 8])?; let mut hashes = [0u64; 2]; keys.hashes(42, &mut hashes, &cpu)?; assert_eq!(hashes, [sz::hash_with_seed("ada", 42), sz::hash_with_seed("grace", 42)]); ``` ## utf8_count Checks that every string is well-formed UTF-8, reporting the first one that is not as [`StringsError::Malformed`](#Malformed), and writes each one's rune count into `runes`, unless it is `None`, leaving the counts from the malformed string on unspecified. ```rust pub fn utf8_count(&self, runes: Option<&mut [usize]>, stream: &Stream) -> Result<(), StringsError> ``` `runes` holds exactly `len()` entries, or [`Status::UnexpectedDimensions`](#UnexpectedDimensions) is returned. A sequence crossing from one string of a tape into the next is ill-formed. ## StringsIter The strings of a [`StringsView`](#StringsView) in order, borrowing what the view borrows. ```rust pub struct StringsIter<'a> ``` Implements: `Clone`, `Iterator`, `ExactSizeIterator` ## SemVer A simple semantic version structure. ```rust pub struct SemVer ``` - `major` `i32`: The major version. - `minor` `i32`: The minor version. - `patch` `i32`: The patch version. Implements: `Debug`, `Copy`, `Clone`, `StructuralPartialEq`, `PartialEq`, `Eq` ## new Builds a version from its three components. ```rust pub const fn new(major: i32, minor: i32, patch: i32) -> Self ``` ## Status Why a StringZilla call failed: C's `sz_status_t`, each variant holding the header's value. ```rust pub enum Status ``` Success has no variant, being the `Ok` side of every `Result` this crate returns. A code the header this crate was built from does not list reads as [`Status::Unrecognized`](#Unrecognized). - `BadAlloc` = `-10`: `sz_bad_alloc_k`: a memory allocation failed. - `InvalidUtf8` = `-12`: `sz_invalid_utf8_k`: an input that must be UTF-8 is not. - `ContainsDuplicates` = `-13`: `sz_contains_duplicates_k`: a collection of unique elements holds duplicates. - `OverflowRisk` = `-14`: `sz_overflow_risk_k`: an input is too large for the integer types the algorithm counts in. - `UnexpectedDimensions` = `-15`: `sz_unexpected_dimensions_k`: input and output sizes contradict each other. - `MissingGpu` = `-16`: `sz_missing_gpu_k`: no GPU device answers, so nothing can be prepared or scheduled on one. - `DeviceCodeMismatch` = `-17`: `sz_device_code_mismatch_k`: no kernel for the call, like a device engine handed to a host verb or the reverse, no GPU runtime compiled in, or a launch the device refused. - `DeviceMemoryMismatch` = `-18`: `sz_device_memory_mismatch_k`: an operand lies in memory the device cannot address. - `AuthenticationFailed` = `-19`: `sz_authentication_failed_k`: a tag does not match the ciphertext it accompanies. - `MissingKernel` = `-20`: `sz_missing_kernel_k`: no capability in the mask the call dispatched on has its kernel. - `MissingLibrary` = `-21`: `sz_missing_library_k`: a dispatch point reached from a header-only build, which links no library. - `InvalidSyntax` = `-22`: `sz_invalid_syntax_k`: a pattern, path or document that breaks its grammar, or needs a construct no automaton runs. - `Unknown` = `-1`: `sz_status_unknown_k`: a failure the C core has no specific code for. - `Unrecognized` = `i32::MIN`: A code outside this list, which only a library built from another header returns. Implements: `Clone`, `Copy`, `Debug`, `StructuralPartialEq`, `PartialEq`, `Eq`, `Hash`, `Display`, `Error` ## Byteset A set of byte values, held as a 256-bit mask. ```rust pub struct Byteset ``` Implements: `Debug`, `Clone`, `Copy`, `Default`, `From` ## new Initializes a bit-set to an empty collection, banning all characters. ```rust pub const fn new() -> Self ``` ## new_ascii Initializes a bit-set to contain all ASCII characters. ```rust pub const fn new_ascii() -> Self ``` ## add Adds a byte to the set. ```rust pub const fn add(&mut self, byte: u8) ``` ## invert Inverts the bit-set so that all set bits become unset and vice versa. ```rust pub const fn invert(&mut self) ``` ## inverted Returns a new Byteset with all bits inverted, leaving self unchanged. ```rust pub const fn inverted(&self) -> Self ``` ## from_bytes Constructs a Byteset from a slice of bytes. ```rust pub const fn from_bytes(bytes: &[u8]) -> Self ``` Usable in a `const`, so a hot loop can name a set instead of rebuilding one per call: ``` use stringzilla::Byteset; const WHITESPACE: Byteset = Byteset::from_bytes(b" \t\r\n"); assert_eq!(stringzilla::find_byteset("ab cd", WHITESPACE), Some(2)); ``` ## ITERATORS_DEFAULT_STEPS Default number of boundaries the UTF-8 segmentation iterators buffer per call. ```rust pub const ITERATORS_DEFAULT_STEPS: usize = 64; ``` Buffering this many amortizes the per-item dispatch overhead without an unbounded buffer; a full buffer simply resumes where it left off on the next call.