Unicode Segmentation

Splitting Unicode text into the units a human actually perceives is deceptively hard, and getting it wrong corrupts data. A single emoji can be dozens of bytes and several codepoints yet one perceived character, CJK runs carry no spaces, and a period is not always the end of a sentence. StringZilla ships the real algorithms — UAX-29 graphemes, words, and sentences, plus UAX-14 line-break opportunities — as lazy ranges that borrow from the source and yield one sz::string_view_t per segment with no per-segment allocation.

A string’s length is really three different numbers, and only the grapheme count matches what a user sees:

#include <stringzilla/stringzilla.hpp>
#include <cassert>
namespace sz = ashvardanian::stringzilla;

int main() {
    sz::string_view_t family = "👨‍👩‍👧‍👦"; // one emoji, one perceived character

    std::size_t codepoints = 0;
    for (sz_rune_t r : family.utf8_runes()) (void)r, ++codepoints;
    std::size_t graphemes = 0;
    for (sz::string_view_t g : family.utf8_graphemes()) (void)g, ++graphemes;

    assert(family.size() == 25); // 25 bytes on the wire
    assert(codepoints == 7); // four people joined by three zero-width joiners
    assert(graphemes == 1); // one grapheme cluster
    return 0;
}

This is exactly why truncating by bytes or codepoints tears emoji apart — it can slice a skin-tone modifier off its base, or split a flag in half. Iterating graphemes lets you cut on a real boundary:

sz::string_view_t msg = "🙏🏽👍🏿✨ thanks";
std::size_t cut = 0, seen = 0;
for (sz::string_view_t g : msg.utf8_graphemes()) { if (seen++ == 3) break; cut += g.size(); }
assert(msg.sub(0, cut) == "🙏🏽👍🏿✨"); // the three emoji keep their skin-tone modifiers

Word and sentence boundaries are just as treacherous: contractions, CJK, decimals, and URLs all defeat naive splitting. UAX-29 keeps can't whole while separating CJK ideographs that no space divides, and refuses to break a sentence inside $9.99:

sz::string_view_t phrase = "can't stop 北京!";
assert(*phrase.utf8_wordbreaks().begin() == "can't"); // apostrophe is interior; 北 and 京 split apart

sz::string_view_t prose = "She paid $9.99. Cheap.";
std::size_t sentences = 0;
for (sz::string_view_t s : prose.utf8_sentences()) (void)s, ++sentences;
assert(sentences == 2); // the dot inside $9.99 is not a sentence break

The full family of ranges, each borrowing from the source and yielding sz::string_view_t segments, or sz_rune_t for runes. Naming follows one rule: the bare name (newlines/whitespaces/delimiters) yields the separators the kernel finds, while split_* yields the content between them:

• utf8_runes() — every codepoint as a decoded sz_rune_t UTF-32 scalar, with U+FFFD for ill-formed bytes and a truncated tail.
• utf8_graphemes() — UAX-29 grapheme clusters, the user-perceived characters.
• utf8_wordbreaks() — all UAX-29 word segments (words and the separators between them).
• utf8_sentences() — UAX-29 sentence boundaries.
• utf8_linebreaks() — UAX-14 line-break opportunities, both hard breaks and soft wrap points.
• utf8_split_newlines() / utf8_newlines() — content between the seven Unicode newlines (CRLF as one), or the newline runs themselves.
• utf8_split_whitespaces() / utf8_whitespaces() — content between Unicode “White_Space” runs (tokens), or the whitespace runs.
• utf8_split_delimiters() / utf8_delimiters() — content between any Unicode punctuation/symbol/separator, or the delimiter runs.

On any split_* range, .skip_empty() drops empty segments and .with_separators() interleaves segments and separators losslessly (concatenation reconstructs the input). The utf8_wordbreaks, utf8_graphemes, utf8_sentences, and utf8_linebreaks ranges tile the input — every byte belongs to exactly one segment, with no gaps and no empty slices. Because every range borrows and walks the buffer once, segmenting a multi-megabyte document into graphemes or words allocates nothing.

In C, sz_utf8_graphemes_best and its three tiling siblings write one length per segment, each segment starting where the previous one ended, so after a full batch the caller resumes at the sum of its lengths. The newline, whitespace and delimiter finders write offsets and lengths of their matches, and after a full batch bytes_consumed ends exactly at the last match; otherwise it is the whole input. sz_utf8_decode_best treats the end of the text as the end of the input, decoding a truncated final sequence to one U+FFFD, so a caller feeding it chunks splits them at a rune boundary.