# Unicode Segmentation

> StringZilla for C++

Splitting Unicode text into the units a human actually perceives is deceptively hard, and getting it wrong corrupts data.
A single emoji can be dozens of bytes and several codepoints yet one perceived character, CJK runs carry no spaces, and a period is not always the end of a sentence.
StringZilla ships the real algorithms — UAX-29 graphemes, words, and sentences, plus UAX-14 line-break opportunities — as lazy ranges that borrow from the source and yield one `sz::string_view_t` per segment with no per-segment allocation.

A string's length is really three different numbers, and only the grapheme count matches what a user sees:

```cpp
#include <stringzilla/stringzilla.hpp>
#include <cassert>
namespace sz = ashvardanian::stringzilla;

int main() {
    sz::string_view_t family = "👨‍👩‍👧‍👦"; // one emoji, one perceived character

    std::size_t codepoints = 0;
    for (sz_rune_t r : family.utf8_runes()) (void)r, ++codepoints;
    std::size_t graphemes = 0;
    for (sz::string_view_t g : family.utf8_graphemes()) (void)g, ++graphemes;

    assert(family.size() == 25); // 25 bytes on the wire
    assert(codepoints == 7); // four people joined by three zero-width joiners
    assert(graphemes == 1); // one grapheme cluster
    return 0;
}
```

This is exactly why truncating by bytes or codepoints tears emoji apart — it can slice a skin-tone modifier off its base, or split a flag in half.
Iterating graphemes lets you cut on a real boundary:

```cpp
sz::string_view_t msg = "🙏🏽👍🏿✨ thanks";
std::size_t cut = 0, seen = 0;
for (sz::string_view_t g : msg.utf8_graphemes()) { if (seen++ == 3) break; cut += g.size(); }
assert(msg.sub(0, cut) == "🙏🏽👍🏿✨"); // the three emoji keep their skin-tone modifiers
```

Word and sentence boundaries are just as treacherous: contractions, CJK, decimals, and URLs all defeat naive splitting.
UAX-29 keeps `can't` whole while separating CJK ideographs that no space divides, and refuses to break a sentence inside `$9.99`:

```cpp
sz::string_view_t phrase = "can't stop 北京!";
assert(*phrase.utf8_wordbreaks().begin() == "can't"); // apostrophe is interior; 北 and 京 split apart

sz::string_view_t prose = "She paid $9.99. Cheap.";
std::size_t sentences = 0;
for (sz::string_view_t s : prose.utf8_sentences()) (void)s, ++sentences;
assert(sentences == 2); // the dot inside $9.99 is not a sentence break
```

The full family of ranges, each borrowing from the source and yielding `sz::string_view_t` segments, or `sz_rune_t` for runes.
Naming follows one rule: the bare name (`newlines`/`whitespaces`/`delimiters`) yields the __separators__ the kernel finds, while `split_*` yields the content __between__ them:

- `utf8_runes()` — every codepoint as a decoded `sz_rune_t` UTF-32 scalar, with U+FFFD for ill-formed bytes and a truncated tail.
- `utf8_graphemes()` — UAX-29 grapheme clusters, the user-perceived characters.
- `utf8_wordbreaks()` — all UAX-29 word segments (words and the separators between them).
- `utf8_sentences()` — UAX-29 sentence boundaries.
- `utf8_linebreaks()` — UAX-14 line-break opportunities, both hard breaks and soft wrap points.
- `utf8_split_newlines()` / `utf8_newlines()` — content between the seven Unicode newlines (CRLF as one), or the newline runs themselves.
- `utf8_split_whitespaces()` / `utf8_whitespaces()` — content between Unicode "White_Space" runs (tokens), or the whitespace runs.
- `utf8_split_delimiters()` / `utf8_delimiters()` — content between any Unicode punctuation/symbol/separator, or the delimiter runs.

On any `split_*` range, `.skip_empty()` drops empty segments and `.with_separators()` interleaves segments and separators losslessly (concatenation reconstructs the input).
The `utf8_wordbreaks`, `utf8_graphemes`, `utf8_sentences`, and `utf8_linebreaks` ranges _tile_ the input — every byte belongs to exactly one segment, with no gaps and no empty slices.
Because every range borrows and walks the buffer once, segmenting a multi-megabyte document into graphemes or words allocates nothing.

In C, `sz_utf8_graphemes_best` and its three tiling siblings write one length per segment, each segment starting where the previous one ended, so after a full batch the caller resumes at the sum of its lengths.
The newline, whitespace and delimiter finders write offsets and lengths of their matches, and after a full batch `bytes_consumed` ends exactly at the last match; otherwise it is the whole input.
`sz_utf8_decode_best` treats the end of the text as the end of the input, decoding a truncated final sequence to one U+FFFD, so a caller feeding it chunks splits them at a rune boundary.

`#include <stringzilla/stringzilla.hpp>`

## utf8_segments_view

A range of string slices that __tile__ the input, as a segmentation kernel reports them.

```cpp
template <utf8_segmenter_best_t segmenter_,
          utf8_segmenter_serial_t segmenter_serial_,
          typename string_type_,
          std::size_t steps_ = sz_iterators_default_steps_k>
class utf8_segments_view
```

The shared engine behind `utf8_wordbreaks_view`, `utf8_graphemes_view`, `utf8_sentences_view`, and `utf8_linebreaks_view`. The kernel returns the tiling units directly: every byte belongs to exactly one unit, so consecutive units are contiguous and no empty segments arise.

## utf8_runes_view

A range view over UTF-8 characters (codepoints) in a string.

```cpp
template <typename string_type_,
          std::size_t steps_ = sz_iterators_default_steps_k>
class utf8_runes_view
```

Iterates over UTF-32 codepoints decoded from UTF-8 bytes using efficient batched decoding. Each refill decodes up to `steps_` codepoints in a single [`sz_utf8_decode_best`](https://www.unum.cloud/docs/stringzilla/c/unicode-segmentation.md#sz_utf8_decode_best) call (the decoder fills the whole buffer regardless of script width), then yields them one by one. Ill-formed bytes, a truncated sequence at the end included, decode to U+FFFD.

## utf8_runes_view

```cpp
utf8_runes_view() noexcept=default;
```

```cpp
utf8_runes_view(string_type haystack) noexcept;
```

## size

Count UTF-8 characters in the string.

```cpp
size_type size() const noexcept;
```

## to

Copies the characters into a container.

```cpp
template <typename container_>
void to(container_ & container);
```

```cpp
template <typename container_>
container_ to(container_ && container = {});
```

Copies the characters into a consumed container, returning it at the end.