Unicode Segmentation

Emits one length per segment and their count; each segment starts where the previous one ended.

Emits parallel (offset, length) arrays for each match, their count, and a resume bytes_consumed.

Skipping to character 1000, e.g. for pagination:

char const *position;
sz_utf8_seek_best(text, length, 1000, &position, capabilities, NULL);
if (!position) {
    // String has fewer than 1000 characters
}

Truncating to 280 characters, Twitter-style:

char const *end;
sz_utf8_seek_best(text, length, 280, &end, capabilities, NULL);
size_t truncated_bytes = end ? (end - text) : length;

This function is designed for streaming-like decoding with smart iterators built on top of it. The iterator would unpack a continuous slice of UTF-8 text into UTF-32 codepoints in chunks, yielding them upstream - only one at a time. This avoids allocating large buffers for the entire UTF-32 string, which can be 4x the size of the UTF-8 input.

This functionality is similar to the simdutf library's UTF-8 to UTF-32 conversion routines, and leverages an assumption that the absolute majority of written text doesn't mix codepoints of every length in each register-sized chunk:

• English text and source code is predominantly 1-byte ASCII characters.
• Broader European languages with diacritics: mostly 2-byte characters, 1-byte punctuation.
• Chinese & Japanese mostly use 3-byte characters with rare 1- or 3-byte punctuation.
• Korean uses 3-byte characters with 1-byte spaces; word are 2-6 syllables or 6-16 bytes.

It's a different story for emoji-heavy texts, which can mix 4-byte characters more frequently.

Every backend shares one contract, the basis for every codepoint iterator built on it:

• Fill-or-drain: emits runes until runes_capacity is reached or text is exhausted, looping internally - one call fills the buffer however many byte-widths the text mixes.
• Total and safe: never reads past text + length; substitutes one U+FFFD per maximal ill-formed subpart (overlong, surrogate, out-of-range, or a stray byte) and resyncs.
• Valid scalar values: every emitted rune is a Unicode scalar value (incl. U+FFFD), so callers may convert without re-validation.

End of text is end of input: a well-formed but truncated final sequence decodes to one U+FFFD over its maximal subpart and is consumed, so "a\xE2\x82" yields {'a', U+FFFD} over all 3 bytes. Only a full runes buffer stops early, so bytes_consumed falls short of length only when runes_count reaches runes_capacity, and then at a rune boundary. Callers decoding a stream in chunks split it at a rune boundary.

Enumerates every newline delimiter in a single sweep: writes the byte offset and byte length of each match into the parallel match_offsets and match_lengths arrays, a "\r\n" CRLF being one match of length 2, and their number into matches_count. A caller resuming from text + *bytes_consumed after a full output obtains the identical remainder.

Here are all the UTF-8 newline characters we are looking for (7 characters + CRLF):

• single-byte chars (4 total):

  • U+000A for "\n" (LINE FEED)
  • U+000B for "\v" (VERTICAL TAB / LINE TABULATION)
  • U+000C for "\f" (FORM FEED)
  • U+000D for "\r" (CARRIAGE RETURN)

• double-byte chars (1 total):

  • U+0085 for 0xC285 (NEXT LINE)

• triple-byte chars (2 total):

  • U+2028 for 0xE280A8 (LINE SEPARATOR)
  • U+2029 for 0xE280A9 (PARAGRAPH SEPARATOR)

• double-character sequence:

  • U+000D U+000A for "\r\n" that should be treated as a single new line!

U+001C, U+001D, U+001E (FILE/GROUP/RECORD SEPARATOR) are not included. These are data structure delimiters used in formats like USV (Unicode Separated Values), not line breaks. Use sz_find_byte_best if you need to find these separators.

Enumerates every whitespace delimiter in a single sweep, with the same contract as sz_utf8_newlines_best: writes the byte offset and byte length of each match into the parallel match_offsets and match_lengths arrays, their number into matches_count, and sets bytes_consumed to the resume offset. Each whitespace codepoint is one match: CR and LF are independent length-1 matches, as there is no CRLF merging in the whitespace set.

Implements the Unicode White_Space property, 25 characters in total. Per the Unicode standard, whitespace includes all newline characters plus horizontal spaces. Matches the behavior of ICU's u_isspace() and Python's str.isspace().

• single-byte chars (6 total):

  • U+0009 tab "\t" (CHARACTER TABULATION)
  • U+000A "\n" (LINE FEED - newline)
  • U+000B "\v" (LINE TABULATION - newline)
  • U+000C "\f" (FORM FEED - newline)
  • U+000D "\r" (CARRIAGE RETURN - newline)
  • U+0020 (SPACE)

• double-byte chars (2 total):

  • U+0085 0xC285 (NEXT LINE - newline)
  • U+00A0 0xC2A0 (NO-BREAK SPACE)

• triple-byte chars (17 total):

  • U+1680 0xE19A80 (OGHAM SPACE MARK)
  • U+2000 0xE28080 (EN QUAD)
  • U+2001 0xE28081 (EM QUAD)
  • U+2002 0xE28082 (EN SPACE)
  • U+2003 0xE28083 (EM SPACE)
  • U+2004 0xE28084 (THREE-PER-EM SPACE)
  • U+2005 0xE28085 (FOUR-PER-EM SPACE)
  • U+2006 0xE28086 (SIX-PER-EM SPACE)
  • U+2007 0xE28087 (FIGURE SPACE)
  • U+2008 0xE28088 (PUNCTUATION SPACE)
  • U+2009 0xE28089 (THIN SPACE)
  • U+200A 0xE2808A (HAIR SPACE)
  • U+2028 0xE280A8 (LINE SEPARATOR - newline)
  • U+2029 0xE280A9 (PARAGRAPH SEPARATOR - newline)
  • U+202F 0xE280AF (NARROW NO-BREAK SPACE)
  • U+205F 0xE2819F (MEDIUM MATHEMATICAL SPACE)
  • U+3000 0xE38080 (IDEOGRAPHIC SPACE)

The last one, the IDEOGRAPHIC SPACE (U+3000), is commonly used in East Asian typography, like Japanese formatted text or Chinese traditional poetry alignments.

Some implementations treat more codepoints as whitespace, but these are not included:

• U+001C, U+001D, U+001E, U+001F (FILE/GROUP/RECORD/UNIT SEPARATOR) are data structure delimiters for formats like USV (Unicode Separated Values). Only Java's Character.isWhitespace() includes them; Unicode, ICU, and Python do not.
• U+200B, U+200C, U+200D (ZERO WIDTH SPACE/NON-JOINER/JOINER) are Format characters, not whitespace. They have no width and affect rendering rather than spacing.

The general superset of sz_utf8_newlines_best and sz_utf8_whitespaces_best: every codepoint whose Unicode general category is a punctuation (P*), symbol (S*), or separator (Z*) is a delimiter. Shares the contract of sz_utf8_newlines_best: writes the byte offset and byte length of each match into the parallel match_offsets and match_lengths arrays, their number into matches_count, and sets bytes_consumed to the resume offset. A byte that does not begin a well-formed codepoint is skipped and never reported.

Walks the input left-to-right and writes the byte length of each word into lengths. Words are the spans between consecutive TR29 boundaries and tile the input: each starts where the previous one ended, the first at text. A call covers all of text unless it fills lengths, in which case it covers the sum of the lengths it wrote, and the caller resumes from text advanced by that sum.

Walks the input left-to-right and writes the byte length of each grapheme cluster into lengths. Clusters are the spans between consecutive UAX-29 grapheme boundaries (GB1-GB999) and tile the input: each starts where the previous one ended, the first at text. A call covers all of text unless it fills lengths, in which case it covers the sum of the lengths it wrote, and the caller resumes from text advanced by that sum.

Walks the input left-to-right and writes the byte length of each sentence into lengths. Sentences are the spans between consecutive UAX-29 sentence boundaries (SB1-SB998) and tile the input: each starts where the previous one ended, the first at text. A call covers all of text unless it fills lengths, in which case it covers the sum of the lengths it wrote, and the caller resumes from text advanced by that sum.

Walks the input left-to-right and writes the byte length of each line segment into lengths. Segments are the spans between consecutive UAX-14 break opportunities and tile the input: each starts where the previous one ended, the first at text. A call covers all of text unless it fills lengths, in which case it covers the sum of the lengths it wrote, and the caller resumes from text advanced by that sum.

This emits every wrap opportunity (both the mandatory LB4/LB5 hard breaks and the allowed soft-wrap points). To split only on the hard breaks (the str.splitlines behavior) use sz_utf8_newlines_best instead, which enumerates exactly the LB4/LB5 break positions.