# Unicode Segmentation > StringZilla for Python `String`, along with the matching module-level functions, exposes lazy iterators over Unicode boundaries. Each yields `String` views into the original buffer, so segmentation stays allocation-free, except `utf8_codepoints`, which yields `int` code points. | Method / function | Standard | Yields | | :------------------------------------------------------------------------ | :------------------------------- | :------------------------------------------------------------------------------------- | | `utf8_codepoints(string)` | scalar values | `int` code points; ill-formed bytes decode to `U+FFFD`, so iteration never raises. | | `utf8_graphemes(string, skip_empty=False)` | UAX-29 grapheme clusters | user-perceived characters such as a base plus combining marks, or emoji ZWJ sequences. | | `utf8_wordbreaks(string, skip_empty=False)` | UAX-29 word boundaries | all UAX-29 word segments (words and the separators between them; they tile). | | `utf8_sentences(string, skip_empty=False)` | UAX-29 sentence boundaries | sentences. | | `utf8_linebreaks(string, skip_empty=False)` | UAX-14 line-break opportunities | soft-wrap segments. | | `utf8_split_newlines(string, skip_empty=False, with_separators=False)` | 7 Unicode newlines + CRLF | content BETWEEN hard newlines (LF, VT, FF, CR, NEL, `U+2028`, `U+2029`, CRLF). | | `utf8_newlines(string, skip_empty=False)` | 7 Unicode newlines + CRLF | the newline runs themselves (the separators). | | `utf8_split_whitespaces(string, skip_empty=False, with_separators=False)` | 25 Unicode `"White_Space"` | content BETWEEN whitespace runs, like `str.split()` with no separator. | | `utf8_whitespaces(string, skip_empty=False)` | 25 Unicode `"White_Space"` chars | the whitespace runs themselves (the separators). | | `utf8_split_delimiters(string, skip_empty=False, with_separators=False)` | punctuation/symbol/separator | content BETWEEN any Unicode delimiter (superset of whitespace). | | `utf8_delimiters(string, skip_empty=False)` | punctuation/symbol/separator | the delimiter runs themselves (the separators). | Naming follows one rule: the bare name (`newlines`/`whitespaces`/`delimiters`) yields the __separators__, while `split_*` yields the content __between__ them. `skip_empty` drops empty segments, and `with_separators=True` interleaves both losslessly, so concatenating the result reproduces the input. ```python import stringzilla as sz assert list(sz.utf8_codepoints("AB")) == [65, 66] assert [str(g) for g in sz.String("a👍🏽b").utf8_graphemes()] == ["a", "👍🏽", "b"] [str(w) for w in sz.utf8_wordbreaks("Hi, world")] # all UAX-29 segments assert sum(1 for _ in sz.String("first\nsecond\nthird").utf8_split_newlines()) == 3 assert [str(t) for t in sz.utf8_split_whitespaces("foo bar baz", skip_empty=True)] == ["foo", "bar", "baz"] assert "".join(str(s) for s in sz.utf8_split_newlines("a\nb", with_separators=True)) == "a\nb" ``` ### Counting Code Points `utf8_count(string)` counts Unicode characters rather than bytes, unlike `len`, which counts bytes, and `Strings.utf8_count()` counts every string of a collection in one pass: ```python import stringzilla as sz assert sz.utf8_count("hello") == 5 assert sz.utf8_count("é") == 1 assert len(sz.String("é")) == 2 # bytes ``` ### Unicode Case Folding for Uncased Matching These apply Unicode case folding, correctly handling one-to-many expansions such as German `ß` matching `SS`. - `utf8_uncased_fold(text, validate=False)` — return the case-folded UTF-8 string as `bytes`. - `utf8_uncased_search(haystack, needle, start=0, end=len, validate=False)` — index of the first uncased match, or `-1`. For `str` inputs `start`/`end` and the result are codepoint offsets; for `bytes` inputs they are byte offsets. - `utf8_uncased_order(a, b, validate=False)` — uncased lexicographic comparison: negative, zero, or positive `int`. - `utf8_uncased_matches(haystack, needle, include_overlapping=False)` — iterate over all uncased matches, yielding each matched region as a `String` view whose length may differ from `needle` due to folding expansions; an empty `needle` matches at every codepoint boundary, the end included. Pass `validate=True` to validate UTF-8 before processing. ```python import stringzilla as sz assert sz.utf8_uncased_fold("HELLO") == b"hello" assert sz.utf8_uncased_fold("Straße") == b"strasse" assert sz.utf8_uncased_search("Hello World", "WORLD") == 6 assert sz.utf8_uncased_order("hello", "HELLO") == 0 assert [str(m) for m in sz.utf8_uncased_matches("Hello HELLO hello", "hello")] == ["Hello", "HELLO", "hello"] ``` ### Unicode Normalization These mirror `unicodedata.normalize` but operate on raw UTF-8 bytes. - `utf8_norm(text, form, validate=False)` — normalize to one of `'NFC'`, `'NFD'`, `'NFKC'`, `'NFKD'`, `'NFKC_Casefold'`; returns `bytes`. `'NFKC_Casefold'` is the Unicode toNFKC_Casefold, the keys search engines match on: NFKC with full case folding and without default-ignorable codepoints. - `utf8_find_denormalized(text, form)` — return the byte offset of the first codepoint that breaks the given normalization form, or `None` if the string is already fully normalized. ```python import stringzilla as sz assert sz.utf8_norm("café", "NFD") == b"cafe\xcc\x81" # decomposed assert sz.utf8_norm("fi", "NFKD") == b"fi" # ligature expanded assert sz.utf8_find_denormalized("café", "NFC") is None # already NFC assert sz.utf8_norm("Straße­", "NFKC_Casefold") == b"strasse" # folded, soft hyphen dropped ``` `import stringzilla` ## Utf8Graphemes UTF-8 aware grapheme cluster boundary iterator per Unicode UAX-29 algorithm. ```python class Utf8Graphemes ``` Yields grapheme clusters, the user-perceived characters. Created by: - String.utf8_graphemes() - sz.utf8_graphemes() UAX-29 Grapheme_Cluster_Break rules implemented: - GB3: CR x LF, no break - GB4-GB5: Control/CR/LF breaks - GB6-GB8: Hangul syllable sequences - GB9-GB9c: Extend/ZWJ/SpacingMark, Indic conjuncts - GB11-GB12: Emoji ZWJ and Regional Indicator pairs Examples: ``` >>> len(list(sz.utf8_graphemes('abc'))) == 3 True ``` ## Utf8Linebreaks UTF-8 aware line-break-opportunity iterator per Unicode UAX-14 algorithm. ```python class Utf8Linebreaks ``` Yields String views for each line-break-opportunity segment, a soft wrap. For hard-line splitting (str.splitlines()), use utf8_split_newlines(). Created by: - String.utf8_linebreaks() - sz.utf8_linebreaks() UAX-14 Line_Break rules implemented: - LB4-LB6: Mandatory breaks at BK, CR, LF, NL - LB7-LB8: Spaces and ZWSP break opportunities - LB9-LB14: Combining marks, opening/closing punctuation - LB15-LB31: Quotation, numbers, words, CJK rules Examples: ``` >>> len(list(sz.utf8_linebreaks('a\nb'))) >= 2 True ``` ## Utf8Codepoints UTF-8 aware code point iterator yielding Unicode scalar values as int. ```python class Utf8Codepoints ``` Ill-formed bytes decode to U+FFFD, so iteration is total. Created by: - String.utf8_codepoints() - sz.utf8_codepoints() Examples: ``` >>> list(sz.utf8_codepoints('AB')) [65, 66] ``` ## Utf8Sentences UTF-8 aware sentence boundary iterator per Unicode UAX-29 algorithm. ```python class Utf8Sentences ``` Yields sentences, the text segments between consecutive sentence boundaries. Created by: - String.utf8_sentences() - sz.utf8_sentences() UAX-29 Sentence_Break rules implemented: - SB3: CR x LF, no break - SB4: ParaSep breaks - SB6-SB8: ATerm/STerm sentence-final sequences - SB9-SB11: Close/Sp/ParaSep continuations - SB998: Otherwise no break Examples: ``` >>> len(list(sz.utf8_sentences('Hi. Bye.'))) >= 2 True ``` ## Utf8Wordbreaks UTF-8 aware word boundary iterator per Unicode UAX-29 algorithm. ```python class Utf8Wordbreaks ``` Yields words, the text segments between consecutive word boundaries. Created by: - String.utf8_wordbreaks() - sz.utf8_wordbreaks() UAX-29 Word_Break rules implemented: - WB3: CR x LF, no break - WB4: Ignore Extend/Format/ZWJ - WB5-WB13: Letter, number, punctuation rules - WB15-WB16: Regional Indicator pairs Examples: ``` >>> len(list(sz.utf8_wordbreaks('Hi there'))) >= 2 True ``` ## Utf8SplitNewlines ```python class Utf8SplitNewlines ``` ## Utf8Newlines ```python class Utf8Newlines ``` ## Utf8SplitWhitespaces ```python class Utf8SplitWhitespaces ``` ## Utf8Whitespaces ```python class Utf8Whitespaces ``` ## Utf8SplitDelimiters ```python class Utf8SplitDelimiters ``` ## Utf8Delimiters ```python class Utf8Delimiters ``` ## utf8_split_newlines Create an iterator over the lines, the content between Unicode newlines. ```python def utf8_split_newlines( text: _StringLike, skip_empty: bool = False, with_separators: bool = False, *, capabilities: Capability | None = None, ) -> Utf8SplitNewlines ``` Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF. Unlike splitlines(), this returns an iterator for memory-efficient processing. Returns: iterator: An iterator yielding the lines as String objects. - `text`: The string object. - `skip_empty`: Skip empty lines, defaulting to False. - `with_separators`: Interleave the separators losslessly, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Recognized newlines: LF (\n), VT (\v), FF (\f), CR (\r), NEL (U+0085), LINE SEPARATOR (U+2028), PARAGRAPH SEPARATOR (U+2029), CRLF (\r\n) Examples: ``` >>> sum(1 for _ in sz.String('a\nb').utf8_split_newlines()) 2 ``` ## utf8_newlines Create an iterator over the Unicode newline separators, one per codepoint, CR+LF kept together. ```python def utf8_newlines( text: _StringLike, skip_empty: bool = False, *, capabilities: Capability | None = None, ) -> Utf8Newlines ``` Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF. Yields each newline separator; utf8_split_newlines() yields the lines between them. Returns: iterator: An iterator yielding the newline separators as String objects. - `text`: The string object. - `skip_empty`: Skip empty segments, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Recognized newlines: LF (\n), VT (\v), FF (\f), CR (\r), NEL (U+0085), LINE SEPARATOR (U+2028), PARAGRAPH SEPARATOR (U+2029), CRLF (\r\n) Examples: ``` >>> sum(1 for _ in sz.String('a\nb').utf8_newlines()) 1 ``` ## utf8_split_whitespaces Create an iterator over the tokens, the content between Unicode whitespace. ```python def utf8_split_whitespaces( text: _StringLike, skip_empty: bool = False, with_separators: bool = False, *, capabilities: Capability | None = None, ) -> Utf8SplitWhitespaces ``` Uses SIMD-accelerated detection of all 25 Unicode White_Space characters. Splits on each whitespace codepoint; pass skip_empty=True for str.split()-style tokens. Returns: iterator: An iterator yielding the non-whitespace tokens as String objects. - `text`: The string object. - `skip_empty`: Skip empty segments, defaulting to False. - `with_separators`: Interleave the separators losslessly, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Recognized whitespace: ASCII: TAB, LF, VT, FF, CR, SPACE Latin-1: NEXT LINE, NO-BREAK SPACE General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc. CJK: IDEOGRAPHIC SPACE (U+3000) Examples: ``` >>> sum(1 for _ in sz.String('foo bar baz').utf8_split_whitespaces()) 3 ``` ## utf8_whitespaces Create an iterator over Unicode whitespace separators, one per codepoint, CR+LF kept together. ```python def utf8_whitespaces( text: _StringLike, skip_empty: bool = False, *, capabilities: Capability | None = None, ) -> Utf8Whitespaces ``` Uses SIMD-accelerated detection of all 25 Unicode White_Space characters. Yields each whitespace codepoint; utf8_split_whitespaces() yields the tokens between them. Returns: iterator: An iterator yielding the whitespace separators as String objects. - `text`: The string object. - `skip_empty`: Skip empty segments, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Recognized whitespace: ASCII: TAB, LF, VT, FF, CR, SPACE Latin-1: NEXT LINE, NO-BREAK SPACE General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc. CJK: IDEOGRAPHIC SPACE (U+3000) Examples: ``` >>> sum(1 for _ in sz.String('foo bar').utf8_whitespaces()) 1 ``` ## utf8_split_delimiters Create an iterator over the fields, the content between Unicode delimiters. ```python def utf8_split_delimiters( text: _StringLike, skip_empty: bool = False, with_separators: bool = False, *, capabilities: Capability | None = None, ) -> Utf8SplitDelimiters ``` Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and separator/whitespace (Z*) codepoint - the superset of utf8_split_whitespaces(). Splits on delimiters, one separator per codepoint, and yields the segments between them. Returns: iterator: An iterator yielding the non-delimiter segments as String objects. - `text`: The string object. - `skip_empty`: Skip empty segments, defaulting to False. - `with_separators`: Interleave the separators losslessly, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Examples: ``` >>> list(str(t) for t in sz.String('Hi, world').utf8_split_delimiters(skip_empty=True)) ['Hi', 'world'] ``` ## utf8_delimiters Create an iterator over Unicode delimiter separators, one per codepoint, CR+LF kept together. ```python def utf8_delimiters( text: _StringLike, skip_empty: bool = False, *, capabilities: Capability | None = None, ) -> Utf8Delimiters ``` Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and separator/whitespace (Z*) codepoint. Yields each delimiter codepoint; utf8_split_delimiters() yields the fields between them. Returns: iterator: An iterator yielding the delimiter separators as String objects. - `text`: The string object. - `skip_empty`: Skip empty segments, defaulting to False. - `capabilities`: Capabilities to run, defaulting to the CPU's enabled ones. Examples: ``` >>> list(str(d) for d in sz.String('a.b').utf8_delimiters()) ['.'] ``` ## utf8_wordbreaks Return an iterator yielding words per Unicode UAX-29 word boundary rules. ```python def utf8_wordbreaks( string: _StringLike, /, *, capabilities: Capability | None = None, ) -> Utf8Wordbreaks ``` Unlike str.split(), this is UAX-29 compliant and supports all Unicode scripts. - `string`: The input UTF-8 string to split into words. - `capabilities`: Capabilities to run, by default the CPU's enabled ones. Examples: ``` >>> # Stream UAX-29 word tokens lazily: >>> 'world' in (str(w) for w in sz.utf8_wordbreaks('Hi, world')) True ``` ## utf8_codepoints Return an iterator yielding Unicode code points as int, decoded from UTF-8. ```python def utf8_codepoints( string: _StringLike, /, *, capabilities: Capability | None = None, ) -> Utf8Codepoints ``` Ill-formed bytes decode to U+FFFD, the replacement character, so iteration is total and never raises on malformed input. - `string`: The input UTF-8 string to decode into code points. - `capabilities`: Capabilities to run, by default the CPU's enabled ones. Examples: ``` >>> list(sz.utf8_codepoints('AB')) [65, 66] ``` ## utf8_graphemes Return an iterator yielding grapheme clusters per Unicode UAX-29 rules. ```python def utf8_graphemes( string: _StringLike, /, *, capabilities: Capability | None = None, ) -> Utf8Graphemes ``` A grapheme cluster is a user-perceived character (e.g. a base plus combining marks, or an emoji ZWJ sequence) and may span several code points. - `string`: The input UTF-8 string to split into grapheme clusters. - `capabilities`: Capabilities to run, by default the CPU's enabled ones. Examples: ``` >>> # Stream UAX-29 grapheme clusters lazily: >>> [str(g) for g in sz.utf8_graphemes('abc')] ['a', 'b', 'c'] ``` ## utf8_sentences Return an iterator yielding sentences per Unicode UAX-29 sentence boundary rules. ```python def utf8_sentences( string: _StringLike, /, *, capabilities: Capability | None = None, ) -> Utf8Sentences ``` UAX-29 compliant and Unicode-script aware, unlike naive period splitting. - `string`: The input UTF-8 string to split into sentences. - `capabilities`: Capabilities to run, by default the CPU's enabled ones. Examples: ``` >>> # Stream UAX-29 sentences lazily: >>> len(list(sz.utf8_sentences('Hi. Bye.'))) >= 2 True ``` ## utf8_linebreaks Return an iterator yielding segments at line-break opportunities per UAX-14. ```python def utf8_linebreaks( string: _StringLike, /, *, capabilities: Capability | None = None, ) -> Utf8Linebreaks ``` Each segment ends at a line-break opportunity, a soft wrap point. For hard-line splitting (str.splitlines()), use utf8_split_newlines(). - `string`: The input UTF-8 string to split at line-break opportunities. - `capabilities`: Capabilities to run, by default the CPU's enabled ones. Examples: ``` >>> # Stream UAX-14 line-break opportunities lazily: >>> len(list(sz.utf8_linebreaks('a\nb'))) >= 2 True ```