Unicode Segmentation | StringZilla for Python

String, along with the matching module-level functions, exposes lazy iterators over Unicode boundaries. Each yields String views into the original buffer, so segmentation stays allocation-free, except utf8_codepoints, which yields int code points.

Method / function	Standard	Yields
`utf8_codepoints(string)`	scalar values	`int` code points; ill-formed bytes decode to `U+FFFD`, so iteration never raises.
`utf8_graphemes(string, skip_empty=False)`	UAX-29 grapheme clusters	user-perceived characters such as a base plus combining marks, or emoji ZWJ sequences.
`utf8_wordbreaks(string, skip_empty=False)`	UAX-29 word boundaries	all UAX-29 word segments (words and the separators between them; they tile).
`utf8_sentences(string, skip_empty=False)`	UAX-29 sentence boundaries	sentences.
`utf8_linebreaks(string, skip_empty=False)`	UAX-14 line-break opportunities	soft-wrap segments.
`utf8_split_newlines(string, skip_empty=False, with_separators=False)`	7 Unicode newlines + CRLF	content BETWEEN hard newlines (LF, VT, FF, CR, NEL, `U+2028`, `U+2029`, CRLF).
`utf8_newlines(string, skip_empty=False)`	7 Unicode newlines + CRLF	the newline runs themselves (the separators).
`utf8_split_whitespaces(string, skip_empty=False, with_separators=False)`	25 Unicode `"White_Space"`	content BETWEEN whitespace runs, like `str.split()` with no separator.
`utf8_whitespaces(string, skip_empty=False)`	25 Unicode `"White_Space"` chars	the whitespace runs themselves (the separators).
`utf8_split_delimiters(string, skip_empty=False, with_separators=False)`	punctuation/symbol/separator	content BETWEEN any Unicode delimiter (superset of whitespace).
`utf8_delimiters(string, skip_empty=False)`	punctuation/symbol/separator	the delimiter runs themselves (the separators).

Naming follows one rule: the bare name (newlines/whitespaces/delimiters) yields the separators, while split_* yields the content between them. skip_empty drops empty segments, and with_separators=True interleaves both losslessly, so concatenating the result reproduces the input.

import stringzilla as sz

assert list(sz.utf8_codepoints("AB")) == [65, 66]
assert [str(g) for g in sz.String("a👍🏽b").utf8_graphemes()] == ["a", "👍🏽", "b"]
[str(w) for w in sz.utf8_wordbreaks("Hi, world")]    # all UAX-29 segments
assert sum(1 for _ in sz.String("first\nsecond\nthird").utf8_split_newlines()) == 3
assert [str(t) for t in sz.utf8_split_whitespaces("foo  bar baz", skip_empty=True)] == ["foo", "bar", "baz"]
assert "".join(str(s) for s in sz.utf8_split_newlines("a\nb", with_separators=True)) == "a\nb"

§ Counting Code Points

utf8_count(string) counts Unicode characters rather than bytes, unlike len, which counts bytes, and Strings.utf8_count() counts every string of a collection in one pass:

import stringzilla as sz

assert sz.utf8_count("hello") == 5
assert sz.utf8_count("é") == 1
assert len(sz.String("é")) == 2      # bytes

§ Unicode Case Folding for Uncased Matching

These apply Unicode case folding, correctly handling one-to-many expansions such as German ß matching SS.

utf8_uncased_fold(text, validate=False) — return the case-folded UTF-8 string as bytes.
utf8_uncased_search(haystack, needle, start=0, end=len, validate=False) — index of the first uncased match, or -1. For str inputs start/end and the result are codepoint offsets; for bytes inputs they are byte offsets.
utf8_uncased_order(a, b, validate=False) — uncased lexicographic comparison: negative, zero, or positive int.
utf8_uncased_matches(haystack, needle, include_overlapping=False) — iterate over all uncased matches, yielding each matched region as a String view whose length may differ from needle due to folding expansions; an empty needle matches at every codepoint boundary, the end included.

Pass validate=True to validate UTF-8 before processing.

import stringzilla as sz

assert sz.utf8_uncased_fold("HELLO") == b"hello"
assert sz.utf8_uncased_fold("Straße") == b"strasse"
assert sz.utf8_uncased_search("Hello World", "WORLD") == 6
assert sz.utf8_uncased_order("hello", "HELLO") == 0
assert [str(m) for m in sz.utf8_uncased_matches("Hello HELLO hello", "hello")] == ["Hello", "HELLO", "hello"]

§ Unicode Normalization

These mirror unicodedata.normalize but operate on raw UTF-8 bytes.

utf8_norm(text, form, validate=False) — normalize to one of 'NFC', 'NFD', 'NFKC', 'NFKD', 'NFKC_Casefold'; returns bytes. 'NFKC_Casefold' is the Unicode toNFKC_Casefold, the keys search engines match on: NFKC with full case folding and without default-ignorable codepoints.
utf8_find_denormalized(text, form) — return the byte offset of the first codepoint that breaks the given normalization form, or None if the string is already fully normalized.

import stringzilla as sz

assert sz.utf8_norm("café", "NFD") == b"cafe\xcc\x81"   # decomposed
assert sz.utf8_norm("ﬁ", "NFKD") == b"fi"   # ligature expanded
assert sz.utf8_find_denormalized("café", "NFC") is None   # already NFC
assert sz.utf8_norm("Straße", "NFKC_Casefold") == b"strasse"   # folded, soft hyphen dropped

API

Types

Utf8GraphemesUTF-8 aware grapheme cluster boundary iterator per Unicode UAX-29 algorithm.

__init__.pyi:2383 ↗

class Utf8Graphemes

Yields grapheme clusters, the user-perceived characters.

Created by

String.utf8_graphemes()
sz.utf8_graphemes()

UAX-29 Grapheme_Cluster_Break rules implemented

GB3: CR x LF, no break
GB4-GB5: Control/CR/LF breaks
GB6-GB8: Hangul syllable sequences
GB9-GB9c: Extend/ZWJ/SpacingMark, Indic conjuncts
GB11-GB12: Emoji ZWJ and Regional Indicator pairs

Examples

>>> len(list(sz.utf8_graphemes('abc'))) == 3
True

Utf8LinebreaksUTF-8 aware line-break-opportunity iterator per Unicode UAX-14 algorithm.

__init__.pyi:2409 ↗

class Utf8Linebreaks

Yields String views for each line-break-opportunity segment, a soft wrap. For hard-line splitting (str.splitlines()), use utf8_split_newlines().

Created by

String.utf8_linebreaks()
sz.utf8_linebreaks()

UAX-14 Line_Break rules implemented

LB4-LB6: Mandatory breaks at BK, CR, LF, NL
LB7-LB8: Spaces and ZWSP break opportunities
LB9-LB14: Combining marks, opening/closing punctuation
LB15-LB31: Quotation, numbers, words, CJK rules

Examples

>>> len(list(sz.utf8_linebreaks('a\nb'))) >= 2
True

Utf8CodepointsUTF-8 aware code point iterator yielding Unicode scalar values as int.

__init__.pyi:2435 ↗

class Utf8Codepoints

Ill-formed bytes decode to U+FFFD, so iteration is total.

Created by

String.utf8_codepoints()
sz.utf8_codepoints()

Examples

>>> list(sz.utf8_codepoints('AB'))
[65, 66]

Utf8SentencesUTF-8 aware sentence boundary iterator per Unicode UAX-29 algorithm.

__init__.pyi:2453 ↗

class Utf8Sentences

Yields sentences, the text segments between consecutive sentence boundaries.

Created by

String.utf8_sentences()
sz.utf8_sentences()

UAX-29 Sentence_Break rules implemented

SB3: CR x LF, no break
SB4: ParaSep breaks
SB6-SB8: ATerm/STerm sentence-final sequences
SB9-SB11: Close/Sp/ParaSep continuations
SB998: Otherwise no break

Examples

>>> len(list(sz.utf8_sentences('Hi. Bye.'))) >= 2
True

Utf8WordbreaksUTF-8 aware word boundary iterator per Unicode UAX-29 algorithm.

__init__.pyi:2479 ↗

class Utf8Wordbreaks

Yields words, the text segments between consecutive word boundaries.

Created by

String.utf8_wordbreaks()
sz.utf8_wordbreaks()

UAX-29 Word_Break rules implemented

WB3: CR x LF, no break
WB4: Ignore Extend/Format/ZWJ
WB5-WB13: Letter, number, punctuation rules
WB15-WB16: Regional Indicator pairs

Examples

>>> len(list(sz.utf8_wordbreaks('Hi there'))) >= 2
True

Utf8SplitNewlines

__init__.pyi:2527 ↗

class Utf8SplitNewlines

Utf8Newlines

__init__.pyi:2532 ↗

class Utf8Newlines

Utf8SplitWhitespaces

__init__.pyi:2537 ↗

class Utf8SplitWhitespaces

Utf8Whitespaces

__init__.pyi:2542 ↗

class Utf8Whitespaces

Utf8SplitDelimiters

__init__.pyi:2547 ↗

class Utf8SplitDelimiters

Utf8Delimiters

__init__.pyi:2552 ↗

class Utf8Delimiters

Functions

utf8_split_newlinesCreate an iterator over the lines, the content between Unicode newlines.

__init__.pyi:1582 ↗

def utf8_split_newlines(
    text: _StringLike,
    skip_empty: bool = False,
    with_separators: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8SplitNewlines

Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF. Unlike splitlines(), this returns an iterator for memory-efficient processing.

Returns: iterator: An iterator yielding the lines as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty lines, defaulting to False.
`with_separatorsbool`		Interleave the separators losslessly, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Recognized newlines

LF (\n), VT (\v), FF (\f), CR (\r), NEL (U+0085), LINE SEPARATOR (U+2028), PARAGRAPH SEPARATOR (U+2029), CRLF (\r\n)

Examples

>>> sum(1 for _ in sz.String('a\nb').utf8_split_newlines())
2

utf8_newlinesCreate an iterator over the Unicode newline separators, one per codepoint, CR+LF kept together.

__init__.pyi:1607 ↗

def utf8_newlines(
    text: _StringLike,
    skip_empty: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8Newlines

Uses SIMD-accelerated detection of all 7 Unicode newline characters plus CRLF. Yields each newline separator; utf8_split_newlines() yields the lines between them.

Returns: iterator: An iterator yielding the newline separators as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty segments, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Recognized newlines

LF (\n), VT (\v), FF (\f), CR (\r), NEL (U+0085), LINE SEPARATOR (U+2028), PARAGRAPH SEPARATOR (U+2029), CRLF (\r\n)

Examples

>>> sum(1 for _ in sz.String('a\nb').utf8_newlines())
1

utf8_split_whitespacesCreate an iterator over the tokens, the content between Unicode whitespace.

__init__.pyi:1631 ↗

def utf8_split_whitespaces(
    text: _StringLike,
    skip_empty: bool = False,
    with_separators: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8SplitWhitespaces

Uses SIMD-accelerated detection of all 25 Unicode White_Space characters. Splits on each whitespace codepoint; pass skip_empty=True for str.split()-style tokens.

Returns: iterator: An iterator yielding the non-whitespace tokens as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty segments, defaulting to False.
`with_separatorsbool`		Interleave the separators losslessly, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Recognized whitespace

ASCII: TAB, LF, VT, FF, CR, SPACE Latin-1: NEXT LINE, NO-BREAK SPACE General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc. CJK: IDEOGRAPHIC SPACE (U+3000)

Examples

>>> sum(1 for _ in sz.String('foo bar baz').utf8_split_whitespaces())
3

utf8_whitespacesCreate an iterator over Unicode whitespace separators, one per codepoint, CR+LF kept together.

__init__.pyi:1658 ↗

def utf8_whitespaces(
    text: _StringLike,
    skip_empty: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8Whitespaces

Uses SIMD-accelerated detection of all 25 Unicode White_Space characters. Yields each whitespace codepoint; utf8_split_whitespaces() yields the tokens between them.

Returns: iterator: An iterator yielding the whitespace separators as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty segments, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Recognized whitespace

ASCII: TAB, LF, VT, FF, CR, SPACE Latin-1: NEXT LINE, NO-BREAK SPACE General Punctuation: EN/EM QUAD/SPACE, THIN SPACE, etc. CJK: IDEOGRAPHIC SPACE (U+3000)

Examples

>>> sum(1 for _ in sz.String('foo bar').utf8_whitespaces())
1

utf8_split_delimitersCreate an iterator over the fields, the content between Unicode delimiters.

__init__.pyi:1684 ↗

def utf8_split_delimiters(
    text: _StringLike,
    skip_empty: bool = False,
    with_separators: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8SplitDelimiters

Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and separator/whitespace (Z*) codepoint - the superset of utf8_split_whitespaces(). Splits on delimiters, one separator per codepoint, and yields the segments between them.

Returns: iterator: An iterator yielding the non-delimiter segments as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty segments, defaulting to False.
`with_separatorsbool`		Interleave the separators losslessly, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> list(str(t) for t in sz.String('Hi, world').utf8_split_delimiters(skip_empty=True))
['Hi', 'world']

utf8_delimitersCreate an iterator over Unicode delimiter separators, one per codepoint, CR+LF kept together.

__init__.pyi:1706 ↗

def utf8_delimiters(
    text: _StringLike,
    skip_empty: bool = False,
    *,
    capabilities: Capability | None = None,
) -> Utf8Delimiters

Uses SIMD-accelerated detection of every punctuation (P*), symbol (S*), and separator/whitespace (Z*) codepoint. Yields each delimiter codepoint; utf8_split_delimiters() yields the fields between them.

Returns: iterator: An iterator yielding the delimiter separators as String objects.

Parameters

`text_StringLike`		The string object.
`skip_emptybool`		Skip empty segments, defaulting to False.
`capabilitiesCapability \| None`		Capabilities to run, defaulting to the CPU's enabled ones.

Examples

>>> list(str(d) for d in sz.String('a.b').utf8_delimiters())
['.']

utf8_wordbreaksReturn an iterator yielding words per Unicode UAX-29 word boundary rules.

__init__.pyi:1727 ↗

def utf8_wordbreaks(
    string: _StringLike,
    /,
    *,
    capabilities: Capability | None = None,
) -> Utf8Wordbreaks

Unlike str.split(), this is UAX-29 compliant and supports all Unicode scripts.

Parameters

`string_StringLike`		The input UTF-8 string to split into words.
`capabilitiesCapability \| None`		Capabilities to run, by default the CPU's enabled ones.

Returns

Iterator yielding String objects for each word.

Examples

>>> # Stream UAX-29 word tokens lazily:
>>> 'world' in (str(w) for w in sz.utf8_wordbreaks('Hi, world'))
True

utf8_codepointsReturn an iterator yielding Unicode code points as int, decoded from UTF-8.

__init__.pyi:1747 ↗

def utf8_codepoints(
    string: _StringLike,
    /,
    *,
    capabilities: Capability | None = None,
) -> Utf8Codepoints

Ill-formed bytes decode to U+FFFD, the replacement character, so iteration is total and never raises on malformed input.

Parameters

`string_StringLike`		The input UTF-8 string to decode into code points.
`capabilitiesCapability \| None`		Capabilities to run, by default the CPU's enabled ones.

Returns

Iterator yielding int code points, one per Unicode scalar value.

Examples

>>> list(sz.utf8_codepoints('AB'))
[65, 66]

utf8_graphemesReturn an iterator yielding grapheme clusters per Unicode UAX-29 rules.

__init__.pyi:1767 ↗

def utf8_graphemes(
    string: _StringLike,
    /,
    *,
    capabilities: Capability | None = None,
) -> Utf8Graphemes

A grapheme cluster is a user-perceived character (e.g. a base plus combining marks, or an emoji ZWJ sequence) and may span several code points.

Parameters

`string_StringLike`		The input UTF-8 string to split into grapheme clusters.
`capabilitiesCapability \| None`		Capabilities to run, by default the CPU's enabled ones.

Returns

Iterator yielding String objects for each grapheme cluster.

Examples

>>> # Stream UAX-29 grapheme clusters lazily:
>>> [str(g) for g in sz.utf8_graphemes('abc')]
['a', 'b', 'c']

utf8_sentencesReturn an iterator yielding sentences per Unicode UAX-29 sentence boundary rules.

__init__.pyi:1788 ↗

def utf8_sentences(
    string: _StringLike,
    /,
    *,
    capabilities: Capability | None = None,
) -> Utf8Sentences

UAX-29 compliant and Unicode-script aware, unlike naive period splitting.

Parameters

`string_StringLike`		The input UTF-8 string to split into sentences.
`capabilitiesCapability \| None`		Capabilities to run, by default the CPU's enabled ones.

Returns

Iterator yielding String objects for each sentence.

Examples

>>> # Stream UAX-29 sentences lazily:
>>> len(list(sz.utf8_sentences('Hi. Bye.'))) >= 2
True

utf8_linebreaksReturn an iterator yielding segments at line-break opportunities per UAX-14.

__init__.pyi:1808 ↗

def utf8_linebreaks(
    string: _StringLike,
    /,
    *,
    capabilities: Capability | None = None,
) -> Utf8Linebreaks

Each segment ends at a line-break opportunity, a soft wrap point. For hard-line splitting (str.splitlines()), use utf8_split_newlines().

Parameters

`string_StringLike`		The input UTF-8 string to split at line-break opportunities.
`capabilitiesCapability \| None`		Capabilities to run, by default the CPU's enabled ones.

Returns

Iterator yielding String objects for each line-break-opportunity segment.

Examples

>>> # Stream UAX-14 line-break opportunities lazily:
>>> len(list(sz.utf8_linebreaks('a\nb'))) >= 2
True