interface Utf8Segments extends Iterable<Buffer>Four iterable classes lazily split a UTF-8 buffer into TR29 and UAX14 segments: Utf8Wordbreaks for words, Utf8Graphemes for grapheme clusters, Utf8Sentences for sentences, and Utf8Linebreaks for line-break opportunities.
Each is constructed with the source buffer and an optional validate flag, and is its own iterator, yielding zero-copy subarray views into the buffer it keeps alive.
const words = [...new sz.Utf8Wordbreaks(Buffer.from("Hello world!"))].map((b) => b.toString());
assert.deepStrictEqual(words, ["Hello", " ", "world", "!"]);
const graphemes = [...new sz.Utf8Graphemes(Buffer.from("a👩👩👧👦b"))].map((b) => b.toString());
assert.deepStrictEqual(graphemes, ["a", "👩👩👧👦", "b"]); // the ZWJ family is one clusterAPI
InterfaceUtf8SegmentsIterator over segments of a UTF-8 buffer, as zero-copy subarray views of the buffer it keeps alive.1
nextThe next segment, or done once the buffer is exhausted.
next(): IteratorResult<Buffer, undefined>InterfaceUtf8SegmentsConstructorConstructs a segment iterator over a UTF-8 buffer.1
interface Utf8SegmentsConstructorUtf8SegmentsConstructor
new Utf8SegmentsConstructor(
buffer: Buffer,
validate?: boolean,
): Utf8SegmentsParameters
bufferBuffer | UTF-8 encoded input | |
|---|---|---|
validateboolean | If true, validates UTF-8 and throws on invalid input |