C++ · Splitting and Partitioning

Splitting and Partitioning

#include <stringzilla/stringzilla.hpp> 6.0.0-rc.1 · 91a27bdd

This is a C++-only convenience layer; the C ABI provides the underlying scans sz_find_best and sz_find_byteset_best that you compose by hand.

partition and rpartition return a three-way sz::string_partition_result struct with before, match, and after members around the first or last occurrence of a pattern. The pattern can be a view, a single character, or a byte-set.

split, rsplit, and splitlines return lazy ranges of string_views. split(view) yields a sz::find_splits_view, rsplit(view) yields a sz::rfind_splits_view, and the byte-set overloads split(byteset) and rsplit(byteset) yield the matching character-splitter views. split(byteset) and rsplit(byteset) default to whitespace, and splitlines() defaults to the newline set. A trailing separator yields a final empty segment, and an empty separator never splits, yielding the whole text as the only segment.

The key property is that these ranges are lazy — they borrow from the source buffer and walk it one string_view at a time, so the next boundary is computed only when you ask for it. Splitting a large document into millions of tokens therefore costs no per-token allocation: each iteration step advances a cursor and hands back a borrowed view. This is the decisive contrast with the idiomatic STL approach, where splitting builds a std::vector<std::string> and heap-allocates a fresh string for every element, copying its bytes out of the source.

#include <stringzilla/stringzilla.hpp>
namespace sz = ashvardanian::stringzilla;

int main() {
    sz::string_view_t csv = "alice,30,engineer";

    sz::string_partition_result<sz::string_view_t> first = csv.partition(',');
    sz::string_view_t name = first.before;
    assert(name == "alice");
    assert(first.match == ",");
    assert(first.after == "30,engineer");

    std::size_t fields = 0;
    for (sz::string_view_t field : csv.split(",")) // "alice", "30", "engineer"
        ++fields;                                // no string is allocated per field
    assert(fields == 3);

    sz::string_view_t doc = "line one\nline two\nline three";
    for (sz::string_view_t line : doc.splitlines()) (void)line;

    return 0;
}

Each range exposes begin/end, dereferences to a sz::string_view_t, and can be consumed in a single pass. When you genuinely need the tokens to outlive the source you can still materialize them into a std::vector<sz::string_view_t>, but only then do you pay for storage — the default path stays allocation-free.

API

Types

find_splits_viewA range of string slices for different splits of the data.
stringzilla.hpp:819 ↗
template <typename string_type_,
          typename matcher_type_>
class find_splits_view

In some sense, represents the inverse operation to find_matches_view, as it reports not the search matches but the data between them. Meaning that for N search matches, there will be N+1 elements in the range, the last one empty when the text ends with a separator. Unlike ::find_matches_view, this range can't be empty, and an empty separator yields the whole text as its only segment. It also can't report overlapping intervals.

Notes

  • Lifetime semantics: Stores forwarded objects, owning strings included, to keep them alive. Iterators receive lightweight views only, so iteration is safe from ownership bugs.
  • For-loop optimized: Iterators are lightweight views with minimal register pressure, ideal for high-performance applications where cache efficiency and register allocation matter.
  • Sentinel support: Supports sentinel-based iteration via operator==(end_sentinel_t) for efficient termination without constructing full end iterators.
  • Compatible with C++20 ranges, std::string_view, and of course, StringZilla.
  • Similar to a pair of boost::algorithm::split_iterator.
rfind_splits_viewA range of string slices for different splits of the data in reverse-order.
stringzilla.hpp:930 ↗
template <typename string_type_,
          typename matcher_type_>
class rfind_splits_view

In some sense, represents the inverse operation to find_matches_view, as it reports not the search matches but the data between them. Meaning that for N search matches, there will be N+1 elements in the range, the last one empty when the text starts with a separator. Unlike ::find_matches_view, this range can't be empty, and an empty separator yields the whole text as its only segment. It also can't report overlapping intervals.

Notes

  • Lifetime semantics: Stores forwarded objects, owning strings included, to keep them alive. Iterators receive lightweight views only, so iteration is safe from ownership bugs.
  • For-loop optimized: Iterators are lightweight views with minimal register pressure, ideal for high-performance applications where cache efficiency and register allocation matter.
  • Sentinel support: Supports sentinel-based iteration via operator==(end_sentinel_t) for efficient termination without constructing full end iterators.
  • Compatible with C++20 ranges, std::string_view, and of course, StringZilla.
  • Similar to a pair of boost::algorithm::split_iterator.
split_parts_tWhich parts of the split a utf8_split_view yields at each separator the kernel reports.
stringzilla.hpp:1154 ↗
enum split_parts_t

Values

between_kThe default: the segments between separators, as in split_newlines, split_whitespaces, and split_delimiters.
separators_kThe separator runs themselves: newlines, whitespace, or delimiter codepoints.
both_kInterleaved segment, separator, segment, and so on, so rejoining them yields the input.
empty_segments_tWhether a utf8_split_view keeps or drops empty (zero-length) segments.
stringzilla.hpp:1168 ↗
enum empty_segments_t

Values

keep_k
skip_k
string_partition_resultA result of split a string once, containing the string slice ::before, the ::match itself, and the slice ::after.
stringzilla.hpp:1804 ↗
template <typename string_>
struct string_partition_result

Fields

beforestring_
matchstring_
afterstring_
Classutf8_split_viewA range of string slices split on the delimiter codepoints that a token kernel reports to it.4
stringzilla.hpp:1216 ↗
template <utf8_tokenizer_best_t tokenizer_,
          utf8_tokenizer_serial_t tokenizer_serial_,
          typename string_type_,
          std::size_t steps_ = sz_iterators_default_steps_k,
          split_parts_t parts_ = split_parts_t::between_k,
          empty_segments_t empties_ = empty_segments_t::keep_k>
class utf8_split_view

The shared engine behind the between-segment views utf8_split_newlines_view, utf8_split_whitespaces_view, and utf8_split_delimiters_view, and the delimiter-run views utf8_newlines_view, utf8_whitespaces_view, and utf8_delimiters_view. The kernel reports the byte spans of the delimiters, and parts_ selects what the view yields:

  • between_k: the N+1 segments between delimiters, the friendly lines, tokens, or fields;
  • separators_k: the N delimiter runs themselves, the newlines, whitespace, or delimiters;
  • both_k: the 2N+1 items interleaved, so concatenating them reconstructs the input exactly.

Each batch's separator endpoints are expanded once into a single boundary array, so every span is just bounds_[i] .. bounds_[i+1] and parts_ reduces to a compile-time (first,stride) over it: the iterator's hot path is one branchless formula shared by all three modes. empties_, a compile-time switch reachable via .skip_empty(), drops empty segments, and both_k is lossless only when empties are kept.

utf8_split_view
stringzilla.hpp:1228 ↗
utf8_split_view() noexcept=default;
utf8_split_view(string_type haystack) noexcept;
skip_emptyThe same split with empty segments dropped (compile-time, branchless).
stringzilla.hpp:1324 ↗
utf8_split_view<tokenizer_, tokenizer_serial_, string_type_, steps_, parts_, empty_segments_t::skip_k> skip_empty() const noexcept;
with_separatorsThe same split yielding segments and delimiters interleaved, losslessly, turning between_k into both_k.
stringzilla.hpp:1332 ↗
utf8_split_view<tokenizer_, tokenizer_serial_, string_type_, steps_, split_parts_t::both_k, empties_> with_separators() const noexcept;
toCopies the items into a container.
stringzilla.hpp:1338 ↗
template <typename container_>
void to(container_ & container);
template <typename container_>
container_ to(container_ && container = {});

Copies the items into a consumed container, returning it at the end.

Functions

splitSplits a string around every non-overlapping inclusion of the second string.
stringzilla.hpp:1707 ↗
template <typename haystack_type_, typename needle_type_>
find_splits_view<range_haystack_type<haystack_type_>, matcher_find<range_view_type<haystack_type_>, exclude_overlaps_t, range_needle_type<haystack_type_, needle_type_>>> split(
    haystack_type_ && h,
    needle_type_ && n) noexcept;

Notes

An empty delimiter never splits, yielding the whole string as the only segment.

rsplitSplits a string around every non-overlapping inclusion of the second string in reverse order.
stringzilla.hpp:1726 ↗
template <typename haystack_type_, typename needle_type_>
rfind_splits_view<range_haystack_type<haystack_type_>, matcher_rfind<range_view_type<haystack_type_>, exclude_overlaps_t, range_needle_type<haystack_type_, needle_type_>>> rsplit(
    haystack_type_ && h,
    needle_type_ && n) noexcept;

Notes

An empty delimiter never splits, yielding the whole string as the only segment.

See also

split_charactersSplits a string around every character from the second string.
stringzilla.hpp:1740 ↗
template <typename haystack_type_, typename needle_type_>
find_splits_view<range_haystack_type<haystack_type_>, matcher_find_first_of<range_view_type<haystack_type_>>> split_characters(
    haystack_type_ && h,
    needle_type_ const & n) noexcept;
rsplit_charactersSplits a string around every character from the second string in reverse order.
stringzilla.hpp:1752 ↗
template <typename haystack_type_, typename needle_type_>
rfind_splits_view<range_haystack_type<haystack_type_>, matcher_find_last_of<range_view_type<haystack_type_>>> rsplit_characters(
    haystack_type_ && h,
    needle_type_ const & n) noexcept;
split_other_charactersSplits a string around every character except the ones from the second string.
stringzilla.hpp:1764 ↗
template <typename haystack_type_, typename needle_type_>
find_splits_view<range_haystack_type<haystack_type_>, matcher_find_first_of<range_view_type<haystack_type_>>> split_other_characters(
    haystack_type_ && h,
    needle_type_ const & n) noexcept;
rsplit_other_charactersSplits a string around every character except the ones from the second string in reverse order.
stringzilla.hpp:1778 ↗
template <typename haystack_type_, typename needle_type_>
rfind_splits_view<range_haystack_type<haystack_type_>, matcher_find_last_of<range_view_type<haystack_type_>>> rsplit_other_characters(
    haystack_type_ && h,
    needle_type_ const & n) noexcept;
TypesMember types5
utf8_split_view::string_typestring_type_
utf8_split_view::string_view_typetypename string_view_for<string_type>::type
utf8_split_view::value_typestring_view_type
utf8_split_view::size_typestd::size_t
utf8_split_view::difference_typestd::ptrdiff_t