Platform Notes | StringZilla for Java

§ Zero-Copy with Apache Lucene

Lucene terms are UTF-8 already (BytesRef), so StringZilla reads the backing bytes directly. Value operations such as hash and compare run with no copy; pointer searches copy a heap BytesRef once (see Memory Operations). This is ideal in custom codecs, comparators, or analysis components.

import org.apache.lucene.util.BytesRef;
import java.lang.foreign.MemorySegment;

BytesRef term = termsEnum.term(); // {bytes, offset, length}
MemorySegment view = MemorySegment.ofArray(term.bytes).asSlice(term.offset, term.length); // zero-copy view
long termHash = StringZilla.hash(view, 0);
long position = StringZilla.indexOf(view, "prefix".getBytes(UTF_8));

For index data read through MMapDirectory, the bytes are already off-heap MemorySegments, so they can be passed straight in for fully zero-copy scans.

§ Zero-Copy with Apache Spark

Spark’s UTF8String (Tungsten) is UTF-8 and exposes its backing memory. Wrap it into a MemorySegment to run StringZilla inside Catalyst expressions or a native execution engine.

import org.apache.spark.unsafe.types.UTF8String;
import java.lang.foreign.MemorySegment;

UTF8String value = row.getUTF8String(column);
MemorySegment view = (value.getBaseObject() == null)
        ? MemorySegment.ofAddress(value.getBaseOffset()).reinterpret(value.numBytes())  // off-heap (Tungsten)
        : MemorySegment.ofArray((byte[]) value.getBaseObject())                         // on-heap
                .asSlice(value.getBaseOffset() - 16, value.numBytes());                 // 16 = Unsafe.ARRAY_BYTE_BASE_OFFSET
long digest = StringZilla.hash(view, 0);

UTF8String lives in org.apache.spark.unsafe, a low-level surface best used at the Catalyst-expression or native-engine level rather than from ordinary UDFs. A plain UDF receives a decoded java.lang.String, which is UTF-16 and would transcode.