Skip to main content

Module string

Module string 

Source
Expand description

GenericStringArray — the variable-length UTF-8 column, in its Utf8 (32-bit offsets) and LargeUtf8 (64-bit offsets) forms.

The layout is identical to crate::array::GenericBinaryArray — this type is one, wrapped — plus one invariant: every slot is valid UTF-8.

§Validating untrusted bytes

A payload arriving over the wire is untrusted, and two independent things can be wrong with it:

  1. the concatenated value region is not valid UTF-8; or
  2. the region is valid UTF-8, but an offset lands in the middle of a multi-byte code point, so an individual slot is not.

Checking only the first is the classic bug: "é" is [0xC3, 0xA9], so offsets [0, 1, 2] split a perfectly valid two-byte region into two invalid slots. GenericStringArray::try_new checks both — a single str::from_utf8 pass over the whole region, then str::is_char_boundary on every offset.

use astrs_data::array::{Array, StringArray};
use astrs_data::{Buffer, DataError, ScalarBuffer};

// Valid region, invalid split.
let err = StringArray::try_new(
    ScalarBuffer::from_slice(&[0, 1, 2]),
    Buffer::from_slice("é".as_bytes()),
    None,
)
.unwrap_err();
assert!(matches!(err, DataError::OffsetNotCharBoundary { index: 1, offset: 1 }));

// The same region, split correctly.
let ok = StringArray::try_new(
    ScalarBuffer::from_slice(&[0, 2]),
    Buffer::from_slice("é".as_bytes()),
    None,
)?;
assert_eq!(ok.get(0), Some("é"));

§Access cost

Value access returns &str through a checked str::from_utf8 rather than an unchecked reinterpretation. That keeps unsafe confined to the two buffer modules (crate::buffer::aligned and crate::buffer::scalar), and the check is a vectorised scan over one — usually short — value. Hot paths that do not need &str can read the raw bytes through GenericStringArray::value_bytes and skip it entirely.

Structs§

GenericStringArray
A column of variable-length UTF-8 strings.

Type Aliases§

LargeStringArray
LargeUtf8 column — variable-length UTF-8 with 64-bit offsets.
StringArray
Utf8 column — variable-length UTF-8 with 32-bit offsets.