Expand description
GenericStringArray — the variable-length UTF-8 column, in its Utf8
(32-bit offsets) and LargeUtf8 (64-bit offsets) forms.
The layout is identical to crate::array::GenericBinaryArray — this type
is one, wrapped — plus one invariant: every slot is valid UTF-8.
§Validating untrusted bytes
A payload arriving over the wire is untrusted, and two independent things can be wrong with it:
- the concatenated value region is not valid UTF-8; or
- the region is valid UTF-8, but an offset lands in the middle of a multi-byte code point, so an individual slot is not.
Checking only the first is the classic bug: "é" is [0xC3, 0xA9], so
offsets [0, 1, 2] split a perfectly valid two-byte region into two
invalid slots. GenericStringArray::try_new checks both — a single
str::from_utf8 pass over the whole region, then
str::is_char_boundary on every offset.
use astrs_data::array::{Array, StringArray};
use astrs_data::{Buffer, DataError, ScalarBuffer};
// Valid region, invalid split.
let err = StringArray::try_new(
ScalarBuffer::from_slice(&[0, 1, 2]),
Buffer::from_slice("é".as_bytes()),
None,
)
.unwrap_err();
assert!(matches!(err, DataError::OffsetNotCharBoundary { index: 1, offset: 1 }));
// The same region, split correctly.
let ok = StringArray::try_new(
ScalarBuffer::from_slice(&[0, 2]),
Buffer::from_slice("é".as_bytes()),
None,
)?;
assert_eq!(ok.get(0), Some("é"));§Access cost
Value access returns &str through a checked str::from_utf8 rather
than an unchecked reinterpretation. That keeps unsafe confined to the two
buffer modules (crate::buffer::aligned and crate::buffer::scalar),
and the check is a vectorised scan over one — usually short — value. Hot
paths that do not need &str can read the raw bytes through
GenericStringArray::value_bytes and skip it entirely.
Structs§
- Generic
String Array - A column of variable-length UTF-8 strings.
Type Aliases§
- Large
String Array LargeUtf8column — variable-length UTF-8 with 64-bit offsets.- String
Array Utf8column — variable-length UTF-8 with 32-bit offsets.