Crate lance_jsonb 

JSONB encoding for Lance JSON columns.

Lance stores lance.json columns as JSONB values in LargeBinary arrays. This crate owns that encoding: it parses JSON text into JSONB (parse_value, OwnedJsonb), reads JSONB values (RawJsonb, from_raw_jsonb), and evaluates SQL/JSONPath expressions over them (jsonpath).

use lance_jsonb::OwnedJsonb;
use lance_jsonb::jsonpath::{Selector, parse_json_path};

let jsonb: OwnedJsonb = r#"{"user":{"name":"Ada","tags":["a","b"]}}"#.parse().unwrap();
let path = parse_json_path(b"$.user.tags[1]").unwrap();
let value = Selector::new(jsonb.as_raw()).select_value(&path).unwrap().unwrap();
assert_eq!(value.to_string(), r#""b""#);

The implementation is adapted from databendlabs/jsonb and keeps only what Lance uses. See the crate README for the upstream revision and the differences from it.

Compatibility

The byte layout below is persisted in Lance data files. Every change must keep reading values written by released Lance versions, and writers must keep producing bytes that those versions can read. tests/it/compat.rs pins the bytes produced for representative inputs.

Encoding format

The JSONB encoding format is a tree-like structure. Each node contains a container header, a number of JEntry headers, and nested encoding values.

• 32-bit container header. 3 bits identify the type of value, including scalar, object and array, and 29 bits identify the number of JEntries in the array or object. The root node of the jsonb value is always a container header.
  • scalar container header: 0x20000000
  • object container header: 0x40000000
  • array container header: 0x80000000
• 32-bit JEntry header. 1 bit identifies whether the JEntry stores a length or an offset, 3 bits identify the type of value, including null, string, number, false, true and container, and the remaining 28 bits identify the length or offset of the encoding value.
  • null JEntry header: 0x00000000
  • string JEntry header: 0x10000000
  • number JEntry header: 0x20000000
  • false JEntry header: 0x30000000
  • true JEntry header: 0x40000000
  • container JEntry header 0x50000000
• Encoding value. Different types of JEntry header have different encoding values.
  • null, true, false: no encoding value, identified by the JEntry header.
  • string: a normal UTF-8 string.
  • number: an encoded number to represent uint64s, int64s and float64s.
  • container: a nested json value with a recursive structure.

An encoding example

// JSON value
[false, 10, {"k":"v"}]

// JSONB encoding
0x80000003    array container header (3 JEntries)
0x30000000    false JEntry header (no encoding value)
0x20000002    number JEntry header (encoding value length 2)
0x5000000e    container JEntry header (encoding value length 14)
0x500a        number encoding value (10)
0x40000001    object container header (1 JEntry)
0x10000001    string key JEntry header (encoding value length 1)
0x10000001    string value JEntry header (encoding value length 1)
0x6b          string encoding value ("k")
0x76          string encoding value ("v")

Modules

jsonpath

Structs

Date

Represents a calendar date (year, month, day) without time component.

Decimal64

Represents a decimal number with 64-bit precision.

Decimal128

Represents a decimal number with 128-bit precision.

Decimal256

Represents a decimal number with 256-bit precision.

Interval

Represents a time interval or duration.

OwnedJsonb

Represents a JSONB data that owns its underlying data.

RawJsonb

Represents JSONB data wrapped around a raw, immutable slice of bytes.

Timestamp

Represents a timestamp (date and time) without timezone information.

TimestampTz

Represents a timestamp with timezone information.

Enums

Error

Number

Represents a JSON number with multiple possible internal representations.

Value

Represents a JSON or extended JSON value.

Functions

from_raw_jsonb

Deserializes a RawJsonb into a Rust data structure using Serde.

parse_value

Parse JSON text to JSONB Value with extended mode. The parser will follow extended JSON syntax rules like leading plus signs, multiple leading zeros, decimal points without digits, and empty array elements. Numeric values are preferentially parsed as decimal values to ensure that precision is not lost.

Type Aliases

Object