Module schema 

Schema encoding types and traits for building custom frontends

This module provides the intermediate representation (IR) layer for schema encoding, allowing library users to implement custom parsers (YAML, CSV, TOML, etc.) and serializers that leverage the binary encoding backend.

Architecture

The schema encoding pipeline has three layers:

1. Input layer: Parse custom formats into IR

   • Implement InputParser trait
   • Reference: JsonParser

2. Binary layer: Pack/unpack IR to/from binary

   • pack() - IR to binary bytes
   • unpack() - Binary bytes to IR
   • encode_framed() - Binary to display96 with delimiters
   • decode_framed() - Display96 to binary

3. Output layer: Serialize IR to custom formats

   • Implement OutputSerializer trait
   • Reference: JsonSerializer

Example: Custom CSV Parser

use base_d::schema::{
    InputParser, IntermediateRepresentation, SchemaHeader, FieldDef,
    FieldType, SchemaValue, SchemaError, pack, encode_framed,
};

struct CsvParser;

impl InputParser for CsvParser {
    type Error = SchemaError;

    fn parse(input: &str) -> Result<IntermediateRepresentation, Self::Error> {
        // Parse CSV headers
        let lines: Vec<&str> = input.lines().collect();
        let headers: Vec<&str> = lines[0].split(',').collect();

        // Infer types and build fields
        let fields: Vec<FieldDef> = headers.iter()
            .map(|h| FieldDef::new(h.to_string(), FieldType::String))
            .collect();

        // Parse rows
        let row_count = lines.len() - 1;
        let mut values = Vec::new();
        for line in &lines[1..] {
            for cell in line.split(',') {
                values.push(SchemaValue::String(cell.to_string()));
            }
        }

        let header = SchemaHeader::new(row_count, fields);
        IntermediateRepresentation::new(header, values)
    }
}

// Encode CSV to schema format
let csv = "name,age\nalice,30\nbob,25";
let ir = CsvParser::parse(csv)?;
let binary = pack(&ir);
let encoded = encode_framed(&binary);

IR Structure

The IntermediateRepresentation consists of:

• Header: Schema metadata

  • Field definitions (name + type)
  • Row count
  • Optional root key
  • Optional null bitmap

• Values: Flat array in row-major order

  • [row0_field0, row0_field1, row1_field0, row1_field1, ...]

Type System

Supported field types:

• U64 - Unsigned 64-bit integer
• I64 - Signed 64-bit integer
• F64 - 64-bit floating point
• String - UTF-8 string
• Bool - Boolean
• Null - Null value
• Array(T) - Homogeneous array of type T
• Any - Mixed-type values

Compression

Optional compression algorithms:

• SchemaCompressionAlgo::Brotli - Best ratio
• SchemaCompressionAlgo::Lz4 - Fastest
• SchemaCompressionAlgo::Zstd - Balanced

See Also

• SCHEMA.md - Full format specification
• encode_schema() / decode_schema() - High-level JSON functions

Structs

FieldDef

Field definition with name and type

IntermediateRepresentation

The format-agnostic intermediate representation

JsonParser

JsonSerializer

SchemaHeader

Schema header containing metadata about the encoded data

Enums

FieldType

Field types supported in schema encoding

SchemaCompressionAlgo

Compression algorithms for schema encoding

SchemaError

Errors that can occur during schema encoding/decoding

SchemaValue

Schema value representing a single data element

Traits

InputParser

Trait for parsing input formats into intermediate representation

OutputSerializer

Trait for serializing intermediate representation to output formats

Functions

decode_framed

Remove frame delimiters and decode display96 back to binary

decode_schema

Decode schema format to JSON: framed → display96 → [decompress] → binary → IR → JSON

encode_framed

Encode binary data with display96 and wrap in frame delimiters

encode_schema

Encode JSON to schema format: JSON → IR → binary → [compress] → display96 → framed

pack

Pack intermediate representation into binary format

unpack

Unpack binary data into intermediate representation