Crate serializer

Expand description

§DX Serializer

A high-performance serialization library optimized for Humans, LLMs, AND Machines.

§Two Primary Formats

Format	Use Case	Performance
DX LLM	Text format for humans & LLMs	26.8% more efficient than TOON
DX Machine	Binary format for runtime	0.70ns field access

§Quick Start

The simplest way to use dx-serializer is with the convenience functions:

use serializer::{serialize, deserialize, DxDocument, DxLlmValue};

// Create a document
let mut doc = DxDocument::new();
doc.context.insert("name".to_string(), DxLlmValue::Str("MyApp".to_string()));
doc.context.insert("version".to_string(), DxLlmValue::Str("1.0.0".to_string()));

// Serialize to LLM format (text)
let text = serialize(&doc);

// Deserialize back
let parsed = deserialize(&text).unwrap();

§Advanced Configuration

For more control over formatting and output options, use the builder pattern:

use serializer::{SerializerBuilder, DxDocument, DxLlmValue};

let mut doc = DxDocument::new();
doc.context.insert("name".to_string(), DxLlmValue::Str("MyApp".to_string()));

// Configure with builder pattern
let serializer = SerializerBuilder::new()
    .indent_size(4)
    .expand_keys(true)
    .validate_output(false)
    .for_humans()  // Preset for human readability
    .build();

// Use configured serializer
let text = serializer.serialize(&doc);
let human_format = serializer.format_human_unchecked(&doc);

§Format-Specific APIs

For more control, use the format-specific functions:

use serializer::{DxDocument, DxLlmValue};
use serializer::{document_to_llm, llm_to_document};  // LLM format
use serializer::zero::DxZeroBuilder;                  // Machine format

// Create a document
let mut doc = DxDocument::new();
doc.context.insert("name".to_string(), DxLlmValue::Str("MyApp".to_string()));

// Convert to LLM format (text, 26.8% better than TOON)
let llm_text = document_to_llm(&doc);

// Convert to Machine format (binary, 0.70ns access)
let mut buffer = Vec::new();
let mut builder = DxZeroBuilder::new(&mut buffer, 8, 1);
builder.write_u64(0, 12345);
builder.write_string(8, "MyApp");
builder.finish();

§Thread Safety

All public types in dx-serializer are designed to be thread-safe:

§Types that implement `Send + Sync`

The following types can be safely shared between threads:

Type	Send	Sync	Notes
`DxDocument`	✓	✓	Immutable sharing is safe; clone for mutation
`DxLlmValue`	✓	✓	All variants are thread-safe
`DxSection`	✓	✓	Contains only thread-safe types
`DxValue`	✓	✓	Core value type, fully thread-safe
`DxArray`	✓	✓	Vector-backed, thread-safe
`DxObject`	✓	✓	HashMap-backed, thread-safe
`DxError`	✓	✓	Error type, fully thread-safe
`Mappings`	✓	✓	Global singleton, read-only after init
`SerializerBuilder`	✓	✓	Builder pattern, thread-safe
`Serializer`	✓	✓	Configured serializer instance

§Stateless Parsing

All parsing functions are stateless and can be called concurrently from multiple threads without synchronization:

use std::thread;
use serializer::parse;

let handles: Vec<_> = (0..4).map(|i| {
    thread::spawn(move || {
        let input = format!("key{}:value{}", i, i);
        parse(input.as_bytes())
    })
}).collect();

for handle in handles {
    let result = handle.join().unwrap();
    assert!(result.is_ok());
}

§Thread Safety Guarantees

No global mutable state: Parsers create fresh state for each invocation
Immutable singletons: Mappings::get() returns a read-only reference
No interior mutability: Types use standard Rust ownership semantics
Safe concurrent reads: All types support concurrent read access

§Types NOT Thread-Safe

The following types contain mutable state and should not be shared:

Type	Reason	Alternative
`Parser`	Contains mutable parsing state	Create one per thread
`DxZeroBuilder`	Writes to mutable buffer	Create one per thread
`StreamCompressor`	Maintains compression state	Create one per thread
`StreamDecompressor`	Maintains decompression state	Create one per thread

These types implement Send (can be moved between threads) but not Sync (cannot be shared via &T). Create separate instances for each thread.

§Triple Format Architecture (2026 Update)

DX seamlessly converts between three formats:

Human Format (Front-facing files: .sr, .dx) - Beautiful, readable, on disk
LLM Format (.dx/serializer/*.llm) - Token-efficient, 26.8% better than TOON
Machine Format (.dx/serializer/*.machine) - Binary, 0.70ns access

§New Architecture (January 2026)

Front-facing .sr/.dx files now contain Human format (readable)
LLM format moved to .dx/serializer/*.llm (token-optimized for AI)
Machine format remains in .dx/serializer/*.machine (binary performance)

§Key Features

Base62 integers (%x): 320→5A, 540→8k
Auto-increment (%#): Sequential IDs generated automatically
Holographic inflate/deflate for editor integration
Binary format using RKYV zero-copy architecture

§Safety

This crate uses unsafe code in specific, well-documented locations for performance-critical operations. All unsafe code follows these principles:

Minimal scope: Unsafe blocks are as small as possible
Documented invariants: Every unsafe block has a // SAFETY: comment
Validated preconditions: Safe wrappers validate before unsafe operations

§Unsafe Operations by Module

Module	Operation	Justification
`zero/deserialize`	Zero-copy pointer cast	Validated size and alignment
`zero/safe_deserialize`	Safe wrapper for casts	Validates bounds and alignment before cast
`zero/quantum`	Unchecked field access	Compile-time offsets, caller validates bounds
`zero/prefetch`	CPU cache prefetch hints	Hint-only, no memory access
`zero/simd`	SSE4.2/AVX2 intrinsics	Target feature guards ensure CPU support
`zero/simd512`	AVX-512 intrinsics	Target feature guards ensure CPU support
`zero/mmap`	Memory-mapped access	Caller validates offset and type
`zero/arena`	Arena allocation	Capacity checked before allocation
`zero/inline`	UTF-8 unchecked	UTF-8 validated on construction
`utf8`	UTF-8 unchecked	Manual UTF-8 validation precedes conversion
`safety`	Safe cast wrappers	Validates size and alignment before cast

§Safe Wrappers

For most use cases, prefer the safe wrappers in the safety module:

safety::safe_cast - Validates size and alignment before casting
safety::safe_read_slice - Validates bounds before reading a slice
safety::check_bounds - Validates offset and length are within bounds
safety::check_alignment - Validates pointer alignment for a type

The zero::safe_deserialize::SafeDeserializer provides a fully safe API for zero-copy deserialization with automatic bounds and alignment checking.

§SIMD Safety

SIMD operations are guarded by #[target_feature] attributes and cfg blocks:

Compile-time: #[cfg(target_feature = "avx512f")] ensures the code is only compiled when the target supports the feature
Runtime: is_x86_feature_detected! macro checks CPU capabilities
Fallback: Portable implementations are always available

§Memory Safety Guarantees

No undefined behavior: All unsafe code maintains Rust’s safety invariants
No data races: No shared mutable state in unsafe code
No use-after-free: Lifetime annotations ensure references remain valid
No buffer overflows: Bounds are validated before unsafe access

Re-exports§

pub use base62::decode_base62;
pub use base62::encode_base62;
pub use binary_output::BinaryConfig;
pub use binary_output::get_binary_path;
pub use binary_output::hash_path;
pub use binary_output::is_cache_valid;
pub use binary_output::read_binary;
pub use binary_output::write_binary;
pub use biome_config::DxBiomeConfig;
pub use biome_config::DxBiomeConfigEntry;
pub use biome_config::DxBiomeConfigError;
pub use biome_config::DxBiomeTarget;
pub use biome_config::biome_config_from_document;
pub use biome_config::biome_config_from_source;
pub use biome_config::load_biome_config;
pub use compress::compress_to_writer;
pub use compress::format_machine;
pub use converters::convert_to_dx;
pub use converters::dx_to_toon;
pub use converters::toon_to_dx;
pub use converters::json_to_document;
pub use converters::json_to_dx;
pub use converters::toml_to_document;
pub use converters::toml_to_dx;
pub use converters::yaml_to_dx;
pub use encoder::Encoder;
pub use encoder::encode;
pub use encoder::encode_to_writer;
pub use error::DxError;
pub use error::Result;
pub use formatter::HumanFormatter as BinaryHumanFormatter;
pub use formatter::format_human;
pub use mappings::Mappings;
pub use optimizer::optimize_key;
pub use optimizer::optimize_path;
pub use parser::Parser;
pub use parser::parse;
pub use parser::parse_stream;
pub use schema::Schema;
pub use schema::TypeHint;
pub use types::DxArray;
pub use types::DxObject;
pub use types::DxValue;
pub use utf8::Utf8ValidationError;
pub use utf8::validate_string_input;
pub use utf8::validate_utf8;
pub use utf8::validate_utf8_detailed;
pub use utf8::validate_utf8_owned;
pub use llm::AbbrevDict;
pub use llm::ConvertError;
pub use llm::DxDocument;
pub use llm::DxLlmValue;
pub use llm::DxSection;
pub use llm::HumanFormatConfig;
pub use llm::HumanFormatter;
pub use llm::HumanParseError;
pub use llm::HumanParser;
pub use llm::LlmParser;
pub use llm::LlmSerializer;
pub use llm::MachineFormat;
pub use llm::ParseError as LlmParseError;
pub use llm::document_to_human;
pub use llm::document_to_llm;
pub use llm::document_to_machine;
pub use llm::human_to_document;
pub use llm::human_to_llm;
pub use llm::human_to_machine;
pub use llm::human_to_machine_uncompressed;
pub use llm::is_llm_format;
pub use llm::llm_to_document;
pub use llm::llm_to_human;
pub use llm::llm_to_machine;
pub use llm::machine_bytes_to_document;
pub use llm::machine_to_document;
pub use llm::machine_to_human;
pub use llm::machine_to_llm;
pub use llm::try_document_to_machine_with_compression;
pub use llm::SerializerOutput;
pub use llm::SerializerOutputConfig;
pub use llm::SerializerOutputError;
pub use llm::SerializerPaths;
pub use llm::SerializerResult;
pub use llm::CacheConfig;
pub use llm::CacheError;
pub use llm::CacheGenerator;
pub use llm::CachePaths;
pub use llm::CacheResult;
pub use llm::PrettyPrintError;
pub use llm::PrettyPrinter;
pub use llm::PrettyPrinterConfig;
pub use llm::TableWrapper;
pub use llm::TableWrapperConfig;
pub use llm::ModelType;
pub use llm::TokenCounter;
pub use llm::TokenInfo;
pub use llm_models::LLM_MODELS;
pub use llm_models::LlmModel;
pub use llm_models::Provider;
pub use llm_models::TokenAnalysis;
pub use llm_models::analyze_all_models;
pub use llm_models::format_cost;
pub use llm_models::format_tokens;
pub use llm_models::models_by_provider;
pub use llm::LlmParser as DxSerializerParser;
pub use llm::LlmSerializer as DxSerializerSerializer;
pub use llm::ParseError as DxSerializerParseError;
pub use wasm::DxSerializer;
pub use wasm::SerializerConfig;
pub use wasm::TransformResult;
pub use wasm::ValidationResult;
pub use wasm::smart_quote;
pub use builder::Serializer;
pub use builder::SerializerBuilder;

Modules§

base62: Base62 encoding/decoding for DX ∞
binary_output: DX Binary Output (.sr) Module
biome_config: DX-native Biome configuration parsed from the extensionless root dx file.
builder: Serializer Builder Pattern
compress: Compression helpers for compact machine-format output.
converters: Converters from other formats to DX format
encoder: Encoder for DX Machine format
error: Comprehensive error types for dx-serializer
formatter: Human-readable formatter for DX data
llm: DX Serializer LLM Format Module
llm_models: LLM Model Definitions with Pricing
machine: DX-Machine: Pure RKYV + LZ4 Implementation
mappings: Key and path abbreviation mappings used by text and machine encoders.
optimizer: Key and path optimization helpers for compact DX output.
parser: Parser for DX Machine format
safety: Safety Validation Utilities
schema: Schema definition and type hints for DX format
tokenizer: Zero-copy tokenizer for DX Machine format
types: Core data types for DX format
utf8: UTF-8 validation utilities for dx-serializer
wasm: WASM Bindings for DX Serializer VS Code Extension
watch: File Watcher for .sr and dx config files
zero: Compatibility exports for the documented DX-Zero machine-format API.

Structs§

IndexMap: A hash table where the iteration order of the key-value pairs is independent of the hash values of the keys.

Functions§

deserialize: Deserialize LLM text format to a DxDocument.
serialize: Serialize a DxDocument to the LLM text format.