lnmp-codec

Parser and encoder implementations for LNMP (LLM Native Minimal Protocol) v0.3 text format and v0.4 binary format.

Features

Text Format (v0.3)

• Deterministic serialization: Fields always sorted by FID for consistent output
• Canonical format: Newline-separated, no extra whitespace (v0.2)
• Type hints: Optional type annotations (:i, :f, :b, :s, :sa, :r, :ra)
• Nested structures: Parse and encode nested records and arrays (v0.3)
• Semantic checksums: Optional SC32 checksums for drift prevention (v0.3)
• Value normalization: Canonical value transformations (v0.3)
• Equivalence mapping: Synonym recognition (v0.3)
• Semantic dictionary (optional): Apply lnmp-sfe dictionaries during parse/encode to map values to canonical equivalents
• Strict mode: Validates canonical format compliance
• Loose mode: Accepts format variations (default)
• Lenient sanitizer: Optional pre-parse repair layer shared with lnmp-sanitize for LLM-facing inputs
• Round-trip stability: parse(encode(parse(x))) == parse(encode(x))

Binary Format (v0.4)

• Efficient encoding: 30-50% size reduction compared to text format
• Zero-copy design: Fast serialization and deserialization
• Bidirectional conversion: Seamless text ↔ binary conversion
• Canonical binary: Fields sorted by FID, deterministic encoding
• VarInt encoding: Space-efficient integer representation
• Type safety: Explicit type tags for all values
• Version validation: Protocol version checking
• Interoperability: Compatible with v0.3 text format for supported types

Quick Start

Text Format

use lnmp_codec::{Parser, Encoder};

// Parse LNMP text
let input = "F12=14532\nF7=1\nF23=[admin,dev]";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

// Encode to canonical format
let encoder = Encoder::new();
let output = encoder.encode(&record);
// Output: F7=1\nF12=14532\nF23=[admin,dev]  (sorted by FID)

Semantic Dictionary Normalization (optional)

use lnmp_codec::{Parser, Encoder};
use lnmp_sfe::SemanticDictionary;

// Build a dictionary: map Admin/ADMIN -> admin for field 23
let mut dict = SemanticDictionary::new();
dict.add_equivalence(23, "Admin".to_string(), "admin".to_string());

// Parse with dictionary (applies equivalence during parse)
let mut parser = Parser::with_config(
    "F23=[Admin]",
    lnmp_codec::config::ParserConfig {
        semantic_dictionary: Some(dict.clone()),
        ..Default::default()
    },
)
.unwrap();
let record = parser.parse_record().unwrap();

// Encode with the same dictionary (ensures canonical output)
let encoder = Encoder::with_config(
    lnmp_codec::config::EncoderConfig::new().with_semantic_dictionary(dict),
);
let output = encoder.encode(&record);
assert_eq!(output, "F23=[admin]");

Binary Format (v0.4)

use lnmp_codec::binary::{BinaryEncoder, BinaryDecoder};

// Encode text to binary
let text = "F7=1\nF12=14532\nF23=[admin,dev]";
let encoder = BinaryEncoder::new();
let binary = encoder.encode_text(text).unwrap();

// Decode binary to text
let decoder = BinaryDecoder::new();
let decoded_text = decoder.decode_to_text(&binary).unwrap();
// Output: F7=1\nF12=14532\nF23=[admin,dev]  (canonical format)

// Round-trip conversion maintains data integrity
assert_eq!(text, decoded_text);

v0.3 Quick Start - Nested Structures

use lnmp_codec::{Parser, Encoder};

// Parse nested record
let input = "F50={F12=1;F7=1}";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

// Parse nested array
let input = "F60=[{F1=alice},{F1=bob}]";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

// Encode with checksums
use lnmp_codec::EncoderConfig;
let config = EncoderConfig {
    enable_checksums: true,
    ..Default::default()
};
let encoder = Encoder::with_config(config);
let output = encoder.encode(&record);
// Output: F12=14532#36AAE667  (with checksum)

Lenient LLM-Friendly Parsing

use lnmp_codec::{Parser, TextInputMode, ParsingMode};
use lnmp_codec::binary::BinaryEncoder;

let messy = r#"F1=hello "world"; F2 = yes;F3=00042"#;

// Parser profile geared for LLM output
let mut parser = Parser::with_config(
    messy,
    lnmp_codec::config::ParserConfig {
        text_input_mode: TextInputMode::Lenient,
        mode: ParsingMode::Loose,
        normalize_values: true,
        ..Default::default()
    },
).unwrap();
let record = parser.parse_record().unwrap();

// Binary encoder also provides lenient/strict helpers
let encoder = BinaryEncoder::new();
let bytes = encoder.encode_text_llm_profile(messy).unwrap();

// For M2M strict flows use `Parser::new_strict` or `encode_text_strict_profile`.

LNMP v0.2 Features

Deterministic Serialization

Fields are always sorted by FID, ensuring consistent output:

let mut record = LnmpRecord::new();
record.add_field(LnmpField { fid: 100, value: LnmpValue::Int(3) });
record.add_field(LnmpField { fid: 5, value: LnmpValue::Int(1) });
record.add_field(LnmpField { fid: 50, value: LnmpValue::Int(2) });

let encoder = Encoder::new();
let output = encoder.encode(&record);
// Output: F5=1\nF50=2\nF100=3  (sorted)

Type Hints

Optional type annotations for explicit typing:

use lnmp_codec::{Encoder, EncoderConfig};

let config = EncoderConfig {
    include_type_hints: true,
    canonical: true,
};
let encoder = Encoder::with_config(config);
let output = encoder.encode(&record);
// Output: F12:i=14532\nF5:f=3.14\nF7:b=1

Strict vs Loose Parsing

use lnmp_codec::{Parser, ParsingMode};

// Loose mode (default): accepts format variations
let mut parser = Parser::new("F3=test;F1=42").unwrap();  // Unsorted, semicolons OK

// Strict mode: requires canonical format
let mut parser = Parser::with_mode("F1=42\nF3=test", ParsingMode::Strict).unwrap();

// Strict input mode (no sanitizer)
let mut strict_input_parser = Parser::new_strict("F1=42\nF3=test").unwrap();

v0.3 Features

Nested Structures

Parse and encode hierarchical data:

use lnmp_codec::{Parser, Encoder};

// Nested record: F50={F12=1;F7=1}
let input = "F50={F12=1;F7=1}";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

// Nested array: F60=[{F1=alice},{F1=bob}]
let input = "F60=[{F1=alice},{F1=bob}]";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

// Deep nesting
let input = "F100={F1=user;F2={F10=nested;F11=data}}";
let mut parser = Parser::new(input).unwrap();
let record = parser.parse_record().unwrap();

Compliance & Lenient Test Suite

• tests/compliance/rust contains the cross-language suite for strict flows.
• tests/compliance/rust/test-cases-lenient.yaml mirrors the shared sanitizer behavior (auto-quote, comment trimming, nested repairs).
• Run cargo test -p lnmp-codec --tests test-driver -- --nocapture to execute both strict and lenient suites.

The lenient path uses the lnmp-sanitize crate under the hood so SDKs (Rust/TS/Go/Python) can apply identical repair logic before calling strict parsers.

Recommended SDK Profiles

Profile	Parser Config	Binary Encoder	Intended Use
LLM-facing	text_input_mode = Lenient, mode = ParsingMode::Loose, normalize_values = true	encode_text_llm_profile	Repair user/LLM text before strict parsing
M2M strict	Parser::new_strict() or ParserConfig { text_input_mode = Strict, mode = ParsingMode::Strict }	encode_text_strict_profile	Deterministic machine-to-machine pipelines

• Rust exposes helpers (Parser::new_lenient, Parser::new_strict, binary profile methods).
• TypeScript/Go/Python SDKs mirror the same defaults: LLMProfile (Lenient+Loose) for agent/model traffic and M2MProfile (Strict+Strict) for canonical pipelines.
• All SDKs rely on the same sanitizer rules from lnmp-sanitize, ensuring identical repairs across languages.

Nested Structure Rules:

• Nested records use {...} syntax with semicolon separators
• Nested arrays use [{...},{...}] syntax
• Fields sorted by FID at every nesting level
• Arbitrary nesting depth supported

Semantic Checksums (SC32)

Enable checksums for drift prevention:

use lnmp_codec::{Encoder, EncoderConfig};

let config = EncoderConfig {
    enable_checksums: true,
    ..Default::default()
};
let encoder = Encoder::with_config(config);
let output = encoder.encode(&record);
// Output: F12:i=14532#36AAE667

// Parse and validate checksums
use lnmp_codec::{Parser, ParserConfig};
let config = ParserConfig {
    validate_checksums: true,
    ..Default::default()
};
let mut parser = Parser::with_config(input, config).unwrap();
let record = parser.parse_record().unwrap();  // Validates checksums

Value Normalization

Canonical value transformations:

use lnmp_codec::{ValueNormalizer, NormalizationConfig};

let config = NormalizationConfig {
    string_case: StringCaseRule::Lower,
    remove_trailing_zeros: true,
    ..Default::default()
};
let normalizer = ValueNormalizer::new(config);

// Normalizes: true → 1, -0.0 → 0.0, 3.140 → 3.14
let normalized = normalizer.normalize(&value);

Equivalence Mapping

Synonym recognition:

use lnmp_codec::EquivalenceMapper;

let mut mapper = EquivalenceMapper::new();
mapper.add_mapping(7, "yes".to_string(), "1".to_string());
mapper.add_mapping(7, "true".to_string(), "1".to_string());

// Maps "yes" → "1" for field 7
let canonical = mapper.map(7, "yes");  // Some("1")

Canonical Format Rules

v0.3 canonical format:

• ✓ Fields sorted by FID at all nesting levels
• ✓ Newline-separated (no semicolons at top level)
• ✓ Semicolons required in nested records
• ✓ No whitespace around equals signs
• ✓ No spaces after commas in arrays
• ✓ No comments (except in explain mode)
• ✓ Checksums appended as #XXXXXXXX when enabled

Configuration Options

EncoderConfig

pub struct EncoderConfig {
    pub canonical: bool,              // Use canonical format
    pub include_type_hints: bool,     // Add type hints
    pub enable_checksums: bool,       // Append SC32 checksums (v0.3)
    pub normalization_config: Option<NormalizationConfig>,  // Value normalization (v0.3)
    pub equivalence_mapper: Option<EquivalenceMapper>,      // Synonym mapping (v0.3)
}

ParserConfig

pub struct ParserConfig {
    pub mode: ParsingMode,            // Strict or Loose
    pub validate_checksums: bool,     // Validate SC32 checksums (v0.3)
    pub equivalence_mapper: Option<EquivalenceMapper>,  // Synonym mapping (v0.3)
}

NormalizationConfig

pub struct NormalizationConfig {
    pub string_case: StringCaseRule,      // Lower, Upper, None
    pub float_precision: Option<usize>,   // Decimal places
    pub remove_trailing_zeros: bool,      // Remove trailing zeros
}

Migration from v0.2

v0.3 is backward compatible with v0.2. New features:

Feature	v0.2	v0.3
Nested structures	Not supported	Supported
Checksums	Not supported	Optional SC32
Value normalization	Not supported	Configurable
Equivalence mapping	Not supported	Configurable
Type hints	:i, :f, :b, :s, :sa	+ :r, :ra

Migration Guide

1. Parsing: No changes needed - v0.3 parser accepts v0.2 format
2. Encoding: New optional features (checksums, normalization)
3. Nested structures: Use new NestedRecord and NestedArray variants
4. Tests: Update for new value types if using nested structures

// v0.2 code (still works)
let encoder = Encoder::new();

// v0.3 code with new features
let config = EncoderConfig {
    enable_checksums: true,
    normalization_config: Some(NormalizationConfig::default()),
    ..Default::default()
};
let encoder = Encoder::with_config(config);

Performance Notes

• Sorting overhead: Minimal - uses stable sort on encode
• Memory: Sorted fields are cloned, original record unchanged
• Parsing: Loose mode has same performance as v0.1

Binary Format Details (v0.4)

Binary Frame Structure

┌─────────┬─────────┬─────────────┬──────────────────────┐
│ VERSION │  FLAGS  │ ENTRY_COUNT │      ENTRIES...      │
│ (1 byte)│(1 byte) │  (VarInt)   │     (variable)       │
└─────────┴─────────┴─────────────┴──────────────────────┘

Each entry contains:

┌──────────┬──────────┬──────────────────┐
│   FID    │  THTAG   │      VALUE       │
│ (2 bytes)│ (1 byte) │   (variable)     │
└──────────┴──────────┴──────────────────┘

Supported Types

• Integer (0x01): VarInt encoded signed 64-bit integers
• Float (0x02): IEEE 754 double-precision (8 bytes, little-endian)
• Boolean (0x03): Single byte (0x00 = false, 0x01 = true)
• String (0x04): Length-prefixed UTF-8 (length as VarInt + bytes)
• String Array (0x05): Count-prefixed array of length-prefixed strings

Binary Encoding Example

use lnmp_codec::binary::BinaryEncoder;
use lnmp_core::{LnmpRecord, LnmpField, LnmpValue};

let mut record = LnmpRecord::new();
record.add_field(LnmpField {
    fid: 7,
    value: LnmpValue::Bool(true),
});
record.add_field(LnmpField {
    fid: 12,
    value: LnmpValue::Int(14532),
});

let encoder = BinaryEncoder::new();
let binary = encoder.encode(&record).unwrap();
// Binary format: [0x04, 0x00, 0x02, ...] (version, flags, entry count, entries)

Configuration Options

use lnmp_codec::binary::{BinaryEncoder, BinaryDecoder, EncoderConfig, DecoderConfig};

// Encoder configuration
let encoder_config = EncoderConfig::new()
    .with_validate_canonical(true)
    .with_sort_fields(true);
let encoder = BinaryEncoder::with_config(encoder_config);

// Decoder configuration
let decoder_config = DecoderConfig::new()
    .with_validate_ordering(true)  // Enforce canonical field order
    .with_strict_parsing(true);    // Detect trailing data
let decoder = BinaryDecoder::with_config(decoder_config);

Performance Characteristics

• Space Efficiency: 30-50% size reduction compared to text format
• Encoding Speed: < 1μs per field for simple types
• Decoding Speed: < 1μs per field for simple types
• Round-trip: < 10μs for typical 10-field record

Examples

See the examples/ directory for complete examples:

v0.2 Examples:

• type_hints.rs: Type hint usage
• strict_vs_loose.rs: Parsing mode comparison
• deterministic_serialization.rs: Canonical format demo

v0.3 Examples:

• nested_structures.rs: Nested records and arrays
• semantic_checksums.rs: SC32 checksum usage
• explain_mode.rs: Explain mode encoding
• shortform.rs: ShortForm encoding/parsing
• structural_canonicalization.rs: Structural canonicalization

v0.4 Examples (Binary Format):

• binary_encoding.rs: Basic binary encoding and decoding
• binary_roundtrip.rs: Round-trip conversion and data integrity

Run examples:

# v0.2 examples
cargo run --example type_hints
cargo run --example strict_vs_loose
cargo run --example deterministic_serialization

# v0.3 examples
cargo run --example nested_structures
cargo run --example semantic_checksums
cargo run --example explain_mode
cargo run --example shortform
cargo run --example structural_canonicalization

# v0.4 examples (binary format)
cargo run --example binary_encoding
cargo run --example binary_roundtrip

License

MIT OR Apache-2.0