tokenless-semantic

Semantic-aware JSON field compression with ONNX embeddings.

Part of the tokenless toolkit.

Features

use tokenless_semantic::SemanticCompressor;

let compressor = SemanticCompressor::new();

// Level 1 (default): keyword-based context detection
let result = compressor.compress_fields(
    &json_value,
    &["temperature", "humidity", "wind_speed"],
);

// Level 2 (onnx feature): ONNX embedding similarity
let result = compressor.compress_with_context(
    &json_value,
    "weather forecast for Tokyo",
);

Levels

Enable Level 2:

[dependencies]
tokenless-semantic = { version = "0.4", features = ["onnx"] }

Minimum Rust Version

Rust 2024 edition, MSRV 1.89.

License: Apache-2.0