tokenless-semantic 1.0.0

Semantic-aware field compression: context-driven keep/drop decisions for JSON responses
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

# tokenless-semantic

[![Crates.io](https://img.shields.io/crates/v/tokenless-semantic.svg)](https://crates.io/crates/tokenless-semantic)
[![Docs](https://docs.rs/tokenless-semantic/badge.svg)](https://docs.rs/tokenless-semantic)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/TokenFleet-AI/tokenless/blob/master/LICENSE)

![tokenless](https://raw.githubusercontent.com/TokenFleet-AI/tokenless/master/assets/tokenless.svg)

Semantic-aware JSON field compression with ONNX embeddings.

Part of the [tokenless](https://github.com/TokenFleet-AI/tokenless) toolkit.

## Why tokenless-semantic?

Compressing JSON responses by dropping irrelevant fields saves tokens — but "relevance" depends on context. A `temperature` field is critical for weather queries but noise for git operations. This crate uses keyword matching (Level 1) or ONNX embeddings (Level 2) to decide which fields to keep based on the user's task description.

## Quick Start

```toml
[dependencies]
tokenless-semantic = "0.4"

# With ONNX support:
# tokenless-semantic = { version = "0.4", features = ["onnx"] }
```

```rust
use tokenless_semantic::SemanticCompressor;

let compressor = SemanticCompressor::new();

// The `compress` method works for both Level 1 and Level 2.
// It keeps fields whose names are semantically relevant to the given context.
let filtered = compressor.compress(&json_value, "weather forecast for Tokyo");
```

### Levels

| Level | Feature | Description |
|-------|---------|-------------|
| **Level 1** | default | Keyword rules from compiled-in TOML profiles (weather, devops, database, git, default). Zero deps, no model download. |
| **Level 2** | `onnx` | ONNX model (`all-MiniLM-L6-v2`, ~86 MB). Cosine similarity between field names and task context. Falls back to Level 1 automatically if the model is unavailable. |

To use Level 2, call `load_onnx()` first (requires the `onnx` feature):

```rust
let mut compressor = SemanticCompressor::new();
match compressor.load_onnx() {
    Ok(true) => println!("ONNX model loaded"),
    Ok(false) => eprintln!("Model not found, using Level 1 fallback"),
    Err(e) => eprintln!("Failed to load model: {e}"),
}
```

Model files are auto-downloaded on first use from GitHub Releases and cached in `~/.tokenless/models/`.

## API

| Method | Description |
|--------|-------------|
| `SemanticCompressor::new()` | Create compressor (Level 1, or `Default::default()`) |
| `compress(&self, value, context) -> Value` | Keep only fields relevant to `context` |
| `is_field_kept(&self, field_name, context) -> bool` | Check if a single field would be kept |
| `detect_category(&self, context) -> &str` | Classify context into a predefined category |
| `load_onnx(&mut self) -> Result<bool, EmbedderError>` | Load ONNX model (requires `onnx` feature) |

### `EmbedderError`

Variants: `ModelNotFound`, `SessionError`, `ModelDownloadError`, `CacheError`, `IoError`, `TokenizerError`, `ThreadPanic`.

## Minimum Rust Version

Rust 2024 edition, MSRV 1.89.

License: Apache-2.0