Crate stowken 

Stowken

Compressed storage and retrieval of LLM token sequences.

Stowken applies segment-level deduplication, variable-width integer encoding, and zstd dictionary compression to achieve 70–85% storage savings over naive Vec<u32> storage.

Licensed under the Apache License 2.0. Enterprise features (cloud backends, compliance tooling, advanced analytics) are available in stowken-enterprise.

Quick start

use stowken::{
    Stowken,
    types::{Conversation, Message, MessageContent, StowkenConfig},
    storage::MemoryBackend,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let vault = Stowken::new(MemoryBackend::new(), StowkenConfig::default()).await?;

    let conversation = Conversation {
        id: None,
        application: Some("my-app".to_string()),
        model: "gpt-4".to_string(),
        tokenizer: "cl100k_base".to_string(),
        messages: vec![
            Message {
                role: "system".to_string(),
                content: MessageContent::Tokens(vec![1, 2, 3]),
                name: None,
                tool_call_id: None,
            },
            Message {
                role: "user".to_string(),
                content: MessageContent::Tokens(vec![4, 5, 6]),
                name: None,
                tool_call_id: None,
            },
            Message {
                role: "assistant".to_string(),
                content: MessageContent::Tokens(vec![7, 8, 9]),
                name: None,
                tool_call_id: None,
            },
        ],
        metadata: None,
    };

    let result = vault.store(conversation).await?;
    println!("Stored {} — {} segments, {} deduped",
        result.id, result.total_segments, result.deduped_segments);

    let retrieved = vault.retrieve(&result.id).await?;
    println!("Retrieved {} segments", retrieved.segments.len());

    Ok(())
}

Re-exports

pub use tokenizer::get_tokenizer;

pub use vault::Stowken;

pub use vault::StowkenError;

Modules

clustering

In-process k-means++ clustering for L2-normalized vectors.

compression

Token compression: varint encoding + zstd (optionally with a dictionary).

dedup

Deduplication strategies for token segments.

dict_registry

Per-vault registry of zstd compression dictionaries.

export

Data export modules.

index

Metadata index and analytics.

near_dedup

Near-duplicate detection via MinHash + LSH, plus a compact streaming delta encoding for storing variants against a canonical segment.

segmenter

Conversation → typed Segment decomposition.

semantic

Semantic similarity search & analytics.

storage

Storage backend trait and implementations.

substring_registry

Per-vault registry of promoted token substrings for v0.5 token-level shared-substring dedup.

tokenizer

Built-in tokenizer implementations.

types

Core data types for Stowken.

vault

Stowken — the main orchestrator.