Crate every_other_token

Expand description

§every-other-token

Watch an LLM answer token by token: intercept the OpenAI or Anthropic stream, score every token’s confidence and perplexity, and rewrite every other token live.

This crate sits between the caller and the model, intercepts the token stream as it arrives over SSE, applies a transform (reverse, uppercase, noise, delete, synonym, chains and more) to tokens at configurable positions, scores model confidence at each position from the logprobs the API returns, and routes the enriched TokenEvents to a terminal renderer, a zero-dependency web UI, JSON lines, and an optional WebSocket collaboration room.

§Example

Runs offline against the built-in mock provider (no API key):

use every_other_token::{providers::Provider, transforms::Transform, TokenInterceptor};

let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
let mut interceptor = TokenInterceptor::new(
    Provider::Mock,
    Transform::Reverse,
    "mock-fixture-v1".to_string(),
    false, // visual
    false, // heatmap
    false, // orchestrator
)
.unwrap()
.with_web_tx(tx);
interceptor.intercept_stream("Why is the sky blue?").await.unwrap();

let first = rx.recv().await.unwrap(); // "The", kept
let second = rx.recv().await.unwrap(); // " quick", reversed
assert_eq!(first.text, "The");
assert_eq!(second.text.trim(), "kciuq");
assert!(second.confidence.is_some()); // exp(logprob) from the stream

§New interpretability modules

Module	Description
`attention`	Attribution matrix arithmetic over full and masked-context logprobs (feed it from `local_model`)
`local_model`	Feature `local`: a model running in-process (candle, SmolLM2 by default) with exact per-token logprobs and real occlusion attribution
`entropy`	Prompt entropy analyzer — Shannon entropy, perplexity estimation, repetition detection, multi-turn timeline
`fingerprint`	Model fingerprinting — statistical signatures for blind A/B testing and model identification
`hallucination`	Hallucination detector — identifies perplexity spikes and confident-but-fragile token positions

§Feature flags

Flag	Description
`sqlite-log`	Persist experiment runs to a local SQLite database via `store::ExperimentStore`.
`self-tune`	Enable the self-improvement telemetry bus and tuning controller.
`self-modify`	Enable snapshot-based parameter mutation (requires `self-tune`).
`intelligence`	Reserved namespace for future interpretability features.
`evolution`	Reserved namespace for future evolutionary optimisation.
`helix-bridge`	HTTP bridge that polls `/api/stats` and pushes config patches.
`redis-backing`	Write-through Redis persistence for agent memory and snapshots.
`wasm`	WASM target bindings via `wasm-bindgen`.

§Quickstart

export OPENAI_API_KEY="sk-..."
cargo run -- "What is consciousness?" --visual
cargo run -- "What is consciousness?" --web
cargo run -- "Explain recursion" --research --runs 20 --output results.json

Modules§

adaptive: Quality-adaptive compression for token sequences.
adaptive_sampling: Adaptive token sampling strategies for diverse and controlled generation.
attention: Causal Attention Tracer
attribution: Token Attribution Exporter
batch: Batch processing pipeline for token sequences.
bayesian: Bayesian A/B Testing with Thompson Sampling
benchmark: Compression Benchmark
chain_of_thought: Chain-of-Thought
chain_validator: Chain-of-thought step validation.
checkpoint: Experiment Checkpointing
chunking: Chunking strategies for token sequences.
circuits: Token Circuit Discovery
citation_manager: Source tracking and citation management.
cli: Command-line argument definitions and helper functions.
code_executor: Sandboxed code block extraction and simulation.
collab: Multiplayer collaboration: room state, participant management, WebSocket handling.
comparison: Cross-model structural comparison.
compression: LZ77-style token sequence deduplication.
config: Optional configuration file support (#16).
context: Context Window Manager
context_compressor: Sliding-window summarisation and importance scoring for context windows.
context_optimizer: Context window optimizer: deduplication, compression, and priority-based budgeting.
context_retriever: BM25 retrieval from a document store.
conversation_memory: Persistent conversation memory with relevance decay over time.
cost_aware_router: Cost-aware request router.
cross_model: Cross-Model Mechanistic Comparison
debate_engine: Multi-agent debate framework with position synthesis.
dialogue_manager: Multi-turn dialogue state manager.
divergence: Divergence Detection Engine.
document_parser: Document text extraction and structure detection.
entropy: Prompt Entropy Analyzer
error: Crate-level error type for Every-Other-Token.
evaluation_framework: LLM output evaluation metrics.
experiments: Experiment Manager
fact_checker: Claim extraction and source-based fact verification.
few_shot_selector: BM25-based few-shot example retrieval with MMR-style diversity re-ranking.
fingerprint: Model Fingerprinting
format_detector: Format detector: classify structured content in token sequences.
hallucination: Hallucination Detector
hallucination_detector: Fact consistency checking for LLM outputs.
heatmap: Per-position token confidence heatmap exporter.
importance: Token importance scorer.
info_entropy: Information-Theoretic Token Analysis
intent_classifier: Zero-shot intent classification for LLM input text.
intervention: Token Intervention Mode.
knowledge_extractor: Named entity extraction and knowledge graph construction.
local_modellocal: A model running in-process (candle): exact logprobs and real occlusion attribution. A small language model running in-process with candle: exact per-token log-probabilities, the real top alternatives at every step, and occlusion attribution computed by actually re-running the model.
logit_lens: Logit Lens
memory_retrieval: Memory Retrieval
model_switcher: Runtime model switching based on task complexity and budget.
multi_turn_tracker: Multi-turn conversation state tracking.
mutation_lab: Structured prompt mutation experiment runner.
output_parser: Structured output parser for LLM responses. Extracts JSON objects, code blocks, lists, and key-value pairs.
output_validator: LLM output validation and schema checking.
patching: Activation Patching
persona_manager: System persona management and style enforcement.
pipeline: Processing Pipeline Combinator
prompt_compression: Lossy and lossless prompt compression for token budget management.
prompt_library: Reusable prompt template library with categorization and search.
prompt_mutation: Systematic prompt mutation for exploring the prompt space.
providers: Provider plugin system and SSE wire types.
quality_scorer: Multi-dimensional response quality scorer.
query_rewriter: Query expansion and reformulation for improved retrieval.
reasoning_tracer: Chain-of-thought extraction and validation.
render: Terminal rendering helpers extracted from the core TokenInterceptor.
replay
request_router: Route requests to different processing pipelines based on content analysis.
reranker: Token sequence reranker using multiple scoring criteria.
research: Headless research mode and batch experiment execution.
response_enhancer: Score and suggest improvements for response clarity.
result_aggregator: Aggregate and merge partial results from multiple model calls.
router: Semantic routing: select compression/truncation strategy based on token signals.
semantic_cache: Semantic cache for prompt-response deduplication using cosine similarity.
semantic_heatmap: Semantic similarity heatmap — TF-IDF cosine similarity across token windows.
semantic_similarity: Semantic similarity computations without external ML dependencies.
sensitivity: Prompt Sensitivity Analyzer
similarity: Semantic similarity scoring using TF-IDF vectors.
stats: Token Statistics Analyzer
steering: Steering Vector Interface
store: SQLite-backed persistence for experiment sessions and per-run metrics.
stream_compress: Streaming token compression with configurable backpressure.
summarizer: Extractive summarization by scoring and selecting top sentences.
template_gen: Template-based prompt generation with variable substitution and conditional blocks.
token_budget: Token budget manager with per-request and per-session tracking.
token_budget_manager: Dynamic token budget allocation for multi-turn conversations.
token_dictionary: Automated token sentiment dictionary with persistence and interactive feedback.
tokenizer: BPE Tokenizer
transforms: Token transform pipeline.
tuiNon-WebAssembly: Full-screen terminal view (--tui). Full-screen terminal view (--tui), built on ratatui.
vocab: Vocabulary Analyzer
web: Embedded web UI server and HTTP request handling.

Structs§

ResearchSession: Aggregated statistics from one or more headless inference runs.
TokenAlternative: One alternative token and its probability (for top-K logprob display).
TokenEvent: A single processed token emitted by the streaming pipeline.
TokenInterceptor: The core streaming engine that sits between the caller and the LLM.

Functions§

run_research_headless: Run runs headless inference calls, collect all TokenEvents, and return an aggregated ResearchSession. Call sites must provide a constructed interceptor (no web_tx set — events are returned via the mpsc channel).