1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
//! `logprobe` — detect normalization errors, entropy bias, and truncation
//! artifacts in LLM logprob data.
//!
//! This crate backs the `logprobe` command-line tool. It parses logprob output
//! from several providers (OpenAI, vLLM, Gemini, Ollama, and JSONL streams),
//! then computes diagnostics over the parsed [`types::LogprobSequence`]:
//! perplexity and summary statistics, per-token entropy, missing-mass and
//! normalization checks, bits-per-byte, and low-confidence highlighting.
//!
//! The modules mirror the CLI subcommands: [`parse`] reads input, [`metrics`]
//! and [`math`] compute statistics, [`diagnostics`] runs the `diagnose`/
//! `validate` checks, [`filters`] backs `confidence`/`highlight`, and
//! [`output`] renders human-readable and JSON results.
//!
//! # Example
//!
//! ```
//! use logprobe::{diagnostics, metrics, parse};
//!
//! // A tiny two-token sequence in the JSONL-array form.
//! let json = r#"[{"token":"Hi","logprob":-0.5},{"token":"!","logprob":-1.0}]"#;
//! let seq = parse::parse_string(json, None, false).unwrap();
//!
//! let summary = metrics::compute_summary(&seq);
//! assert_eq!(summary.token_count, 2);
//!
//! // Without top_logprobs, normalization cannot be assessed.
//! let report = diagnostics::diagnose_report(&seq);
//! assert_eq!(report.total_positions, 0);
//! ```