Module llm_debugging 

Large Language Model (LLM) Specific Debugging

This module provides specialized debugging capabilities for large language models, focusing on safety, alignment, factuality, toxicity detection, and performance characteristics specific to modern LLMs.

Honesty notes on what these analyzers actually do

SafetyAnalyzer is a rule-based keyword heuristic over the literal response text (see SafetyAnalyzer::find_harmful_keywords) – not a trained safety classifier. Its safety_score and confidence are real functions of measurable properties of the match (which keyword categories fired, and how many), never flat literals.

FactualityChecker does no fact-checking: nothing in this crate queries a knowledge base, so FactualityAnalysisResult::factuality_score is always None. What it does report are deterministic properties of the text under names that say so – claim_like_sentences, uncertainty_indicator_hits and their ratio FactualityAnalysisResult::uncertainty_density.

AlignmentMonitor, BiasDetector, HallucinationDetector and ConversationAnalyzer have no per-response scorers at all. The fixed-value scoring functions they used to carry were deleted rather than kept behind a NOTE:, and the scores they used to fabricate are now Option fields that stay None (AlignmentAnalysisResult::alignment_score, ConversationAnalysisResult::turn_quality, and so on). Scoring any of them for real needs a policy/preference model, which this crate does not ship.

Consequently the aggregate views are Options too: AlignmentMetrics::overall_alignment_score and FactualityMetrics::overall_factuality_score are None rather than the 0.85/0.8 they used to be seeded with, and LLMHealthReport::overall_health_score averages only the terms that exist. Every analyzer’s HealthTracker-backed get_health_summary reports status/trend as a real, live function of whatever score that analyzer actually produced (None / "Unknown (insufficient history)" when it produced none) – never the old hardcoded HealthStatus::Good / "Stable".

Structs

AlignmentAnalysisResult

AlignmentMetrics

Metrics for alignment monitoring

AlignmentMonitor

Alignment monitor for ensuring LLM outputs align with intended behavior

BatchLLMAnalysisReport

BatchMetrics

Aggregate view of one LLMDebugger::analyze_batch run.

BiasAnalysisResult

BiasDetector

Bias detector for identifying various forms of bias in LLM outputs

BiasMetrics

Metrics for bias detection

ConsistencyChecker

Consistency checker for internal consistency in responses

ContextTracker

Context tracking for conversation continuity

ConversationAnalysisResult

ConversationAnalyzer

Conversation analyzer for multi-turn dialog analysis

ConversationTurn

Single turn in a conversation

CriticalIssue

DialogMetrics

Metrics for dialog analysis

EfficiencyMetrics

Metrics for computational efficiency

FactualityAnalysisResult

FactualityChecker

Factuality checker for verifying the accuracy of LLM outputs

FactualityMetrics

Metrics for tracking factual accuracy

GenerationMetrics

Metrics for text generation performance

HallucinationAnalysisResult

HallucinationDetector

Hallucination detector for identifying false or fabricated information

HallucinationMetrics

Metrics for hallucination detection

HealthSummary

HealthTracker

Tracks a bounded window of real, per-call analysis scores so get_health_summary can derive a real HealthStatus and trend direction from actual history, instead of the old hardcoded HealthStatus::Good / trend: "Stable".to_string() that never changed no matter what was analyzed.

LLMAnalysisReport

LLMDebugConfig

Configuration for LLM debugging

LLMDebugger

Main LLM debugging framework

LLMHealthReport

LLMPerformanceProfiler

Performance profiler specific to LLM characteristics

PerformanceAnalysisResult

QualityMetrics

Metrics for output quality

SafetyAnalysisResult

SafetyAnalyzer

Safety analyzer for detecting harmful, toxic, or inappropriate content

SafetyMetrics

Safety metrics for tracking harmful content

ScalabilityMetrics

Metrics for scalability analysis

Enums

AlignmentObjective

Types of alignment objectives for LLMs

AlignmentTrend

Trend in alignment scores over time

BiasCategory

Types of bias to detect in LLM outputs

HarmCategory

Categories of potential harm in LLM outputs

HealthStatus

IssueCategory

IssueSeverity

RiskLevel

SafetyTrend

Trend in safety scores over time

Functions

llm_debugger

Create a new LLM debugger with default configuration

llm_debugger_with_config

Create a new LLM debugger with custom configuration

performance_focused_config

Create a performance-focused LLM debugger configuration

safety_focused_config

Create a safety-focused LLM debugger configuration