Expand description
Diff logs by meaning, not by bytes.
logdelta masks the parts of each log line that vary from run to run (timestamps,
ids, durations, addresses), mines what is left into templates with the Drain
algorithm, and compares how often each template appears in one or more baseline
runs against a target run. The result says which templates are new in the target,
which are gone, which changed in frequency, and which variable took a value the
baselines never showed, and it groups the findings that are one event in the log (a
traceback, the steps a failed job skipped) into blocks.
The logdelta binary is the usual way in (cargo install logdelta); this library
is what it runs on, and what the browser demo compiles to WebAssembly. Build it
with default-features = false to leave out the command-line dependencies.
§Example
analysis::diff_lines works on any source of lines, here two logs held in memory:
use logdelta::analysis::{diff_lines, DiffOptions, FindingKind};
fn lines(log: &str) -> impl Iterator<Item = std::io::Result<String>> + '_ {
log.lines().map(|line| Ok(line.to_string()))
}
let good = "\
2024-01-01T00:00:00Z start
2024-01-01T00:00:01Z ok 12ms
2024-01-01T00:00:02Z ok 7ms";
let bad = "\
2024-01-01T00:00:00Z start
2024-01-01T00:00:01Z ok 9ms
2024-01-01T00:00:02Z FATAL: out of memory";
let diff = diff_lines(vec![lines(good)], lines(bad), &[], &DiffOptions::default())?;
// The timestamps and the durations differ on every line, but only one template is new.
let new: Vec<_> = diff.findings.iter().filter(|f| f.kind == FindingKind::New).collect();
assert_eq!(new.len(), 1);
assert!(new[0].first_target_raw.as_deref().unwrap().ends_with("FATAL: out of memory"));analysis::diff_runs and analysis::mine_run are the same operations over file
paths (plain or gzipped), and output renders a result for a terminal, as
Markdown or as JSON.
Modules§
- analysis
- High-level operations built on top of
crate::mask,crate::drainandcrate::scoring: mining a single run’s templates, and diffing one or more baseline runs against a target run. - blocks
- Findings that are one event in the log, reported as one.
- context
- Collects
-C Ncontext lines around a small set of specific line numbers by making one extra pass over a (re-readable) file with a small ring buffer, rather than loading the whole file into memory. Used bylogdelta diffto show a few lines before/after each finding’s first matching line. - drain
- A Drain-inspired online log template miner.
- io
- Opening log sources (files, stdin, gzip) and iterating their lines.
- mask
- Turns a raw log line into a template-friendly line by replacing the parts that vary
from run to run (timestamps, ids, durations, …) with stable placeholder tokens such as
<TS>or<UUID>. - output
- scoring
- Scores how surprising a template’s frequency change is between baseline run(s) and a target run, using a G-test (log-likelihood-ratio test) on a 2x2 contingency table, with additive smoothing so zero counts don’t blow up and with a variance-based penalty that down-weights templates whose rate already jumps around across multiple passing baselines (flaky noise) instead of treating every jump as equally suspicious.
- values
- Tracks the distinct literal values seen at each (cluster, token-position) pair, per
baseline run and in the target, so
logdelta diffcan flag a value that never appeared in any baseline at an otherwise low-cardinality position — the “content flip” case pure frequency scoring can’t see. The canonical example: a pytest line’s status word flipping fromPASSEDtoFAILEDwhile the line’s overall frequency and position stay identical, so nothing incrate::scoring’s count-based G-test ever moves.