tokencompact-core 0.1.0

Deterministic streaming state machine for bounded log reduction
Documentation
  • Coverage
  • 25.5%
    38 out of 149 items documented0 out of 0 items with examples
  • Size
  • Source code size: 50.7 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 2.4 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 7s Average build duration of successful builds.
  • all releases: 7s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • ewhauser/logcompact
    1 0 17
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • ewhauser

tokencompact-core

tokencompact-core is a synchronous, provider-neutral state machine for turning bounded log streams and trusted structured findings into deterministic diagnostics. It performs no I/O and has no async runtime, command runner, storage, network, environment, or provider-protocol dependency.

The core owns:

  • caller-defined scopes and stdout/stderr/combined/annotation streams;
  • chunk-invariant CR, LF, and CRLF framing with explicit line and scope bounds;
  • a fixed, ordered ParserPlan of trusted native parsers;
  • parser lifecycle events for complete, truncated, cancelled, and interrupted scopes;
  • provenance, exact deduplication, typed ranking, and serialized-output budgets;
  • caller-supplied path mapping and redaction, in that order.

It deliberately contains no compiler or test-framework grammar. Use the tokencompact-builtins crate for the built-in parser pack, or implement Parser for a domain-specific pack.

use tokencompact_core::{
    Budget, Diagnostic, DiagnosticClass, Emitter, EndReason, GenericRanker,
    LogLine, NoPathMapping, NoRedaction, OutputPolicy, Parser, ParserPlan,
    ReductionSession, Scope, SessionOptions, Severity, Stream,
};

struct PanicParser;

impl Parser for PanicParser {
    fn id(&self) -> &'static str {
        "example.panic.v1"
    }

    fn observe(&mut self, line: &LogLine<'_>, output: &mut Emitter<'_>) {
        if line.text.contains("panicked at") {
            output.diagnostic(Diagnostic {
                severity: Severity::Error,
                class: DiagnosticClass::Test,
                code: Some("panic".into()),
                message: line.text.into(),
                location: None,
                provenance: None,
                quality: Default::default(),
                repetition_count: 1,
            });
        }
    }
}

let mut plan = ParserPlan::new();
plan.push(PanicParser)?;

let mut session = ReductionSession::new(
    plan,
    SessionOptions {
        budget: Budget { max_bytes: 4096, max_items: 20 },
        ..SessionOptions::default()
    },
    OutputPolicy::new(&NoRedaction, &NoPathMapping, &GenericRanker),
);
session.begin_scope(Scope::step("unit-tests"));
session.push_chunk("unit-tests", Stream::Stderr, b"thread 'x' panicked at src/lib.rs\n");
session.end_scope("unit-tests", EndReason::Complete);
let result = session.finish();
# Ok::<(), tokencompact_core::ParserPlanError>(())

Parser plans are constructed before input is accepted; empty and duplicate IDs are rejected. Parsers are synchronous and deterministic. The engine bounds retained bytes, line length, open scopes, and emitted candidates before the final item and serialized-byte budgets are applied. ReductionStats contains only counters and never raw log text.

Raw log retention is a caller responsibility. The returned Reduction has already passed through path mapping, redaction, control-character sanitization, deduplication, ranking, and budgeting.