tokencompact-core
tokencompact-core is a synchronous, provider-neutral state machine for
turning bounded log streams and trusted structured findings into deterministic
diagnostics. It performs no I/O and has no async runtime, command runner,
storage, network, environment, or provider-protocol dependency.
The core owns:
- caller-defined scopes and stdout/stderr/combined/annotation streams;
- chunk-invariant CR, LF, and CRLF framing with explicit line and scope bounds;
- a fixed, ordered
ParserPlanof trusted native parsers; - parser lifecycle events for complete, truncated, cancelled, and interrupted scopes;
- provenance, exact deduplication, typed ranking, and serialized-output budgets;
- caller-supplied path mapping and redaction, in that order.
It deliberately contains no compiler or test-framework grammar. Use the
tokencompact-builtins crate for the built-in parser pack, or implement Parser
for a domain-specific pack.
use ;
;
let mut plan = new;
plan.push?;
let mut session = new;
session.begin_scope;
session.push_chunk;
session.end_scope;
let result = session.finish;
# Ok::
Parser plans are constructed before input is accepted; empty and duplicate IDs
are rejected. Parsers are synchronous and deterministic. The engine bounds
retained bytes, line length, open scopes, and emitted candidates before the
final item and serialized-byte budgets are applied. ReductionStats contains
only counters and never raw log text.
Raw log retention is a caller responsibility. The returned Reduction has
already passed through path mapping, redaction, control-character sanitization,
deduplication, ranking, and budgeting.