ccze-rs 0.5.4

Fast streaming log colorizer with native analytics and formally specified protocol verification
Documentation

ccze-rs

ccze-rs is a fast streaming log colorizer with optional anomaly detection, protocol-order verification, and fixed-width metric-vector export. It installs the ccze executable and is designed for pipelines, live journal output, and large log files.

The parser works on borrowed byte slices, reuses its input and output buffers, and preserves non-UTF-8 input. Async standard I/O keeps the Rust layer responsive while native components handle analytics and verified reductions.

Highlights

  • Automatic, syslog, Apache/Nginx access-log, and JSON parsers
  • Terminal-aware color output plus classic ccze -A raw ANSI behavior
  • Rolling anomaly detection implemented in Fortran through iso_c_binding
  • Lifecycle validation for Start -> Authenticate -> Bind -> Ready
  • Total Idris 2 protocol specification and Agda severity-algebra proofs
  • Portable analytics fallback when GFortran is not installed
  • Versioned 64-byte metric-vector records with strict corruption detection
  • Optional Linux system-integration library APIs behind a Cargo feature
  • Rust 1.75 minimum supported version

Install on Arch-based systems

From the Sisyphus Arch repository:

sudo pacman -Syy
sudo pacman -S ccze-rs

The package owns the standard ccze executable path and includes the native analytics backend; no compiler is required at runtime. The supported release and maintenance target is x86_64 Arch Linux and compatible Arch-based systems.

From crates.io:

cargo install ccze-rs

GFortran is optional for crates.io installs. When it is available, the build uses the vector-friendly Fortran analytics engine; otherwise it compiles the equivalent portable C implementation.

To build locally on Arch or an Arch-based distribution:

sudo pacman -S --needed base-devel rust cargo gcc-fortran
makepkg -si

Use

Color a stream when stdout is a terminal:

journalctl -f | ccze

Force ANSI sequences in a pipeline:

tail -f /var/log/messages | ccze -A | less -R

Analyze a stream with a 128-line window:

ccze --analytics --analytics-window 128 < application.log

Verify a service lifecycle:

ccze --verify-protocol < service.log

Encode one lossy metric vector per input record, then inspect those metrics:

ccze --analytics --verify-protocol --vector-encode -o service.cczev < service.log
ccze --vector-decode --vector-input service.cczev

Vector files contain normalized metrics, not the original log payload. Decode mode prints summaries such as length, severity, rate, entropy, z-score, process identifier, and protocol phase; it cannot reconstruct discarded text. Files carry a versioned header, and malformed headers, partial records, non-finite values, and out-of-range features are rejected.

Select or list parsers:

ccze --plugin syslog < /var/log/messages
ccze --list-plugins

Inspect the native implementation selected at build time:

ccze --backend-info

Run ccze --help for all options.

Architecture

stdin -> Rust async I/O -> zero-copy token ranges -> ANSI renderer -> stdout
                          |                      |
                          |                      +-> Agda-specified severity join
                          +-> Fortran analytics +-> Idris-specified protocol step
                          |
                          +-> Fortran state-vector encoder -> versioned file

Rust owns all I/O, buffers, CLI behavior, and parsing. Fortran receives compact arrays of line lengths and error flags through the C ABI, then computes rolling z-scores and binary entropy. Idris defines the only valid service-state transitions. Agda proves that severity reduction is commutative and idempotent. The vector encoder uses the same build-selected Fortran or portable C policy as analytics. Rust owns the on-disk little-endian format and validates every record before exposing it.

Idris 2's RefC backend and Agda's MAlonzo backend do not expose portable, stable C library symbols directly. The repository therefore keeps their total, type-checked sources as the authoritative specifications and ships small C ABI counterparts whose complete input domains are exercised by Rust tests. See ARCHITECTURE.md for the boundary details.

Optional system integration

The system-integration Cargo feature exposes Linux-oriented library modules for cgroup observation, configuration repair, DKMS/module inspection, gossip, LSM policy, rollback coordination, scheduling experiments, seccomp policy, timing analysis, XDP state, and zram management:

cargo build --features system-integration

These APIs are not activated by the ccze command and do not perform privileged changes merely by compiling the crate. Applications must opt in, construct the relevant manager, and provide the operating-system privileges required by each operation. The LSM manager evaluates policy and observes active kernel state; kernel enforcement requires a separately maintained, kernel-compatible module. These modules are experimental Linux integration APIs, not portable log-colorizer behavior.

Build and verify

make check
make test

make proofs requires Idris 2 and Agda. On Arch, install them with sudo pacman -S --needed agda idris2 when formal verification is required.

License

MIT