Crate logreduce_tokenizer

This library provides a tokenizer function for the logreduce project.

The goal is to replace varying words with fixed tokens (e.g. sha256://... is converted to %HASH).

Modules

tokenizer

The tokenizer logic

Macros

tokens_eq

Helper macro to write short tests. tokens_eq!("a", "b") is assert_eq!(process("a"), process("b"))

Functions

PyInit_logreduce_tokenizer^⚠

This autogenerated function is called by the python interpreter when importing the module.