Module preprocessor 

Content preprocessing ported from guard_core/detection_engine/preprocessor.py.

Spec 4.0.2: the shared decode chain produces the processed view (null-byte removal, whitespace collapse, attack-preserving truncation) plus the decoded view handed to the URL-decoded scan pass, the signal-preserving raw view, and the short-base64 additive view.

Constants

DEFAULT_MAX_CONTENT_LENGTH

DEFAULT_MAX_FULL_SCAN_BYTES

Functions

decode_base64_candidates

decode_base64_candidates: candidate tokens are replaced by their decoded form when the printable/UTF-8 gates pass, with run-splitting and sub-floor reassembly fallbacks.

decode_common_encodings

Legacy entry kept for the ledger verification tooling.

decode_common_encodings_with_budget

decode_common_encodings: up to 16 fixed-point decode iterations, then a single SQL-comment strip. Returns (content, decode_budget_exhausted).

decode_overlong_utf8_percent_runs

decode_overlong_utf8_percent_runs: percent-encoded byte runs that do not form valid UTF-8 are re-decoded leniently (overlong lead bytes).

normalize_unicode

NFKC normalization + unicode lookalike replacement (fullwidth forms, zero-width characters, etc.).

preprocess

preprocess: full processed view (legacy entry point).

preprocess_signal_preserving

preprocess_signal_preserving: raw view (no decoding).

preprocess_with_decoded

preprocess_with_decoded: (processed, decoded, decode_budget_exhausted).

short_base64_additive_view

preprocess_short_base64_additive_view: decoded fragments of short base64-looking tokens joined with newlines.

strip_sql_comments

_strip_sql_comments: block comments become body (inner-word and /*!-guarded forms are kept); every --/# marker becomes one space.