Crate symbi_invis_strip

Expand description

Strip invisible / steganographic Unicode code points and renderer-hidden markup from strings.

This crate extracts the sanitize_field helper used by the symbiont-karpathy-loop demo’s knowledge store, packaged as a standalone zero-dep library for reuse by other agent frameworks that write user-influenced strings into long-term memory.

Two entry points:

sanitize_field — strips Unicode-level invisibles only. Conservative; safe to apply to any free-text field.
sanitize_field_with_markup — also strips  HTML comments and triple-backtick fenced blocks. Right for short structured fields (knowledge-store triples, journal entries) where these have no legitimate use; defends against the 2026 GitHub-comment prompt-injection family (multiple agent frameworks parsed Markdown-renderer-hidden payloads from agent context) and the same trick using “just example code” fenced blocks.

§Why it exists

LLM agents that write to a knowledge store are a ready channel for prompt injection and payload smuggling. An attacker-controlled reflector or a confused planner can emit a seemingly-innocent triple whose object field contains zero-width characters, bidi overrides, the Unicode Tag block, ASCII C0 controls, or any of the other canonical steganographic channels. Those payloads survive round-trip into storage and resurface when the next agent reads the store — at which point the knowledge-store content becomes an instruction surface.

The fix is straightforward and content-agnostic: drop every code point that has no legitimate textual use, before it lands. That’s what sanitize_field does.

§Forbidden ranges

U+0000..=U+0008, U+000B..=U+000C, U+000E..=U+001F, U+007F — ASCII C0 controls and DEL (excluding \t, \n, \r).
U+0080..=U+009F — C1 control block.
U+200B..=U+200F — zero-width + LRM/RLM.
U+202A..=U+202E — bidi explicit directional overrides.
U+2060..=U+206F — word joiner, invisible operators, bidi isolates, deprecated format controls.
U+FEFF — BOM / ZWNBSP.
U+180E — Mongolian vowel separator (legacy invisible).
U+1D173..=U+1D17A — musical notation invisible format chars.
U+FE00..=U+FE0F — variation selectors (emoji-VS steg).
U+E0100..=U+E01EF — supplementary variation selectors.
U+E0000..=U+E007F — Unicode Tag block (primary steg channel for “invisible text”).
U+00AD — soft hyphen (invisible word breaker, used to defeat substring marker detection).
U+0300..=U+036F — combining diacritical marks (attach to letters, steganographic mangling of substring matchers).
U+2070..=U+209F — superscript / subscript forms (visual lookalike bypass for substring matchers).

Legitimate printable content — ASCII, CJK, Cyrillic, diacritics, emoji proper, regular whitespace (\t, \n, \r) — survives unchanged.

§Minimal example

use symbi_invis_strip::sanitize_field;

// Zero-width space smuggled between two words:
assert_eq!(sanitize_field("store\u{200B}knowledge"), "storeknowledge");

// Multilingual content roundtrips:
assert_eq!(sanitize_field("Привет 世界"), "Привет 世界");

§Minimal dependencies, `no_std`-compatible core

The crate has one small dependency (unicode-normalization, used by detect_injection_patterns to NFKC-normalise input before marker matching — this closes the fullwidth and math-alphanumeric homoglyph bypass classes). The core strip surface (sanitize_field, is_forbidden) still uses only core-equivalent operations.

§Sync point

The forbidden-range list is also mirrored in two Python scripts in the upstream repo: scripts/audit-knowledge-stores.py (post-sweep DB scanner) and scripts/lint-cedar-policies.py (Cedar-file linter). Drift between this table and those scripts is a bug; update all three together.

Constants§

INJECTION_MARKERS: Canonical prompt-injection / orchestrator-manipulation marker substrings, lowercased. Detected case-insensitively. Exposed as a &[&str] so downstream tools can share the same authoritative list (the same way is_forbidden exposes the Unicode range list).

Functions§

detect_injection_patterns: Detect known prompt-injection markers in s. Returns the list of matched canonical pattern strings (from INJECTION_MARKERS), in the order they were declared. Empty result means no markers were found.
is_forbidden: true iff code is one of the ranges sanitize_field strips. Exposed so callers that want to detect rather than strip (e.g. audit scripts, linters) can share the same authoritative list.
redact_injection_markers: Replace every case-insensitive occurrence of any INJECTION_MARKERS entry with [redacted:injection]. Exposed for callers that want to apply the marker-redaction step independently of the Unicode/markup strip.
sanitize_field: Remove every forbidden code point from s, returning a sanitised String. Idempotent: sanitize_field(sanitize_field(x)) == sanitize_field(x).
sanitize_field_with_markup: Like sanitize_field but also removes renderer-hidden markup: HTML comments () and triple-backtick fenced blocks (``` ... ```).
sanitize_for_downstream_prompt: Sanitise free-text intended to flow into a downstream agent’s prompt as system / context content. Pipeline:
strip_html_comments: Strip balanced/unbalanced  blocks. Exposed for callers that want markup-only or composition.
strip_md_fences: Strip ``` ... ``` fenced blocks (any length, may contain newlines). Unmatched opener strips to end. Inline single-backtick code is left intact.