Module runtime_flags 

§Fase 33.x.h — Process-wide runtime opt-in flags. Today carries the tokenizer_fallback flag that gates BPE-tokenized chunking on the SSE LEGACY path (D9 — opt-in; defaults OFF for v1.24.0 wire byte-compat). §Fase 33.x.h — Process-wide runtime opt-in flags.

Adopter-tunable runtime behaviors that DEFAULT to OFF (to preserve v1.24.0 wire byte-compat) and can be flipped ON for production-time experimentation or vertical-aware enterprise enhancements.

Why not on ServerConfig?

ServerConfig is constructed in 29+ call sites across the test suite; adding fields there means a one-time-but-broad churn. Process-wide flags are simpler for OSS opt-in features that don’t affect the wire format or the auth surface. The std::sync::Mutex<bool> indirection serializes read+write so there’s no torn-write under concurrent test access.

D9 contract (Fase 33.x cycle)

[tokenizer_fallback_enabled] gates the BPE-tokenized chunking that replaces the legacy whitespace 3-word grouping on the SSE LEGACY path. Defaults to OFF — the wire body stays byte- identical with v1.24.0 + with 33.x.b-g for adopters that don’t opt in.

When ON + the LEGACY path activates (flow shape unsupported, backend unknown, etc.), each step’s full output goes through axon_csys::tokens::cl100k_base() and one StepToken event is emitted per BPE-token-decode-boundary. Adopter sees ~1-token granularity that matches real provider chunk size on English prose; non-English degrades to UTF-8-replacement chars at invalid token-boundary slices (rare in practice).

Test isolation

Tests that toggle the flag use the tokenizer_fallback_guard RAII helper or the with_tokenizer_fallback scoped runner. Both restore the previous flag value on drop, so a test that crashes mid-body doesn’t leak state into the next test.

Structs

TokenizerFallbackGuard

RAII guard that restores the flag to its previous value when dropped. Use in tests to scope a flag mutation to a single #[tokio::test] body:

Functions

bpe_chunk_text

§Fase 33.x.h — Tokenize text into BPE chunks via axon_csys::tokens::cl100k_base() and return one String per token (or per safe UTF-8 boundary group when a single token produces non-UTF-8 bytes).

set_tokenizer_fallback

Set the flag explicitly. Returns the previous value so callers can restore it (the TokenizerFallbackGuard RAII helper does this automatically).

tokenizer_fallback_enabled

Read the current flag value. Cheap — single Mutex acquisition. Called once per run_streaming_legacy_path chunking decision (per-flow, not per-token), so the lock contention is negligible.