Module spans 

Span boundary repair — turn model predictions over sub-word pieces into spans over whole words.

A token classifier predicts over wordpieces, not words. “Registeel” is tokenised as something like Reg ##ist ##eel, and if the B-ENT label lands on ##ist the decoded span is the three characters ist. The prediction was right about where the entity is and wrong about where it starts, which is a boundary problem, not a classification problem — so it is repaired here rather than trained around.

Left unrepaired this is corrosive rather than merely untidy: fragments become gazetteer entries, index tokens and clustering inputs, so an entity is stored under a name that appears nowhere in the text and can never be matched again. A real run of the exporter produced itar for Registeel, topolis City for Sootopolis City and 202 for 2025, and the resulting facet codebook was debris.

Kept free of feature gates and model dependencies so every consumer shares one implementation: the ONNX tagger, the tuned tagger, and offline export all need the same repair.

Functions

snap_and_merge

Snap a list of (start, end, kind) ranges and merge neighbours of the same kind that snapping brought into contact.

snap_to_words

Expand [start, end) so both edges sit on word boundaries in text.