Module tokenizer 

Stateful, line-oriented classifier for hashline diff text.

Turns raw patch lines into typed Tokens (section headers, hunk headers, payload rows). The parser consumes the token stream.

Line-ops only (default build): SWAP, DEL, INS.PRE|POST|HEAD|TAIL. Block ops (SWAP.BLK, DEL.BLK, INS.BLK.POST) live behind the block-ops feature gate and are not recognized here.

Ported from omp packages/hashline/src/tokenizer.ts.

Structs

Tokenizer

Stateful, reusable line classifier. Buffer text with feed, flush the remainder with end, and reset with reset before reuse.

Enums

BlockTarget

Where a hunk header lands. Line-ops only in the default build.

Token

One classified line of patch text.

Functions

clone_cursor

Cursor carries only a Copy Anchor; cloning is cheap. Provided for API parity with omp cloneCursor — idiomatic Rust callers reach for Cursor::clone directly.

parse_lid

Parse a bare line-number anchor. Errors on malformed input.

split_hashline_lines

Split text into lines on \n, stripping a trailing \r from each. An empty input yields a single empty line (omp splitHashlineLines).