Module byte_stream 

A byte-oriented CharStream for parsing binary formats.

ANTLR grammars normally consume Unicode text, but many real-world formats are raw bytes: chunk containers (RIFF/WAV), fixed-width records (tar), and self-describing tag streams (CBOR, Standard MIDI). The reference runtimes parse these by treating each byte as a codepoint in U+0000..=U+00FF (a “Latin-1” view) and writing lexer rules over ' '..'ÿ'.

InputStream can do this too, but only after decoding the bytes into a String: any byte >= 0x80 is not valid UTF-8 on its own, so the whole input takes the non-ASCII path and is materialized into a Vec<char> plus a byte-offset table — roughly 12 bytes of heap per input byte, and the compiled-DFA ASCII scanner is disabled.

ByteStream avoids all of that. It is generic over any AsRef<[u8]> backing store, so stream index equals byte offset, lookahead is a single array read, and there is no transcoding or auxiliary allocation.

Mapping to Rust IO primitives

ANTLR parsing needs random access — the lexer and parser seek, look behind with la(-1), and mark/release for prediction — so the bytes must live fully in memory; ByteStream cannot lazily pull from a socket mid-parse. The design instead meets the two IO shapes that matter:

• Bytes you already hold (a network read buffer, an mmap, a slice of a larger frame): borrow them zero-copy with ByteStream::new(&buf[..]). Nothing is copied; the stream lives as long as the borrow.
• A reader (File, TcpStream, Stdin, Cursor): drain it into an owned buffer with ByteStream::from_reader, which is just a thin wrapper over std::io::Read::read_to_end.
• An owned Vec<u8>: hand it over with ByteStream::new(vec) and the stream takes ownership without copying.

// From a file:
let stream = ByteStream::from_reader(std::fs::File::open(path)?)?;
// Zero-copy from an in-memory buffer (e.g. bytes read off a socket):
let stream = ByteStream::new(&packet[..]);

// Feed it to any generated lexer built from a byte-oriented grammar.
let lexer = FooLexer::new(stream);
let tokens = CommonTokenStream::new(lexer);
let mut parser = FooParser::new(tokens);
let tree = parser.entry_rule()?;

Write lexer rules against the byte range, e.g. BYTE : ' ' .. 'ÿ';. A complete worked example — a Standard MIDI File grammar parsed over a ByteStream — lives under tests/fixtures/antlr4-rust-gen/midi-binary/.

Token text is hex

Because the bytes are not text, CharStream::text renders the matched span as a lowercase hex string with no separators ([0xDE, 0xAD] becomes "dead"). Token positions are still exact byte offsets; use IntStream::index or a token’s byte span when you need to slice the original bytes.

Structs

ByteStream

A CharStream backed by raw bytes, where each byte is one symbol in 0..=255 and the stream index is the byte offset.