Expand description
The SIMD island: SSE2 and AVX2 twins of the JSON byte scanners.
§Why this is a separate crate
Vector intrinsics are unsafe, and the workspace denies unsafe_code
everywhere else. Rather than open a hole in the parser, the vector code
lives here, alone, so rusty_json_turbo can keep tightening toward
forbid(unsafe_code) while still reaching a wide scan. Every unsafe in
this crate is a load or a compare, each one bounds-checked by the caller
immediately above it and carrying its invariant in a SAFETY comment.
There are no dependencies and no build script. That is a contract, not an
accident: rusty_json_turbo advertises an empty dependency tree.
§The rule every kernel here follows
The scalar version is written first, it is the oracle, and it stays in
the tree forever. Each vector kernel is tested against it over every one
of the 256 byte values at every offset, and RJT_ISA=scalar runs it in
production so the two can be compared inside one binary. A vector kernel
whose scalar twin has been deleted cannot be checked and cannot be
debugged.
§What the corpus said before any of this was written
Sizing a kernel is a measurement, not a preference for the widest register
available. On JSONCORP, whitespace runs are 16 to 29 bytes on
citm_catalog and nothing in the corpus reaches 32 bytes, so no
32-byte step is ever fully used. That does not make AVX2 worthless – a
wide load that overshoots the end of a run still resolves it in one
iteration – but it does mean the honest figure is steps taken, not bytes
covered: 8-byte steps 140,381, 16-byte 95,424, 32-byte 50,467 on
citm_catalog, and 16,252 against 15,476 on twitter, which is nothing.
So the whitespace kernel is a citm_catalog win and twitter is a
CONTROL. That distinction was paid for once already: brick B1s cost
twitter 10% by taking an 8-byte step over its 1-byte runs, until a
4-byte scalar peel was put in front.
Enums§
- Isa
- Which instruction set the scanners will actually use.
Functions§
- first_
escape - First byte at or after
fromthat needs a JSON escape. - first_
escape_ scalar - THE ORACLE. One byte at a time, and permanent.
- first_
non_ ws - First byte at or after
fromthat is not JSON whitespace. - first_
non_ ws_ scalar - THE ORACLE. One byte at a time, obviously correct, and permanent.
- is_ws
- Is this byte JSON insignificant whitespace?
- isa
- The ISA in force, honouring the
RJT_ISAoverride. - isa_
available - The widest ISA this build can actually reach on this machine.
- needs_
escape - Does this byte need a JSON escape?
Type Aliases§
- Scanner
- A scanner kernel: index of the first interesting byte at or after
from.