Expand description
Write-ahead log. Frame layout, little-endian: 0 len u32 payload length 4 lsn u64 12 kind u8 13 pad u8 x3 16 crc32c u32 over the 16-byte header with crc zeroed, then payload 20 payload
THE TORN TAIL, and the two mechanisms that hold the property.
The predecessor engine lost five rows that had been written AND fsynced. A crash left a complete header with a short payload; the next writer appended at PHYSICAL END OF FILE, so those bytes survived and the new records landed after them. On replay the torn header’s declared length swallowed the new records as its own payload, failed CRC, and stopped replay before reaching them. Every torn-tail test in that engine passed, because none of them wrote anything AFTER recovering.
Here the property is held REDUNDANTLY, by two independent mechanisms, and either one alone is sufficient:
opentruncates to the end of the last CRC-valid frame only when the next header declares a frame that physically crosses EOF, or when every remaining byte is zero. A complete CRC-failed frame and a non-zero fragment shorter than a header are damage, never endings.appendwrites atself.end— the offsetscan’s CRC walk established — and never at physical EOF, so a new frame is laid down ON TOP of any torn bytes and there is nothing left to swallow it.
This redundancy is a maintenance hazard and must be treated as one. Removing either mechanism alone breaks NO test, because the other still holds the property. Someone deleting one will see a green suite and reasonably conclude it was dead code; someone later deleting the other will see a green suite too, right up until a power cut. Reproducing the original bug requires removing BOTH — which is exactly what the falsification in this task does.
Because the integration test cannot distinguish the two, each mechanism is
additionally pinned by its own direct test:
an_all_zero_extension_is_a_clean_ending_and_is_truncated and
append_lands_at_the_scanned_end_not_at_physical_eof.
WHY A WALK MUST SAY WHY IT STOPPED (Task 19; Task 17 final review, F4).
scan walks frames until one cannot be accepted. There are two utterly
different reasons that can happen, and for most of this engine’s life
they left through the same door:
- THE LOG ENDS HERE. A writer was interrupted; what follows is a partial frame, or nothing, or zeros. Discarding it costs nothing, because no completed write is in it.
- SOMETHING IS WRONG HERE. A bit rotted, a sector went bad, a header verifies but names a record kind this build does not know. What follows may be hundreds of committed frames.
Treating the second as the first is how one flipped bit at the midpoint
of a log destroyed 1,499 of 3,000 committed rows while open returned
Ok, and how a single injected EIO on a header read erased half a log.
Both measured. The distinction is now carried by Stop, a type – not by
a comment asserting that every break means the same thing, which is
what the previous version of this file did, incorrectly, in four places.
Three rules follow, and all three are load-bearing:
- Only
Stop::Endmay truncate. - A read that FAILED is not an answer about content: it leaves as
Err. “I could not read this” is not “the log ends here”. - Before CRC, the payload bound and physical extent limit reads. The unverified writer shape may only move classification toward DAMAGE; kind and LSN are trusted only after CRC verifies them.
And a fourth rule, about the refusal itself: Stop::Damaged makes
open refuse, which is only half a design. A refusal nothing can clear
is as unrecoverable as a deletion, and Law 5 forbids both. recover()
is the other half – it copies the whole log aside intact,
independently re-reads the copy, then reconstructs committed regions that
can be resynchronised as the live log – and
a_damaged_log_refuses_to_open_and_recover_clears_it pins the two
halves together, because either alone is a defect.
Structs§
- Scan
- What a walk of the log found: how far it got, the LSN to carry on from, and – the part that used to be missing – WHY it stopped.
- Wal