Expand description
Read-time integrity verification for streaming object bodies.
Object-store sources (S3, GCS) read an object body through an async reader and trust the SDK to either deliver the whole body or surface an error. A transfer that terminates early but cleanly (a truncated stream that still yields EOF) would otherwise be parsed and emitted as a complete object — silent data loss with no error (#161).
VerifyingReader wraps the raw body reader (below any decompression
layer, so byte counts and checksums cover the stored bytes, not the
decoded ones) and runs a set of IntegrityChecks once the underlying
reader reaches EOF. A failed check surfaces as an io::Error of kind
InvalidData on the final read, which
the connector maps to FaucetError::Source.
The built-in LengthCheck guards against truncation. Connector crates
implement IntegrityCheck over their own hash libraries for opt-in
checksum verification.
Structs§
- Length
Check - Built-in
IntegrityCheckthat fails when the number of bytes read does not match the length advertised by the object store (Content-Lengthfor S3, objectsizefor GCS). Catches a cleanly-truncated transfer that would otherwise be accepted as a complete object. - Verifying
Reader - An
AsyncReadadapter that observes every byte read frominnerand runs the configuredIntegrityChecks at EOF. Wrap the raw network reader before anyBufReader/ decompression layer.
Traits§
- Integrity
Check - A check fed the raw bytes of an object body as they stream past, and asked to validate once the underlying reader reaches EOF.