Module plain

forest::db::car

Module plain 

Source
Expand description

§Varint frames

CARs are made of concatenations of varint frames. Each varint frame is a concatenation of the body length as an varint, and the frame body itself. unsigned_varint::codec::UviBytes can be used to read frames piecewise into memory.

       varint frame
│◄───────────────────────►│
│                         │
├───────────┬─────────────┤
│varint:    │             │
│body length│frame body   │
└───────────┼─────────────┤
            │             │
frame body ►│◄───────────►│
    offset     =body length

§CARv1 layout and seeking

The first varint frame is a header frame, where the frame body is a [CarHeader] encoded using ipld_dagcbor.

Subsequent varint frames are block frames, where the frame body is a concatenation of a Cid and the block data addressed by that CID.

block frame ►│
body offset  │
             │  =body length
             │◄────────────►│
 ┌───────────┼───┬──────────┤
 │body length│cid│block data│
 └───────────┴───┼──────────┤
                 │◄────────►│
                 │  =block data length
     block data  │
         offset ►│

§Block ordering

… a filecoin-deterministic car-file is currently implementation-defined as containing all DAG-forming blocks in first-seen order, as a result of a depth-first DAG traversal starting from a single root.

§Future work

  • fadvise-based APIs to pre-fetch parts of the file, to improve random access performance.
  • Use an inner Blockstore for writes.
  • Use safe arithmetic for all operations - a malicious frame shouldn’t cause a crash.
  • Theoretically, file-backed blockstores should be clonable (or even Sync) with very low overhead, so that multiple threads could perform operations concurrently.
  • CARv2 support
  • A wrapper that abstracts over car formats for reading.

Structs§

CountRead 🔒
A reader that keeps track of how many bytes it has read.
PlainCar
Note that all operations on this store are blocking.
UncompressedBlockDataLocation
If you seek to offset (from the start of the file), and read length bytes, you should get data that corresponds to a Cid (but NOT the Cid itself).

Functions§

cid_error_to_io_error 🔒
read_block_data_location_and_skip 🔒
Returns (Cid, the block data offset and block data length)
read_v1_header 🔒
read_v2_header
https://ipld.io/specs/transport/car/carv2/#header
read_varint_body_length_or_eof 🔒
Reads body length, leaving the reader at the start of a varint frame, or returns [Ok(None)] if we’ve reached EOF
write_skip_frame_header_async