> [!WARNING]
> These schematics were generated by Claude Code and have received minimal review
> They are useful to get a high-level overview, but may be wrong.
# Icechunk Flatbuffer File Format Schema
This document describes the flatbuffer metadata files used by Icechunk for
storing repository state, snapshots, manifests, and transaction logs.
## Storage Layout
```
<repository-root>/
|
+-- repo # Repository info file (Repo flatbuffer)
|
+-- config.yaml # Configuration file (YAML)
|
+-- snapshots/ # Snapshot files
| +-- <snapshot-id> # Individual snapshot (Snapshot flatbuffer)
| +-- ...
|
+-- manifests/ # Manifest files
| +-- <manifest-id> # Individual manifest (Manifest flatbuffer)
| +-- ...
|
+-- chunks/ # Chunk data files
| +-- <chunk-id> # Raw chunk data (compressed bytes)
| +-- ...
|
+-- transactions/ # Transaction log files
+-- <snapshot-id> # Transaction log (TransactionLog flatbuffer)
+-- ...
```
## File Header Format (39 bytes)
All flatbuffer files (except chunks) have a 39-byte header:
```
+---+---+---+---+---+---+---+---+---+---+---+---+--- ... ---+---+---+---+
0 1 2 3 4 5 6 7 8 9 10 11 12 35 36 37 38
Bytes 0-11: Magic bytes "ICE🧊CHUNK" (12 bytes: "ICE"=3, 🧊=4 UTF-8, "CHUNK"=5)
Bytes 12-35: Implementation name, padded to 24 bytes (e.g., "ic-0.1.0")
Byte 36: Spec version (1=V1.0, 2=V2.0)
Byte 37: File type (1=Snapshot, 2=Manifest, 3=Attributes,
4=TransactionLog, 5=Chunk, 6=RepoInfo)
Byte 38: Compression (0=None, 1=Zstd)
```
## Object ID Types
```
ObjectId12 (12 bytes) ObjectId8 (8 bytes)
+-------------------------+ +------------------+
Used for: SnapshotId, Used for: NodeId
ManifestId, (arrays, groups)
ChunkId,
AttributesId
```
IDs are encoded as Crockford Base32 strings for display/paths.
## Repo File (Root Repository State)
The `repo` file is the entry point to an Icechunk repository.
It contains all branches, tags, and a summary of all snapshots.
```
+==================================================================+
| spec_version: u8 # Format version (1 or 2) |
+------------------------------------------------------------------+
| tags: [Ref] # Named tag references |
| +-- name: string |
| +-- snapshot_index: u32 # Index into snapshots[] |
+------------------------------------------------------------------+
| branches: [Ref] # Named branch references |
| +-- name: string |
| +-- snapshot_index: u32 # Index into snapshots[] |
+------------------------------------------------------------------+
| deleted_tags: [string] # Names of deleted tags |
+------------------------------------------------------------------+
| snapshots: [SnapshotInfo] # All snapshots in repo |
| +-- id: ObjectId12 |
| +-- parent_offset: i32 # Offset to parent in this array |
| +-- flushed_at: u64 # Microseconds since Unix epoch |
| +-- message: string # Commit message |
| +-- metadata: [MetadataItem]? |
+------------------------------------------------------------------+
| status: RepoStatus # Repository availability |
| +-- availability: enum # Online/ReadOnly/Offline |
| +-- set_at: u64 # When status was set |
| +-- limited_availability_reason: string? |
+------------------------------------------------------------------+
| metadata: [MetadataItem]? # Repository-level metadata |
+------------------------------------------------------------------+
| latest_updates: [Update] # Recent update history |
+------------------------------------------------------------------+
| repo_before_updates: string? # Path to previous repo state |
+==================================================================+
```
### Ref (Branch/Tag Reference)
```
+--------------------------+
+--------------------------+
```
## Snapshot File
Snapshot files contain the complete state of a repository version,
including all groups and arrays with their metadata and manifest refs.
```
+==================================================================+
| id: ObjectId12 # Unique snapshot identifier |
+------------------------------------------------------------------+
| parent_id: ObjectId12? # Previous snapshot (if any) |
| # DEPRECATED: v2.0 uses Repo file |
+------------------------------------------------------------------+
| nodes: [NodeSnapshot] # All groups and arrays |
+------------------------------------------------------------------+
| flushed_at: u64 # Microseconds since Unix epoch |
+------------------------------------------------------------------+
| message: string # Commit message |
+------------------------------------------------------------------+
| metadata: [MetadataItem] # Snapshot metadata |
+------------------------------------------------------------------+
| manifest_files: [ManifestFileInfo] # Summary of manifests |
| +-- id: ObjectId12 |
| +-- size_bytes: u64 |
| +-- num_chunk_refs: u32 |
+==================================================================+
```
### NodeSnapshot (Array or Group)
```
+====================================+
| user_data: [u8] # Zarr attrs |
| node_data: NodeData (union) |
| +-- ArrayNodeData |
| +-- GroupNodeData |
+====================================+
```
Note: Flatbuffers automatically generates a `node_data_type` discriminator field
(NONE=0, Array=1, Group=2) to identify which union variant is stored.
### ArrayNodeData and GroupNodeData
```
ArrayNodeData: GroupNodeData:
+-------------------------------+ +----------------+
| +-- chunk_length: u64 |
+-------------------------------+
| dimension_names: [DimensionName]?
+-------------------------------+
| manifests: [ManifestRef] |
| +-- object_id: ObjectId12 --|---> manifests/<id>
| +-- extents: [ChunkIndexRange]
+-------------------------------+
```
## Manifest File
Manifest files map chunk coordinates to chunk locations.
Each manifest contains chunk references for one or more arrays.
```
+==================================================================+
| id: ObjectId12 # Manifest file identifier |
+------------------------------------------------------------------+
| arrays: [ArrayManifest] # Per-array chunk mappings |
+==================================================================+
```
### ArrayManifest
```
+=========================================+
| node_id: ObjectId8 # Which array |
+-----------------------------------------+
```
### ChunkRef (Chunk Reference)
```
+==================================================+
+--------------------------------------------------+
| inline: [u8]? # Inline chunk data |
| # (small chunks) |
+--------------------------------------------------+
| offset: u64 # Byte offset in file |
+--------------------------------------------------+
| length: u64 # Chunk size in bytes |
+--------------------------------------------------+
| chunk_id: ObjectId12? # --> chunks/<id> |
+--------------------------------------------------+
| location: string? # Virtual chunk URL |
| # (external reference) |
+--------------------------------------------------+
| checksum_etag: string? # S3 ETag for verify |
+--------------------------------------------------+
| checksum_last_modified: u32 # Last modified |
+==================================================+
```
ChunkRef has three mutually exclusive storage modes:
1. **inline** - Data embedded directly in manifest
2. **chunk_id** - Reference to `chunks/<id>` file
3. **location** - URL to external virtual chunk (HDF5, NetCDF, etc.)
## Transaction Log File
Transaction logs record the delta between snapshots for conflict
detection and efficient change tracking.
```
+==================================================================+
| id: ObjectId12 # Same as snapshot id |
+------------------------------------------------------------------+
| new_groups: [ObjectId8] # Created groups |
+------------------------------------------------------------------+
| new_arrays: [ObjectId8] # Created arrays |
+------------------------------------------------------------------+
| deleted_groups: [ObjectId8] # Deleted groups |
+------------------------------------------------------------------+
| deleted_arrays: [ObjectId8] # Deleted arrays |
+------------------------------------------------------------------+
| updated_arrays: [ObjectId8] # Metadata-modified arrays |
+------------------------------------------------------------------+
| updated_groups: [ObjectId8] # Metadata-modified groups |
+------------------------------------------------------------------+
| updated_chunks: [ArrayUpdatedChunks] # Per-array chunk deltas |
| +-- node_id: ObjectId8 |
| +-- chunks: [ChunkIndices] # List of modified chunk coords |
+------------------------------------------------------------------+
| moved_nodes: [MoveOperation]? # Renamed/moved nodes (v2.0+) |
| +-- from: string # Source path |
| +-- to: string # Destination path |
+==================================================================+
```
## Relationships Diagram
```
+--------+
| REPO |
+--------+
|
+----------------------+----------------------+
| | |
branches[] tags[] snapshots[]
| | |
v v |
+----------+ +----------+ |
| Ref | | Ref | |
| name | | name | |
| snap_idx-|--------->| snap_idx-|---------------->|
+----------+ +----------+ |
v
+--------------+
| SnapshotInfo |
| id, message |
| parent_off |
+--------------+
|
| (id lookup)
v
+-------------------------------------------------------------------------+
| +====================================================================+ |
| | SNAPSHOT | |
| +====================================================================+ |
| | id, parent_id, message, flushed_at | |
| +--------------------------------------------------------------------+ |
| | nodes: [NodeSnapshot] | |
| | +-- id: ObjectId8 | |
| | +-- path: "/group/array" | |
| | +-- ArrayNodeData | |
| | +-- manifests: [ManifestRef] | |
| | +-- object_id -------------------------------------------|----+
| +--------------------------------------------------------------------+ | |
| | manifest_files: [ManifestFileInfo] | | |
| | +-- id, size_bytes, num_chunk_refs | | |
| +====================================================================+ | |
| | |
+---------+---------------------------------------------------------------+ |
| |
| +----------------------------------------+
| |
| v
+---------+--------------------+ manifests/<id>
| +==========================+ | +==========================================+
| | TRANSACTION LOG | | | id: ObjectId12 |
| +==========================+ | +------------------------------------------+
| | new_groups, new_arrays | | | arrays: [ArrayManifest] |
| | deleted_*, updated_* | | | +-- node_id: ObjectId8 |
| | updated_chunks | | | +-- refs: [ChunkRef] |
| | +-- node_id | | | +-- index: [0, 5, 12] |
| | +-- chunk coords | | | +-- chunk_id ----------------------|----+
| | moved_nodes | | | +-- offset, length | |
| | +-- from, to | | | +-- inline, location | |
| +==========================+ | +==========================================+ |
+------------------------------+ |
|
v
chunks/<chunk-id>
+================+
| Raw chunk data |
| (compressed) |
+================+
```
## Data Flow on Write
1. Session modifies arrays and chunks
2. Changed chunks written to `chunks/<chunk-id>`
3. New Manifest files created with updated ChunkRefs
4. New Snapshot created pointing to Manifests
5. TransactionLog created recording the delta
6. Repo file atomically updated with new SnapshotInfo
```
+----------+ +----------+ +------------+ +------+
+----------+ +----------+ +------------+ +------+
```
## Data Flow on Read
1. Read Repo file to get branch/tag -> snapshot mapping
2. Fetch Snapshot file by id
3. For each array, read ManifestRefs
4. Fetch Manifest files, find ChunkRefs by coordinates
5. Fetch chunk data from `chunks/<id>` or inline
```
Repo --> SnapshotInfo --> Snapshot --> ManifestRef --> Manifest --> ChunkRef --> Chunk
```
## Supporting Types
### MetadataItem
```
+--------------------+
+--------------------+
```
### DimensionShape (16 bytes, packed struct)
```
+----------------------+
| array_length: u64 | # Size of array in this dimension
| chunk_length: u64 | # Size of chunks in this dimension
+----------------------+
```
### ManifestFileInfo (32 bytes, packed struct)
```
+--------------------+
| size_bytes: u64 | # bytes 16-23
| num_chunk_refs: u32| # bytes 24-27
| (padding) | # bytes 28-31
+--------------------+
```
### ChunkIndices
```
+--------------------+
```
### ChunkIndexRange (8 bytes, packed struct)
```
+--------------------+
+--------------------+
```
### MoveOperation
```
+--------------------+
+--------------------+
```
### ArrayUpdatedChunks
```
+-----------------------------+
| node_id: ObjectId8 | # Which array was modified
```
### DimensionName
```
+--------------------+
```
## Update Types (in Repo)
Update union type for tracking recent repository changes:
| `RepoInitializedUpdate` | Repository created |
| `RepoMigratedUpdate` | Format migration |
| `ConfigChangedUpdate` | Config modified |
| `MetadataChangedUpdate` | Repo metadata changed |
| `TagCreatedUpdate` | New tag |
| `TagDeletedUpdate` | Tag removed |
| `BranchCreatedUpdate` | New branch |
| `BranchDeletedUpdate` | Branch removed |
| `BranchResetUpdate` | Branch pointer moved |
| `NewCommitUpdate` | New snapshot committed |
| `CommitAmendedUpdate` | Snapshot amended |
| `NewDetachedSnapshotUpdate` | Detached snapshot |
| `GCRanUpdate` | Garbage collection ran |
| `ExpirationRanUpdate` | Expiration policy ran |
## Notes
- ObjectIds are content-addressed (typically random or hash-based)
- Snapshot `parent_id` is deprecated in v2.0; use `Repo.snapshots[].parent_offset` instead
- Manifests can be shared across snapshots (copy-on-write)
- Chunks are immutable and deduplicated by id
- Transaction logs enable conflict detection for concurrent writes
- Virtual chunks (`location` field) reference external data (HDF5, NetCDF)
- Inline chunks store small data directly in manifest (no separate file)
- All numeric values are little-endian in flatbuffers
- All timestamps (`flushed_at`, `set_at`, etc.) are microseconds since Unix epoch
- Files are typically zstd-compressed after the 39-byte header