Expand description
Metadata index contracts and adapters for Shardline.
This crate models the metadata Shardline needs to reconstruct files from immutable content-addressed objects:
FileIdandStoredObjectIdwrap protocol hashes with domain-specific meaning.FileReconstructionstores the ordered recipe for one file.IndexStoreandRecordStoredescribe the synchronous metadata contracts used by local and Postgres-backed deployments.AsyncIndexStoreexposes the same boundary for async server workflows.
§Quick start
Create an upload intent and follow its state machine, then validate the
reconstruction plan of a stored FileRecord:
use shardline_index::{
FileChunkRecord, FileRecord, StorageRepresentation, UploadIntent, UploadIntentState,
};
// A durable upload intent starts in the `Created` state and can only move
// forward through the persistence boundaries.
let intent = UploadIntent::new(
"intent-1".to_owned(),
"chunks/aa/bb/example.xorb".to_owned(),
"a".repeat(64),
1024,
);
assert_eq!(intent.state(), UploadIntentState::Created);
assert!(intent.state().can_transition_to(UploadIntentState::Storing));
assert!(!intent.state().can_transition_to(UploadIntentState::Visible));
// A stored file record must describe a contiguous, ordered chunk plan.
let hash = "a".repeat(64);
let record = FileRecord {
file_id: "assets/logo.png".to_owned(),
content_hash: hash.clone(),
total_bytes: 1024,
chunk_size: 1024,
storage_repr: StorageRepresentation::FixedChunkV1,
repository_scope: None,
chunks: vec![FileChunkRecord {
hash,
offset: 0,
length: 1024,
range_start: 0,
range_end: 1,
packed_start: 0,
packed_end: 1,
}],
};
record.validate_reconstruction_plan()?;Modules§
Structs§
- Dedupe
Shard Mapping - Durable mapping from a chunk hash to one retained shard object that contains it.
- File
Chunk Record - One chunk term stored for a file version record.
- FileId
- Content-addressed file identifier.
- File
Reconstruction - File reconstruction recipe.
- File
Record - Durable file-version or latest-file metadata record.
- LfsLock
Record - Repository-scoped file lock and its authenticated owner identity.
- Local
Index Store - Local SQLite implementation of
IndexStore. - Local
Record Locator - Opaque local SQLite file-record locator.
- Local
Record Store - Local SQLite implementation of the record-store contract.
- Memory
Index Store - In-memory implementation of
crate::IndexStore. - Memory
Record Locator - Opaque in-memory file-record locator.
- Memory
Record Store - In-memory implementation of
crate::RecordStore. - OciManifest
Subject - The subject extracted from one digest-verified immutable manifest.
- OciObject
Key - Repository-qualified identity of an immutable OCI object.
- OciObject
Kind Parse Error - An invalid durable OCI object-kind representation.
- OciObject
Tombstone - One durable OCI logical-deletion generation eligible for later GC.
- OciTag
Entry - One mutable OCI tag pointer.
- Postgres
Index Store - Postgres-compatible implementation of the asynchronous index-store contract.
- Postgres
Provider Mutation - All durable metadata changes produced by one repository rename or deletion webhook.
- Postgres
Record Locator - Opaque Postgres file-record locator.
- Postgres
Record Store - Postgres-compatible implementation of the record-store contract.
- Postgres
Remote Delete Permit - Identity of one durably admitted remote delete.
- Postgres
Resource Fence - One durable fencing identity that must still match when a provider mutation commits.
- Postgres
Task Artifact - One bounded artifact read.
Nonebytes with a known length means oversized evidence; no blob above the requested bound was transferred to the client. - Provider
Repository Key - Primary-key identity for provider repository lifecycle state.
- Provider
Repository State - Durable provider-derived repository lifecycle state.
- Quarantine
Candidate - Durable quarantine state for one unreachable object.
- Reconstruction
Term - A term in a file reconstruction recipe.
- Remote
Delete Pending - A later mutation cannot safely protect or publish this object yet.
- Remote
Delete Receipt - A signed observation of a successful single-attempt remote delete reply. This is historical evidence, not authority to issue another storage request.
- RepoKey
- A (provider, owner, repo) repository identity without a revision.
- Repository
Record Scope - Repository identity used to scope record traversal across all revisions.
- Resource
Lock Key - Canonical identity of a mutable resource protected by a distributed fence.
- Resumable
Completion Fence - Ownership token for one resumable-session completion attempt.
- Resumable
Part Range - Validated half-open byte range owned by a staged resumable part.
- Resumable
Session - Durable metadata for one resumable session.
- Resumable
Session Attributes - Validated protocol-specific metadata attached to a resumable session.
- Resumable
Session GcInventory - Consistent metadata snapshot used by object-store staging reclamation.
- Resumable
Session Part - Immutable staged object selected by a session’s authoritative part map.
- Retention
Hold - Durable retention hold for one object.
- Revision
Record - A revision registry row.
- S3Object
Entry - One indexed S3 object row for the S3 frontend’s listing index.
- Stored
Object Id - Content-addressed stored-object identifier.
- Stored
Record - Stored record bytes together with adapter locator and modification time.
- Tree
Entry - A path ->
file_idmapping row within a revision tree. - Tree
Entry Outcome - Outcome of an upsert that reports whether a prior mapping existed.
- TreeKey
- A per-(provider, owner, repo, revision) tree namespace identity.
- Upload
Intent - A durable upload intent record.
- Upload
Intent Conflict Error - An idempotency key was reused for a different upload operation.
- Webhook
Delivery - Durable record of one processed provider webhook delivery.
- XetFile
Identity - Maps a Hub/LFS SHA-256 identity to its native Xet file identity.
- XetXorb
Chunk - One verified member’s content identity and serialized/decoded boundaries.
- XetXorb
Descriptor - Complete immutable descriptor, published only after content verification.
Enums§
- Create
LfsLock Outcome - Atomic creation result.
- Create
Resumable Session Outcome - Outcome of bounded durable-session creation.
- File
Record Invariant Error - File-record reconstruction-plan invariant failure.
- File
Record Storage Layout - File-record reconstruction storage layout.
- Local
Index Store Error - Local metadata-store failure.
- Memory
Index Store Error - Memory index-store failure.
- Memory
Record Store Error - Memory record-store failure.
- OciObject
Kind - Immutable OCI object category recorded by the logical deletion index.
- Postgres
Metadata Store Error - Postgres metadata-store failure.
- Postgres
Provider Mutation Outcome - Result of attempting one fenced provider metadata transaction.
- Publish
Resumable Part Outcome - Outcome of transactionally publishing one staged part.
- Quarantine
Candidate Error - Quarantine candidate construction failure.
- Remote
Delete Receipt Settlement - Outcome of reconciling a verified saved reply with current metadata.
- Resource
Lock Domain - Typed namespace for a mutable resource protected by a distributed fence.
- Resumable
Session Error - Invalid durable session data.
- Resumable
Session Protocol - Protocol owning a durable resumable upload session.
- Resumable
Session State - Compatibility name for the canonical reliability resumable-session state. Durable lifecycle states for LFS, OCI, and S3 resumable sessions.
- Retention
Hold Error - Invalid retention-hold input.
- Revision
Creation Outcome - Outcome of insert-only revision creation with an atomic repository limit.
- S3Object
Scan Start - Inclusive or exclusive lower bound for a raw S3 key scan.
- S3Publish
Condition - Metadata precondition applied at the S3 object’s publication transaction.
- Storage
Representation - Storage representation used for a file’s physical data.
- Tree
Registration Outcome - Atomic tree registration result, including repository capacity checks.
- Unlock
LfsOutcome - Atomic ownership check and deletion result.
- Upload
Intent State - Compatibility name for the canonical reliability lifecycle state.
- Webhook
Delivery Error - Invalid processed-webhook-delivery input.
Constants§
- MAX_
XET_ DESCRIPTOR_ CHUNKS - Maximum members in a standard Xet Xorb’s bounded lookup descriptor.
Traits§
- Async
Index Store - Asynchronous metadata index adapter contract.
- Dedupe
Store - Dedupe shard mapping queries.
- Index
Store - Metadata index adapter contract.
- LfsLock
Store - Transactional LFS locking for SQLite and PostgreSQL deployments.
- Lifecycle
Store - Lifecycle metadata: quarantine, retention, webhooks, and provider state.
- OciObject
Store - Durable OCI logical-deletion contract.
- OciTag
Store - Durable OCI tag storage contract.
- Reconstruction
Store - File reconstruction and stored-object presence queries.
- Record
Mutation - Write/delete record mutation capability.
- Record
Store - Combined read + write record-store contract.
- Record
Traversal - Read-only record traversal capability.
- Repository
- A complete repository providing all storage capabilities.
- S3Object
Index Store - S3 object listing-index storage contract.
- Tree
Store - Tree metadata storage contract.
- Upload
Intent Store - Durable storage for upload intent records.
Functions§
- parse_
xet_ hash_ hex - Parses the reordered lowercase hash text used by Xet API paths and persisted Xet metadata.
- resumable_
state_ digest - Computes the canonical StateChronicle digest for the complete materialized resumable-session state, including all persisted part metadata.
- s3_
prefix_ successor - Returns the exclusive UTF-8 lexical upper bound of a prefix. Empty or all-largest-scalar prefixes have no upper bound.
- xet_
hash_ hex_ string - Returns the reordered lowercase hash text used by Xet API paths and persisted Xet metadata.
Type Aliases§
- Index
Store Future - Boxed asynchronous index-store operation.
- Local
Task Artifact Page - Bounded key/body pairs;
Noneretains an oversized row without loading it. - Record
Store Future - Boxed asynchronous record-store operation.
- XorbId
- Backward-compatible Xet alias for
StoredObjectId.