Skip to main content

Crate shardline_index

Crate shardline_index 

Source
Expand description

Metadata index contracts and adapters for Shardline.

This crate models the metadata Shardline needs to reconstruct files from immutable content-addressed objects:

§Quick start

Create an upload intent and follow its state machine, then validate the reconstruction plan of a stored FileRecord:

use shardline_index::{
    FileChunkRecord, FileRecord, StorageRepresentation, UploadIntent, UploadIntentState,
};

// A durable upload intent starts in the `Created` state and can only move
// forward through the persistence boundaries.
let intent = UploadIntent::new(
    "intent-1".to_owned(),
    "chunks/aa/bb/example.xorb".to_owned(),
    "a".repeat(64),
    1024,
);
assert_eq!(intent.state(), UploadIntentState::Created);
assert!(intent.state().can_transition_to(UploadIntentState::Storing));
assert!(!intent.state().can_transition_to(UploadIntentState::Visible));

// A stored file record must describe a contiguous, ordered chunk plan.
let hash = "a".repeat(64);
let record = FileRecord {
    file_id: "assets/logo.png".to_owned(),
    content_hash: hash.clone(),
    total_bytes: 1024,
    chunk_size: 1024,
    storage_repr: StorageRepresentation::FixedChunkV1,
    repository_scope: None,
    chunks: vec![FileChunkRecord {
        hash,
        offset: 0,
        length: 1024,
        range_start: 0,
        range_end: 1,
        packed_start: 0,
        packed_end: 1,
    }],
};
record.validate_reconstruction_plan()?;

Modules§

hub

Structs§

DedupeShardMapping
Durable mapping from a chunk hash to one retained shard object that contains it.
FileChunkRecord
One chunk term stored for a file version record.
FileId
Content-addressed file identifier.
FileReconstruction
File reconstruction recipe.
FileRecord
Durable file-version or latest-file metadata record.
LfsLockRecord
Repository-scoped file lock and its authenticated owner identity.
LocalIndexStore
Local SQLite implementation of IndexStore.
LocalRecordLocator
Opaque local SQLite file-record locator.
LocalRecordStore
Local SQLite implementation of the record-store contract.
MemoryIndexStore
In-memory implementation of crate::IndexStore.
MemoryRecordLocator
Opaque in-memory file-record locator.
MemoryRecordStore
In-memory implementation of crate::RecordStore.
OciManifestSubject
The subject extracted from one digest-verified immutable manifest.
OciObjectKey
Repository-qualified identity of an immutable OCI object.
OciObjectKindParseError
An invalid durable OCI object-kind representation.
OciObjectTombstone
One durable OCI logical-deletion generation eligible for later GC.
OciTagEntry
One mutable OCI tag pointer.
PostgresIndexStore
Postgres-compatible implementation of the asynchronous index-store contract.
PostgresProviderMutation
All durable metadata changes produced by one repository rename or deletion webhook.
PostgresRecordLocator
Opaque Postgres file-record locator.
PostgresRecordStore
Postgres-compatible implementation of the record-store contract.
PostgresRemoteDeletePermit
Identity of one durably admitted remote delete.
PostgresResourceFence
One durable fencing identity that must still match when a provider mutation commits.
PostgresTaskArtifact
One bounded artifact read. None bytes with a known length means oversized evidence; no blob above the requested bound was transferred to the client.
ProviderRepositoryKey
Primary-key identity for provider repository lifecycle state.
ProviderRepositoryState
Durable provider-derived repository lifecycle state.
QuarantineCandidate
Durable quarantine state for one unreachable object.
ReconstructionTerm
A term in a file reconstruction recipe.
RemoteDeletePending
A later mutation cannot safely protect or publish this object yet.
RemoteDeleteReceipt
A signed observation of a successful single-attempt remote delete reply. This is historical evidence, not authority to issue another storage request.
RepoKey
A (provider, owner, repo) repository identity without a revision.
RepositoryRecordScope
Repository identity used to scope record traversal across all revisions.
ResourceLockKey
Canonical identity of a mutable resource protected by a distributed fence.
ResumableCompletionFence
Ownership token for one resumable-session completion attempt.
ResumablePartRange
Validated half-open byte range owned by a staged resumable part.
ResumableSession
Durable metadata for one resumable session.
ResumableSessionAttributes
Validated protocol-specific metadata attached to a resumable session.
ResumableSessionGcInventory
Consistent metadata snapshot used by object-store staging reclamation.
ResumableSessionPart
Immutable staged object selected by a session’s authoritative part map.
RetentionHold
Durable retention hold for one object.
RevisionRecord
A revision registry row.
S3ObjectEntry
One indexed S3 object row for the S3 frontend’s listing index.
StoredObjectId
Content-addressed stored-object identifier.
StoredRecord
Stored record bytes together with adapter locator and modification time.
TreeEntry
A path -> file_id mapping row within a revision tree.
TreeEntryOutcome
Outcome of an upsert that reports whether a prior mapping existed.
TreeKey
A per-(provider, owner, repo, revision) tree namespace identity.
UploadIntent
A durable upload intent record.
UploadIntentConflictError
An idempotency key was reused for a different upload operation.
WebhookDelivery
Durable record of one processed provider webhook delivery.
XetFileIdentity
Maps a Hub/LFS SHA-256 identity to its native Xet file identity.
XetXorbChunk
One verified member’s content identity and serialized/decoded boundaries.
XetXorbDescriptor
Complete immutable descriptor, published only after content verification.

Enums§

CreateLfsLockOutcome
Atomic creation result.
CreateResumableSessionOutcome
Outcome of bounded durable-session creation.
FileRecordInvariantError
File-record reconstruction-plan invariant failure.
FileRecordStorageLayout
File-record reconstruction storage layout.
LocalIndexStoreError
Local metadata-store failure.
MemoryIndexStoreError
Memory index-store failure.
MemoryRecordStoreError
Memory record-store failure.
OciObjectKind
Immutable OCI object category recorded by the logical deletion index.
PostgresMetadataStoreError
Postgres metadata-store failure.
PostgresProviderMutationOutcome
Result of attempting one fenced provider metadata transaction.
PublishResumablePartOutcome
Outcome of transactionally publishing one staged part.
QuarantineCandidateError
Quarantine candidate construction failure.
RemoteDeleteReceiptSettlement
Outcome of reconciling a verified saved reply with current metadata.
ResourceLockDomain
Typed namespace for a mutable resource protected by a distributed fence.
ResumableSessionError
Invalid durable session data.
ResumableSessionProtocol
Protocol owning a durable resumable upload session.
ResumableSessionState
Compatibility name for the canonical reliability resumable-session state. Durable lifecycle states for LFS, OCI, and S3 resumable sessions.
RetentionHoldError
Invalid retention-hold input.
RevisionCreationOutcome
Outcome of insert-only revision creation with an atomic repository limit.
S3ObjectScanStart
Inclusive or exclusive lower bound for a raw S3 key scan.
S3PublishCondition
Metadata precondition applied at the S3 object’s publication transaction.
StorageRepresentation
Storage representation used for a file’s physical data.
TreeRegistrationOutcome
Atomic tree registration result, including repository capacity checks.
UnlockLfsOutcome
Atomic ownership check and deletion result.
UploadIntentState
Compatibility name for the canonical reliability lifecycle state.
WebhookDeliveryError
Invalid processed-webhook-delivery input.

Constants§

MAX_XET_DESCRIPTOR_CHUNKS
Maximum members in a standard Xet Xorb’s bounded lookup descriptor.

Traits§

AsyncIndexStore
Asynchronous metadata index adapter contract.
DedupeStore
Dedupe shard mapping queries.
IndexStore
Metadata index adapter contract.
LfsLockStore
Transactional LFS locking for SQLite and PostgreSQL deployments.
LifecycleStore
Lifecycle metadata: quarantine, retention, webhooks, and provider state.
OciObjectStore
Durable OCI logical-deletion contract.
OciTagStore
Durable OCI tag storage contract.
ReconstructionStore
File reconstruction and stored-object presence queries.
RecordMutation
Write/delete record mutation capability.
RecordStore
Combined read + write record-store contract.
RecordTraversal
Read-only record traversal capability.
Repository
A complete repository providing all storage capabilities.
S3ObjectIndexStore
S3 object listing-index storage contract.
TreeStore
Tree metadata storage contract.
UploadIntentStore
Durable storage for upload intent records.

Functions§

parse_xet_hash_hex
Parses the reordered lowercase hash text used by Xet API paths and persisted Xet metadata.
resumable_state_digest
Computes the canonical StateChronicle digest for the complete materialized resumable-session state, including all persisted part metadata.
s3_prefix_successor
Returns the exclusive UTF-8 lexical upper bound of a prefix. Empty or all-largest-scalar prefixes have no upper bound.
xet_hash_hex_string
Returns the reordered lowercase hash text used by Xet API paths and persisted Xet metadata.

Type Aliases§

IndexStoreFuture
Boxed asynchronous index-store operation.
LocalTaskArtifactPage
Bounded key/body pairs; None retains an oversized row without loading it.
RecordStoreFuture
Boxed asynchronous record-store operation.
XorbId
Backward-compatible Xet alias for StoredObjectId.