Crate shardline_server

Expand description

HTTP server runtime and operator workflows for Shardline.

This crate owns the deployable server boundary: HTTP routing, configuration, local and Postgres metadata backends, provider token issuance, integrity checks, repair flows, garbage collection, backup manifests, and storage migration.

Most embedders only need ServerConfig and serve. Operational tooling can use the same library entry points exposed by the shardline CLI, such as run_fsck, run_lifecycle_repair, run_gc, and run_storage_migration.

§Example

use shardline_server::{ServerConfig, serve};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ServerConfig::from_env()?;
    serve(config).await?;
    Ok(())
}

Re-exports§

pub use app::ProtocolMetrics;
pub use app::AppState;
pub use app::MAX_PROVIDER_NAME_BYTES;
pub use app::MAX_PROVIDER_SUBJECT_BYTES;
pub use app::MAX_PROVIDER_TOKEN_REQUEST_BODY_BYTES;
pub use app::MAX_PROVIDER_WEBHOOK_BODY_BYTES;
pub use app::acquire_chunk_transfer_permit;
pub use app::full_byte_stream_response;
pub use download_stream::STREAM_READ_BUFFER_BYTES;
pub use download_stream::ServerByteStream;
pub use app::serve;
pub use app::serve_with_listener;

Modules§

app
download_stream
metrics
test_fixtures
test_invariant_error

Structs§

BackupManifestReport: Adapter-neutral backup manifest summary.
BatchReconstructionResponse: Batch reconstruction response for multiple file identifiers.
BenchmarkBackend: Public benchmark-facing backend wrapper that resolves the active metadata and object adapters without exposing the private server runtime enum.
ConfigCheckReport: Result of validating a Shardline runtime configuration.
DatabaseMigration: One Shardline schema migration.
DatabaseMigrationOptions: Database-migration runtime options.
DatabaseMigrationReport: Database-migration execution report.
DatabaseMigrationStatusEntry: One migration row in the status report.
FileReconstructionResponse: File reconstruction response.
FileReconstructionV2Response: V2 file reconstruction response.
FuzzBazelHttpFrontendSummary: Summary of Bazel HTTP cache frontend validation used by fuzz targets.
FuzzLfsFrontendSummary: Summary of Git LFS frontend validation used by fuzz targets.
FuzzLifecycleRepairSummary: Summary of lifecycle-repair classifications used by fuzz targets.
FuzzOciFrontendSummary: Summary of OCI frontend parsing and validation used by fuzz targets.
FuzzProtocolFrontendSummary: Summary of protocol-frontend parser and key validation used by fuzz targets.
FuzzReconstructionResponseSummary: Summary of reconstruction response shape used by fuzz targets.
FuzzRetainedShardSummary: Summary of chunk hashes retained by a shard payload.
FuzzValidatedXorbSummary: Summary of a normalized and validated xorb payload used by fuzz targets.
GitLfsAuthenticateResponse: Git LFS authenticate response carrying Xet custom-transfer bootstrap headers.
HealthResponse: Health response returned by the HTTP server.
LifecycleRepairOptions: Lifecycle-repair execution options.
LifecycleRepairReport: Lifecycle-repair report.
LocalBackend: Local filesystem backend for file chunk storage and reconstruction metadata.
LocalGcDiagnostics: Detailed GC diagnostics intended for operators and automation.
LocalGcOptions: Local filesystem garbage-collection execution options.
LocalGcReport: Local filesystem garbage-collection report.
ProviderTokenIssueRequest: Provider-backed CAS token issuance request.
ProviderTokenIssueResponse: Provider-backed CAS token issuance response.
ProviderWebhookResponse: Provider webhook handling response.
ReadyResponse: Readiness response returned by the HTTP server.
ReconstructionCacheBenchReport: Benchmarks one cold reconstruction load followed by one hot cache hit.
ReconstructionCacheService: Runtime reconstruction-cache service.
ServerConfig: Public server configuration.
ServerFrontendParseError: Invalid server frontend token.
ServerRoleParseError: Invalid server role token.
ServerStatsResponse: Storage stats response.
ShardMetadataLimits: Bounded-parser limits for native Xet shard metadata. Bounded-parser limits for native Xet shard metadata.
StorageMigrationOptions: Storage-migration runtime options.
StorageMigrationReport: Storage-migration summary.
TransferLimiter: Weighted transfer concurrency limiter based on chunk-equivalent cost.
XetCasTokenResponse: Xet CAS access-token response consumed by reference clients.
XorbUploadResponse: Upload result for a single xorb.

Enums§

AuthProviderKind: Authentication provider selection.
BazelCacheKind
DatabaseMigrationCommand: Requested database-migration action.
DatabaseMigrationError: Database-migration failure.
FsckIssueDetail: Integrity issue detail.
FsckIssueKind: Integrity issue kinds for the checker.
FsckReconstructionPlanDetail: Reconstruction-plan detail for fsck issues.
IndexRebuildIssueDetail: Index-rebuild issue detail.
IndexRebuildReconstructionPlanDetail: Index-rebuild reconstruction-plan issue detail.
ObjectStorageAdapter: Immutable object-storage adapter selection.
ProviderRepositoryStateTimestampField: Provider repository state timestamp field.
ServerBackend
ServerConfigError: Server configuration loading failure.
ServerError: Server runtime failure.
ServerFrontend: Runtime protocol frontend selection.
ServerObjectStore: Unified object-store backend that delegates to local, S3, or blackhole storage.
ServerRole: Runtime server role selection.
StorageMigrationEndpoint: Object-storage endpoint used by storage migration.

Constants§

DEFAULT_LOCAL_GC_RETENTION_SECONDS: Default retention window for new local quarantine candidates.
DEFAULT_WEBHOOK_DELIVERY_RETENTION_SECONDS: Default retention for processed webhook delivery claims before repair prunes them.

Functions§

apply_database_migrations: Applies pending Shardline migrations to an existing Postgres pool.
bazel_cache_object_key: Returns the storage object key for a Bazel cache entry.
benchmark_memory_reconstruction_cache: Measures cold and hot reconstruction-cache behavior over the local backend path.
bundled_database_migrations: Returns the bundled Shardline migration list in application order.
chunk_hash
clear_repository_reference_probe_filter
decode_serialized_xorb_chunks: Decodes the packed chunk stream of a previously validated serialized xorb.
fuzz_bazel_http_frontend_summary: Validates Bazel HTTP cache key derivation for fuzzing.
fuzz_lfs_frontend_summary: Validates Git LFS object identity and key determinism for fuzzing.
fuzz_lifecycle_repair_summary: Classifies lifecycle-repair decisions for fuzzed metadata states.
fuzz_normalize_and_validate_xorb: Normalizes a raw uploaded Xorb payload and validates the normalized result.
fuzz_oci_frontend_summary: Validates OCI path parsing and identity derivation for fuzzing.
fuzz_protocol_frontend_summary: Parses protocol frontend selectors and validates protocol-specific object keys.
fuzz_reconstruction_response_summary: Builds a reconstruction response and checks protocol-shape invariants for fuzzing.
fuzz_retained_shard_chunk_hashes: Parses a serialized shard with bounded metadata limits and reports the retained dedupe chunk hashes.
ingest_without_storage_with_parallelism: Runs the upload ingestion pipeline without storage using explicit chunk parallelism.
lfs_object_key: Returns the storage object key for an LFS object.
lock_repository_reference_probe_test
oci_blob_key: Errors
oci_manifest_key: Errors
oci_manifest_media_type_key: Errors
repository_reference_probe_count
reset_repository_reference_probe_count_for_hash
run_config_check: Validates the selected runtime configuration and backend reachability.
run_database_migration: Executes a Shardline database-migration command against Postgres.
run_fsck: Runs integrity checks against the configured metadata backend and local chunk storage.
run_gc: Runs garbage collection against the configured metadata backend and local chunk storage.
run_gc_diagnostics: Runs garbage collection and returns operator diagnostics.
run_index_rebuild: Rebuilds latest-record state against the configured metadata backend.
run_lifecycle_repair: Repairs stale lifecycle metadata against the configured metadata backend.
run_local_fsck: Runs local filesystem integrity checks over Shardline metadata and chunk storage.
run_local_index_rebuild: Rebuilds local latest-record state from immutable version records.
run_local_lifecycle_repair: Repairs stale lifecycle metadata for one local storage root.
run_storage_migration: Copies immutable payload objects between object-storage adapters.
shared_sha256_object_key
try_for_each_serialized_xorb_chunk: Decodes the packed chunk stream of a previously validated serialized xorb and passes each decoded chunk to visitor before decoding the next chunk.
validate_serialized_xorb: Validates a serialized Xet xorb against the requested content hash and extracts trusted chunk metadata.
write_backup_manifest: Writes an adapter-neutral backup manifest for the configured deployment.

Type Aliases§

LocalFsckIssue: Backward-compatible local fsck issue alias.
LocalFsckIssueKind: Backward-compatible local fsck issue-kind alias.
LocalFsckReport: Backward-compatible local fsck report alias.
LocalIndexRebuildIssue: Backward-compatible local index-rebuild issue alias.
LocalIndexRebuildIssueKind: Backward-compatible local index-rebuild issue-kind alias.
LocalIndexRebuildReport: Backward-compatible local index-rebuild report alias.

Crate shardline_server

Crate shardline_server Copy item path

§Example

Re-exports§

Modules§

Structs§

Enums§

Constants§

Functions§

Type Aliases§

Crate shardline_server