Skip to main content

Crate shardline_s3_adapter

Crate shardline_s3_adapter 

Source
Expand description

S3-compatible protocol adapter for the Shardline server ecosystem.

This crate provides the S3 frontend’s protocol primitives:

  • bucket ({owner}.{name}) encoding and decoding and the storage object-key layout (protocols/s3/{scope_namespace}/{key}) — S3ObjectKey;
  • the S3 XML wire models and error envelope — ListBucketResult and S3Error;
  • the SigV4 → bearer-token auth bridge and bucket-scope binding — AuthScheme and CredentialScope;
  • the query-param sub-resource dispatch set, ETag formatting, and byte-range parsing — S3SubResource and parse_s3_range.

§Quick start

use shardline_s3_adapter::{decode_bucket, encode_bucket, s3_object_key};

// A bucket is the single dotted segment `{owner}.{name}` of the token scope.
let bucket = encode_bucket("acme", "models");
assert_eq!(
    decode_bucket(&bucket)?,
    ("acme".to_owned(), "models".to_owned())
);

// Object keys live under the `protocols/s3/{scope_namespace}/` prefix.
let key = s3_object_key("global", "data/model.pt")?;
assert_eq!(key.as_str(), "protocols/s3/global/data/model.pt");

// Errors serialize to the S3 XML envelope with the correct status.
use axum::response::IntoResponse;
let response = shardline_s3_adapter::S3Error::no_such_key("data/model.pt")
    .into_response();
assert_eq!(response.status(), 404);

Structs§

CompleteMultipartUploadResult
The CompleteMultipartUpload response envelope.
CompleteParts
The parsed CompleteMultipartUpload request body.
Contents
One listed object entry.
CopyObjectResult
The CopyObject response envelope.
CopySource
A parsed x-amz-copy-source value.
CredentialScope
The five /-separated segments of a SigV4 Credential scope.
Delimiter
A validated ListObjectsV2 grouping delimiter (exactly one character).
EntityTag
An entity tag with its comparison strength preserved.
HeadObjectHeaders
Response headers for HeadObject.
InitiateMultipartUploadResult
The InitiateMultipartUpload response envelope.
InvalidS3HeaderValue
A malformed S3 conditional-header (If-Match / If-None-Match) or x-amz-copy-source value.
ListBucketResult
The ListObjectsV2 response envelope.
ListBucketResultV1
The ListObjects (v1) response envelope.
ListBucketsResult
The ListAllMyBucketsResult (service-level GET /) response envelope.
ListObjectsV1Params
Parsed ListObjects (v1) request parameters.
ListObjectsV2Params
Parsed ListObjectsV2 request parameters.
ListObjectsV2ResponseContext
Request values echoed by a ListObjectsV2 response. Filtering and cursors use raw values; encoding applies only when serializing the response.
ListPage
One grouped page of a ListObjectsV2 listing.
MultipartPart
One stored part of a multipart upload session.
MultipartUploadSession
A disk-persisted S3 multipart upload session.
PartQuotaLimits
Aggregate and per-session limits for one staged part validation.
PutObjectResponseHeaders
Response headers for a successful PutObject.
QueryCredential
The presigned-URL X-Amz-Credential query parameter.
S3Error
A ready-to-send S3 error envelope.
S3ErrorBody
The S3 error envelope body (<Error><Code>…</Code>…).
S3ObjectKey
A validated client-facing S3 object key.
S3SessionPartFileLock
Cross-process advisory lock protecting one multipart session’s part files.
S3UploadSessionLock
A held session-store advisory file lock.
SigV4ChunkVerifier
Checks the selected streaming mode against verified request headers. Buffers at most one 1 KiB framing line, independent of chunk or upload size. Signed trailer mode additionally checks declared checksums and their signature. Unsigned trailer mode checks checksums without a body MAC and requires TLS.
SigV4Credential
Signing material resolved by the server, never from untrusted request data.
SigV4PostPolicyAuth
Authentication fields supplied by an AWS Signature Version 4 POST form.
VerifiedSigV4
A verified request commitment, not proof that its body was received intact.
VerifiedSigV4PostPolicy
A verified SigV4 browser POST policy and its permitted form conditions.
WireConditionalHeader
A validated S3 precondition retaining opaque entity-tag bytes.

Enums§

AuthScheme
The typed Authorization header scheme.
BucketDecodeError
S3 bucket decode failure.
ConditionalHeader
A parsed S3 conditional request header.
EntityTagSet
The entity-tag set of an S3 conditional header.
S3ErrorClass
Coarse server-error classification used by the S3 frontend to translate a server-layer failure into an S3 error envelope.
S3KeyError
S3 object key construction failure.
S3SessionError
S3 multipart upload session persistence failure.
S3SubResource
A recognized S3 sub-resource (the ?subresource query dispatch set).

Constants§

MAX_LIST_KEYS
The S3 ListObjectsV2 maximum page size.
MAX_S3_DELETE_KEYS
The maximum number of keys in a single DeleteObjects request.
MAX_S3_PART_NUMBER
The S3 protocol maximum part number.

Traits§

S3ErrorClassify
Classifies a server-layer error into an S3ErrorClass.

Functions§

acquire_session_part_lock
Returns the legacy per-session part-write lock for an upload id, creating it on first use.
acquire_session_part_lock_for_root
Returns the per-session part-write lock scoped to one local deployment root, creating it on first use.
classify
Classifies the query parameters of an S3 request into the recognized sub-resources, in query order.
count_active_sessions
Counts the currently active (unexpired) sessions.
create_session
Creates a new multipart upload session and returns its upload id.
decode_bucket
Decodes a {owner}.{name} S3 bucket into its parts.
decode_continuation_token
Decodes an opaque base64 continuation token back to the raw cursor key.
delete_session
Deletes a session and all of its part files.
delete_session_locked
The lock-free counterpart of delete_session: the caller must hold the session lock (lock_upload_sessions) for the duration of the mutation.
encode_bucket
Encodes a repository owner and name into the {owner}.{name} S3 bucket.
encode_continuation_token
Encodes a raw cursor key as an opaque base64 continuation token.
etag_header
Formats a content hash as the S3-style quoted ETag header value.
extract_access_key
Extracts the S3 access key — the Shardline bearer token — from request headers.
extract_access_key_from_query
Extracts the S3 access key from the query string (presigned-URL form).
format_iso8601
Formats unix seconds as an S3 LastModified ISO-8601 UTC string (2026-08-13T09:51:00Z).
group_page
Groups a sorted, prefix-filtered batch of index rows into one page.
is_expired
Returns whether a session has expired against the TTL.
list_bucket_sessions
Lists live sessions for a bucket and namespace without opening part payloads.
lock_session_parts
Acquires the cross-process advisory lock for one session’s part files.
lock_upload_sessions
Acquires the upload-root advisory file lock.
new_upload_id
Generates a new random upload id (32 lowercase hex chars).
parse_complete_multipart_parts
Parses ordered parts and their echoed ETags from completion XML.
parse_content_md5
Parses an optional Content-MD5 header without allocating request-sized data.
parse_copy_source
Parses an S3 x-amz-copy-source header value (/{bucket}/{key}, with the leading slash optional) into its typed bucket + key parts.
parse_delete_object_keys
Parses the <Key> values from a DeleteObjects request body.
parse_list_objects_url_encoding
Returns whether ListObjects response key-like fields should be URL encoded. The existing v1/v2 parameter structs keep their public literal shape.
parse_list_objects_v1_params
Parses the ListObjects (v1) query parameters from a decoded query map.
parse_list_objects_v2_params
Parses the ListObjectsV2 query parameters from a decoded query map.
parse_s3_range
Parses an S3 Range header against a resource length.
parse_subresource
Parses one query pair into its typed sub-resource, if the name is a recognized sub-resource.
part_file_path
The part-{n} file path for one part of a session.
read_conditional_headers
Reads the S3 conditional headers (If-Match / If-None-Match) from a request’s header map.
read_session
Reads a session, treating missing or expired sessions as NotFound.
read_session_locked
Reads a session while the caller owns lock_upload_sessions.
read_wire_conditional_headers
Parses all repeated conditional fields, preserving If-Match precedence.
repair_session_evidence
Explicitly establishes missing reliability baselines for one legacy session after operator approval.
require_s3_bucket_binding
Requires the bucket to bind exactly to the token’s repository scope.
s3_object_key
Builds the storage object key for an S3 object.
session_dir
The per-session directory (<root>/s3-uploads/<upload_id>).
session_metadata_path
The session metadata path (session.json inside the session directory).
store_part
Records a stored part’s size in the session metadata.
store_part_locked
The lock-free counterpart of store_part: the caller must hold the session lock (lock_upload_sessions) for the duration of the mutation so a concurrent sweep cannot remove the session directory mid-write.
store_versioned_part_locked
Publishes an already persisted immutable content-addressed part.
store_versioned_part_with_checksum_locked
Publishes an immutable part with its negotiated checksum in the same integrity-protected session snapshot.
stored_part_file_path
Resolves a recorded part without permitting path traversal or cross-part references.
sweep_expired_sessions
Removes every expired (or unreadable) session directory, returning the number of sessions removed.
upload_dir
The upload root directory (<root>/s3-uploads).
validate_part_number
Validates an S3 part number against the protocol bounds.
validate_part_quota_for_session_locked
Validates quotas using a session already loaded under the caller’s session lock. This avoids re-reading the same session metadata on request paths that already needed it for ownership and body-size validation.
validate_part_quota_locked
Validates a part against the per-session and aggregate byte quotas and the global active-part-file cap WITHOUT persisting anything.
validate_upload_id
Validates that an upload id is a safe path component (hex or -).
verify_sigv4
Verifies header or query authentication with a configured S3 region.
verify_sigv4_post_policy
Verifies an AWS Signature Version 4 browser POST policy.
versioned_part_file_name
Builds the immutable filename for a BLAKE3-addressed part.
visit_bucket_sessions
Visits verified live session metadata one session at a time, without part I/O. The synchronous visitor runs under the upload-session lock and must not do blocking work or acquire that lock again.
xml_escape
Escapes XML text, omitting characters XML 1.0 cannot represent.

Type Aliases§

QueryMap
An ordered query-parameter list.