Expand description
S3 Lifecycle execution — per-bucket rule evaluation + manager skeleton (v0.6 #37).
AWS S3 Lifecycle attaches a list of rules to a bucket; each rule may request that S3
- Expire an object once its age (or the calendar date) crosses a
threshold (
Expiration { Days | Date }), - Transition an object to a different storage class (
Transition { Days, StorageClass }—STANDARD_IA,GLACIER_IR, …), - Expire noncurrent versions in a versioning-enabled bucket
(
NoncurrentVersionExpiration { NoncurrentDays }).
Until v0.6 #37 the matching PutBucketLifecycleConfiguration /
GetBucketLifecycleConfiguration / DeleteBucketLifecycle handlers
in crates/s4-server/src/service.rs were pure passthroughs (the s3s
framework’s default backend stored them but nothing read the rules).
This module owns the in-memory configuration store + the rule
evaluator that decides, for any single object, whether an action
should fire right now.
§responsibilities (v0.6 #37)
- in-memory
bucket -> LifecycleConfigmap with JSON snapshot round-trip (mirroringversioning.rs/object_lock.rs/inventory.rs’s shape so--lifecycle-state-fileis a one-line addition inmain.rs). - per-bucket action counters (
actions_total) — bumped by the future scanner when an Expiration / Transition / NoncurrentExpiration action is taken, surfaced via Prometheus (s4_lifecycle_actions_total, seemetrics.rs). LifecycleManager::evaluate— given one (bucket, key, age, size, tags) tuple, walk the bucket’s rules in declaration order and return the first matching action. ReturnsNonewhen no rule matches (or when the matching rule isDisabled).evaluate_batch— batched form for the test path: walks a slice of(key, age, size, tags)tuples and returns the (key, action) pairs that should fire. The actual backend invocation (S3.delete_object / metadata rewrite) is the caller’s job.
§scope limitations (v0.6 #37)
- Background scanner is a skeleton only.
main.rs’s--lifecycle-scan-interval-hoursflag spawns a tokio task that logs the bucket list and stamps a “would-have-run” marker; walking the source bucket vialist_objects_v2and actually invokingdelete_object/ metadata rewrite for each evaluated action is deferred to v0.7+. Wiring the scheduler to walk a real bucket end-to-end requires a back-reference from the scheduler intoS4Servicefor thelist_objects_v2walk and that reshuffle is out of scope for this issue. Thecrate::S4Service::run_lifecycle_once_for_testentry covers the in-memory equivalent so the unit + E2E tests exercise the evaluator end-to-end. AbortIncompleteMultipartUploadis parsed and stored on theLifecycleRule(so PutBucketLifecycleConfiguration round-trips the field) but not enforced — multipart abort sweeping is a separate scanner that lives next to the multipart upload manager (v0.7+).expiration_date(calendar date) is supported in the evaluator: a rule withexpiration_datepastnowfires Expiration immediately. Same wire form as AWS S3.- Multi-instance replication. All state is single-instance
in-memory;
--lifecycle-state-file <PATH>provides restart recovery via JSON snapshot, matching the--versioning-state-fileshape. - Object Lock interplay: the evaluator does NOT consult the
ObjectLockManagerdirectly (the evaluator API is object-tags-and-size only); the scanner caller is expected to skip locked objects — see theevaluate_batch_skips_lockedtest for the canonical pattern. Locking always wins over Lifecycle. - Versioning interplay: the evaluator treats noncurrent
versions as a separate input — pass
is_noncurrent = truetoLifecycleManager::evaluate_with_flagsfor noncurrent version expiration matching. The legacyevaluateshorthand defaultsis_noncurrent = false(current version) so existing call sites stay one-liners.
Structs§
- Evaluate
Flags - Flags for
LifecycleManager::evaluate_with_flags. Default is “current-version object, evaluator picksUtc::now()for the date comparison”. Tests overridenowfor determinism. - Lifecycle
Config - Per-bucket lifecycle configuration (ordered list of rules — first match wins, matching AWS S3 semantics).
- Lifecycle
Filter - Per-rule object filter. AWS S3 represents the filter as one of
Prefix,Tag,ObjectSizeGreaterThan,ObjectSizeLessThan, orAnd(= AND of any subset of those predicates). For internal storage we flatten the “And” form into a struct of optional fields plus a vector of (key, value) tags — every present field must match (logical AND). An empty filter (all fieldsNone/ emptytags) matches every object in the bucket. - Lifecycle
Manager - Per-bucket lifecycle configuration manager.
- Lifecycle
Rule - One lifecycle rule. AWS S3’s
LifecycleRuleflattened into the subset the v0.6 #37 evaluator handles.idis the operator-supplied label and makes Get / Put round-trips non-lossy. - Multipart
Upload Candidate - v0.8.3 #69: one in-flight multipart upload the lifecycle scanner
considers for abort. Mirrors the (subset of)
MultipartUploadfields the rule evaluator needs (key, upload_id, initiated).tagsis kept in the shape the existing object-path evaluator uses (Vec<(String, String)>) so a future enhancement that surfaces upload-time tags fromMultipartStateStorecan flow through the same filter check without API churn — AWS S3 itself does not attach tags to in-flight multipart uploads, so for the scanner-driven path the slice is always empty (the filter’s prefix / size predicates still apply viaLifecycleFilter::matches, passing size = 0). - Scan
Report - Per-invocation scanner counters returned by
run_scan_once. Useful for tests, the--lifecycle-scan-interval-hourslog line, and any future/admin/lifecycle/scanintrospection endpoint. Operators see the same numbers via Prometheus (s4_lifecycle_actions_total{action="expire"|"transition"}). - Transition
Rule - A single transition step (object age threshold + target storage class).
daysis days since the object was created. AWS S3 also acceptsDatefor transitions but Lifecycle deployments overwhelmingly useDays; theDateform is omitted here on purpose to keep the evaluator narrow (operators wanting calendar transitions can synthesise a one-shot rule at the cadence of their scanner).
Enums§
- Lifecycle
Action - The action a single rule wants to take right now for a candidate object.
- Lifecycle
Status - Whether a rule is currently being applied. Mirrors AWS S3
ExpirationStatus("Enabled"/"Disabled").
Functions§
- evaluate_
batch - Test-driven scan entry: walks a list of
EvaluateBatchEntrytuples and produces (key, action) pairs for every object that should fire an action right now. The actual backend invocation (S3.delete_object / metadata rewrite) is the caller’s job. Used by both unit tests and the E2E test intests/roundtrip.rs; the future background scanner will reuse the same entry once the bucket-walk is wired through the backend. - run_
scan_ once - Walk every bucket that has a lifecycle configuration attached, list
its objects via
list_objects_v2(continuation-token pagination), and for each object evaluate the rule set + execute the matching Expiration / Transition action. Object-Lock-protected objects are skipped (the Lock always wins over Lifecycle). Versioning chains are intentionally out of scope for v0.7 #45 — see the module-level limitations note.
Type Aliases§
- Evaluate
Batch Entry - One object the evaluator considers in a batch:
(key, object_age, object_size, object_tags). Defined as a type alias soevaluate_batch/crate::S4Service::run_lifecycle_once_for_testdon’t trip clippy’stype-complexitylint, and so callers building the list have a single canonical shape to reach for.