# Saddle Log File Plan
## Status and scope
This document defines the Observability-owned file layout that follows the V1
structured Log and Trace model. It deliberately does not define a log storage
platform, remote transport, Metrics, alerting, or dashboards.
The current implementation writes every record to one injected `Write`
destination. The target design replaces that single physical stream with a
small, fixed set of streams whose routing is owned by Observability. Applications
must not choose a file per event, service, tenant, trace, or module.
File mode uses bounded asynchronous backpressure. It never performs file I/O on
an application task, but it also never treats a dropped Access, Trace, Event, or
System record as a successful emission. Sampling and drop-on-full are not
enterprise defaults.
## Fixed streams
Each record is written to exactly one stream. All streams use newline-delimited
JSON and retain the existing trace and span correlation fields.
| Stream | Active file | Records | Collection purpose |
|---|---|---|---|
| Access | `saddle-access.jsonl` | `framework.call.*` with `kind=external_request` | Request rate, latency, outcome, and entry-point analysis |
| Trace | `saddle-trace.jsonl` | `framework.call.*` with `kind=service`, `database`, or `transaction` | In-process call graph and dependency analysis |
| Event | `saddle-event.jsonl` | `domain.event` | Domain-event collection and business correlation |
| System | `saddle-system.jsonl` | Observability/runtime lifecycle, degradation, and internal diagnostic records | Runtime health and collection-pipeline diagnosis |
Four is a fixed upper bound for V1, not an application setting. In particular:
- errors stay in the stream of the operation that produced them;
- start and finish records for one call stay in the same stream;
- stream assignment uses typed record metadata, never string-prefix matching at
the file writer;
- no stream is split by log level, service, database, tenant, or trace ID;
- records are never duplicated into an aggregate "all" file.
The System stream is intentionally reserved even if an initial implementation
emits few records to it. Output failure cannot reliably be reported back into
the failed stream, so a short, rate-limited emergency diagnostic may also be
written to stderr. Business payloads must never be copied to that fallback.
## Application configuration
File mode requires one application-supplied value:
```text
log_directory
```
The application chooses the writable root because only the deployment knows its
volume layout. Observability owns all other behavior:
- the four file names and routing rules;
- JSON Lines encoding and schema version;
- file creation and append behavior;
- rotation thresholds and retention;
- flushing, shutdown, and failure reporting.
Application identity is already carried by `CallContext` and must not be
duplicated in logger configuration. Queue sizes, per-stream file names, rotation
thresholds, and retention counts are not application-facing knobs in V1.
The public shape should therefore be a file-specific initializer taking a path.
Stdout may remain an explicit deployment option, but it uses the same typed
streams, bounded asynchronous admission, failure wakeup, and sticky health state
as file mode. Its physical output is multiplexed and every record carries its
logical stream; it is not a weaker synchronous fallback.
Arbitrary writer injection is not a supported deployment contract. A private
test-only writer factory exercises the same queue, admission, health, flush, and
shutdown state machine as production writers. It cannot be selected by business
applications or preserve the current drop-new behavior.
## Directory ownership and recovery
The configured directory is an exclusive Observability resource, not a shared
append target. File-mode initialization must acquire and hold an operating-system
advisory exclusive lock on a well-known lock file in that directory for the
entire observer lifetime. Initialization fails with a stable
`directory-in-use` error if another process owns the lock, and fails as
unsupported if the filesystem cannot provide the required lock semantics.
Process termination releases the operating-system lock; PID inspection or
blind deletion of a stale-looking lock file is never used as an ownership test.
Rotation uses same-directory, no-replace filesystem operations. A rotated name
contains the stream, timestamp, and a newly generated collision-resistant
suffix. The operation must fail or choose another name if any destination entry
already exists; it must never overwrite an existing file. The active name is
published only after the old contents have an immutable rotated name.
After acquiring the directory lock, startup reconciles an interrupted rotation
before accepting records:
- an intact active file is reopened in append mode;
- a completed immutable rotated file is retained as-is;
- an unambiguously staged old active file is promoted to a fresh, non-colliding
rotated name;
- a missing active file is created with create-new semantics;
- ambiguous aliases, unsupported file types, or evidence that two files may
claim the same generation cause initialization to fail rather than delete or
overwrite data.
Retention only considers immutable files matching Saddle's exact generated-name
grammar. It never follows symlinks and never removes the active file, lock file,
unknown entries, or ambiguous recovery artifacts.
## Autonomous file lifecycle
Initial defaults:
- rotate an active file when it reaches 128 MiB;
- retain at most 8 rotated files per stream;
- also remove rotated files older than 7 days;
- apply both limits, deleting the oldest eligible files first;
- generate collision-resistant rotated names and use no-replace publication;
- reopen active files in append mode after process restart;
- create the configured directory if it does not exist.
The maximum retained rotated data is approximately 4 GiB across four streams,
plus active files. These are framework defaults and can evolve with a versioned
Observability release rather than becoming per-application tuning.
Initialization is atomic from the caller's perspective: if any required stream
cannot be opened, file-mode initialization fails and does not silently fall back
to a combined file or stdout. After initialization, healthy writers continue
draining already accepted records if one stream fails, but any unrecoverable
stream, rotation, retention, sync, or disk-capacity failure becomes a sticky
Observer health failure. It is visible immediately through admission and health
interfaces, not deferred until final shutdown. New external requests are
rejected once the Observer is unhealthy.
Each stream needs an independent writer path so a slow or failed destination
does not create head-of-line blocking across unrelated record classes.
## Bounded asynchronous backpressure
Each of the four streams owns a bounded queue and independent capacity permits.
Acquiring capacity is asynchronous; it may suspend the calling task but never
runs or blocks on file I/O. A successful admission means only that the record's
memory capacity is reserved, not that the record has reached the kernel or
durable storage.
Framework calls reserve the complete balanced pair before beginning work:
- starting an external call asynchronously reserves two Access permits for its
start and finish records;
- starting a Service, Database, or Transaction child call asynchronously
reserves two Trace permits;
- the active-call guard owns the finish permit, allowing success, failure,
cancellation, and `Drop` abandonment to enqueue the finish record without
waiting;
- recording a domain event asynchronously acquires one Event permit;
- System producers acquire System capacity before committing the lifecycle
action represented by the record.
This reservation bounds both queued records and in-flight calls while preserving
balanced traces. No successful framework admission can later lose its finish
record merely because the queue became full.
The System queue withholds three permits from general producers for
`shutdown.started`, `admission.stopped`, and `shutdown.finalizing`. The Observer
owns these control permits from initialization. Starting shutdown first consumes
the three reserved permits, materializes the first two bounded control records,
and retains the final permit before closing general capacity acquisition. Normal
System traffic cannot consume the control reserve, so a full System queue cannot
prevent shutdown from starting.
`shutdown.finalizing` is materialized from the retained permit after request
drain and all non-System component results are known. Its fields describe only
those known results and explicitly identify the remaining
`observability_output_finalize` stage. It is delivered before the System
writer's final durability barrier and join, so it never claims that Observer or
application shutdown has completed successfully. Final success is represented
only by the shutdown return value and may be confirmed by an out-of-process
supervisor; no record written through the writer being closed can prove its own
successful final synchronization and join.
If a writer terminates or enters sticky failure, all capacity waiters are woken
with a stable error. They never wait indefinitely. The external-entry adapter
uses that result to stop admission and reject new requests. After materializing
its control records, shutdown stops new capacity acquisition, wakes unadmitted
waiters with a shutdown error, and allows already admitted requests to converge
using their reserved permits. It then drains all accepted records, applies the
durability barrier, and joins the writers.
If a writer has already failed, reserved call and System control records still
materialize in bounded memory and release their permits deterministically. The
sticky error aggregates every undeliverable stream and stage into shutdown's
result; shutdown never waits for failed capacity to recover. The framework never
reports observability success for an undeliverable record.
All supported outputs share this contract. Sampling, priority dropping, silent
fallback, and a synchronous drop-new constructor are not part of the public
Observer API.
## Durability and visible failure
File mode defines these states:
- **admitted**: bounded memory is reserved and the record can be enqueued;
- **written**: the complete JSON line has been accepted by the file write
operation, normally into the operating-system page cache;
- **durable**: all earlier records for the stream are covered by a successful
file data synchronization barrier.
The first write after a successful synchronization starts a one-second deadline
owned by that stream's writer. The deadline is independent of later traffic:
becoming idle does not cancel it, and the writer performs the barrier no later
than the deadline even if no additional command arrives. Reaching 4 MiB since
the last barrier triggers it earlier. A barrier failure becomes sticky health
immediately and wakes admission waiters.
`flush` is a cross-stream barrier: it waits for all records admitted before the
call, flushes userspace buffers, performs file data synchronization on every
stream, and succeeds only if they are durable. Order between different streams
is not promised. `shutdown` stops admission after reserving its System control
records, drains admitted records, performs the same durability barrier, and
joins every writer.
A sudden process crash may lose records still admitted or queued but not
written. A host or power failure may additionally lose written records since the
last successful synchronization barrier. Therefore the bounded loss window is
at most each stream's bounded admitted backlog plus the smaller of one second or
4 MiB of already written data. A successful `flush` or `shutdown` closes that
window for all records preceding its barrier, subject to the storage device's
truthful synchronization guarantees.
Serialization, write, newline, rotation, synchronization, disk-full, and
retention failures are sticky and identify the stream and output stage.
Retention deletion failure is not ignored: it marks the Observer unhealthy,
wakes admission waiters, rejects new external requests, and remains visible from
health, `flush`, and `shutdown`. A short rate-limited stderr diagnostic without
business payload may announce the failure, but stderr is never a record
fallback.
## Stable collection contract
Collectors may rely on:
- stable active file names;
- one JSON record per line;
- exactly one logical stream per record;
- `timestamp_ms`, `level`, `event`, `trace_id`, `span_id`, and
`parent_span_id` retaining their current meanings;
- call outcome and duration remaining on call-finish records;
- rotated files being immutable.
Collectors must discover rotated files by the stable stream prefix rather than
parsing an application-selected filename.
Future schema changes must add an explicit schema version before changing field
meaning or removing fields. File routing must remain based on record type and
call kind so event-name evolution does not silently move data between streams.
## Implementation stages
1. Introduce an internal typed stream classification and test the routing table.
2. Implement directory locking, interrupted-rotation recovery, and the four-file
no-replace rotating writer behind a single-directory initializer.
3. Implement per-stream asynchronous capacity reservation, balanced call
permits, failure wakeup, health propagation, and shutdown convergence.
4. Add deterministic durability-barrier, sticky-failure, and bounded-backpressure
tests, including one write followed by complete inactivity past the sync
deadline and a synchronization failure at that deadline.
5. Add integration tests for same-directory dual initialization, restart name
collision, interrupted rotation recovery, immutable no-overwrite publication,
retention failure, disk-full/write failure, System queue saturation when
shutdown starts, failed-System-writer shutdown aggregation, and a successfully
written `shutdown.finalizing` record followed by final durability-barrier
failure. The latter must keep the phase name and pending-stage field truthful
while shutdown returns the final synchronization error. Also verify trace
correlation across Access and Trace files.
6. Document collector glob patterns and migration from the current stdout
stream.
## Final ingress deadline diagnostic
`ingress_deadline_bound` 是入口预算最终绑定后、业务之前的单次事实；`ingress_deadline_unix_ms` 为实际 Runtime 权威 UTC 毫秒期限。记录复用现有诊断文件和输出状态，入队 `Enqueued` 不是写入或持久化证明，不新增测试专用通道。
单根路径 `schema_version=2` 的 `context` 读取原请求执行视图，兼容路径 `schema_version=1` 读取已建立的 Call/Event 投影。兼容投影的 request 显示带安全元数据，脱敏或截断的 request 不允许产生可关联入口事实，返回 `EncodingFailed` 并保留既有隐私边界。不同 schema 的字段层级不能混读。采集者先按 schema 解码完整请求身份，再核入站原件及出站 Invoke 的请求/父子调用关联，同 trace 的不同请求分别匹配。
消费者冻结并回读实际原件，要求每个已成功绑定请求有唯一入口事实，再把其期限与该请求每次实际 Invoke 的期限逐一比较。缺失、重复冲突、错请求或被改写的出站期限都不能通过；不能从发送时刻范围、源码或外呼自比较补证。输出故障不会更改业务及预算，也不保证失败标记自身必达。
按现有字段封套读取身份：schema 2 的请求值位于 `context.request.value`；schema 1 的安全显示值位于 `context.request.value.value`，并要求其 `redacted` 与 `truncated` 均为 false。trace/rpc/span 也必须按该 schema 的 present 状态读取，不能用缺省空值匹配。原件文件读回和现有 writer snapshot 的失败状态共同决定观察是否成立；仅返回入队状态或仅看到其中一个 Invoke 都不足以证明完整传播。
## Process diagnostic output final record
The existing Facade process exit emits one independent stderr JSON line from
Runtime's actual diagnostic exit (including unhealthy exits). It does not submit
this record to the closed diagnostic logger. The schema is
`diagnostic-output-final/1`, event `diagnostic_output_final`, and `process_id` is
the emitting OS PID. The five wrapper fields are `schema`, `event`, `process_id`,
`shutdown`, and `snapshot`. `shutdown` is `pending`, `finished`, or
`worker_panicked`; it preserves the actual shutdown result.
The complete snapshot contains these ten fields:
| Field | Meaning |
| --- | --- |
| `initialized` | Output initialization completed. |
| `first_failure` | Null or the first failure object: `stage`, `kind`, `os_code` (nullable). |
| `enqueued` | Records accepted by the output queue. |
| `written` | Records whose write completed; no durability guarantee. |
| `dropped` | Records dropped by the existing output policy. |
| `output_failed` | Output failure counter. |
| `stderr_failed` | Existing emergency stderr failure counter. |
| `stderr_suppressed` | Existing emergency stderr suppression counter. |
| `truncated` | Existing truncated-record counter. |
| `closed` | Output submission was closed. |
Collectors capture exclusive stdout/stderr from process startup through EOF and
bind the captured PID, startup instance, and fixed log directory to the same
run. PID alone cannot distinguish PID reuse. A complete version-1 final record
requires exactly one record, the exact wrapper and snapshot fields, `finished`,
initialized and closed output, null first failure, equal integer written/enqueued
counts, and zero dropped/output-failed/stderr-failed/stderr-suppressed/truncated
counts. Missing, truncated, duplicate (even identical), conflicting, foreign-PID,
unknown-version, pending, panicked, uninitialized, or short-write records cannot
prove completion. Added fields or changed meanings require a new schema version.
This proves only the reported output state. Actual ingress originals and every
correlated Invoke still require independent file readback; the final line alone
cannot prove request correctness or filesystem durability. No record is fabricated
when no diagnostic exit exists. Abrupt interruption may produce no line. The old
pre-coordinator owner exit retains its original one-shot shutdown policy, so a
pending result remains pending rather than being upgraded to success.
Encoding uses a fixed 2048-byte stack buffer including the newline at process
exit, without adding request-path storage or a second shutdown deadline. Failure
to encode or write stderr is ignored for the primary process result, with no
recursive diagnostic submission; the collector then has insufficient evidence.
There is no guarantee against OS stderr blocking or failure, and no guarantee
that failure reporting itself reaches the collector.
## Request task and physical account finalization
Two independent events in the existing emergency diagnostic stream use
`schema="request-finalization/1"`: `request_task_reclaimed` and
`request_account_released`. Both contain `process_id`, `process_instance`,
`completion`, `context`, and `account`. The instance is an opaque exact integer
initialized once in this process from startup observation time; preserve its
integer precision and associate it with exclusive process capture, native PID
start generation and the fixed log directory. It is not a request clock or a
cross-process ordering clock.
`context` is the existing schema-2 request view. Correlate the complete published
request identity (`publication=1`, `request.value`, `trace_id`, rpc/span present
states), `local_request`, and process instance. Trace alone is insufficient.
`account` contains `account_slot`, `account_generation`, and `audit_generation`;
slot reuse must never associate a new generation with an older request. A task
without a normal account has null `account` and cannot prove normal account
release. `context.task.value` is the actual Tokio task identity for a reclaimed
task, scoped to this process instance; no task identity is generated at recovery.
A task event follows a matched actual join result and removal of its supervisor
collection entry. Its completion is `completed`, `failed`, `cancelled`, or
`panicked` (the latter for a native join panic). Captured source/cleanup failures
remain separate original diagnostics, including panics caught inside the request
boundary. Unsubmitted withdrawal produces no reclaimed-task event. This event
proves recovery of that execution slot, not destruction of all retained task
storage, response delivery, database disposal or overall request completion.
The account event has `completion="physically_released"`; its account object
also contains `healthy`, sampled from the existing Admission health latch.
It is generated only after the exact account generation has been removed,
storage and object charges, terminal commitment and DB/RPC occupancy have ended,
and existing signal/audit closure has run. Merely setting closing or returning
from reconcile does not emit it. The last physical storage/RPC/byte holder can
trigger the same event after a delayed release. Its context is an immutable
bounded identity copy made at entry publication: the copied lifecycle/task
fields describe that binding site, not the last execution phase or the last task.
The copy and output handle retain no request root or account capability.
Admission holds a neutral observer allocated under the existing process-log
storage domain, independently of the request being released. Its actual boxed
layout and permit are prepaid. Account backing and task status layouts include
the added fields. The fixed context buffer is 8192 bytes, and notification links
reuse the prepaid observer nodes. Output submission and observer destruction
happen after ledger unlock; original capacity wakeup precedes diagnostics.
No request deadline, capacity policy or release ownership is changed. Preparation,
encoding, queue closure/fullness and writer failure leave observation insufficient
and preserve the primary product result. Missing observer preparation does not
promise a failure record of its own. Enqueued remains different from Written,
and neither final event proves durability.
Collectors freeze and verify actual source files and process EOF plus the healthy
complete diagnostic exit snapshot. Require the independently identified request's
relevant task roster and exact account release generation, along with existing
business-stop, response and database physical-disposal originals as required by
the scenario. Missing, unknown-version, unpublished/foreign identity, wrong
process/generation, duplicate or conflicting facts cannot prove full cleanup.
Global zero counters, handler completion and process exit are insufficient.
A known native product crash remains a product failure even when these facts are
missing. Do not require facts for tasks/accounts that were never created.
The causal order is scoped to actual ownership: execution borrow ends before
that task's matched recovery; the last physical holder ends before account
release. A retained root for a particular task must outlive its recovery before
that account can release. Preserve such same-source/ownership relations; do not
invent a total order between response completion, independent DB sources and
other tasks from file order or unrelated clocks. Adding fields or changing these
meanings requires a new finalization schema version.