# POSIX Semantics
Hyperfile aims to follow POSIX file semantics as closely as the
underlying object-storage substrate allows. This doc enumerates,
flag by flag and operation by operation, what Hyperfile actually
does today.
When Hyperfile diverges from POSIX, the divergence is called out
explicitly with a "Divergence" note. Anything not mentioned here is
unimplemented or undefined; treat it as a known gap.
## Open flags (`FileFlags` / `O_*`)
`FileFlags::from(libc::c_int)` accepts any combination of the flags
below. The crate-internal `HyperFileFlags::from_flags` translates
the bitset into structured fields and feeds them through `Hyper::
open` / `Hyper::create`.
### Access mode (mutually exclusive)
| Flag | Behaviour | Notes |
| --- | --- | --- |
| `O_RDONLY` | Read-only handle. | Every write-side operation (`fs_write`, `fs_write_zero`, `fs_write_batch`, `fs_write_aligned_batch`, `fs_truncate`, and the `fh_*` / tokio equivalents) fails with `EBADF`. See [Access mode enforcement](#access-mode-enforcement). |
| `O_WRONLY` | Write-only handle. | `fs_read` (and `fh_read` / the tokio `AsyncRead` impl) fails with `EBADF`. Writes, truncate and `lseek` all work — including operations that read internally, such as a sub-block write or a shrink to a non-aligned size. |
| `O_RDWR` | Read+write handle. | Recommended for any mutation flow. |
### Access mode enforcement
An operation that the handle's access mode does not permit fails with
`EBADF`:
| Handle | Rejected | Allowed |
| --- | --- | --- |
| `O_RDONLY` | `write`, `write_zero`, `write_batch`, `write_aligned_batch`, `truncate` | `read`, `lseek`, `fsync`/`fdatasync`, `getattr`, `chmod`/`chown` |
| `O_WRONLY` | `read` | everything else, including writes and `lseek` |
* POSIX `write()` lists `[EBADF] The fildes argument is not a valid
file descriptor open for writing` as a mandatory ("shall fail")
error, and `read()` the same for `... open for reading`.
* POSIX `ftruncate()` permits either `[EBADF]` or `[EINVAL]` for the
same condition. Hyperfile uses `EBADF` so that every access-mode
violation reports one errno.
The check keys off the access mode only: `O_APPEND` does not grant
write access on its own, matching Linux, where
`open(O_RDONLY | O_APPEND)` followed by a `write` fails with `EBADF`.
`lseek` — including the `SEEK_DATA` / `SEEK_HOLE` extensions behind
`fs_seek_data` / `fs_seek_hole`, or `fh_seek_data` / `fh_seek_hole` on the
handler — requires no particular access mode
and is never gated.
Operations that read internally are not affected by the read gate. A
sub-block write performs a read-modify-write of the target block, and
a shrink to a non-block-aligned size reads the tail block to zero its
remainder; both use lower-level block loaders rather than the public
read path, so they still work on an `O_WRONLY` handle. Symmetrically,
WAL crash recovery replays previously acknowledged writes while
opening the file and bypasses the write gate, so a read-only open of a
file that crashed mid-flush still presents correct contents.
Because `std::io::ErrorKind` has no `EBADF` variant, the error is
constructed with [`std::io::Error::from_raw_os_error`]. Callers should
test it with `err.raw_os_error() == Some(libc::EBADF)`; `err.kind()`
is the unmatchable `Uncategorized` and must not be used. (The
alternatives were rejected as inaccurate: `PermissionDenied` maps to
`EACCES`, which POSIX reserves for permission-bit failures at `open`
time, and `InvalidInput` maps to `EINVAL`, conformant for `ftruncate`
but not for `read` / `write`.)
### File-creation flags
| Flag | Implemented | Behaviour |
| --- | --- | --- |
| `O_CREAT` | ✅ | Use `Hyper::fs_open_or_create_*` (or the reactor / tokio equivalents). The function performs an `open`, and on `NotFound` falls back to `create`. The bare `Hyper::fs_open` does **not** honour `O_CREAT`. |
| `O_EXCL` | ✅ | Honoured on the `fs_open_or_create_*` family of entry points: when paired with `O_CREAT`, the open MUST create a new file and must fail with `ErrorKind::AlreadyExists` if one already exists. Implementation: `Hyper::do_open_or_create` short-circuits to `Hyper::create()` when `O_EXCL` is set, skipping the try-open dance, so the create-or-fail attempt is a single S3 round trip (a HEAD on the inode, then a conditional PUT on miss). On the bare `fs_open` path (no `O_CREAT`) the bit is silently ignored, matching Linux behaviour where POSIX leaves the combination undefined. The `fs_create` entry point is always exclusive regardless of the flag — hyperfile has no "open existing or create new" mode that lacks an exclusivity check on missing-file. |
| `O_TRUNC` | ✅ | Honoured by `Hyper::open` (and therefore `fs_open`, `fs_open_opt`, `fh_open*`, `HyperFileTokio::open*`, plus the existing-file path of `fs_open_or_create_*`). When set together with `O_WRONLY` or `O_RDWR`, the file's length is set to 0 immediately after the open succeeds, before the handle is returned. The truncation runs through the normal `truncate(0)` path: `mtime`/`ctime` are updated and the change is staged for the next flush. **Divergence**: the truncation is not durable until you flush or release the file. POSIX requires `O_TRUNC` to be effectively immediate; Hyperfile defers persistence the same way it defers writes. |
| `O_TRUNC` (without write access) | ✅ no-op | If the access mode is `O_RDONLY`, `O_TRUNC` is silently ignored, matching glibc / Linux. |
### Access pattern flags
| Flag | Implemented | Behaviour |
| --- | --- | --- |
| `O_APPEND` | ✅ | The offset argument to `fs_write` / `fh_write` / `fs_write_zero` / `fh_write_zero` is ignored; the call writes at the current `i_size` and atomically advances `i_size` by the number of bytes written. Atomicity is enforced through the per-file write serializer: in the direct API, `&mut self` enforces it; in the reactor without `range-lock`, the handler dispatches one ctx at a time; in the reactor with `range-lock`, concurrent appenders compute identical-or-overlapping byte ranges starting at the current `i_size` and only one wins the range lock per turn — the loser is re-queued via send_highprio and re-evaluates `i_size` on its next dispatch. The result is whole, contiguous payloads laid down in some serial order; never a torn interleave. **Divergence**: cross-instance / cross-process append (two `Hyper`s opened against the same URI) is **not** atomic — it is governed by the same conflict policy as ordinary writes, not by O_APPEND. |
| `O_APPEND` (batch APIs) | n/a | `fs_write_aligned_batch` / `fs_write_batch` accept an explicit `offset` per data block; that is the entire point of the batch APIs. They ignore O_APPEND on the open flag set. Use the single-write entry points if you want POSIX append semantics. |
| `O_DIRECT` | ✅ | Disables the in-memory data block cache. Without WAL: every write triggers an immediate flush. With WAL: writes still go to the WAL synchronously but the data-block cache is sized to zero so there is no in-memory accumulation. |
| `O_SYNC` / `O_DSYNC` | ✅ | Triggers a flush on every write. With WAL enabled, the WAL persistence already provides the same crash-consistency guarantee, so the explicit flush is skipped. The two flags are treated identically at write time: every write in hyperfile already updates `i_size` (which `fdatasync` is required to persist), so there is no work that O_DSYNC could legitimately skip but O_SYNC must do. The fsync vs. fdatasync **distinction is exposed at the syscall-equivalent layer** instead — see `fs_flush` (= `fsync`) and `fs_fdatasync` (= `fdatasync`) below. |
| `O_NOATIME` | ✅ | Read paths skip `update_atime` on the in-memory inode. Other timestamps (`mtime`, `ctime`) are unaffected. |
| `O_NONBLOCK` / `O_NDELAY` | ❌ unsupported | These flags govern the read/write blocking discipline of file descriptors in the kernel, where a non-blocking read on an empty pipe returns `EAGAIN` instead of suspending. Hyperfile has no equivalent state machine: every async fn already returns control to the runtime when waiting on S3 I/O, so there is nothing to flip on. Parsed for display only; setting the bit changes no behaviour and there is no plan to add semantics for it. |
| `O_ASYNC` | ❌ unsupported | Requests SIGIO / SIGURG signal-driven I/O on POSIX file descriptors. Hyperfile is a library, not a process running under a kernel fd; signal delivery is outside the model. Parsed for display only and explicitly **not** going to be implemented. Use the standard async/await flow against the existing `fs_*` / `fh_*` / `HyperFileTokio` APIs instead. |
### Path resolution flags
`O_DIRECTORY`, `O_NOFOLLOW`, `O_NOCTTY`, `O_PATH`, `O_LARGEFILE`,
`O_CLOEXEC` are all parsed for display but Hyperfile does not
operate on directories or file descriptors in the kernel sense, so
they are no-ops.
## Sync mode quick reference
The same information as the table above, presented as a behaviour
matrix for the three flush-affecting flags. WAL changes the
"persistence on write" column.
### WAL disabled
| Flag | Data block cache size | Immediate flush after write | Persistence on `fs_write` ack |
| --- | --- | --- | --- |
| (none) | from config | False | None until flush |
| `O_DIRECT` | 0 | True | Yes (flush completed) |
| `O_SYNC` / `O_DSYNC` | from config | True | Yes (flush completed) |
### WAL enabled
| Flag | Data block cache size | Immediate flush after write | Persistence on `fs_write` ack |
| --- | --- | --- | --- |
| (none) | from config | False | Yes (WAL PUT completed; full flush deferred) |
| `O_DIRECT` | 0 | False | Yes (WAL PUT completed) |
| `O_SYNC` / `O_DSYNC` | from config | False | Yes (WAL PUT completed) |
## Inode timestamps
Hyperfile stores `atime`, `mtime`, `ctime` in the on-disk inode
with second + nanosecond resolution. They update as follows:
| Operation | atime | mtime | ctime |
| --- | --- | --- | --- |
| `read` (cache hit, `bytes_read > 0`) | updated unless `O_NOATIME` | — | — |
| `read` (cache miss) | updated unless `O_NOATIME` | — | — |
| `write` / `write_zero` / `write_aligned_batch` / `write_batch` | — | updated | updated |
| `truncate` (size changed) | — | updated | updated |
| `chmod` | — | — | updated |
| `chown` | — | — | updated |
| `setattr` | from caller (if differs) | from caller (if differs) | updated to NOW (caller's `st_ctime` is ignored — POSIX says `ctime` is not user-settable) |
| `flush` / `release` | — | — | — (timestamps are persisted as-is) |
**Divergences from POSIX**:
- `atime` updates only land on the in-memory inode. They are
flushed lazily as part of the next data-or-metadata flush. A
read on a read-only handle that gets dropped without flushing
will lose the `atime` advance. Linux behaves the same way under
default `relatime` mounts but differs from `strictatime`.
## Sync APIs (`fsync` vs `fdatasync`)
POSIX distinguishes two flush primitives:
- `fsync(fd)` — persist data and **all** metadata (data, file
size, atime, mtime, ctime, mode, uid, gid).
- `fdatasync(fd)` — persist data and metadata that's required
for the data to be read correctly, allowed to skip pure
attribute updates that don't affect retrieval (atime, mtime,
ctime, mode, uid, gid).
Hyperfile exposes both:
| Hyperfile API | POSIX equivalent | Behaviour |
| --- | --- | --- |
| `Hyper::fs_flush` / `HyperFileHandler::fh_flush` / `HyperFileTokio::flush_ext` | `fsync` | Always persists. If only attrs are dirty, issues an inode-only PUT (overwrites the same S3 key, no new segment, `last_cno` unchanged). If data or bmap is dirty, builds and uploads a new segment. |
| `Hyper::fs_fdatasync` / `HyperFileHandler::fh_fdatasync` / `HyperFileTokio::fdatasync_ext` | `fdatasync` | Skips when only attrs are dirty (no S3 PUT, no `last_cno` change). When data or bmap is dirty, falls through to the same path as `fs_flush`: hyperfile keeps `i_size` in the inode, so any data flush already needs a new segment that carries the inode along. |
Observable difference: a read on an otherwise-clean file dirties
`atime` in memory. `fs_flush` writes the inode out; subsequent
opens see the advanced atime. `fs_fdatasync` skips; the atime
change is lost if the handle is dropped without an `fs_flush` /
`fs_release`. This matches the POSIX rule.
When data is dirty, both calls do the same work, including
issuing a new segment. There is no "data-only segment" path;
`i_size`, mtime, and the bmap travel together with data in the
on-disk format.
## Truncate behaviour
`fs_truncate(new_size)`:
| Case | Effect |
| --- | --- |
| `new_size == cur_size` | No-op (returns immediately). |
| `new_size < cur_size` (shrink, mid-block) | Data blocks beyond `new_size` are dropped from the cache and from the persisted block index on next flush. The block containing byte `new_size - 1` is the **last partial block**; its bytes `[new_size % block_size, block_size)` are zero-padded so a future extending write or read across that boundary cannot expose stale data. |
| `new_size < cur_size` (shrink, exact block boundary) | Same as above except no partial-block zeroing happens — the last fully-retained block (`new_size / block_size - 1`) is kept untouched. (Pre-fix bug: the code used to call the partial-block path with `offset_to_discard = 0`, wiping the block.) |
| `new_size > cur_size` (extend) | The file size is extended; the new range reads as zeros. No data blocks are allocated for the extended range — they are sparse until written. `st_blocks` does not change. |
| `new_size == 0` | All data blocks discarded; bmap fully truncated; `i_blocks` goes to 0. |
Truncate updates `mtime`. It does **not** itself flush; pair with
`fs_flush` or `fs_release` to persist.
## Read on a zero-length file
A `fs_read` issued on a file whose `i_size` is 0 returns `Ok(0)`
immediately, before consulting the bmap or staging. The same
applies for an offset at or past `i_size`.
## Reads and the data cache
Byte reads do not populate the data cache; block access does, and so
does an explicit read-ahead. Every one of them reads from it.
| entry point | populates the data cache |
|---|---|
| `fs_read` / `fh_read` | **no** |
| `fs_read_ahead` / `fh_read_ahead` | **yes** |
| `fs_block` / `fh_with_block` | **yes** |
| `fs_block_mut` / `fh_with_block_mut` | **yes** |
Reading the same bytes twice through `fs_read` fetches them from
staging twice. Borrowing the same block twice fetches it once. A byte
read after a block borrow of the same data is served from memory; a
block borrow after a byte read is not.
`fs_read_ahead` exists for a caller that knows what will be asked for
next. Fetching ahead through `fs_read` cannot work — the bytes have
nowhere to live, so the read that wants them fetches them again, and the
speculation is pure cost. Read-ahead through this entry point warms the
cache instead, with its requests coalesced the way a read of the same
range would be, so a wide window costs a handful of requests rather than
one per block. Ordinary reads still do not fill the cache, so they cannot
evict what the write path is holding.
`read_timing` (or `fh_read_timing` on the reactor surface) reports
this: `data_gets` counts fetches, `cache_hits` counts reads served from
the cache. See
[Block API](block-api.md#interaction-with-the-data-cache).
## Sparse holes (write past EOF without O_APPEND)
`fs_write(off, buf)` where `off > i_size` (and the handle was not
opened with O_APPEND) is allowed. The intervening range is sparse: a subsequent `fs_read` of that range
returns zeros, and the persisted block index records no allocation
for the gap. `i_size` becomes `off + buf.len()`. This is the
standard POSIX "sparse hole" behaviour.
## `stat()` field semantics
`fs_getattr` / `fh_getattr` populate a `libc::stat`. Field rules:
| Field | Source / behaviour |
| --- | --- |
| `st_dev`, `st_rdev` | Always 0. Not modelled — hyperfile is not a device-backed filesystem. (Tracked TODO.) |
| `st_ino` | The inode number from the on-disk inode. |
| `st_nlink` | Always 1; hyperfile has no concept of hard links. |
| `st_mode` | File-type bits (`S_IFREG` / `S_IFDIR`) plus permission bits as last set by `chmod` / `setattr`. |
| `st_uid`, `st_gid` | As last set by `chown` / `setattr`. Default: 1000 / 1000 at creation. Not enforced by the library; see "Permissions and ownership" below. |
| `st_size` | Logical file length in bytes (`i_size`). For sparse files this is the **virtual** size, not the storage footprint. |
| `st_blksize` | The data-block size from the file's `HyperFileMetaConfig` (default 4096). Acts as the I/O hint that POSIX intends. |
| `st_blocks` | Number of 512-byte units actually allocated. **Sparse-aware**: a 1 GiB sparse file with one 4 KiB written block reports `st_blocks = 8` (= 4096 / 512), not 2 097 152. Maintained incrementally: each newly-allocated bmap entry adds `data_block_size / 512` to the counter; truncate-shrink subtracts the units of the entries it discards. |
| `st_atime`, `st_mtime`, `st_ctime` (+ `_nsec`) | See the "Inode timestamps" section. |
**Divergence**: `st_dev` and `st_rdev` are not yet plumbed; both
report 0. If you need a stable device id (e.g. for a FUSE adapter)
you must inject one in the layer above.
## Rename
Hyperfile exposes `Hyper::fs_rename(client, src_uri, dst_uri)`
and the `HyperFileHandler::fh_rename` mirror, but **the
operation is not yet implemented** — the call returns
[`ErrorKind::Unsupported`] today.
The interface is reserved at the API layer so that callers can
write code today that targets the eventual rename without
reaching for a different namespace later. Any such code will
fail with `Unsupported` until the implementation lands; treat
this as a hard signal, not a transient error.
### Why it isn't done yet
A POSIX-faithful `rename(2)` requires:
1. **Atomic move of every backing object.** A hyperfile lives
as several S3 objects under a shared URI prefix (the inode
key, segment keys, optional WAL chunks, optional local cache
files). S3 has no native cross-key atomic-rename primitive;
the natural implementation is a copy-each-object,
then-delete-the-originals loop, which is **not** atomic.
2. **Atomic replacement of the destination.** POSIX requires
that if `dst_uri` already exists, the old destination is
atomically replaced. Under copy-then-delete, a concurrent
reader on either prefix can observe an intermediate state.
3. **Coordination with open handles.** An in-flight
`fs_write` / `fs_flush` against either URI would race the
rename. Either rename has to fail when handles are open, or
the handles need to be re-pointed atomically (currently they
bind to a fixed URI at construction).
4. **Feature interactions.** With WAL on, the WAL prefix lives
at a sibling URI and would need to move in lock-step with
the file. With a local cache, cached blocks under the source
URI need to be invalidated or rewritten under the destination
URI.
Each item is solvable on its own; the open question is which
non-strict atomicity guarantees are acceptable, and that's a
design decision rather than a coding task.
### Recommended workaround until rename lands
The closest hand-rolled alternative is application-driven:
1. Open the source URI with `Hyper::fs_open`, read the data,
close.
2. Create the destination URI with
`Hyper::fs_open_or_create_with_default_opt`, write the data,
release.
3. Call `Hyper::fs_unlink(source_uri)` once the destination is
safely persisted.
This is **not atomic**, makes a full data copy, and breaks any
open handles to the source. It is fine for offline /
single-tenant relocations of small-to-medium files; do not
build "atomic rename" semantics on top of it.
## Unlink
`Hyper::fs_unlink(uri)` removes a file. It reports
`ErrorKind::NotFound` (`ENOENT`) when the file does not exist, as
POSIX `unlink(2)` requires. The check is a `HEAD` on the inode
object in `S3Staging::from`, which runs before any deletion: the
inode is the authoritative "this file exists" marker, and
`load_inode` returning `NotFound` is likewise what makes
`fs_open` report `ENOENT`.
On success, every object under the URI prefix is removed — the
inode, all segments, and any WAL chunks that live under the same
prefix.
### Known gap: orphaned objects are not reclaimable
`unlink` deletes the inode **first** and the remaining objects
**second**. If it fails in between — throttling, a partially
failed `DeleteObjects`, a crash — the prefix is left holding
segments with no inode. `unlink` returns an error when that
happens, including when a `DeleteObjects` succeeded but listed
keys it could not delete, and logs each such key; until 0.10.2
it reported success. That state cannot be cleaned up through
the library:
* `fs_unlink` returns `ENOENT` from the inode `HEAD` before the
list-and-delete sweep runs, so re-running it does not help.
* `hyperfile-cleaner`'s `prune` cannot help either: it reads
`inode.get_last_seq()` to locate the latest checkpoint before
walking the metadata tree.
Both reclaim paths key off the inode, so the leftover objects are
unreachable and keep costing storage. Recovering them currently
requires deleting the prefix out of band, e.g.:
```bash
aws s3 rm --recursive s3://<your-bucket>/<prefix-of-the-file>
```
Be sure the prefix really is an abandoned hyperfile before doing
this; the command is not selective.
This is a design gap rather than a simple bug — it needs a
decision on which component owns orphan collection (a `force`
mode on `unlink` that skips the inode check, a prefix-level sweep
in the cleaner that does not need an inode, or an external
reaper) — so it is recorded here rather than worked around.
## Permissions and ownership
Hyperfile **does not enforce** the POSIX permission model. The
`mode`, `uid`, and `gid` fields stored in the inode are
**opaque metadata** from the library's perspective: hyperfile
records what callers ask it to record, persists it across
flushes, and surfaces it back through `fs_getattr`, but every
read / write / truncate / chmod / chown / setattr succeeds
**regardless of the bits stored**. There is no `EACCES` /
`PermissionDenied` code path anywhere in the read/write
pipeline.
Concretely:
- A handle opened with `FileFlags::rdonly()` will accept
`fs_write` calls (the access-mode bits influence cache
sizing and sync-flush behaviour, but they do not gate
mutation operations).
- A file with `mode = 0o000` and `uid = 0` is just as
read/writable through the hyperfile API as `0o777`.
- `fs_chmod` / `fs_chown` succeed even when the caller is not
the file's nominal owner.
This is by design. Hyperfile is a Rust library, not a process
running as a specific user; it has no notion of a "current
user" to check the bits against, and inventing one
(thread-local "set-uid" or per-call credentials) would only
provide self-attestation, not security.
### How to enforce, if you need to
Build the access-control layer **outside** hyperfile:
- **FUSE adapters**: rely on the kernel's `default_permissions`
mode or its FUSE-protocol equivalent. The kernel does the
uid/gid comparison against the inode bits hyperfile returns
via `fs_getattr` before any read/write reaches hyperfile.
- **Multi-tenant services**: use IAM / app-level auth to
decide whether a request gets to call hyperfile at all. The
hyperfile mode bits then become a documentation /
reflection-only field, not a security boundary.
- **Single-tenant tools**: typically don't need enforcement;
the process owns its files and the POSIX bits are
cosmetic.
If you build a layer that DOES check, remember:
- Hyperfile has no `O_EXEC`-equivalent gate. A reader-only
layer must reject writes itself; hyperfile won't.
- `fs_chmod` on a file you "shouldn't be able to chmod" still
succeeds at the hyperfile level. Either don't surface the
call, or check before forwarding.
- `umask`, supplementary groups, set-uid/set-gid bits,
capabilities, ACLs, and POSIX file capabilities are all
out of scope.
## Related reading
- [`docs/concurrency.md`](concurrency.md) — multi-handle / multi-process consistency and the `range-lock` feature.
- [`docs/wal.md`](wal.md) — the durability contract added by `--features wal`.
- [`tests/README.md`](../tests/README.md) — test layout including
the smoke suite that exercises `O_TRUNC` and `O_NOATIME`.