gfeh: object storage with many faces

Source: https://github.com/town-os/gfeh | Part of Town OS

gfeh is a virtual routing layer over a managed filesystem subtree. It presents one collection of bytes through several protocol views at once -- write a file over SMB from your laptop, read it back over S3 from a backup tool, fetch it by content identifier over IPFS, and see it listed in a Google Drive client. There is one object underneath all of it, with one set of permissions.

gfeh can also federate: log in to a real upstream service and splice it into the namespace as a subtree. An Amazon S3 bucket or a real Google Drive account mounted at archive/ is then browsable over SMB and fetchable over IPFS, because federation happens below the protocol layer rather than inside any one view.

Table of Contents

• Features
• Where gfeh Fits
• Protocol Views
• Federation
• Users, Permissions, and Sharing
• Partitions and Networks
• Building and Testing
• Makefile Targets
• Known Limitations
• License

Features

• Multiple protocol views into the same collection -- S3, SMB/CIFS, IPFS, Google Drive, and plain HTTP, all reading and writing the same objects.
• Native implementations. The SMB view is a Rust SMB2/3 server, not a Samba configuration generator. The Drive view is a Drive API v3 emulator, not a sync client. Nothing is shelled out to a second daemon.
• Federation with four modes -- mount an upstream as a subtree, use it as a read-through cache, proxy it transparently, or mirror it two ways. Toggleable per mount, with credentials sealed at rest.
• Hierarchical permissions. Users create sub-users and delegate subsets of their own access. Delegation can never exceed the granting user's own grant, and revoking a grant recursively revokes everything derived from it.
• Sharing by mapped path -- a subtree of one user's namespace appears inside another's without copying bytes, and shows up in every protocol view at once.
• Per-file plain-HTTP exposure, toggled per file, with public or unguessable-token links.
• Content addressing without duplication. IPFS leaf blocks are slices of the stored file rather than copies, so the IPFS view costs about 1% overhead instead of doubling storage.
• Asynchronous, journalled federation with crash recovery, retry with backoff, and an audit log readable by the users whose data it concerns.

Where gfeh Fits

Town OS's storage layer manages volumes: it creates, resizes, snapshots, and deletes btrfs subvolumes and enforces quotas. gfeh provides object storage: the namespace, the metadata, the permissions, the sharing, and the protocols. The storage layer does not handle object storage at all -- gfeh is the responsible party.

gfeh ships with Town OS as a core system service, alongside the DNS resolver and the ingress router. It is not an installable package, which matters for a practical reason: Town OS reconciles its ingress routes and DNS records declaratively and deletes anything it did not derive. Because gfeh is part of the platform, Town OS knows about gfeh's hostnames and routes and re-creates them on every reconcile, instead of gfeh having to defend them.

Protocol Views

Every object has one stable identity, and each view projects it differently: an S3 key, an SMB handle, a Drive file ID, an IPFS content identifier. One consequence worth knowing about is that these interact -- if an SMB client holds an exclusive lease on a file, an S3 PUT to that object is refused rather than silently racing it.

Federation

Each mount binds a subtree to an upstream service in one of four modes:

Mutations are journalled, so they survive a crash, retry with backoff, and land in the audit log. Transparent-proxy reads are the deliberate exception: they bypass the journal and are served synchronously, because a read has no durable intent to record.

Disabling a mount does not make its path disappear -- the path stays and reports that the upstream is unavailable. A path that vanishes breaks SMB clients and sync jobs far more disruptively than an error does.

Users, Permissions, and Sharing

Town OS administrators create the roots of the permission forest. Below a root, everything is self-service: a user creates sub-users and grants them subsets of what they themselves hold.

Two invariants are enforced structurally rather than by convention:

1. A delegated grant can never exceed the granting user's own grant. Checked when the grant is written, and again when it is resolved.
2. Revoking a grant revokes everything derived from it, recursively, as a property of the schema. A grant that outlives the authority it came from is not representable.

Sharing places a subtree of one user's namespace inside another's, symlink-style, without copying data. Because composition happens below the protocol layer, a share is visible over S3, SMB, Drive, IPFS, and HTTP simultaneously.

Partitions and Networks

A partition is one btrfs subvolume, one quota, one metadata index, and one snapshot unit. Each partition is bound to a Town OS network, which determines the TLD its name is published under and whether its DNS records are LAN-visible or scoped to a WireGuard overlay. A peer joined to one network resolves that network's storage names and not a sibling network's -- the partition is a real boundary, not just a naming convention.

Per-partition quotas are btrfs qgroup limits. Per-user quotas within a partition are enforced by gfeh in software, not by btrfs.

Building and Testing

Requires a Rust toolchain and podman. Podman is not optional: make test drives every view and every protocol emulator with real third-party clients in containers, and those are the tests that catch the mistakes our own tests share.

make build             # compile
make test              # everything: format, lint, contract check, and every test
make conformance-test  # a shortcut for re-running just one protocol's client tests

make test is deliberately one task. Linting and testing are not separate gates: a change is not done until formatting, clippy with -D warnings, and the tests all agree.

make test runs every test there is -- nothing is #[ignore]d and there is no second tier to remember. That includes the third-party conformance tests, which drive the views and the emulators with real clients in containers, so make test needs podman. Each of those tests starts and removes its own containers, so runs in separate checkouts do not collide.

There are no cargo features in this workspace. Every crate always builds in full. That means one build configuration to reason about, no combination that compiles for one person and not another, and no feature matrix to test.

One storage trait, one conformance suite

Every protocol view sits on a single trait, and every backend implements it: the in-memory reference, a local filesystem subtree, and one per federated upstream. Above the trait nothing can tell them apart, which is what makes federation a composition of stores rather than a second set of code paths.

That substitutability is asserted rather than assumed. gfeh-store's conformance suite is a library module, not a test target, so every backend runs the identical cases -- what a rename does to an identity, whether opening a missing file creates it, how a prefix and a delimiter interact, whether paging repeats or drops an entry. A backend that answers differently fails the same case the reference passes.

The suite is also run against deliberately broken stores, which is the part that keeps it honest: a store that renames by copying, one that lists out of order, and one that drops the tail of a listing each have to fail the specific cases covering that defect. A suite whose cases all passed vacuously would look exactly like a working one.

One view, three stores underneath it

Each of the five views carries a suite of its own, on the same pattern and for the same reason. A view is written against the storage trait and cannot tell which store it has, so every case runs three times: against the in-memory reference, against a real filesystem partition with a real SQLite index, and against the composed namespace a daemon actually serves from. That third one matters most, because a view meets composition in production and a bare store in every other test.

The cases speak the protocol over a socket -- signed SigV4 requests, real SMB2 frames, real multipart uploads -- rather than calling the functions behind it. And each suite is run once more against a store that refuses every operation, where the only cases still allowed to pass are the ones about the protocol itself: a handshake, a version probe, a status for a handle nobody opened. Anything else surviving is a case that has quietly stopped touching the store.

That run also settles a question the ordinary backends cannot ask: what a view says when the store cannot answer. The SMB view used to say the file was not there -- which is not a hedge but a claim, and a client acts on it by creating one, so a store that was merely unreachable becomes one that has quietly replaced a file with an empty one.

Checking against real clients, not against ourselves

Our own tests speak each protocol the way we understand it, so they cannot catch a misreading of a specification -- they share it. make conformance-test runs the five views and the four emulators against clients written by people who had to interoperate with the real thing: aws, mc and rclone for S3; rclone's go-smb2 for SMB; kubo's own CLI for IPFS; curl for the plain-HTTP view and for the HTTP layer of Drive.

Both halves are needed and neither substitutes for the other. An emulator stands in for an upstream to federate against; a view is what the daemon serves. Assuming one was evidence about the other cost real time -- see the ETag row below, where the emulator was right and the view was wrong and every test in the tree passed.

Every one of those protocols failed the first time this ran, and every failure was a real defect that our own tests passed:

The sharpest of them is the CID cross-check, and there are two. A CID is a claim about bytes that the whole network verifies, so the only way to know the chunker is right is to ask a real implementation: identical bytes go into a kubo container and into gfeh, and the roots have to match at every size where the DAG changes shape. The second runs the bytes through the IPFS view's /api/v0/add first, so it covers the route that parses the upload as well -- a correct DAG builder reached through a route that keeps one byte of multipart framing publishes a perfectly valid identifier for content nobody asked to store.

That found a real one too. The DAG builder emitted a single flat node with a link per chunk, so any file over 174 chunks -- about 43 MiB -- got a root no gateway would ever resolve. Below that boundary a flat node and a balanced tree are indistinguishable, which is why it agreed with kubo on every test anyone would think to write.

Testing against Town OS without Town OS

A real Town OS needs root, systemd, podman, and a btrfs filesystem, which makes it a poor thing to require of every test run. gfeh-townos-emulator replicates the parts gfeh actually uses -- account authentication and volume provisioning -- as an in-process HTTP server, so integration tests exercise the real client over real HTTP against a faithful contract.

An emulator that has drifted from the real thing is worse than no emulator: it turns a broken deployment into a green test suite. So the contract is written down in TOWNOS_CONTRACT.md and verified two ways. Nothing is pinned to a revision, deliberately — a recorded revision no script reads is a claim nobody is maintaining, and it goes stale silently while failing loudly on Town OS commits that changed nothing gfeh depends on.

• make check-townos-sync runs as part of make test against a local checkout, skipping cleanly when there is none (TOWNOS_DIR= points it elsewhere).
• make check-townos-sync-release fetches HEAD of the canonical Gitea repository and verifies against that, resolving HEAD at the moment it runs. make publish and make publish-dry both depend on it.

Either way the check resolves Town OS's reserved-prefix constants and route table out of its Go source rather than matching literals, so it survives a rename and still catches a real change.

The release gate is the one that matters: the everyday check verifies against whatever Town OS is on the machine, which may be months old, while a release verifies against what Town OS actually is right now. It also means the pin expires at every release, so it stays a claim somebody maintains rather than one nobody revisits.

That document also records the routes gfeh needs that Town OS does not have yet, kept separate from the ones it replicates, so nothing quietly pretends to be integrated when it is not.

make test is also safe to run concurrently in the same checkout. Container names, ports, and temporary paths all derive from an instance ID computed from the working directory.

Makefile Targets

Run make help for the full list.

• test -- the whole sweep: lint, the Town OS contract check, every test, the doc tests, and the coverage floor. TEST_PATTERN= scopes it; TEST_LINEAR=1 forces single-threaded execution.
• test-log -- the same, tee'd to a timestamped log printed even on failure.
• lint -- format check, clippy with -D warnings, and the crate-header check.
• fmt -- rewrite sources with cargo fmt.
• all-tests -- every compiled test, in one invocation. unit-test and integration-test are subset shortcuts for iterating; make test already runs both.
• doc-test -- the examples embedded in documentation, all of which assert something rather than merely compiling. Neither --lib nor --tests selects these, so without it a documented example is compiled by cargo doc and executed by nothing.
• conformance-test -- third-party conformance; needs podman. Takes a protocol name, so make/conformance.sh ipfs runs just that view and its emulator against kubo and curl.
• coverage, coverage-html, coverage-files -- coverage with a floor that exists to catch a regression, not to be satisfied. coverage fails below 90% per crate and across the workspace, because an average hides a crate.
• install -- installs gfehd (the daemon) and gfeh (its administrative client).
• build, release-build, doc, clean.
• publish-dry, publish -- crates.io release in dependency order.

Known Limitations

Stated plainly, because each of these will otherwise surprise someone:

• The S3 view verifies header signatures only. Presigned URLs -- the query-string form -- are not verified, so a client that relies on them cannot use this endpoint yet. Request bodies are buffered rather than streamed, so a single PutObject is bounded by memory; multipart upload is the path for anything larger.

• An S3 key normalizes as a path. cat.jpg and cat.jpg/ name the same object, where S3 would treat them as two. S3 keys are opaque strings; a gfeh key is a path that the SMB and Drive views also walk, and keeping them distinct would need a node whose name ends in a separator. That is the price of one object being reachable everywhere.

• Drive has no OAuth flow. A bearer token is resolved to a principal through a table the daemon is configured with; minting one is the account system's business.

• The IPFS view does not join the swarm. No libp2p, bitswap, or DHT: content is reachable over the gateway and the api/v0 surface, and nothing here announces itself to the network. The identifiers are real -- computed by the same code the kubo cross-check holds to what ipfs add produces -- but reaching them means asking this server.

• The SMB view verifies NTLMv2 only when it is given a credential table. With smb.users configured, a client must prove possession of an account's NT hash against a nonce generated for that connection, and the session runs as that account's principal. With smb.users empty the exchange completes unchecked, the session is flagged a guest, and every session acts as the one configured principal -- so reaching the port is the only requirement. The second is a deliberate option for a trusted segment and a bad one anywhere else, and gfehd warns at startup when it is in use.

  Neither mode does a MIC check or derives a session key, so there is no signing or sealing in either.

• The daemon serves one partition. A partition is a subvolume with its own quota, index, and change sequence; serving several from one process would produce a feed whose page tokens mean different things depending on which partition moved last.

• The administrative surface is JSON over a Unix socket, not gRPC. The invariant -- admin reachable only through the filesystem, users only through the network -- holds either way, and this keeps cargo install gfehd free of a protoc dependency.

• SMB needs its own password. NTLM authentication requires a password hash that cannot be derived from the hash Town OS stores, so enabling SMB access means setting an SMB-specific credential. This is the same reason Samba keeps a separate password database.

• The Drive view is Drive-compatible, and very few clients can be pointed at it. This limitation is narrower than it first appears, so it is worth stating precisely. A client has to let you move the API endpoint, and most do not: rclone cannot, despite being the obvious candidate -- its Drive backend offers --drive-auth-url and --drive-token-url and no way to change where the API itself lives. Google's gdrive CLI is the same, and the Google Drive desktop application hard-codes Google's endpoints. What does work is Google's own client libraries, which accept an api_endpoint, and code you wrote yourself.

• Transparent-proxy subtrees have no content identifier. There is no local content to address, so the IPFS view cannot serve them.

• Content identifiers are computed lazily. Immediately after a write, the IPFS view may report that a file's identifier is not ready yet rather than serving a stale one.

• Per-user quotas inside a partition are enforced in software, not by btrfs.

License

gfeh is licensed under the GNU Affero General Public License v3.0 only.