# Qubit FS User Manual
## 1. What Qubit FS is
`qubit-fs` is a provider-neutral filesystem abstraction for Rust. It defines the common API surface for filesystems such as local filesystems, FTP, WebDAV, OSS, S3, HDFS, in-memory test filesystems, and private storage services.
The crate is intentionally only the core abstraction layer. It does not ship a concrete local, OSS, FTP, HDFS, or WebDAV implementation. Concrete implementations should live in separate crates, for example:
- `qubit-fs-local`
- `qubit-fs-webdav`
- `qubit-fs-oss`
- `qubit-fs-hdfs`
- `qubit-fs-memory`
This separation keeps `qubit-fs` extensible. Adding a new backend should not require modifying the root crate.
## 2. Installation
Add `qubit-fs` to `Cargo.toml`:
```toml
[dependencies]
qubit-fs = "0.1"
```
If you implement or register providers, you will usually also need `qubit-spi`:
```toml
[dependencies]
qubit-fs = "0.1"
qubit-spi = "0.3"
```
If your implementation stores rich metadata, use `qubit-metadata`:
```toml
[dependencies]
qubit-metadata = "0.5"
```
Optional feature:
```toml
[dependencies]
qubit-fs = { version = "0.1", features = ["registry-cache"] }
```
The current public API is synchronous and object-safe. Async providers can be built as separate crates or future extension traits, but the core `FileSystem` trait is intentionally not tied to a runtime.
## 3. Core concepts
### 3.1 `FsUri`
`FsUri` is the full resource locator. It contains:
- `scheme`: provider selector such as `file`, `oss`, `s3`, `webdav`, or `hdfs`
- `authority`: optional host, bucket, namespace, endpoint, or user hint
- `path`: provider-local `FsPath`
- `query`: non-sensitive options stored in `qubit_metadata::Metadata`
Example:
```rust
use qubit_fs::FsUri;
fn main() -> qubit_fs::FsResult<()> {
let uri = FsUri::parse("oss://my-bucket/reports/2026/a.csv?region=cn-hangzhou")?;
assert_eq!("oss", uri.scheme);
assert_eq!("/reports/2026/a.csv", uri.path.as_str());
let authority = uri.authority.expect("bucket should exist");
assert_eq!("my-bucket", authority.host);
Ok(())
}
```
Do not store passwords, access keys, secret keys, or tokens in URI strings. Use `CredentialRef` or provider-specific secure configuration.
### 3.2 `FsPath`
`FsPath` is the path inside one filesystem instance. It is not `std::path::Path`.
`FsPath` rules:
- uses UTF-8 strings
- uses `/` as the only separator
- can represent absolute and relative paths
- rejects empty paths
- rejects NUL bytes
- normalizes repeated `/` and `.`
- rejects paths that escape above their root with `..`
Example:
```rust
use qubit_fs::FsPath;
fn main() -> qubit_fs::FsResult<()> {
let path = FsPath::parse("/a//b/./c.txt")?;
assert_eq!("/a/b/c.txt", path.as_str());
assert_eq!(Some("c.txt"), path.file_name());
assert_eq!(Some("txt"), path.file_extension());
let parent = path.parent().expect("parent should exist");
assert_eq!("/a/b", parent.as_str());
let joined = parent.join("d.txt")?;
assert_eq!("/a/b/d.txt", joined.as_str());
Ok(())
}
```
Use `std::path::Path` only inside local filesystem providers. It is not a portable cross-provider path model.
### 3.3 `FileSystem`
`FileSystem` is the central provider-neutral trait:
```rust
use qubit_fs::{
CopyOptions, CopyOutcome, CreateDirOptions, DeleteOptions, DirectoryStream,
FileMetadata, FileReader, FileSystem, FileSystemMetadata, FileWriter, FsPath,
FsResult, ListOptions, ReadOptions, RenameOptions, WriteOptions,
};
pub trait FileSystem: std::fmt::Debug + Send + Sync {
fn metadata(&self) -> FileSystemMetadata;
fn path_metadata(&self, path: &FsPath) -> FsResult<FileMetadata>;
fn exists(&self, path: &FsPath) -> FsResult<bool>;
fn list(&self, path: &FsPath, options: &ListOptions) -> FsResult<Box<dyn DirectoryStream>>;
fn open_reader(&self, path: &FsPath, options: &ReadOptions) -> FsResult<Box<dyn FileReader>>;
fn open_writer(&self, path: &FsPath, options: &WriteOptions) -> FsResult<Box<dyn FileWriter>>;
fn create_dir(&self, path: &FsPath, options: &CreateDirOptions) -> FsResult<()>;
fn delete(&self, path: &FsPath, options: &DeleteOptions) -> FsResult<()>;
fn rename(&self, from: &FsPath, to: &FsPath, options: &RenameOptions) -> FsResult<()>;
fn copy(&self, from: &FsPath, to: &FsPath, options: &CopyOptions) -> FsResult<CopyOutcome>;
}
```
Implementations must not pretend all backends are POSIX filesystems. Unsupported operations should return `FsErrorKind::UnsupportedOperation` and capabilities should report the limitation.
### 3.4 Provider registry
`qubit-fs` uses `qubit-spi` for provider registration. It does not define a fixed enum like `FsKind`.
A provider declares an id and optional aliases. Example:
- provider id: `local`
- alias: `file`
- URI accepted by registry: `file:///tmp/a.txt`
This keeps the core crate open for third-party providers.
## 4. Opening filesystems and resources
A provider must be registered before a URI can be resolved. Application code
normally uses the process-wide `FileSystems` facade:
```rust
use qubit_fs::{FileSystems, FsResult};
fn configure_filesystems() -> FsResult<()> {
// Example only. The concrete provider type would come from a backend crate.
// FileSystems::register(LocalFileSystemProvider::new())?;
// FileSystems::register(OssFileSystemProvider::new())?;
Ok(())
}
```
Use `FileSystems::fs()` when you want the filesystem selected from a URI string:
```rust
use qubit_fs::{FileSystems, FsResult};
fn open_filesystem() -> FsResult<()> {
let fs = FileSystems::fs("file:///var/data/report.csv")?;
let caps = fs.capabilities();
println!("directories supported: {}", caps.directories);
Ok(())
}
```
If the URI has already been parsed, use `FileSystems::fs_for_uri()`.
If only a scheme is available, `FileSystems::fs_for_scheme()` resolves the
minimal URI `{scheme}:///`; this only works for providers that can be created
from default authority, root path, and default options.
Use `FileSystems::resource()` when you need both the filesystem and the
provider-local path. This is the common resource-oriented API:
```rust
use qubit_fs::{FileSystems, FsResult};
fn resolve_and_check() -> FsResult<bool> {
let resource = FileSystems::resource("oss://bucket/reports/a.csv")?;
resource.exists()
}
```
If the URI has already been parsed, use `FileSystems::resource_for_uri()`.
`FileSystemRegistry` is still available for isolated registries in tests,
plugins, or embedded runtimes. It only accepts parsed `FsUri` values:
```rust
use qubit_fs::{FileSystemRegistry, FsResult, FsUri};
fn isolated_registry() -> FsResult<()> {
let mut registry = FileSystemRegistry::new();
// registry.register(MemoryFileSystemProvider::new())?;
let uri = FsUri::parse("mem:///hello.txt")?;
let resource = registry.resource(&uri)?;
println!("{}", resource.path().as_str());
Ok(())
}
```
## 5. Common operations
### 5.1 Read all bytes
Use `FileSystemExt::read_all()` for small or medium resources that can fit in memory:
```rust
use qubit_fs::{FileSystem, FileSystemExt, FsPath, FsResult};
fn read_config(fs: &dyn FileSystem) -> FsResult<Vec<u8>> {
let path = FsPath::parse("/config/app.toml")?;
fs.read_all(&path)
}
```
For large files, use `open_reader()` and stream data yourself:
```rust
use std::io::Read;
use qubit_fs::{FileSystem, FsPath, FsResult, ReadOptions};
fn stream_read(fs: &dyn FileSystem) -> FsResult<Vec<u8>> {
let path = FsPath::parse("/large.bin")?;
let mut reader = fs.open_reader(&path, &ReadOptions::default())?;
let mut buf = Vec::new();
reader.read_to_end(&mut buf).map_err(|error| {
qubit_fs::FsError::with_source(
qubit_fs::FsErrorKind::Io,
qubit_fs::FsOperation::OpenReader,
"failed to read stream",
error,
).with_path(path.clone())
})?;
Ok(buf)
}
```
### 5.2 Range read
Providers that support `range_read` should honor `ReadOptions.offset` and `ReadOptions.length`:
```rust
use qubit_fs::{FileSystem, FsPath, FsResult, ReadOptions};
fn open_range(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/video.bin")?;
let options = ReadOptions {
offset: Some(1024),
length: Some(4096),
..ReadOptions::default()
};
let _reader = fs.open_reader(&path, &options)?;
Ok(())
}
```
If a backend cannot support range reads, it should either ignore the range only when documented as safe or return `UnsupportedOperation`. Prefer returning an explicit error when honoring the option is required by caller semantics.
### 5.3 Write all bytes
Use `FileSystemExt::write_all()` for simple writes:
```rust
use qubit_fs::{FileSystem, FileSystemExt, FsPath, FsResult};
fn write_report(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/reports/today.txt")?;
fs.write_all(&path, b"report content")?;
Ok(())
}
```
For advanced writes, use `open_writer()`, write bytes, then call `commit()`:
```rust
use std::io::Write;
use qubit_fs::{FileSystem, FsPath, FsResult, WriteMode, WriteOptions};
fn create_new(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/reports/new.txt")?;
let options = WriteOptions {
create_parent: true,
mode: WriteMode::CreateNew,
content_type: Some("text/plain".to_owned()),
..WriteOptions::default()
};
let mut writer = fs.open_writer(&path, &options)?;
if let Err(error) = writer.write_all(b"hello") {
let _ = writer.abort();
return Err(qubit_fs::FsError::with_source(
qubit_fs::FsErrorKind::Io,
qubit_fs::FsOperation::OpenWriter,
"failed to write data",
error,
).with_path(path));
}
writer.commit()?;
Ok(())
}
```
Always call either `commit()` or `abort()` for writers. Remote providers may allocate multipart uploads, temporary objects, or server sessions.
### 5.4 Conditional write
Use `WriteMode::ConditionalReplace` when replacing only a known version:
```rust
use qubit_fs::{FileSystem, FsPath, FsResult, WriteMode, WriteOptions};
fn replace_if_version_matches(fs: &dyn FileSystem, etag: String) -> FsResult<()> {
let path = FsPath::parse("/state.json")?;
let options = WriteOptions {
mode: WriteMode::ConditionalReplace { etag },
..WriteOptions::default()
};
let writer = fs.open_writer(&path, &options)?;
writer.commit()?;
Ok(())
}
```
Providers should return `FsErrorKind::PreconditionFailed` when the condition does not match.
### 5.5 Metadata and existence
```rust
use qubit_fs::{FileSystem, FileKind, FsPath, FsResult};
fn inspect(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/data/input.csv")?;
if !fs.exists(&path)? {
return Ok(());
}
let metadata = fs.path_metadata(&path)?;
match metadata.kind {
FileKind::File | FileKind::Object => println!("file-like resource"),
FileKind::Directory | FileKind::Prefix => println!("container-like resource"),
FileKind::Symlink => println!("symbolic link"),
FileKind::Other(_) => println!("provider-specific resource"),
}
if let Some(len) = metadata.len {
println!("size: {len}");
}
if let Some(etag) = metadata.etag {
println!("etag: {etag}");
}
Ok(())
}
```
`exists()` should not hide authentication, permission, timeout, or network errors. It should return `Ok(false)` only when absence is confirmed.
### 5.6 Listing
`list()` returns a `DirectoryStream`, so large remote listings can page internally:
```rust
use qubit_fs::{DirectoryStreamExt, FileSystem, FsPath, FsResult, ListOptions};
fn list_once(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/reports")?;
let options = ListOptions {
include_metadata: true,
page_size: Some(100),
..ListOptions::default()
};
let entries = fs.list(&path, &options)?.collect_entries()?;
for entry in entries {
println!("{}", entry.path);
}
Ok(())
}
```
For very large directories, consume the stream incrementally:
```rust
use qubit_fs::{FileSystem, FsPath, FsResult, ListOptions};
fn list_streaming(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/big-prefix")?;
let mut stream = fs.list(&path, &ListOptions::default())?;
while let Some(entry) = stream.next_entry()? {
println!("{}", entry.path);
}
Ok(())
}
```
### 5.7 Create directory or collection
```rust
use qubit_fs::{CreateDirOptions, FileSystem, FsPath, FsResult};
fn create_reports_dir(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/reports/2026")?;
fs.create_dir(
&path,
&CreateDirOptions {
recursive: true,
exists_ok: true,
..CreateDirOptions::default()
},
)?;
Ok(())
}
```
Object stores may model directories as prefixes or marker objects. Check `fs.capabilities().empty_directories` before relying on empty directory persistence.
### 5.8 Delete
```rust
use qubit_fs::{DeleteOptions, FileSystem, FsPath, FsResult};
fn delete_tree(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/tmp/job-123")?;
fs.delete(
&path,
&DeleteOptions {
recursive: true,
missing_ok: true,
..DeleteOptions::default()
},
)?;
Ok(())
}
```
`missing_ok = true` applies only to confirmed absence. It must not swallow permission or authentication errors.
### 5.9 Rename or move
```rust
use qubit_fs::{AtomicityRequirement, FileSystem, FsPath, FsResult, RenameOptions};
fn atomic_publish(fs: &dyn FileSystem) -> FsResult<()> {
let tmp = FsPath::parse("/out/report.tmp")?;
let final_path = FsPath::parse("/out/report.csv")?;
fs.rename(
&tmp,
&final_path,
&RenameOptions {
overwrite: true,
atomic: AtomicityRequirement::Required,
},
)?;
Ok(())
}
```
When `atomic = Required`, providers must fail with `UnsupportedOperation` if atomic rename cannot be guaranteed. Object stores usually cannot provide true atomic rename.
## 6. Copy model
`copy()` handles file-like, object-like, directory tree, prefix tree, or collection copy within one filesystem instance.
### 6.1 Copy one file or object
```rust
use qubit_fs::{CopyConflictPolicy, CopyOptions, FileSystem, FsPath, FsResult};
fn copy_file(fs: &dyn FileSystem) -> FsResult<()> {
let from = FsPath::parse("/input/a.csv")?;
let to = FsPath::parse("/archive/a.csv")?;
let mut options = CopyOptions::file();
options.conflict = CopyConflictPolicy::Overwrite;
options.create_parent = true;
let outcome = fs.copy(&from, &to, &options)?;
println!("copied bytes: {}", outcome.stats.bytes);
println!("method: {:?}", outcome.method);
Ok(())
}
```
### 6.2 Copy a tree
```rust
use qubit_fs::{CopyOptions, FileSystem, FsPath, FsResult, MetadataPreservePolicy};
fn copy_tree(fs: &dyn FileSystem) -> FsResult<()> {
let from = FsPath::parse("/dataset")?;
let to = FsPath::parse("/backup/dataset")?;
let mut options = CopyOptions::tree();
options.create_parent = true;
options.preserve_metadata = MetadataPreservePolicy::Portable;
options.continue_on_error = false;
let outcome = fs.copy(&from, &to, &options)?;
println!("files: {}", outcome.stats.files);
println!("directories: {}", outcome.stats.directories);
println!("objects: {}", outcome.stats.objects);
println!("prefixes: {}", outcome.stats.prefixes);
Ok(())
}
```
### 6.3 Require server-side copy
```rust
use qubit_fs::{CopyOptions, FileSystem, FsPath, FsResult, ServerSidePreference};
fn server_side_copy_only(fs: &dyn FileSystem) -> FsResult<()> {
let from = FsPath::parse("/a.bin")?;
let to = FsPath::parse("/b.bin")?;
let mut options = CopyOptions::file();
options.server_side = ServerSidePreference::Require;
fs.copy(&from, &to, &options)?;
Ok(())
}
```
If server-side copy is not supported, the provider should return `FsErrorKind::UnsupportedOperation`.
### 6.4 Cross-filesystem copy
`FileSystem::copy()` copies within one filesystem instance. Cross-provider copy should be implemented at a higher layer by resolving both URIs as `FileResource` values and streaming from source to destination.
```rust
use std::io::{Read, Write};
use qubit_fs::{FileSystems, FsError, FsErrorKind, FsOperation, FsResult, ReadOptions, WriteOptions};
fn copy_between(from_uri: &str, to_uri: &str) -> FsResult<()> {
let from = FileSystems::resource(from_uri)?;
let to = FileSystems::resource(to_uri)?;
let mut reader = from.open_reader(&ReadOptions::default())?;
let mut writer = to.open_writer(&WriteOptions::default())?;
let mut buffer = [0_u8; 64 * 1024];
loop {
let n = reader.read(&mut buffer).map_err(|error| {
FsError::with_source(FsErrorKind::Io, FsOperation::OpenReader, "read failed", error)
.with_path(from.path().clone())
})?;
if n == 0 {
break;
}
if let Err(error) = writer.write_all(&buffer[..n]) {
let _ = writer.abort();
return Err(FsError::with_source(
FsErrorKind::Io,
FsOperation::OpenWriter,
"write failed",
error,
).with_path(to.path().clone()));
}
}
writer.commit()?;
Ok(())
}
```
A production cross-filesystem copier should also support checksums, progress reporting, metadata preservation, cancellation, retry, and cleanup on failure.
## 7. Temporary resources
Temporary resources model ownership and cleanup responsibility. They are useful because they provide `Drop` best-effort cleanup in addition to explicit methods.
### 7.1 Create temporary files and directories
```rust
use std::sync::Arc;
use qubit_fs::{
FileSystem,
FileSystemExt,
FsPath,
FsResult,
PersistOptions,
TempDir,
TempDirOptions,
TempFile,
TempFileOptions,
TempResources,
};
fn temp_file_publish(fs: Arc<dyn FileSystem>) -> FsResult<()> {
let temp: Box<dyn TempFile> =
TempResources::create_file(fs.clone(), &TempFileOptions::default())?;
let staging_path = temp.path().clone();
fs.write_all(&staging_path, b"generated report\n")?;
let target = FsPath::parse("/published/final.txt")?;
temp.persist(&target, &PersistOptions::default())?;
Ok(())
}
fn temp_dir_workspace(fs: Arc<dyn FileSystem>) -> FsResult<()> {
let workspace: Box<dyn TempDir> =
TempResources::create_dir(fs.clone(), &TempDirOptions::default())?;
let part_file = workspace.path().join("part-0001.csv")?;
fs.write_all(&part_file, b"id,value\n1,42\n")?;
let target = FsPath::parse("/published/report-parts")?;
workspace.persist(&target, &PersistOptions::default())?;
Ok(())
}
```
`TempResources::create_file(fs, options)` works as follows:
- It first asks the current `FileSystem` instance for `fs.temp_resource_factory()`.
- It calls `create_file(fs, options)` on that factory, allowing the filesystem to return either a native `TempFile` or a core fallback.
- The default factory is `ManagedTempResourceFactory`: it generates a temporary path, reserves an empty file with `open_writer(..., CreateNew)`, and returns `ManagedTempFile`.
- Factory implementations may reuse `TempResourceFactory::make_temp_path()` for the common naming format, or use provider-specific naming rules.
`TempResources::create_dir(fs, options)` follows the same rule:
- It first asks the current `FileSystem` instance for `fs.temp_resource_factory()`.
- It calls `create_dir(fs, options)` on that factory, allowing the filesystem to return either a native `TempDir` or a core fallback.
- The default factory is `ManagedTempResourceFactory`: it generates a temporary path, creates the directory with `create_dir(..., recursive=true)`, and returns `ManagedTempDir`.
- Factory implementations may reuse `TempResourceFactory::make_temp_path()` for the common naming format, or use provider-specific naming rules.
`TempResources` also provides convenience helpers:
```rust
let file1 = TempResources::create_default_file(fs.clone())?;
let file2 = TempResources::create_file_with_prefix(fs.clone(), "upload-")?;
let dir1 = TempResources::create_default_dir(fs.clone())?;
let dir2 = TempResources::create_dir_with_prefix(fs.clone(), "job-")?;
```
`TempFile` and `TempDir` share the same usage pattern:
- Create a cleanup-owning handle with `TempResources::create_file()` or `TempResources::create_dir()`.
- Get the temporary path with `path()`, then write content or child files through normal `FileSystem` APIs.
- Call `persist()` to publish the resource to its final target.
- If the resource should not be published, call `cleanup()` explicitly or call `keep()` to disable automatic cleanup and transfer the temporary path to another component.
`ManagedTempFile` and `ManagedTempDir` are the fallback default implementations. They use the underlying filesystem to reserve temporary paths. If the handle is dropped before `cleanup()`, `persist()`, or `keep()`, it attempts best-effort cleanup.
### 7.2 Custom temporary file name
```rust
use std::sync::Arc;
use qubit_fs::{FileSystem, FsPath, FsResult, TempFileOptions, TempResources};
fn create_named_temp(fs: Arc<dyn FileSystem>) -> FsResult<()> {
let parent = FsPath::parse("/tmp")?;
let options = TempFileOptions {
parent: Some(parent),
prefix: "upload-".to_owned(),
suffix: ".part".to_owned(),
};
let temp = TempResources::create_file(fs, &options)?;
println!("temporary path: {}", temp.path());
temp.cleanup()?;
Ok(())
}
```
### 7.3 Keep a temporary resource
```rust
use std::sync::Arc;
use qubit_fs::{FileSystem, FsResult, TempDirOptions, TempResources};
fn keep_temp_dir(fs: Arc<dyn FileSystem>) -> FsResult<()> {
let temp = TempResources::create_dir(fs, &TempDirOptions::default())?;
let retained_path = temp.keep()?;
println!("kept temp dir at {}", retained_path);
Ok(())
}
```
`keep()` disables automatic cleanup and returns the temporary path. Use it when ownership is intentionally transferred to another component.
### 7.4 Persist with copy-delete fallback
```rust
use std::sync::Arc;
use qubit_fs::{AtomicityRequirement, FileSystem, FsPath, FsResult, PersistOptions, TempResources};
fn persist_with_fallback(fs: Arc<dyn FileSystem>) -> FsResult<()> {
let temp = TempResources::create_file(fs, &Default::default())?;
let target = FsPath::parse("/final/object.bin")?;
let options = PersistOptions {
overwrite: true,
atomic: AtomicityRequirement::BestEffort,
allow_copy_delete: true,
..PersistOptions::default()
};
temp.persist(&target, &options)?;
Ok(())
}
```
If `rename()` returns `UnsupportedOperation` and `allow_copy_delete` is true, `ManagedTempFile` can fall back to `copy()` followed by `delete()`.
## 8. Errors
Most operations return `FsResult<T>`, which is an alias for `Result<T, FsError>`.
Important error kinds:
| `NotFound` | The provider confirmed the resource does not exist. |
| `AlreadyExists` | Creation failed because the target exists. |
| `InvalidPath` | Path or URI is invalid. |
| `PermissionDenied` | The identity is known but lacks permission. |
| `AuthenticationFailed` | Credentials are missing or invalid. |
| `ProviderUnavailable` | No matching provider is registered or the provider cannot run. |
| `UnsupportedOperation` | The backend model does not support the operation. |
| `Conflict` | State conflict, such as deleting a non-empty directory without recursion. |
| `PreconditionFailed` | ETag or version condition failed. |
| `Timeout` | Provider operation timed out. |
| `QuotaExceeded` | Quota or capacity limit exceeded. |
| `DataCorruption` | Checksum or data integrity validation failed. |
| `Io` | Local or stream I/O error. |
| `Other` | Provider-specific fallback. |
Example:
```rust
use qubit_fs::{FileSystem, FsErrorKind, FsPath, FsResult};
fn delete_if_supported(fs: &dyn FileSystem) -> FsResult<()> {
let path = FsPath::parse("/old")?;
match fs.delete(&path, &Default::default()) {
Ok(()) => Ok(()),
Err(error) if error.kind() == FsErrorKind::UnsupportedOperation => {
println!("delete is not supported by this backend");
Ok(())
}
Err(error) => Err(error),
}
}
```
Provider implementations should preserve source errors with `FsError::with_source()` and attach path, target, and provider context when possible.
## 9. Metadata model
`FileMetadata` contains stable cross-provider fields and extension metadata.
Common fields include:
- `kind`: `File`, `Directory`, `Symlink`, `Object`, `Prefix`, or provider-specific `Other`
- `len`: optional byte length
- `modified_at`, `created_at`, `accessed_at`: optional timestamps
- `etag`: optional provider version or entity tag
- `content_type`: optional media type
- `checksum`: optional checksum
- `user_metadata`: user-defined metadata
- `provider_metadata`: backend-specific metadata
Example:
```rust
use qubit_fs::{FileMetadata, FileKind};
fn is_container(metadata: &FileMetadata) -> bool {
metadata.is_directory_like()
|| matches!(metadata.kind, FileKind::Directory | FileKind::Prefix)
}
```
Use `provider_metadata` for fields that are meaningful only for one backend, such as HDFS replication, OSS storage class, FTP permission text, or WebDAV custom properties.
## 10. Capabilities
Always inspect capabilities before depending on non-basic behavior:
```rust
use qubit_fs::FileSystem;
fn explain_capabilities(fs: &dyn FileSystem) {
let caps = fs.capabilities();
println!("directories: {}", caps.directories);
println!("range read: {}", caps.range_read);
println!("append: {}", caps.append);
println!("atomic rename: {}", caps.atomic_rename);
println!("server-side copy: {}", caps.server_side_copy);
println!("temporary files: {}", caps.temp_file);
}
```
Capabilities are not authorization. A provider may support recursive delete in general, while the current user still lacks permission for a specific path.
## 11. Implementing a new backend
This section shows how to build a new provider crate. The example is an in-memory filesystem because it is small enough to show the moving parts. Real providers follow the same shape.
### 11.1 Suggested crate layout
```text
rs-fs-memory/
Cargo.toml
src/
lib.rs
memory_file_system.rs
memory_provider.rs
memory_reader.rs
memory_writer.rs
memory_directory_stream.rs
error_mapping.rs
tests/
memory_file_system_tests.rs
```
`Cargo.toml`:
```toml
[package]
name = "qubit-fs-memory"
version = "0.1.0"
edition = "2024"
[dependencies]
qubit-fs = "0.1"
qubit-spi = "0.3"
qubit-metadata = "0.5"
```
### 11.2 Define the filesystem type
```rust
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
#[derive(Debug, Default, Clone)]
pub struct MemoryFileSystem {
state: Arc<Mutex<MemoryState>>,
}
#[derive(Debug, Default)]
struct MemoryState {
files: BTreeMap<String, Vec<u8>>,
}
```
A production backend would hold an SDK client, connection pool, root configuration, credential handle, and provider-specific options.
### 11.3 Implement metadata and capabilities
```rust
use qubit_fs::{
FileMetadata, FileSystem, FileSystemCapabilities, FileSystemMetadata, FileKind,
FsError, FsErrorKind, FsOperation, FsPath, FsResult,
};
impl FileSystem for MemoryFileSystem {
fn metadata(&self) -> FileSystemMetadata {
let mut metadata = FileSystemMetadata::new("memory");
metadata.schemes.push("mem".to_owned());
metadata.capabilities = FileSystemCapabilities {
hierarchical_paths: true,
directories: true,
empty_directories: true,
symlinks: false,
range_read: true,
append: false,
random_write: false,
atomic_rename: true,
atomic_replace: true,
conditional_write: false,
server_side_copy: false,
recursive_delete: true,
temp_file: true,
temp_dir: true,
temp_persist: true,
temp_persist_atomic: true,
native_metadata: false,
};
metadata
}
fn path_metadata(&self, path: &FsPath) -> FsResult<FileMetadata> {
let state = self.state.lock().expect("memory state should not be poisoned");
let Some(bytes) = state.files.get(path.as_str()) else {
return Err(FsError::new(FsErrorKind::NotFound, FsOperation::Metadata, "file not found")
.with_path(path.clone())
.with_provider("memory"));
};
let mut metadata = FileMetadata::new(FileKind::File);
metadata.len = Some(bytes.len() as u64);
Ok(metadata)
}
fn exists(&self, path: &FsPath) -> FsResult<bool> {
let state = self.state.lock().expect("memory state should not be poisoned");
Ok(state.files.contains_key(path.as_str()))
}
// Other trait methods are shown below.
}
```
Capability rules:
- report backend model capabilities, not current permissions
- return `UnsupportedOperation` when callers request unsupported behavior
- do not silently downgrade operations when the option says `Required`
### 11.4 Implement reading
Because `FileReader` has a blanket implementation for `Read + Send`, a `Cursor<Vec<u8>>` can be returned directly.
```rust
use std::io::Cursor;
use qubit_fs::{FileReader, ReadOptions};
impl MemoryFileSystem {
fn read_bytes(&self, path: &FsPath, options: &ReadOptions) -> FsResult<Vec<u8>> {
let state = self.state.lock().expect("memory state should not be poisoned");
let Some(bytes) = state.files.get(path.as_str()) else {
return Err(FsError::new(FsErrorKind::NotFound, FsOperation::OpenReader, "file not found")
.with_path(path.clone())
.with_provider("memory"));
};
let start = options.offset.unwrap_or(0) as usize;
let end = match options.length {
Some(length) => start.saturating_add(length as usize).min(bytes.len()),
None => bytes.len(),
};
Ok(bytes.get(start..end).unwrap_or_default().to_vec())
}
}
impl FileSystem for MemoryFileSystem {
fn open_reader(&self, path: &FsPath, options: &ReadOptions) -> FsResult<Box<dyn FileReader>> {
let bytes = self.read_bytes(path, options)?;
Ok(Box::new(Cursor::new(bytes)))
}
// Other methods omitted in this excerpt.
}
```
For real remote providers, map HTTP, SDK, FTP, or HDFS errors into `FsErrorKind` and keep the original error as `source` when safe.
### 11.5 Implement writing
Writers must support `commit()` and `abort()`.
```rust
use std::io::{Result as IoResult, Write};
use qubit_fs::{FileWriter, WriteOutcome};
#[derive(Debug)]
struct MemoryWriter {
fs: MemoryFileSystem,
path: FsPath,
buffer: Vec<u8>,
}
impl Write for MemoryWriter {
fn write(&mut self, bytes: &[u8]) -> IoResult<usize> {
self.buffer.extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> IoResult<()> {
Ok(())
}
}
impl FileWriter for MemoryWriter {
fn commit(self: Box<Self>) -> FsResult<WriteOutcome> {
let mut state = self.fs.state.lock().expect("memory state should not be poisoned");
let bytes_written = self.buffer.len() as u64;
state.files.insert(self.path.as_str().to_owned(), self.buffer);
Ok(WriteOutcome {
bytes_written: Some(bytes_written),
etag: None,
diagnostics: qubit_metadata::Metadata::new(),
})
}
fn abort(self: Box<Self>) -> FsResult<()> {
Ok(())
}
}
```
Then wire it into `open_writer()`:
```rust
use qubit_fs::{WriteMode, WriteOptions};
impl FileSystem for MemoryFileSystem {
fn open_writer(&self, path: &FsPath, options: &WriteOptions) -> FsResult<Box<dyn FileWriter>> {
if matches!(options.mode, WriteMode::Append) {
return Err(FsError::new(
FsErrorKind::UnsupportedOperation,
FsOperation::OpenWriter,
"append is not supported",
).with_path(path.clone()).with_provider("memory"));
}
Ok(Box::new(MemoryWriter {
fs: self.clone(),
path: path.clone(),
buffer: Vec::new(),
}))
}
// Other methods omitted in this excerpt.
}
```
Real object storage writers typically start a multipart upload, buffer parts, commit the upload in `commit()`, and abort the upload in `abort()` or `Drop`.
### 11.6 Implement listing
```rust
use qubit_fs::{DirEntry, DirectoryStream, ListOptions};
#[derive(Debug)]
struct MemoryDirectoryStream {
entries: Vec<DirEntry>,
}
impl DirectoryStream for MemoryDirectoryStream {
fn next_entry(&mut self) -> FsResult<Option<DirEntry>> {
Ok(self.entries.pop())
}
}
impl FileSystem for MemoryFileSystem {
fn list(&self, path: &FsPath, _options: &ListOptions) -> FsResult<Box<dyn DirectoryStream>> {
let state = self.state.lock().expect("memory state should not be poisoned");
let prefix = if path.as_str() == "/" {
"/".to_owned()
} else {
format!("{}/", path.as_str().trim_end_matches('/'))
};
let entries = state.files.keys()
.filter(|key| key.starts_with(&prefix))
.filter_map(|key| FsPath::parse(key).ok())
.map(|path| DirEntry::new(path, FileKind::File))
.collect();
Ok(Box::new(MemoryDirectoryStream { entries }))
}
// Other methods omitted in this excerpt.
}
```
Remote providers should fetch pages lazily inside `next_entry()` instead of loading everything upfront when directories may be large.
### 11.7 Implement delete, rename, and copy
```rust
use qubit_fs::{CopyMethod, CopyOptions, CopyOutcome, CopyStats, DeleteOptions, RenameOptions};
impl FileSystem for MemoryFileSystem {
fn delete(&self, path: &FsPath, options: &DeleteOptions) -> FsResult<()> {
let mut state = self.state.lock().expect("memory state should not be poisoned");
let existed = state.files.remove(path.as_str()).is_some();
if !existed && !options.missing_ok {
return Err(FsError::new(FsErrorKind::NotFound, FsOperation::Delete, "file not found")
.with_path(path.clone())
.with_provider("memory"));
}
Ok(())
}
fn rename(&self, from: &FsPath, to: &FsPath, options: &RenameOptions) -> FsResult<()> {
let mut state = self.state.lock().expect("memory state should not be poisoned");
if !options.overwrite && state.files.contains_key(to.as_str()) {
return Err(FsError::new(FsErrorKind::AlreadyExists, FsOperation::Rename, "target exists")
.with_path(from.clone())
.with_target(to.clone())
.with_provider("memory"));
}
let Some(bytes) = state.files.remove(from.as_str()) else {
return Err(FsError::new(FsErrorKind::NotFound, FsOperation::Rename, "source not found")
.with_path(from.clone())
.with_target(to.clone())
.with_provider("memory"));
};
state.files.insert(to.as_str().to_owned(), bytes);
Ok(())
}
fn copy(&self, from: &FsPath, to: &FsPath, options: &CopyOptions) -> FsResult<CopyOutcome> {
let mut state = self.state.lock().expect("memory state should not be poisoned");
if !options.conflict.eq(&qubit_fs::CopyConflictPolicy::Overwrite)
&& state.files.contains_key(to.as_str())
{
return Err(FsError::new(FsErrorKind::AlreadyExists, FsOperation::Copy, "target exists")
.with_path(from.clone())
.with_target(to.clone())
.with_provider("memory"));
}
let Some(bytes) = state.files.get(from.as_str()).cloned() else {
return Err(FsError::new(FsErrorKind::NotFound, FsOperation::Copy, "source not found")
.with_path(from.clone())
.with_target(to.clone())
.with_provider("memory"));
};
let len = bytes.len() as u64;
state.files.insert(to.as_str().to_owned(), bytes);
Ok(CopyOutcome::new(
CopyStats { files: 1, bytes: len, ..CopyStats::default() },
CopyMethod::Local,
))
}
// Other methods omitted in this excerpt.
}
```
For object stores, `rename()` is usually copy plus delete and should not claim atomic rename. For WebDAV, `MOVE` may be used. For HDFS, rename may be a strong commit primitive.
### 11.8 Implement provider registration
A provider creates filesystem instances from `FileSystemConfig`.
```rust
use qubit_fs::{FileSystem, FileSystemConfig, FileSystemSpec};
use qubit_spi::{ProviderCreateError, ProviderDescriptor, ProviderRegistryError, ServiceProvider};
#[derive(Debug, Default)]
pub struct MemoryFileSystemProvider;
impl ServiceProvider<FileSystemSpec> for MemoryFileSystemProvider {
fn descriptor(&self) -> Result<ProviderDescriptor, ProviderRegistryError> {
ProviderDescriptor::new("memory")?.with_aliases(&["mem"])
}
fn create_box(
&self,
_config: &FileSystemConfig,
) -> Result<Box<dyn FileSystem>, ProviderCreateError> {
Ok(Box::new(MemoryFileSystem::default()))
}
}
pub fn register_provider(registry: &mut qubit_fs::FileSystemRegistry) -> qubit_fs::FsResult<()> {
registry.register(MemoryFileSystemProvider)
}
```
Application usage:
```rust
use qubit_fs::{FileSystemRegistry, FsResult, FsUri};
fn main() -> FsResult<()> {
let mut registry = FileSystemRegistry::new();
qubit_fs_memory::register_provider(&mut registry)?;
let uri = FsUri::parse("mem:///hello.txt")?;
let resource = registry.resource(&uri)?;
resource.write_all(b"hello")?;
let bytes = resource.read_all()?;
assert_eq!(b"hello", bytes.as_slice());
Ok(())
}
```
### 11.9 Error mapping checklist for providers
When implementing a real backend, map native errors consistently:
| 404, missing inode, missing object | `NotFound` |
| destination exists on create-new | `AlreadyExists` |
| invalid key, invalid path, invalid URI | `InvalidPath` |
| 401 or invalid credentials | `AuthenticationFailed` |
| 403 or ACL denial | `PermissionDenied` |
| 405 unsupported method, unsupported SDK operation | `UnsupportedOperation` |
| 409 conflict | `Conflict` |
| 412 ETag mismatch | `PreconditionFailed` |
| 423 locked resource | `Conflict` or provider-specific mapping |
| 429 or service throttle | `ProviderUnavailable` or `Timeout`, depending on semantics |
| checksum mismatch | `DataCorruption` |
| local stream error | `Io` |
Attach context:
```rust
use qubit_fs::{FsError, FsErrorKind, FsOperation, FsPath};
fn map_backend_error(path: &FsPath, source: std::io::Error) -> FsError {
FsError::with_source(
FsErrorKind::Io,
FsOperation::OpenReader,
"backend read failed",
source,
)
.with_path(path.clone())
.with_provider("my-provider")
}
```
### 11.10 Capability checklist for providers
Before publishing a provider, decide and test every capability:
| `hierarchical_paths` | Are paths true directories or lexical keys? |
| `directories` | Can the backend represent directories or collections? |
| `empty_directories` | Can an empty directory survive without children? |
| `symlinks` | Can symlinks be read, copied, or created? |
| `range_read` | Can byte ranges be read efficiently? |
| `append` | Is append native and safe? |
| `random_write` | Can arbitrary offsets be modified? |
| `atomic_rename` | Is rename atomic for the relevant namespace? |
| `atomic_replace` | Can replacement avoid partially visible content? |
| `conditional_write` | Are ETag, generation, or version preconditions supported? |
| `server_side_copy` | Can the backend copy without streaming through the client? |
| `recursive_delete` | Can tree deletion be handled safely? |
| `temp_file` | Can temporary file reservation be implemented? |
| `temp_dir` | Can temporary directory reservation be implemented? |
| `temp_persist` | Can temp resources be persisted to final paths? |
| `temp_persist_atomic` | Can persistence be atomic? |
| `native_metadata` | Does the backend expose native metadata worth preserving? |
## 12. WebDAV provider guidance
A WebDAV provider should map WebDAV methods to `FileSystem` operations:
| `metadata` | `PROPFIND Depth: 0` |
| `list` | `PROPFIND Depth: 1` or deeper paging strategy |
| `open_reader` | `GET` |
| `open_writer` | `PUT` staged until commit when possible |
| `create_dir` | `MKCOL` |
| `delete` | `DELETE` |
| `rename` | `MOVE` |
| `copy` | `COPY` |
Important WebDAV details:
- `ETag` should map to `FileMetadata.etag`.
- `Content-Type` should map to `FileMetadata.content_type`.
- WebDAV properties that have no stable cross-provider field should go to `provider_metadata`.
- `Depth` behavior must be documented because servers differ.
- `MOVE` and `COPY` may be best-effort and not atomic.
- HTTP 401, 403, 404, 405, 409, 412, 423, and 507 should be mapped carefully.
Provider skeleton:
```rust
#[derive(Debug)]
pub struct WebDavFileSystem {
endpoint: String,
client: reqwest::blocking::Client,
}
impl FileSystem for WebDavFileSystem {
fn metadata(&self) -> FileSystemMetadata {
let mut metadata = FileSystemMetadata::new("webdav");
metadata.schemes.push("webdav".to_owned());
metadata.schemes.push("webdavs".to_owned());
metadata.capabilities.directories = true;
metadata.capabilities.empty_directories = true;
metadata.capabilities.server_side_copy = true;
metadata
}
// Map PROPFIND, GET, PUT, MKCOL, DELETE, MOVE, and COPY here.
}
```
## 13. Local provider guidance
A local provider should keep `std::path::PathBuf` inside the provider and expose only `FsPath` through the core API.
Recommended local provider rules:
- configure a root directory
- map every `FsPath` under that root
- reject paths that escape the root
- handle symlink policy explicitly
- use atomic replace for `WriteMode::ReplaceAtomic`
- map `std::io::ErrorKind` to `FsErrorKind`
- use `qubit-local-files` for atomic writes, recursive copy, and local temporary resources where appropriate
Example mapping helper:
```rust
use std::path::{Path, PathBuf};
use qubit_fs::{FsError, FsErrorKind, FsOperation, FsPath, FsResult};
#[derive(Debug)]
pub struct LocalFileSystem {
root: PathBuf,
}
impl LocalFileSystem {
fn to_local_path(&self, path: &FsPath) -> FsResult<PathBuf> {
let relative = path.as_str().trim_start_matches('/');
let candidate = self.root.join(relative);
if candidate.components().any(|component| matches!(component, std::path::Component::ParentDir)) {
return Err(FsError::new(
FsErrorKind::InvalidPath,
FsOperation::ParsePath,
"path escapes the local root",
).with_path(path.clone()).with_provider("local"));
}
Ok(candidate)
}
}
```
## 14. Object storage provider guidance
Object stores such as OSS and S3 should not pretend to be full POSIX filesystems.
Recommended mapping:
| `FsPath` | object key with normalized `/` separators |
| directory | prefix view or optional marker object |
| empty directory | usually unsupported unless marker objects are used |
| `metadata` | HEAD object or list result enrichment |
| `open_reader` | GET object, optionally range GET |
| `open_writer` | PUT object or multipart upload |
| `rename` | copy plus delete, not atomic |
| `copy` | server-side copy when possible |
| `etag` | provider ETag or generation/version id |
| user metadata | provider object metadata |
Key rules:
- `atomic_rename` should usually be false.
- `server_side_copy` should be true if object copy is supported.
- `WriteMode::ConditionalReplace` should map to generation or ETag preconditions if available.
- `TempResources` can work with object stores if `open_writer(CreateNew)` and `delete()` are implemented.
- Multipart upload must be aborted if writing fails.
## 15. Testing a provider
Provider test suites should cover at least:
1. URI parsing and provider registration.
2. Basic write, read, metadata, exists, list, delete.
3. Error mapping for not found, already exists, permission denied, authentication failure, and unsupported operation.
4. `WriteMode::CreateNew`, `CreateOrTruncate`, `ReplaceAtomic`, and provider-supported conditional modes.
5. `ReadOptions` range behavior if `range_read` is true.
6. `RenameOptions.atomic = Required` behavior.
7. `CopyOptions` conflict policies and server-side preference.
8. `TempResources::create_file()` and `create_dir()` if capabilities claim support.
9. `ManagedTempFile::persist()`, `cleanup()`, and `keep()` behavior.
10. Large listing behavior, including paging.
11. Metadata preservation and provider metadata fields.
12. Cleanup after failed writes or failed copy-delete fallback.
A minimal contract test can be written against `Arc<dyn FileSystem>` so the same tests can be reused by local, WebDAV, OSS, and in-memory providers.
## 16. Best practices
Use these rules when consuming or implementing `qubit-fs`:
| Path | Use `FsPath` in public filesystem APIs. Convert to backend-native paths internally. |
| URI | Use `FsUri` only for locating providers and initial configuration. Do not pass full URI strings into every operation. |
| Secrets | Do not put secrets in `FsUri.query`. Use `CredentialRef` or secure provider configuration. |
| Capabilities | Check capabilities before depending on advanced behavior. |
| Unsupported behavior | Return `UnsupportedOperation` rather than silently pretending support. |
| Writing | Always commit or abort writers. |
| Temporary resources | Use `cleanup()`, `persist()`, or `keep()` to make ownership explicit. |
| Copy | Use `CopyOutcome.method` and `CopyStats` for audit and diagnostics. |
| Errors | Preserve operation, path, target, provider, and source error where safe. |
| Provider crates | Keep concrete implementations outside `qubit-fs`. Register through `qubit-spi`. |
## 17. Current limitations
Current `qubit-fs` provides the core abstraction. It does not yet provide:
- a built-in local filesystem provider
- a built-in WebDAV provider
- built-in OSS/S3/HDFS providers
- async traits
- global automatic provider discovery
- cross-provider copy orchestration
- credential resolution implementation
These are expected to be implemented in separate crates or future extension layers.
## 18. Recommended adoption path
For application developers:
1. Add `qubit-fs` and one or more provider crates.
2. Create a `FileSystemRegistry`.
3. Register provider crates explicitly.
4. Resolve URI strings with `FileSystems::resource()`, or parsed `FsUri` values with `FileSystemRegistry::resource()`.
5. Use `FileResource` for resource-oriented operations or `FileSystem` with `FsPath` for lower-level code.
6. Check capabilities before relying on advanced behavior.
7. Treat errors by `FsErrorKind`, not provider-native error types.
For provider authors:
1. Implement `FileSystem` for your backend.
2. Implement `FileReader`, `FileWriter`, and `DirectoryStream` as needed.
3. Report accurate `FileSystemCapabilities`.
4. Map native errors to `FsErrorKind`.
5. Implement `ServiceProvider<FileSystemSpec>`.
6. Expose a `register_provider()` helper.
7. Add contract tests that run against `Arc<dyn FileSystem>`.
8. Document backend-specific semantics, especially atomicity, directory behavior, metadata, and credentials.