[−][src]Crate scfs

SCFS – SplitCatFS

A convenient splitting and concatenating filesystem.

While setting up a cloud based backup and archive solution, I encountered the following phenomenon: Many small files would get uploaded quite fast and – depending on the actual cloud storage provider – highly concurrently, while big files tend to slow down the whole process. The explanation is simple, most cloud storage providers do not support the upload of a single file, sometimes they would not even support resuming a partial upload. You would need to upload it in one go, sequentially byte for byte, it's all or nothing.

Now consider a scenario, where you upload a really big file, like a mirror of your Raspberry Pi's SD card with the system and configuration on it. I have such a file, it is about 4 GB big. Now, while backing up my system, this was the last file to be uploaded. According to ETA calculations, it would have taken several hours, so I let it run overnight. The next morning I found out that after around 95% of upload process, my internet connection vanished for just a few seconds, but long enough that the transfer tool aborted the upload. The temporary file got deleted from the cloud storage, so I had to start from zero again. Several hours of uploading wasted.

I thought of a way to split big files, so that I can upload it more efficiently, but I came to the conclusion, that manually splitting files, uploading them, and deleting them afterwards locally, is not a very scalable solution.

So I came up with the idea of a special filesystem. A filesystem that would present big files as if they were many small chunks in separate files. In reality, the chunks would all point to the same physical file, only with different offsets. This way I could upload chunked files in parallel without losing too much progress, even if the upload gets aborted midway.

SplitFS was born.

If I download such chunked file parts, I would need to call cat * >file afterwards to re-create the actual file. This seems like a similar hassle like manually splitting files. That's why I had also CatFS in mind, when developing SCFS. CatFS will concatenate chunked files transparently and present them as a complete files.

CatFS is included in SCFS since version 0.4.0.

Why Rust?

I am relatively new to Rust and I thought, the best way to deepen my understanding with Rust is to take on a project that would require dedication and a certain knowledge of the language.

Installation

SCFS can be installed easily through Cargo via crates.io:

cargo install scfs

Usage

SplitFS

To mount a directory with SplitFS, use the following form:

scfs --mode=split <base directory> <mount point>

The directory specified as mount point will now reflect the content of base directory, replacing each regular file with a directory that contains enumerated chunks of that file as separate files.

CatFS

To mount a directory with CatFS, use the following form:

scfs --mode=cat <base directory> <mount point>

Please note that base directory needs to be a directory structure that has been generated by SplitFS.

The directory specified as mount point will now reflect the content of base directory, replacing each directory with chunked files in it as single files.

Limitations

Please be aware that this project is merely a raw prototype for now! Specifically:

It only works on Linux for now, maybe even on UNIX. But definitely not on Windows or MacOS.
It can only work with directories and regular files. Every other file type will be ignored or may end end up in a panic!.
The base directory will be mounted read-only in the new mount point and it is expected that it will not be altered while mounted.
CatFS will not check if base directory is a valid SplitFS structure. This might change in the future. For now, please use CatFS only on directory structures that have been generated by SplitFS.

Structs

CatFS
SplitFS