REMOTE SCHEMA

Server File Journal - stores all changes

Namespace Id (NSID) Relative Path in namespace Journal ID (JID): Monotonically increasing within a namespace

BlockServer - can store block or retrieve block

• RocksDB might work

Q:

• where to store chunks? s3 is to expensive for such small files, maybe cheap distributed key/value db?

LOCAL DB SCHEMA

files

jid: integer path // relative to current dir format: text|binary modified: unix timestamp size: integer is_symlink: bool checksum: varchar

USE-CASES

• client needs to update a file from meta server (MS)
  • S during polling receives that file /path/bla was updated
  • sends list request passing namespace and current cursor
  • MS returns all JIDs since passed one and their hashes (maybe except when the same file was updated multiple times, returns only the last one?)
  • S
• client needs to upload a file to server
  • S tries to commit current file it has commit(/path/bla, [h1,h2,h3])
  • MS returns back list of
• program just starts
  • S checks the latest journal_id
  • if local latest journal_id is the same it will do nothing
  • if local latest journal_id
• file was removed locally
• file was moved locally
• file was renamed
• one line in a file was edited
• one line in a file was added
• one line in a file was removed

if latest jid remotely bigger sync dowload from remote if metadata, size is different upload to remote and after commit store into local db

Q:

• do I need hierarchy of services or they should be all independent?
• how sharing should work?
• how to thread it? multiple modules and multiple files
• do I need to sync file metadata as well?

We have separate threads for sniffing the file system, hashing, commit, store_batch, list, retrieve_batch, and reconstruct, allowing us to pipeline parallelize this process across many files. We use compression and rsync to minimize the size of store_batch/retrieve_batch requests.

SYNCER

• checks if database has not assigned jid
• when it finds not assigned jid it will try to commit, after commiting it will update local DB with new jid
• if chunk is not present locally it will try to download it
• if chunk is not present remotely it will try to upload it

commit("breakfast/Mexican Style Burrito.cook", "h1,h2,h3");

Q:

• problem if by line? => seek wont work, need to store block size to do the seek effeftively.
• where to store chunks for not yet assembled file
• how to understand that a new file created remotely
• hot to understand that file was deleted
• how to understand that

INDEXER

• sync between files and local DB on schedule (once a min, f.e.)
• watches changes and triggers sync
• will cleanup DB once a day

Q:

• do I need to copy not changed jid? or just update updated? => it makes sense to update all
• what happens on delete, move?

CHUNKER

Role of Chunker is to deal with persistance of hashes and files. It operates on text files and chunks are not a fixed sized but each chunk is a line of file.

• given path it will produce list of hashes of the file: fn hashify(file_path: String) -> io::Result<Vec<String>>
• given path and list of hashes it will save a new version of a file fn save(file_path: String, Vec<String>) -> io::Result. It should raise an error if cache doesn't have content for a specific chunk hash
• can read content of a specific chunk from cache fn read_chunk(chunk: String) -> io::Result<String>
• can write content of a spefic chunk to cache fn save_chunk(chunk: String, content: String) -> io::Result
• given two vectors of hashes it can compare them if they are the same fn compare_sets(left: Vec<String>, right: Vec<String>) -> bool
• given hash it can check if cache contains content for it or not. fn check_chunk(chunk: String>) -> io::Result<bool>

Q:

• strings will be short, 80-100 symbols. what should be used as hashing function? what size of hash should be? I'd say square root of 10. You can test it!

• empty files should be different from deleted

Usage (iOS/Swift)

Installation

Add the package to your Xcode project:

dependencies: [
    .package(url: "https://github.com/cooklang/cooklang-sync.git", from: "0.3.0")
]

Basic Example

import CooklangSyncClient
import Foundation

// 1. Create a status listener to receive sync updates
class MySyncListener: SyncStatusListener {
    func onStatusChanged(status: SyncStatus) {
        print("Sync status: \(status)")
    }

    func onComplete(success: Bool, message: String?) {
        print("Sync completed. Success: \(success), Message: \(message ?? "none")")
    }
}

// 2. Set up sync context
let context = SyncContext()
let listener = MySyncListener()
context.setListener(listener: listener)

// 3. Configure sync parameters
let storageDir = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
    .appendingPathComponent("recipes")
    .path
let dbFilePath = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask)[0]
    .appendingPathComponent("sync.db")
    .path
let apiEndpoint = "https://api.cooklang.org"
let remoteToken = "your_jwt_token_here"
let namespaceId: Int32 = 1

// 4. Run continuous sync (watches for file changes and syncs automatically)
Task {
    do {
        try run(
            context: context,
            storageDir: storageDir,
            dbFilePath: dbFilePath,
            apiEndpoint: apiEndpoint,
            remoteToken: remoteToken,
            namespaceId: namespaceId,
            downloadOnly: false  // Set to true for read-only sync
        )
    } catch {
        print("Sync error: \(error)")
    }
}

// 5. Cancel sync when needed
context.cancel()

One-Time Sync Operations

For manual control over sync operations:

// Download updates from server (one-time)
Task {
    do {
        try runDownloadOnce(
            storageDir: storageDir,
            dbFilePath: dbFilePath,
            apiEndpoint: apiEndpoint,
            remoteToken: remoteToken,
            namespaceId: namespaceId
        )
        print("Download completed")
    } catch {
        print("Download error: \(error)")
    }
}

// Upload local changes to server (one-time)
Task {
    do {
        try runUploadOnce(
            storageDir: storageDir,
            dbFilePath: dbFilePath,
            apiEndpoint: apiEndpoint,
            remoteToken: remoteToken,
            namespaceId: namespaceId
        )
        print("Upload completed")
    } catch {
        print("Upload error: \(error)")
    }
}

Advanced: Wait for Remote Updates

Use this to implement efficient background sync with server-sent events:

// Wait for remote updates, then download
Task {
    do {
        // This blocks until server notifies of changes or timeout
        try waitRemoteUpdate(apiEndpoint: apiEndpoint, remoteToken: remoteToken)

        // Now download the updates
        try runDownloadOnce(
            storageDir: storageDir,
            dbFilePath: dbFilePath,
            apiEndpoint: apiEndpoint,
            remoteToken: remoteToken,
            namespaceId: namespaceId
        )
    } catch {
        print("Error: \(error)")
    }
}

Sync Status Values

The SyncStatus enum includes:

• .idle - Not syncing
• .syncing - Sync operation in progress
• .uploading - Currently uploading to server
• .downloading - Currently downloading from server
• .error - Sync encountered an error

Best Practices

1. Database Location: Store the database in Application Support directory to persist across app updates
2. Storage Directory: Use a dedicated subdirectory in Documents for synced files
3. Background Sync: On iOS, use Background Tasks API to run periodic syncs
4. Error Handling: Always handle SyncError exceptions and notify users appropriately
5. Cancellation: Call context.cancel() before app termination to clean up resources
6. Read-Only Mode: Set downloadOnly: true if you want to prevent local changes from syncing to server

Usage (Android/Kotlin)

Installation

Add the JNA dependency and the native library to your Android project:

// In build.gradle.kts
dependencies {
    implementation("net.java.dev.jna:jna:5.13.0@aar")
    // Add the generated Kotlin bindings and native library
}

Build the Android library using the Rust toolchain and include it in your project's jniLibs directory.

Basic Example

import org.cooklang.sync.*
import kotlinx.coroutines.*

// 1. Create a status listener to receive sync updates
class MySyncListener : SyncStatusListener {
    override fun onStatusChanged(status: SyncStatus) {
        when (status) {
            is SyncStatus.Idle -> println("Sync idle")
            is SyncStatus.Syncing -> println("Syncing...")
            is SyncStatus.Indexing -> println("Indexing files...")
            is SyncStatus.Downloading -> println("Downloading from server...")
            is SyncStatus.Uploading -> println("Uploading to server...")
            is SyncStatus.Error -> println("Error: ${status.message}")
        }
    }

    override fun onComplete(success: Boolean, message: String?) {
        println("Sync completed. Success: $success, Message: ${message ?: "none"}")
    }
}

// 2. Set up sync context
val context = SyncContext()
val listener = MySyncListener()
context.setListener(listener)

// 3. Configure sync parameters
val storageDir = context.getExternalFilesDir(null)?.resolve("recipes")?.absolutePath
    ?: throw IllegalStateException("Cannot access external files directory")
val dbFilePath = context.getDatabasePath("sync.db").absolutePath
val apiEndpoint = "https://api.cooklang.org"
val remoteToken = "your_jwt_token_here"
val namespaceId = 1

// 4. Run continuous sync in a coroutine (watches for file changes)
lifecycleScope.launch(Dispatchers.IO) {
    try {
        run(
            context = context,
            storageDir = storageDir,
            dbFilePath = dbFilePath,
            apiEndpoint = apiEndpoint,
            remoteToken = remoteToken,
            namespaceId = namespaceId,
            downloadOnly = false  // Set to true for read-only sync
        )
    } catch (e: SyncException) {
        Log.e("Sync", "Sync error", e)
    }
}

// 5. Cancel sync when needed
override fun onDestroy() {
    super.onDestroy()
    context.cancel()
}

One-Time Sync Operations

For manual control over sync operations:

// Download updates from server (one-time)
lifecycleScope.launch(Dispatchers.IO) {
    try {
        runDownloadOnce(
            storageDir = storageDir,
            dbFilePath = dbFilePath,
            apiEndpoint = apiEndpoint,
            remoteToken = remoteToken,
            namespaceId = namespaceId
        )
        Log.d("Sync", "Download completed")
    } catch (e: SyncException) {
        Log.e("Sync", "Download error", e)
    }
}

// Upload local changes to server (one-time)
lifecycleScope.launch(Dispatchers.IO) {
    try {
        runUploadOnce(
            storageDir = storageDir,
            dbFilePath = dbFilePath,
            apiEndpoint = apiEndpoint,
            remoteToken = remoteToken,
            namespaceId = namespaceId
        )
        Log.d("Sync", "Upload completed")
    } catch (e: SyncException) {
        Log.e("Sync", "Upload error", e)
    }
}

Advanced: Wait for Remote Updates

Use this to implement efficient background sync with server-sent events:

// Wait for remote updates, then download
lifecycleScope.launch(Dispatchers.IO) {
    try {
        // This blocks until server notifies of changes or timeout
        waitRemoteUpdate(apiEndpoint = apiEndpoint, remoteToken = remoteToken)

        // Now download the updates
        runDownloadOnce(
            storageDir = storageDir,
            dbFilePath = dbFilePath,
            apiEndpoint = apiEndpoint,
            remoteToken = remoteToken,
            namespaceId = namespaceId
        )
    } catch (e: SyncException) {
        Log.e("Sync", "Error", e)
    }
}

Exception Handling

The library throws SyncException with various subtypes:

try {
    run(...)
} catch (e: SyncException.Unauthorized) {
    // Handle authentication error
    Log.e("Sync", "Unauthorized: ${e.message}")
} catch (e: SyncException.IoException) {
    // Handle I/O error
    Log.e("Sync", "I/O error: ${e.message}")
} catch (e: SyncException) {
    // Handle other errors
    Log.e("Sync", "Sync error: ${e.message}")
}

Best Practices (Android)

1. Database Location: Use context.getDatabasePath() to get proper database location
2. Storage Directory: Use context.getExternalFilesDir() for synced files
3. Background Sync: Use WorkManager for periodic background sync operations
4. Coroutines: Always run sync operations on Dispatchers.IO to avoid blocking main thread
5. Lifecycle: Cancel sync context in onDestroy() or when Activity/Fragment is destroyed
6. Permissions: Request WRITE_EXTERNAL_STORAGE permission if targeting Android < 10
7. Read-Only Mode: Set downloadOnly = true if you want to prevent local changes from syncing

Building bindings

Prepare

Install rustup https://www.rust-lang.org/tools/install.

Then add iOS targets.

rustup target add aarch64-apple-ios
rustup target add x86_64-apple-ios
rustup target add aarch64-apple-ios-sim

Install iOS SDK https://developer.apple.com/xcode/resources/.

Build XCFramework (Recommended)

The easiest way to build the Swift bindings is to use the provided build script:

cd client
./scripts/build_swift_framework.sh cooklang-sync-client cooklang_sync_client CooklangSyncClientFFI

This will:

• Build the Rust library for iOS device and simulator targets
• Generate Swift bindings using UniFFI
• Create a universal XCFramework at ../swift/CooklangSyncClientFFI.xcframework
• Set a valid CFBundleIdentifier (converts underscores to hyphens: org.cooklang.cooklang-sync-client)
• Create a zip archive of the XCFramework
• Calculate the SHA-256 checksum
• Automatically update Package.swift with the current version (from Cargo.toml) and checksum

Note: Apple's CFBundleIdentifier must only contain alphanumeric characters, hyphens, and periods. The build script automatically converts underscores in the library name to hyphens to comply with this requirement.

After building, the script will display the version, checksum, and instructions for publishing a release.

Using the Local XCFramework

To use the locally built XCFramework with the Swift Package instead of the remote release version, set the USE_LOCAL_XCFRAMEWORK environment variable:

export USE_LOCAL_XCFRAMEWORK=1

Then in your Xcode project that depends on this package, clean and rebuild. The Package.swift will automatically use the local XCFramework at swift/CooklangSyncClientFFI.xcframework instead of downloading from GitHub releases.

Manual Build (Advanced)

Build library:

cargo build --lib --target=x86_64-apple-ios --release
cargo build --lib --target=aarch64-apple-ios --release

Build foreign language bindings (this will output Swift code into ./out dir):

cargo run --features="uniffi/cli"  \
  --bin uniffi-bindgen generate \
  --config uniffi.toml \
  --library ../target/x86_64-apple-ios/release/libcooklang_sync_client.a \
  --language swift \
  --out-dir out

See example of a Xcode project here.

Combine into universal library:

mkdir -p ../target/universal/release
mkdir -p ../target/universal_macos/release
lipo -create -output ../target/universal/release/libcooklang_sync_client.a \
  ../target/x86_64-apple-ios/release/libcooklang_sync_client.a \
  ../target/aarch64-apple-ios/release/libcooklang_sync_client.a

lipo -create -output ../target/universal_macos/release/libcooklang_sync_client.a \
  ../target/x86_64-apple-darwin/release/libcooklang_sync_client.a \
  ../target/aarch64-apple-darwin/release/libcooklang_sync_client.a


xcodebuild -create-xcframework \
   -library ../target/aarch64-apple-ios/release/libcooklang_sync_client.a \
   -library ../target/x86_64-apple-ios/release/libcooklang_sync_client.a \
   -output CooklangSyncClientFFI.xcframework


   cp ../target/universal/release/libcooklang_sync_client.a ../swift/CooklangSyncClientFFI.xcframework/ios-arm64/CooklangSyncClientFFI.framework/CooklangSyncClientFFI

   cp ../target/universal/release/libcooklang_sync_client.a ../swift/CooklangSyncClientFFI.xcframework/ios-arm64_x86_64-simulator/CooklangSyncClientFFI.framework/CooklangSyncClient

TODO

• bundling of uploads/downloads

• read-only

• namespaces

• proper error handling

• report error on unexpeted cache behaviour

• don't need to throw unknown error in each non-200 response

• remove clone

• limit max file

• configuration struct

• pull changes first or reindex locally first? research possible conflict scenarios

• extract to core shared datasctuctures

• garbage collection on DB

• test test test

• metrics for monitoring (cache saturation, miss)

• protect from ddos https://github.com/rousan/multer-rs/blob/master/examples/prevent_dos_attack.rs

• auto-update client

open sourcing

• how to keep it available for opensource (one user?)
• add documentation
• draw data-flow