Crate iroh_docs

Crate iroh_docs 

Source
Expand description

§iroh-docs

Multi-dimensional key-value documents with an efficient synchronization protocol.

The crate operates on Replicas. A replica contains an unlimited number of Entries. Each entry is identified by a key, its author, and the replica’s namespace. Its value is the 32-byte BLAKE3 hash of the entry’s content data, the size of this content data, and a timestamp. The content data itself is not stored or transferred through a replica.

All entries in a replica are signed with two keypairs:

  • The Namespace key, as a token of write capability. The public key is the NamespaceId, which also serves as the unique identifier for a replica.
  • The Author key, as a proof of authorship. Any number of authors may be created, and their semantic meaning is application-specific. The public key of an author is the AuthorId.

Replicas can be synchronized between peers by exchanging messages. The synchronization algorithm is based on a technique called range-based set reconciliation, based on this paper by Aljoscha Meyer:

Range-based set reconciliation is a simple approach to efficiently compute the union of two sets over a network, based on recursively partitioning the sets and comparing fingerprints of the partitions to probabilistically detect whether a partition requires further work.

The crate exposes a generic storage interface with in-memory and persistent, file-based implementations. The latter makes use of redb, an embedded key-value store, and persists the whole store with all replicas to a single file.

§Getting Started

The entry into the iroh-docs protocol is the Docs struct, which uses an Engine to power the protocol.

Docs was designed to be used in conjunction with iroh. Iroh is a networking library for making direct connections, these connections are peers send sync messages and transfer data.

Iroh provides a Router that takes an Endpoint and any protocols needed for the application. Similar to a router in webserver library, it runs a loop accepting incoming connections and routes them to the specific protocol handler, based on ALPN.

Docs is a “meta protocol” that relies on the iroh-blobs and iroh-gossip protocols. Setting up Docs will require setting up Blobs and Gossip as well.

Here is a basic example of how to set up iroh-docs with iroh:

use iroh::{protocol::Router, Endpoint};
use iroh_blobs::{BlobsProtocol, store::mem::MemStore, ALPN as BLOBS_ALPN};
use iroh_docs::{protocol::Docs, ALPN as DOCS_ALPN};
use iroh_gossip::{net::Gossip, ALPN as GOSSIP_ALPN};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // create an iroh endpoint that includes the standard discovery mechanisms
    // we've built at number0
    let endpoint = Endpoint::builder().bind().await?;

    // build the blobs protocol
    let blobs = MemStore::default();

    // build the gossip protocol
    let gossip = Gossip::builder().spawn(endpoint.clone());

    // build the docs protocol
    let docs = Docs::memory()
        .spawn(endpoint.clone(), (*blobs).clone(), gossip.clone())
        .await?;

    // create a router builder, we will add the
    // protocols to this builder and then spawn
    // the router
    let builder = Router::builder(endpoint.clone());

    // setup router
    let _router = builder
        .accept(BLOBS_ALPN, BlobsProtocol::new(&blobs, None))
        .accept(GOSSIP_ALPN, gossip)
        .accept(DOCS_ALPN, docs)
        .spawn();

    // do fun stuff with docs!
    Ok(())
}

§License

This project is licensed under either of

at your option.

§Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions. Multi-dimensional key-value documents with an efficient synchronization protocol

The crate operates on Replicas. A replica contains an unlimited number of Entries. Each entry is identified by a key, its author, and the replica’s namespace. Its value is the 32-byte BLAKE3 hash of the entry’s content data, the size of this content data, and a timestamp. The content data itself is not stored or transferred through a replica.

All entries in a replica are signed with two keypairs:

  • The NamespaceSecret key, as a token of write capability. The public key is the NamespaceId, which also serves as the unique identifier for a replica.
  • The Author key, as a proof of authorship. Any number of authors may be created, and their semantic meaning is application-specific. The public key of an author is the AuthorId.

Replicas can be synchronized between peers by exchanging messages. The synchronization algorithm is based on a technique called range-based set reconciliation, based on this paper by Aljoscha Meyer:

Range-based set reconciliation is a simple approach to efficiently compute the union of two sets over a network, based on recursively partitioning the sets and comparing fingerprints of the partitions to probabilistically detect whether a partition requires further work.

The crate exposes a generic storage interface. There is an implementation of this interface, store::fs::Store, that can be used either in-memory or in persistent, file-based mode.

Both modes make use of redb, an embedded key-value store. When used in-memory, the store is backed by a Vec<u8>. When used in persistent mode, the store is backed by a single file on disk.

Re-exports§

pub use self::sync::*;

Modules§

actor
This contains an actor spawned on a separate thread to process replica and store operations.
api
irpc-based RPC implementation for docs.
engine
Handlers and actors to for live syncing replicas.
metrics
Metrics for iroh-docs
net
Network implementation of the iroh-docs protocol
protocol
ProtocolHandler implementation for the docs Engine.
store
Storage trait and implementation for iroh-docs documents
sync
API for iroh-docs replicas

Structs§

Author
Author key to insert entries in a crate::Replica
AuthorHeads
Timestamps of the latest entry for each author.
AuthorId
AuthorPublicKey in bytes
AuthorPublicKey
Identifier for an Author
DocTicket
Contains both a key (either secret or public) to a document, and a list of peers to join.
NamespaceId
NamespacePublicKey in bytes
NamespacePublicKey
The corresponding VerifyingKey for a NamespaceSecret. It is used as an identifier, and can be used to verify Signatures.
NamespaceSecret
Namespace key of a crate::Replica.

Constants§

ALPN
The ALPN identifier for the iroh-docs protocol