Crate yo

Expand description

The embedded API: one file, typed handles, and no query language (15 sections 1 and 2).

Two lines get you a database, and there is no third line. No server to start, no connection string, no schema migration to run first, and nothing to parse at runtime that the compiler could have checked instead.

let db = yo::open(yo::MEMORY)?;
let hits = db.map::<String, u64>("hits")?;

hits.set("home", &1)?;
assert_eq!(hits.get("home")?, Some(1));

§Why there is no query language

A query language is a second language inside the first one, and it costs what a second language costs: strings the compiler cannot check, a parser and a planner on the hot path, types that are yours on one side of the quote and the database’s on the other, and errors that arrive at runtime in production rather than at build time on a laptop. A Map<String, u64> is the same idea with none of that. Your editor completes it, your compiler checks it, and a lookup is a function call.

What replaces the query language for the parts a map cannot do is more handles rather than more syntax. Doc, Vectors, Graph and the rest of the Redis shapes all arrive as types in this crate, and each of them is read the way a collection in your own program is read.

§The type is the schema

The type parameters on a handle are not a convenience the compiler erases. They are written into the collection when it is created, as a description that six languages compute identically (15 section 3), and an open with a different type is refused with a message that says which field moved and whether the change is additive or breaking.

let db = yo::open(yo::MEMORY)?;
let _hits = db.map::<String, u64>("hits")?;

let e = db.map::<String, String>("hits").unwrap_err();
assert_eq!(e.code(), yo::Code::ShapeMismatch);
assert!(e.message().contains("the type changed from u64 to str"));

§Your own struct is the document

Db::docs holds a collection of whatever type you already have, stored as that type. The fields worth looking documents up by say so with an attribute, and the derive writes a constant for each one, so a query is a name the compiler knows rather than a string it does not. The doc module is the whole of it.

use yo::Yo;

#[derive(Yo)]
struct Order {
    #[yo(id)]
    id: u64,
    #[yo(index)]
    status: String,
    #[yo(ordered)]
    total: f64,
}

let db = yo::open(yo::MEMORY)?;
let orders = db.docs::<Order>("orders")?;

orders.put(&Order { id: 1, status: "open".to_owned(), total: 12.5 })?;
assert_eq!(orders.find(Order::STATUS, "open")?.len(), 1);
assert_eq!(orders.range(Order::TOTAL, 0.0..50.0)?.len(), 1);

§The same store the wire talks to

Db::strings is the Redis string keyspace and Db::sets is the set commands over the same one. A program that calls incr here runs the same code an INCR off a socket runs (Y23), without the socket, the parser or the reply, so the embedded API and a Redis client are two doors into one store rather than two stores that agree for now.

let db = yo::open(yo::MEMORY)?;
let hits = db.counter("hits");

hits.incr()?;
assert_eq!(db.strings().get("hits")?.as_deref(), Some(&b"1"[..]));

Where a Redis command works on one key for its whole life, there is a handle that holds the key: Db::counter for a counter and Db::set for a set. Those are sugar and they are worth having, because a name spelled once is a name that cannot be misspelled at the third call site.

let db = yo::open(yo::MEMORY)?;
let online = db.set("online");

online.add("alice")?;
online.add("bob")?;
assert_eq!(online.len()?, 2);

§Vectors are a collection, not a second database

Db::vectors holds embeddings under keys and answers the nearest ones to a query. There is no index to build first, no probe list to tune and no rebuild after a write: the index splits and merges its own partitions as the collection is written to, which is what 10 section 5 is about.

let db = yo::open(yo::MEMORY)?;
let passages = db.vectors_with("passages", 3, yo::Metric::Cosine)?;

passages.put("a", &[1.0, 0.0, 0.0])?;
passages.put("b", &[0.0, 1.0, 0.0])?;

let hits = passages.search(&[0.9, 0.1, 0.0], 1)?;
assert_eq!(hits[0].key, b"a".to_vec());

The searchable form of a vector is a RaBitQ code, a bit per dimension, so a collection of 768 dimensional embeddings is 96 bytes a vector to scan rather than 3072. The candidates it picks are then measured against the full precision vectors, so a hit’s distance is the real distance.

§Zero copy is available, never mandatory

Map::get hands back an owned value because that is what most code wants. Map::with hands the bytes over where they lie, which allocates nothing and is where the point read budget in bench/00 is spent. Same collection, same key, and the choice is made per call rather than per database (Y29).

§What is not here yet

A file. This build holds a database in memory, and a path that is not MEMORY says so rather than pretending. The .yo format arrives in M5 and nothing on this page changes when it does, which is the reason the front door is being built before the room behind it.

Threads. The database runs in inline mode (15 section 7), where the calling thread is the shard and a point read is a call rather than a message. The owned and served modes put this same API over yo-shard’s runtime and arrive with it.

One collection that is two. A document and its embedding are stored in Db::docs and in Db::vectors separately today, under the same key if that is how you write them, and the filtered search that reads a document’s indexed fields inside the vector scan is the rest of M6.

Re-exports§

pub use counter::Counter;
pub use db::Db;
pub use db::MEMORY;
pub use db::open;
pub use doc::Docs;
pub use doc::Document;
pub use doc::Indexed;
pub use doc::Ordered;
pub use doc::Path;
pub use graph::Edge;
pub use graph::Graph;
pub use graph::Hop;
pub use graph::Id;
pub use graph::Node;
pub use graph::Walk;
pub use keys::Keys;
pub use keys::Ttl;
pub use keys::When;
pub use keyspace::Strings;
pub use map::Map;
pub use sets::Set;
pub use sets::Sets;
pub use store::Decode;
pub use store::Encode;
pub use vector::Vectors;

Modules§

counter: Counter, a handle at one key (15 section 2).
db: The database, the one call that opens it, and the handle everything else reaches it through.
doc: Docs<T>, the typed document handle, and the traits #[derive(Yo)] writes (15 sections 2 and 4).
graph: The typed graph surface, where a traversal that does not make sense does not compile (11 section 6).
keys: The keyspace itself: what is there, what type it is, and when it goes away.
keyspace: The Redis string keyspace, from the embedded side.
map: Map<K, V>, the first typed handle (15 section 2).
sets: Sets, from the embedded side.
store: How a value becomes the bytes in a record, and back (15 section 4).
vector: Vectors, the collection of embeddings, and the search over it (10 and 15 section 2).

Structs§

Desc: A canonical description, built by writing and read as bytes.
Error: An error, with everything a caller or an agent needs to act on it.
Match: One answer from a search.
Tag: The 128 bit shape tag.

Enums§

Code: A stable, wire visible condition code.
Kind: What TYPE calls a key, and what the meta byte’s tag holds.
Metric: How a vector is compared. Part of the shape because a collection built for cosine and searched as if it were L2 gives wrong answers quietly.
Moved: What a rename or a copy did.
Str: A stored value, read back out of a record.

Traits§

Shape: A type that can describe itself.

Type Aliases§

Member: A set member: bytes as they lie, or an integer not yet formatted.
Result: The crate wide result type.

Derive Macros§

Yo: Write a type’s shape, its document encoding and the indexes it declares.

Crate yo

Crate yo Copy item path

§Why there is no query language

§The type is the schema

§Your own struct is the document

§The same store the wire talks to

§Vectors are a collection, not a second database

§Zero copy is available, never mandatory

§What is not here yet

Re-exports§

Modules§

Structs§

Enums§

Traits§

Type Aliases§

Derive Macros§

Crate yo