Crate rustdb

Expand description

This crate (rustdb) implements a high-performance database written entirely in Rust.

The SQL-like language is relatively minimal, and does not (currently) include features such as joins or views. Instead it has high performance SET .. FROM … and FOR .. FROM statements to access database tables, generally using an INDEX.

Read-only transactions run immediately and concurrently on a virtual read-only copy of the database, and cannot be blocked. Write transactions run sequentially (and should typically execute in around 100 micro-seconds). The Storage trait allows a variety of underlying storage, including SimpleFileStorage, MemFile and AtomicFile.

Transactions that modify the database can be logged, which allows for database replication.

§Interface

The method Database::run is called to execute an SQL query. This takes a Transaction parameter which accumulates SELECT results and which also has methods for accessing input parameters and controlling output. Custom builtin functions implement CExp and have access to the transaction via an EvalEnv parameter, which can be downcast if necessary.

It is also possible to access the table data directly, see email_loop in example program.

§Example

See here for an example program - a webserver, with timed jobs, password hashing, data compression, email transmission and database replication. Also has a Manual for the SQL-like language, user interface for database browsing/editing etc.

§Features

This crate supports the following cargo features:

gentrans : enables gentrans module ( sample implementation of Transaction ).
serde : enables serialisation of GenQuery via serde crate.
builtin : Allows extra SQL builtin functions to be defined.
table : Allow direct access to database tables.
max : maximal interface, including internal modules (which may not be stable).
verify : Allows database structure to be verified using builtin function VERIFYDB.
pack : Allows database pages to be packed using builtin function REPACKFILE.
renumber : Allows database pages to be renumbered using builtin function RENUMBER, eliminating free pages.
unsafe-optim : Enable unsafe optimisations in release mode.
log : Log “interesting” information about database operation (helps give an idea what is happening).
compact : Default page storage is CompactFile rather than BlockPageStg (can be set explicitly using pstore::SharedPagedData::new_from_ps ).

By default, all features except serde, unsafe-optim and log are enabled.

§General Design of Database

SortedFile stores fixed size Records in a tree of Pages. SortedFile is used to implement:

Database Table storage. Each fixed size record has a 64-bit Id.
Variable length values which are split into fragments, although up to 249 bytes can be stored in the fixed size record.
Index storage - an index record refers back to the main table using the 64-bit Id.

When a page becomes too big, it is split into two pages.

Each page is implemented as a binary tree ( so there is a tree of trees ).

SharedPagedData allows logical database pages to be shared to allow concurrent readers.

AtomicFile ensures that database updates are all or nothing.

The hierarchy overall: Table -> SortedFile -> SharedPagedData -> PageStorage -> AtomicFile -> Storage.

§Test example

    use rustdb::*;
    use std::sync::Arc;
    let stg = AtomicFile::new(MemFile::new(), MemFile::new());

    let spd = SharedPagedData::new(stg);
    let wapd = AccessPagedData::new_writer(spd);

    let mut bmap = BuiltinMap::default();
    standard_builtins(&mut bmap);
    let bmap = Arc::new(bmap);

    let db = Database::new(wapd, "", bmap);
    let mut tr = GenTransaction::default();
    let sql = "
CREATE SCHEMA test GO
CREATE TABLE test.Cust(Name string) GO
INSERT INTO test.Cust(Name) VALUES ('freddy')
SELECT Name FROM test.Cust
";
    db.run(&sql, &mut tr);
    assert!( db.changed() );
    assert!( db.save() > 0 );
    assert!( tr.rp.output == b"freddy" );

Re-exports§

pub use crate::atomfile::AtomicFile;
pub use crate::basicatomfile::BasicAtomicFile;
pub use crate::blockpagestg::BlockPageStg;
pub use crate::builtin::standard_builtins;
pub use crate::pstore::AccessPagedData;
pub use crate::pstore::SharedPagedData;
pub use crate::stg::DummyFile;
pub use crate::stg::MemFile;
pub use crate::stg::MultiFileStorage;
pub use crate::stg::PageStorage;
pub use crate::stg::PageStorageInfo;
pub use crate::stg::SimpleFileStorage;
pub use crate::stg::Storage;
pub use crate::gentrans::GenQuery;
pub use crate::gentrans::GenTransaction;
pub use crate::gentrans::Part;
pub use crate::builtin::check_types;
pub use crate::compile::c_bool;
pub use crate::compile::c_float;
pub use crate::compile::c_int;
pub use crate::compile::c_value;
pub use crate::exec::EvalEnv;
pub use crate::expr::ObjRef;
pub use crate::expr::Block;
pub use crate::expr::DataKind;
pub use crate::expr::Expr;
pub use crate::run::CExp;
pub use crate::run::CExpPtr;
pub use crate::run::CompileFunc;
pub use crate::value::Value;

Modules§

atomfile: AtomicFile - buffered version of BasicAtomicFile.
basicatomfile: BasicAtomicFile - ensures updates are all or nothing.
bench: Benchmark - compare RustDb with competitors!
block: block::BlockStg - divides Storage into relocatable fixed size blocks, basis for dividedstg::DividedStg.
blockpagestg: BlockPageStg - implementation of PageStorage trait based on dividedstg::DividedStg.
buf: Buffering: buf::WriteBuffer and buf::ReadBufStg.
builtin: Compilation of builtin functions, standard_builtins.
bytes: Storage of variable length values : ByteStorage.
cexp: Structs that implement CExp trait.
compact: compact::CompactFile : alternative implementation of PageStorage trait.
compile: Functions to compile parsed expressions, checking types.
dividedstg: dividedstg::DividedStg divides Storage into multiple sub-files of arbitrary size.
exec: EvalEnv : Instruction execution.
expr: Expression types, result of parsing. Expr, DataKind, ObjRef, Block.
gentrans: GenTransaction ( implementation of Transaction ).
heap: Heap with keys that can be modified for tracking least used page.
page: Page of records for SortedFile.
parse: Parser.
pstore: Page storage and sharing, SharedPagedData and AccessPagedData.
run: Instruction and other run time types.
sortedfile: SortedFile : Record storage.
stg: Backing Storage for database. See also AtomicFile.
sys: System table functions.
table: Table, ColInfo, Row and other Table types for direct table access.
test: Test module.
util: Utility functions and macros, SmallSet.
value: Run-time Value.
wmap: wmap::WMap - map of ranges written, for AtomicFile.

Structs§

BTreeMap: An ordered map based on a B-Tree.
Database: Database with SQL-like interface.
Limits: Memory limits.
MData: Mutable Data, copied on write.

Traits§

Transaction: Input/Output message. Query and Response.

Type Aliases§

BuiltinMap: Map that defines SQL pre-defined functions.
DB: Rc<Database>
Data: Arc<Vec<u8>>

Crate rustdbCopy item path