Crate rustdb

Expand description

This crate (rustdb) implements a high-performance database written entirely in Rust.

The SQL-like language is relatively minimal, and does not (currently) include features such as joins or views. Instead it has high performance SET .. FROM … and FOR .. FROM statements to access database tables, generally using an INDEX.

Read-only transactions run immediately and concurrently on a virtual read-only copy of the database, and cannot be blocked. Write transactions run sequentially (and should typically execute in around 100 micro-seconds). The Storage trait allows a variety of underlying storage, including SimpleFileStorage, MemFile and AtomicFile.

Transactions that modify the database can be logged, which allows for database replication.

§Interface

The method Database::run is called to execute an SQL query. This takes a Transaction parameter which accumulates SELECT results and which also has methods for accessing input parameters and controlling output. Custom builtin functions implement CExp and have access to the transaction via an EvalEnv parameter, which can be downcast if necessary.

It is also possible to access the table data directly, see email_loop in example program.

§Example

See here for an example program - a webserver, with timed jobs, password hashing, data compression, email transmission and database replication. Also has a Manual for the SQL-like language, user interface for database browsing/editing etc.

§Features

This crate supports the following cargo features:

gentrans : enables gentrans module ( sample implementation of Transaction ).
serde : enables serialisation of GenQuery via serde crate.
builtin : Allows extra SQL builtin functions to be defined.
table : Allow direct access to database tables.
max : maximal interface, including internal modules (which may not be stable).
verify : Allows database structure to be verified using builtin function VERIFYDB.
pack : Allows database pages to be packed using builtin function REPACKFILE.
renumber : Allows database pages to be renumbered using builtin function RENUMBER, eliminating free pages.
unsafe-optim : Enable unsafe optimisations in release mode.
log : Log “interesting” information about database operation (helps give an idea what is happening).

By default, all features except serde, unsafe-optim, log and compact are enabled.

§General Design of Database

SortedFile stores fixed size Records in a tree of Pages. SortedFile is used to implement:

Database Table storage. Each fixed size record has a 64-bit Id.
Variable length values which are split into fragments, although up to 249 bytes can be stored in the fixed size record.
Index storage - an index record refers back to the main table using the 64-bit Id.

When a page becomes too big, it is split into two pages.

Each page is implemented as a binary tree ( so there is a tree of trees ).

SharedPagedData allows logical database pages to be shared to allow concurrent readers.

AtomicFile ensures that database updates are all or nothing.

The hierarchy overall: Table -> SortedFile -> SharedPagedData -> PageStorage -> AtomicFile -> Storage.

§Test example

    use rustdb::*;
    use std::sync::Arc;
    let stg = MemFile::new();

    let spd = SharedPagedData::new(stg);
    let wapd = spd.new_writer();

    let mut bmap = BuiltinMap::default();
    standard_builtins(&mut bmap);
    let bmap = Arc::new(bmap);

    let db = Database::new(wapd, "", bmap);
    let mut tr = GenTransaction::default();
    let sql = "
CREATE SCHEMA test GO
CREATE TABLE test.Cust(Name string) GO
INSERT INTO test.Cust(Name) VALUES ('freddy')
SELECT Name FROM test.Cust
";
    db.run(&sql, &mut tr);
    assert!( db.changed() );
    assert!( db.save() > 0 );
    assert!( tr.rp.output == b"freddy" );

Re-exports§

pub use crate::builtin::standard_builtins;
pub use crate::gentrans::GenQuery;
pub use crate::gentrans::GenTransaction;
pub use crate::gentrans::Part;
pub use crate::builtin::check_types;
pub use crate::compile::c_bool;
pub use crate::compile::c_float;
pub use crate::compile::c_int;
pub use crate::compile::c_value;
pub use crate::exec::EvalEnv;
pub use crate::expr::ObjRef;
pub use crate::expr::Block;
pub use crate::expr::DataKind;
pub use crate::expr::Expr;
pub use crate::run::CExp;
pub use crate::run::CExpPtr;
pub use crate::run::CompileFunc;
pub use crate::value::Value;

Modules§

bench: Benchmark - compare RustDb with competitors!
builtin: Compilation of builtin functions, standard_builtins.
bytes: Storage of variable length values : ByteStorage.
cexp: Structs that implement CExp trait.
compact: compact::CompactFile : alternative implementation of PageStorage trait.
compile: Functions to compile parsed expressions, checking types.
exec: EvalEnv : Instruction execution.
expr: Expression types, result of parsing. Expr, DataKind, ObjRef, Block.
gentrans: GenTransaction ( implementation of Transaction ).
page: Page of records for SortedFile.
parse: Parser.
run: Instruction and other run time types.
sortedfile: SortedFile : Record storage.
sys: System table functions.
table: Table, ColInfo, Row and other Table types for direct table access.
test: Test module.
util: Utility functions and macros, SmallSet.
value: Run-time Value.

Structs§

AccessPagedData: Access to shared paged data.
AtomicFile: Based on BasicAtomicFile which makes sure that updates are all-or-nothing. Performs commit asyncronously.
BTreeMap: An ordered map based on a B-Tree.
BasicAtomicFile: Basis for crate::AtomicFile ( non-async alternative ). Provides two-phase commit and buffering of writes.
BlockPageStg: Implementation of PageStorage.
Database: Database with SQL-like interface.
DummyFile: Dummy Stg that can be used for Atomic upd file if “reliable” atomic commits are not required.
Limits: Memory limits.
MData: Mutable Data, copied on write.
MemFile: Simple implementation of Storage using Arc<Mutex<Vec<u8>>.
MultiFileStorage: Alternative to SimpleFileStorage that uses multiple SimpleFileStorages to allow parallel reads by different threads.
SharedPagedData: Allows pages to be shared to allow concurrent readers.
SimpleFileStorage: Simple implementation of Storage using std::fs::File.

Enums§

SaveOp: Save or Rollback.

Traits§

PageStorage: Interface for page storage.
PageStorageInfo: Information about page sizes.
Storage: Storage interface - Storage is some kind of “file” storage.
Transaction: Input/Output message. Query and Response.

Type Aliases§

BuiltinMap: Map that defines SQL pre-defined functions.
DB: Rc<Database>
Data: Arc<Vec<u8>>
HashMap: Type alias for a hashmap using the fx hash algorithm.
HashSet: Type alias for a hashmap using the fx hash algorithm.

Crate rustdb

Crate rustdb Copy item path

§Interface

§Example

§Features

§General Design of Database

§Test example

Re-exports§

Modules§

Structs§

Enums§

Traits§

Type Aliases§

Crate rustdb