Rust Dual Interface for q/kdb+
As Rust is becoming a popular programming language for its performance and type safety, the desire to use it with still a maniac time-series database kdb+ is brewing. The aspiration is understandable since we know kdb+ is fast and its interface or a shared library should be fast as well. This interface was created to satisfy such a natural demand, furthermore, in a manner users do not feel any pain to use. The notrious esoteric function names of the q/kdb+ C API is not an interest of Rust developers.
"Give us a Rust interface!!"
Here is your choice.
This interface provides two features:
- IPC interface (Rust client of q/kdb+ process)
- API (build a shared library for q/kdb+)
You can find the detail descriptions of each feature below.
Rust IPC Interface of q/kdb+
As Rust was conceived to address type unsafety of C/C++, replacing C/C++ with Rust can happen if possible. This interface is purposed to be used as a Rust client of q/kdb+ process that sends a query and receives its response. Query to kdb+ is supported in two ways:
- text query
- functional query which is represented by a compound list of kdb+ (See detail of IPC).
Compression/decompression of messages is also implemented following kdb+ implementation.
As for connect method, usually client interfaces of q/kdb+ do not provide a listener due to its protocol. However, sometimes Rust process is connecting to an upstream and q/kdb+ starts afterward or is restarted more frequently. Then providing a listener method is a natural direction and it was achieved here. Following ways are supported to connect to kdb+:
- TCP
- TLS
- Unix domain socket
Furthermore, in order to improve inter-operatability some casting, getter and setter methods are provided.
Environmental Variables
This crate uses q-native or crate-specific environmental variables.
-
KDBPLUS_ACCOUNT_FILE
: A file path to a credential file which an acceptor loads in order to manage access from a q client. This file contains a user name and SHA-1 hashed password in each line which are delimited by':'
without any space. For example, a file containing two credentials"mattew:oracle"
and"reluctant:slowday"
looks like this:mattew:431364b6450fc47ccdbf6a2205dfdb1baeb79412 reluctant:d03f5cc1cdb11a77410ee34e26ca1102e67a893c
The hashed password can be generated with q using a function
.Q.sha1
:q).Q.sha1 "slowday" 0xd03f5cc1cdb11a77410ee34e26ca1102e67a893c
-
KDBPLUS_TLS_KEY_FILE
andKDBPLUS_TLS_KEY_FILE_SECRET
: The pkcs12 file and its password which TLS acceptor uses. -
QUDSPATH
(optional): q-native environmental variable to define an astract namespace. This environmental variable is used by UDS acceptor too. The abstract nameapace will be@${QUDSPATH}/kx.[server process port]
if this environmental variable is defined; otherwise it will be@/tmp/kx.[server process port]
.
Notes:
- Messages will be sent with OS native endian.
- When using this crate for a TLS client you need to set two environmental variables
KX_SSL_CERT_FILE
andKX_SSL_KEY_FILE
on q side to make q/kdb+ to work as a TLS server. For details, see the KX website.
Type Mapping
All types are expressed as K
struct which is quite similar to the K
struct of api
module but its structure is optimized for IPC
usage and for the convenience to interact with. The table below shows the input types of each q type which is used to construct K
object.
Note that the input type can be different from the inner type. For example, timestamp has an input type of chrono::DateTime<Utc>
but
the inner type is i64
denoting an elapsed time in nanoseconds since 2000.01.01D00:00:00
.
q | Rust |
---|---|
bool |
bool |
GUID |
[u8; 16] |
byte |
u8 |
short |
i16 |
int |
i32 |
long |
i64 |
real |
f32 |
float |
f64 |
char |
char |
symbol |
String |
timestamp |
chrono::DateTime<Utc> |
month |
chrono::Date<Utc> |
date |
chrono::Date<Utc> |
datetime |
chrono::DateTime<Utc> |
timespan |
chrono::Duration |
minute |
chrono::Duration |
second |
chrono::Duration |
time |
chrono::Duration |
list |
Vec<Item> (Item is a corrsponding type above) |
compound list |
Vec<K> |
table |
Vec<K> |
dictionary |
Vec<K> |
null |
() |
Examples
Client
use *;
async
Listener
use *;
async
Then q client can connect to this acceptor with the acceptor's host, port and the credential configured in KDBPLUS_ACCOUNT_FILE
:
q)h:hopen `::7000:reluctant:slowday
Installation
Use kdbplus
as a library name in Cargo.toml
with "ipc"
feature.
[]
={="^0.3", =["ipc"]}
Rust Wrapper of q/kdb+ C API
Programming language q (kdb+ is a database written in q) is providing only C API but sometimes an external library provides Rust interface but not C/C++ interface. From the fame of its performance, Rust still should be feasible to build a shared library for kdb+. This library is provided to address such a natural demand (desire, if you will). Since there is no way for everyone but creating a wrapper like this to write a shared library for kdb+, it probably make sense for someone to provide the wrapper, and it was done here.
In order to avoid writing too large unsafe
block which leads to poor optimization, most of native C API functions are provided with a wrapper funtion with a bit of ergonomic safety and with intuitive implementation as a trait method. The only exceptions are knk
and k
which are using elipsis (...
) as its argument. These functions are provided under native
namespace with the other C API functions.
Note: This library is purposed to be used to build a sared library; therefore some unrelated functions are removed. For example, connection functions to kdb+ like khpu
are not included.
Installation
Use kdbplus
as a library name in Cargo.toml
with "api"
feature.
[]
={="^0.3", =["api"]}
Examples
The examples of using C API wrapper are included in api_examples
folder. The examples are mirroring the examples in the document of kdbplus::api
module and the functions are also used for simple tests of the library. The test is conducted in the test.q
under tests/
by loading the functions defined in a shared library built from the examples.
Here are some examples:
C API Style
use qtype;
use *;
use *;
pub extern "C"
pub extern "C"
pub extern "C"
q can use these functions like this:
q)summon:`libc_api_examples 2: (`create_symbol_list; 1)
q)summon[]
`Abraham`Isaac`Jacob`Joseph
q)`Abraham`Isaac`Jacob`Joseph ~ summon[]
q)catchy: `libc_api_examples 2: (`catchy; 2);
q)catchy[$; ("J"; "42")]
42
q)catchy[+; (1; `a)]
error: type
q)behold: `libc_api_examples 2: (`dictionary_list_to_table; 1);
q)behold[]
a b
------
0 0
10 100
20 200
Rust Style
The examples below are written without unsafe
code. You can see how comfortably breathing are the wrapped functions in the code.
use qtype;
use *;
use *;
pub extern "C"
pub extern "C"
And q code is here:
q)summon:`libc_api_examples 2: (`create_symbol_list2; 1)
q)summon[]
`Abraham`Isaac`Jacob`Joseph
q)chill: `libc_api_examples 2: (`no_panick; 2);
q)chill[$; ("J"; "42")]
success!
42
q)chill[+; (1; `a)]
FYI: type
q)climate_change: libc_api_examples 2: (`create_table2; 1);
q)climate_change[]
time temperature
-----------------------------------------
2003.10.10D02:24:19.167018272 22.1
2006.05.24D06:16:49.419710368 24.7
2008.08.12D23:12:24.018691392 30.5
Test
Test is conducted in two ways:
- Using cargo
- Running a q test script
1. Using Cargo
Before starting the test, you need to start a q process on the port 5000:
)
Then fire the cargo test:
Note: Currently 20 tests fails for api
examples in document. This is because the examples do not have main
function by nature of api
but still use #[macro_use]
.
2. Running a q Test Script
Tests are conducted with tests/test.q
by loading the example functions built in api_examples
.
;
)
Projects Using This Library
- qrpc (gRPC client)
- q_comtrade (COMTRADE file parser)
Document
The document of this crate itself is on the crates.io page.
For details of C API itself, check the documents of KX website.