1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
//! Jupiter is a library for providing high throughput ultra low latency services via the RESP
//! protocol as defined by Redis.
//!
//! # Introduction
//! **Jupiter** is a framework for wrapping **compute** or **memory intense** components to
//! provide them as **high throughput** and **ultra low latency** services to
//! applications built on managed runtimes like **node.js**, **Java**, **Ruby**.
//!
//! These managed runtimes are great for building sophisticated web applications but have limited
//! capabilities when raw compute power or optimized memory utilization is required. This
//! on the other hand is an area where **Rust** shines, as it permits to write low-level
//! and highly optimized code which is still safe to run.
//!
//! Therefore all we need is a simple an efficient way of combining the best of both worlds.
//! To minimize the overhead of communication, we use the [RESP Protocol](https://redis.io/topics/protocol)
//! as defined by **Redis**. In contrast to HTTP this is way simpler to parse and handle while
//! also supporting zero-copy operations. Another benefit is, that for nearly every platform
//! there is already a Redis/RESP client available.
//!
//! # SIRIUS / Java
//! We at [scireum](https://www.scireum.de) use **Jupiter** in conjunction with our open source
//! Java framework [SIRIUS](https://github.com/scireum/sirius-kernel) to build web based
//! applications.
//!
//! We use **Jupiter** as an LRU cache for intermediate search results and search metadata. We also
//! store large parts of semi-constant masterdata (think of "all ZIP codes and street names in
//! germany) there. Both of these frameworks are contributed to the open source community and can
//! be either used in own applications or directly by running a
//! [Jupiter IO](https://hub.docker.com/repository/docker/scireum/jupiter-io) instance.
//!
//! Our most central use of this framework lies in **Quasar** which sadly has to remain closed
//! source. This application contains our complete text processing framework which runs everything
//! from syntactical pre-processing to full morphological analysis steps in order to maintain our
//! excellent search experience in our products.
//!
//! # Features
//! * **Ultra fast non allocating parser for RESP queries** (as sent by redis-cli and redis clients).
//! The built-in server will use a single buffer per connection to read, parse and process queries.
//! To deliver a response, a single buffer is allocated to buffer the response to minimize the
//! number of required sys-calls to write a response on the wire.
//! * **100% Async/Await** - the whole server builds upon [tokio](https://tokio.rs/) and async/await
//! primitives as provided by Rust. Also, all commands handlers are build as actors to simplify
//! concurrency correctness and to also minimize any synchronization overheads.
//! * **Reload-aware config facility** which permits to update the configuration during operation.
//! Therefore no restart is ever required, even when changing the IP binding or port. This is kind
//! of important for a in-memory application which might have an expensive startup time.
//! * **Build in management commands**. The *core* module provides a set of management commands to monitor
//! and inspect the state of the system.
//! * **Simple and well documented code base**. After all, Jupiter isn't a large framework at all. This permits
//! every user to browse and understand its source code and when to expect from the system. Also this is due to
//! the fact that Jupiter stands on the shoulders of giants (especially [tokio](https://tokio.rs/)).
//!
//! # Modules
//! * **LRU-Cache**: An size constraint cache with an intelligent refresh strategy which can be used
//!   to maintain low latency response times by employing a coordinated asynchronous cache update
//!   pattern (see `LRU.XGET` or the module documentation of [crate::lru::cache]).
//! * **InfoGraphDB**: Provides a fast and flexible static database for master data. Using the
//!   **Repository** this can be used to load master data from e.g. an S3 Bucket or a git repository
//!   into fast lookup tables or code sets. These permit to perform all kinds of lookups,
//!   reverse-lookups, "search as you type" searches and automatic translation management (even for
//!   tables with thousands of rows / structured documents). More infos: [crate::idb]
//! * **Repository**: The repository is used to fetch files from various sources and invoking
//!   appropriate loaders so that the data can be used (e.g. as IDB table). See [crate::repository]
//!
//! # Examples
//! A complete example of using Jupiter can be found here:
//! [Jupiter IO](https://github.com/scireum/jupiter/tree/master/jupiter-io).
//!
//! Still a short example on how to initialize the library can be found here [Builder](builder::Builder).
//! Some example commands can be found in the implementation of the [core](core) commands.
//!
//! # Using Jupiter
//! **Jupiter** is intended to be used as a framework for your custom application. However the
//! example instance [Jupiter IO](https://github.com/scireum/jupiter/tree/master/jupiter-io) which
//! features an **LRU cache** and an **InfoGraphDB** instance can also be directly used via docker:
//! [Docker image](https://hub.docker.com/repository/docker/scireum/jupiter-io).
//!
//! Further info can be found on [crates.io/crates/jupiter](https://crates.io/crates/jupiter).
//! As well as on GitHub: [github.com/scireum/jupiter](https://github.com/scireum/jupiter)
#![deny(
    warnings,
    missing_docs,
    trivial_casts,
    trivial_numeric_casts,
    unused_extern_crates,
    unused_import_braces,
    unused_results
)]
use simplelog::{format_description, ConfigBuilder, LevelFilter, SimpleLogger};
use std::sync::Once;

pub mod average;
pub mod builder;
pub mod commands;
pub mod config;
pub mod core;
pub mod fmt;
pub mod idb;
pub mod ig;
pub mod lru;
pub mod platform;
pub mod pyrun;
pub mod repository;
pub mod request;
pub mod response;
pub mod server;
pub mod signals;

/// Contains the version of the Jupiter library.
pub const JUPITER_VERSION: &str = "3.1.2";

/// Contains the git commit hash of the Jupiter build being used.
pub const JUPITER_REVISION: &str = "abd9e9aa352a8abf2856d8d8e9a3b2d8d43f46a4";

/// Initializes the logging system.
///
/// Note that most probably the simplest way is to use a [Builder](builder::Builder) to setup the
/// framework, which will also setup logging if enabled.
pub fn init_logging() {
    static INIT_LOGGING: Once = Once::new();

    // We need to do this as otherwise the integration tests might crash as the logging system
    // is initialized several times...
    INIT_LOGGING.call_once(|| {
        if let Err(error) = SimpleLogger::init(
            LevelFilter::Debug,
            ConfigBuilder::new()
                .set_time_format_custom(format_description!(
                    "[year]-[month]-[day]T[hour]:[minute]:[second].[subsecond digits:3]"
                ))
                .set_thread_level(LevelFilter::Trace)
                .set_target_level(LevelFilter::Error)
                .set_location_level(LevelFilter::Trace)
                .build(),
        ) {
            panic!("Failed to initialize logging system: {}", error);
        }
    });
}

/// Provides a simple macro to execute an async lambda within `tokio::spawn`.
///
/// Note that this also applies std::mem::drop on the returned closure to make
/// clippy happy.
///
/// # Example
/// ```rust
/// # #[macro_use] extern crate jupiter;
/// # #[tokio::main]
/// # async fn main() {
/// spawn!(async move {
///     // perform some async stuff here...
/// });
/// # }
#[macro_export]
macro_rules! spawn {
    ($e:expr) => {{
        std::mem::drop(tokio::spawn($e));
    }};
}

#[cfg(test)]
mod testing {
    use redis::{Connection, RedisError};
    use std::sync::Mutex;
    use tokio::time::Duration;

    lazy_static::lazy_static! {
        /// Provides a global lock which has to be acquired if a test operates on shared
        /// resources. This would either be our test port (1503) on which we start our
        /// local server for integrations tests or the repository which operates on the
        /// file system. Using this lock, we can still execute all other tests in parallel
        /// and only block if required.
        pub static ref SHARED_TEST_RESOURCES: Mutex<()> = Mutex::new(());
    }

    /// Executes async code within a single threaded tokio runtime.
    pub fn test_async<F: std::future::Future>(future: F) {
        use tokio::runtime;

        let rt = runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .unwrap();

        let _ = rt.block_on(future);
    }

    /// Executes a blocking Redis query in an async fashion.
    ///
    /// This is required as we must not block tokio in any way. Note that the redis create itself
    /// would permits async queries, however, this seems to rely on a previous version of tokio
    /// which crashes with our version (tokio 1.0.0).
    pub async fn query_redis_async<T, Q>(query: Q) -> Option<T>
    where
        Q: FnOnce(&mut Connection) -> Result<T, RedisError> + Send + Sync + 'static,
        T: Send + 'static,
    {
        let result = tokio::task::spawn_blocking(|| {
            let client = redis::Client::open("redis://127.0.0.1:1503").unwrap();
            let mut con = client
                .get_connection_with_timeout(Duration::from_secs(5))
                .unwrap();
            query(&mut con)
        })
        .await;

        match result {
            Ok(Ok(result)) => Some(result),
            _ => None,
        }
    }
}