drep 0.1.3

drep is grep with dynamic reloadable filter expressions. This allows filtering stream of logs/lines, while changing filters on the fly.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

[![Rust](https://github.com/maxpert/drep/workflows/Rust/badge.svg)](https://github.com/maxpert/drep/actions) [![Version](https://img.shields.io/github/v/release/maxpert/drep)](https://github.com/maxpert/drep/releases)

# drep is dynamic regular expression print

`drep` is `grep` with dynamic reloadable filter expressions. This allows filtering stream of 
logs/lines, while changing filters on the fly.

Filter is either a regex or plain text match, provided via input file.  
Here is an example usage:

```bash
tail -f /var/log/nginx/error.log | drep -f /etc/drep/filters
```

Usually you will end up using this with your servers:

```bash
java -jar my-server.jar | drep -f server-filters
```

or 

```bash
./uwsgi -s :8080 -w my_app  | drep -f server-filters
```

## Filter file syntax

Each line of the filters file is an expression that starts with `~`, `=`, `!=`, or `!~`. The matches will be done 
in the order filters written in the file, and if a filter matches subsequent filters won't be executed. 

 - Any line that starts with `!~` implies does not match regex, e.g: `!~"time": \d+.\d{0,2}`
 - Any line that starts with `~` implies match regex, e.g: `~"time": \d+.\d{3,}`
 - Any line that starts with `!=` implies does not contain text, e.g: `!=INFO`
 - Any line that starts with `=` implies contain text, e.g: `="total-duration"`

Everything else is ignored, as you can see from plain text.
For regular expression documentation please refer to [this document](https://docs.rs/regex/1.3.9/regex/). 

## Why?

While `grep --line-buffered` can do something similar changing regex on the fly is not possible. 
Change filter regex on the fly is extremely useful in server/process environments where it's not possible to restart 
the process just to change the `grep` filter. 

Building on unix philosophy `drep` does only one job well, given bunch of filter from an input file 
it can filter input lines to stdout.

## Features

 - Lightweight on CPU, and memory (~3MB memory foot print, and 2 threads in total).
 - Watch and reload filters file.
 - No GC pauses and memory safe (Written in Rust).
 - Plain text & regex matching (with negation support).
 
## Usage example

Given following simple `fizzbuzz.py`:

```python
import time

i = 1
while True:
    fb = ""
    if i % 3 == 0:
        fb = "fizz"
    if i % 5 == 0:
        fb = "{}buzz".format(fb)

    if fb:
        print("{}. {}".format(i, fb), flush=True)

    i = i + 1
    time.sleep(0.1)
```

We can launch and pipe it's output `python fizzbuzz.py | drep -f filters`. Now if the contents of `filters` are:

```
~\sfizz\n
``` 

drep will only emit logs with fizz. e.g.

```
642. fizz
648. fizz
651. fizz
654. fizz
...
```

While keeping the process running without exiting you can just modify `filters` to:

```
~\sbuzz\n
```

This will change the drep output on the fly to only emit buzz:
```
805. buzz
815. buzz
820. buzz
...
```

## Building

Just clone the repo and run `cargo build --release`.