combot 0.1.1

A utility to parse server access logs and detect bots based on URI paths and User Agents.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

# ComBot 0.1.1

A command line utility to parse server access logs, detect bots, and output a list of detected bots in the specified format.

## Supported Input Formats

- [x] NGINX
- [ ] Apache

## Supported Output Formats

- [x] CSV
- [x] [AbuseIPDB](https://abuseipdb.com) CSV (don't forget to dedpulicate this list before submitting)
- [ ] JSON

## Trigger Lists

This tool supports specifying "trigger lists" or using the built in lists.

List files should be specified in the following format:

```
botname1|trigger string one
botname2|trigger string two
```

The pipe `|` character is used to separate the trigger name from the trigger string, each line is a trigger.

Two lists are supported, these are URI lists and User Agent lists. Trigger strings work best when they are a substring of the respective part of the log entry. For example:

For a URI trigger use `phpunit` not `/test/phpunit/submit.php`.

For a User Agent trigger use `zgrab` not `Mozilla/5.0 zgrab/0.x`.

URIs take precedence over User Agents as they tend to be more specific.

## Help

```
combot 0.1.0	GNU-GPL-3.0
Chad Baxter
A utility to parse server access logs and detect bots based on URI paths and User Agents.

USAGE:
    combot [OPTIONS] <input> <output> --input_format <input_format> --output_format <output_format>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -i, --input_format <input_format>      Select the input format: nginx
    -f, --output_format <output_format>    Select the output format: csv, abuseipdb-csv
    -a, --ua_list <ua_list>                Specify a path to a list of User Agent pieces to trigger on.
    -u, --uri_list <uri_list>              Specify a path to a list of URI pieces to trigger on.

ARGS:
    <input>     The input file path.
    <output>    The output file path (will be overwritten).
```