Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
actiondb
Actiondb is a library and its associated tools to efficiently extract information from unstructured data. It's a tool to parse logs and extract key-value pairs with predefined patterns from them.
The patterns can be specified line-by-line (plain pattern file) or in a JSON serialized file.
The latter one allows you to give a name, a unique identifier (UUID
) and to test messages to each pattern.
This library is intended to used with syslog-ng
. You can find a Docker image which uses this library
as a parser:
Patterns
A pattern is composed of literals and parsers, like:
Jun %{INT:day} %{INT:hour}:%{INT:min}:%{INT:sec} server sshd[%{INT:pid}]: Accepted publickey for joe
It can be used to parse the following log message:
Jun 25 14:09:58 server sshd[26665]: Accepted publickey for joe
Plain pattern files
A plain pattern file contains patterns in each lines as they are defined in Patterns. ActionDB will generate a random UUID for each pattern. If there is a syntax error in one pattern the rest won't be checked and ActionDB returns an error.
JSON pattern files
These files contains patterns and their attributes. A JSON file looks like the following example
It has the following structure:
patterns
: it's a top level array of pattern objects
A pattern object consists of the following key-value pairs:
uuid
: it's a required field and contains a UUID,name
: it's an optional field and contains the name of the pattern. Currently there is no restriction about the valid character set.pattern
: it's the same thing as defined in Patternstest_messages
: it's an array of test messages which can be used to test the patters.
A test message object has the following key-value pairs:
message
: a string message which should be parsed,values
: an object which defines the expected key-value pairs after the parsing. Every key and value must be strings.
An example test message object can be seen in the following example:
Parsers
Parsers can be used to extract data from unstructured text.
Every parser has the following syntax:
%{PARSER_TYPE(required_arg1, required_arg2, optional_arg1="value", optional_arg2=value):parser_instance_name}
If a parser doesn't have extra arguments its parameter list can be omitted:
%{PARSER_TYPE:parser_instance_name}
You can use the _
, .
, [0-9]
, -
and [a-zA-Z]
characters as parser names.
Available parsers
SET
Parses only the characters which was given as its arguments. An optional minimum and maximum length can be specified.
Example
%{SET("abcd",min_len=1,max_len=2):parsed_value_name}
It's identical to the [abcd]{1,2}
regular expression (but faster).
INT
It reuses the SET
parser with the character set of the numbers from 0
to
9
. An optional minimum and maximum length can be specified as in SET.
GREEDY
It tries to fill in the gap between a parser and a literal or two literals. It will use
the next literal as an "end string" condition. If the GREEDY
parser is the last parser
in the pattern it will consume the whole remaining message.
Example
Pattern:
from %{GREEDY:ipaddr}: %{INT:dunno}
Sample message:
from 1.2.3.4: 123
Extracted key-value pairs:
(ipaddr,1.2.3.4)
(dunno,123)
adbtool
adbtool
is a tool which can be used for the following purposes:
- validate patterns,
- parse texts.
It support the validate
and parse
subcommands. For more information check
it's --help
option.
Changelog
Actiondb 0.2.0
User visible changes:
- support JSON pattern files
- nicer and more precise error messages
Internal changes:
Matcher
becomes a trait andParserTrie
implements it- the pattern reading and trie building code is extracted into a
Builder
struct Builder
is able to populate anyMatcher
instance from any type which implements thePatternSource
traitBuildResult = Result<Pattern, BuildError>
BuildError
contains all possibleError
types (IO, pattern parse errors, etc.)PatternSource
is automatically implemented for every type which implementsIterator<Item=BuildResult>
- this makes possible to generalize the
Matcher
building logic: BuildResult
s are being read from aPatternSource
and if they areOk()
then they are added to theMatcher
- in case of an
Err()
value the building process stops and the error is returned Factory
is introduced to createMatcher
instances from files (JSON and plain pattern files)Factory::form_file()
is file extension agnostic and creates aMatcher
instance from the given file- the big modules are split into smaller submodules
- allow
.
character inParser
names - the
JSON
files can contain test messages. They are tested when their pattern is added to theMatcher
. Coveralls.io
checks every modifications- new unit tests are added