Module chewdata::step::validator

Check the consistancy of the data.

If a data is not valid, an error message is stored in the field _error before to share the data to another step and the data is tagged with an error. Use the data_type field of a step to target which kind of data a step can handle.

Actions

1 - Get a crate::Context from the input queue.
2 - Extract the crate::DataResult from the crate::Context.
3 - Validate the data with a list of rules.
4 - Create a new crate::Context and attach the crate::DataResult to it.
5 - Push the new crate::Context into the output queue.
6 - Go to step 1 until the input queue is not empty.

Configuration

key	alias	Description	Default Value	Possible Values
type	-	Required in order to use transformer step	transformer	transformer / transform / t
updater	u	Updater type used as a template engine for transformation	tera	tera
referentials	refs	List of crate::step::Reader indexed by their name. A referential can be use to map object during the validation	null	{"alias_a": READER,"alias_b": READER, etc...}
name	alias	Name step	null	Auto generate alphanumeric value
data_type	data	Type of data used for the transformation. skip other data type	ok	ok / err
concurrency_limit	-	Limit of steps to run in concurrence.	1	unsigned number
rules	-	List of self::Rule indexed by their names	null	{"rule_0": Rule,"rule_1": Rule}
error_separator	-	Separator use to delimite two errors	\r\n	String

Rule

key	Description	Default Value	Possible Values
pattern	Pattern in django template language format used to test a field. If the result of the pattern is not a boolean, an error will raised	null	{% if true %} true {% else %} false {% endif %}
message	Message to display if the render pattern is false. If the message is empty, a default value is rendered	string	My error message

Examples

[
    {
        "type": "validator",
        "updater": {
            "type": "tera"
        },
        "referentials": {
            "mapping_ref": {
                "connector": {
                    "type": "mem",
                    "data": "[{\"mapping_code\":\"value_to_map\",\"mapping_value\":\"value mapped\"},{\"mapping_code\":\"value_to_map_2\",\"mapping_value\":\"value mapped 2\"}]"
                }
            }
        },
        "name": "my_validator",
        "data_type": "ok",
        "concurrency_limit": 1,
        "rules": {
            "number_rule": {
                "pattern": "{% if input.number == 10  %} true {% else %} false {% endif %}",
                "message": "The number field value must be equal to 10"
            },
            "text_rule": {
                "pattern": "{% if input.text is matching('.*hello world.*') %} true {% else %} false {% endif %}",
                "message": "The text field value doesn't contain 'Hello World'"
            },
            "code_rule": {
                "pattern": "{% if mapping_ref | filter(attribute='mapping_code', value=input.code) | length > 0 %} true {% else %} false {% endif %}",
                "message": "The code field value doesn't match with the referential dataset"
            }
        },
        "error_separator": " & "
    }
]

input:

[
    {"number": 100, "text": "my text", "code": "my_code"},
    ...
]

output:

[
    {"number": 100, "text": "my text", "code": "my_code", "_error":"The number field value must be equal to 10 & The text field value doesn't contain 'Hello World' & The code field value doesn't match with the referential dataset"},
    ...
]

Structs

• Rule
• Validator