Module chewdata::step::transformer

Read the data from the input queue and transform it.

Actions

1 - Get a crate::Context from the input queue.
2 - Extract the crate::DataResult from the crate::Context.
3 - Transform the data with a list of crate::updater::Action.
4 - Create a new crate::Context and attach the crate::DataResult to it.
5 - Push the new crate::Context into the output queue.
6 - Go to step 1 until the input queue is not empty.

Configuration

key	alias	Description	Default Value	Possible Values
type	-	Required in order to use transformer step	transformer	transformer / transform / t
updater	u	Updater type used as a template engine for transformation	tera	tera
referentials	refs	List of crate::step::Reader indexed by their name. A referential can be use to map object during the transformation	null	{"alias_a": READER,"alias_b": READER, etc...}
name	alias	Name step	null	Auto generate alphanumeric value
data_type	data	Type of data used for the transformation. skip other data type	ok	ok / err
concurrency_limit	-	Limit of steps to run in concurrence.	1	unsigned number

Action

key	Description	Default Value	Possible Values
field	Json pointer that define the field path created into the output object	/	alphanumeric or json pointer
pattern	Pattern in django template language format used to build the output field. This project use Tera’s methods	null
type	Type of action	merge	merge current result with the output result / replace the output result with the current result / remove the output field

Examples

[
    ...
    {
        "type": "transformer",
        "updater": {
            "type": "tera"
        },
        "referentials": {
            "ref_a": {
                "connector": {
                    "type": "io"
                }
            }
        },
        "name": "transform_a",
        "connector": {
            "type": "io"
        },
        "document": {
            "type": "json"
        },
        "data_type": "ok",
        "concurrency_limit": 1,
        "actions": [
            { # Force to set 'output' with the data in 'input'.
                "pattern": "{{ input | json_encode() }}"
            },
            {}, # Do the same as before.
            { # Create a new field 'my_new_field' in the output and set the value with the 'pattern' expression.
                "field": "my_new_field",
                "pattern": "{{ input.number * output.number * ref_a.number * steps.my_previous_step.number }}",
                "type": "merge"
            },
            { # Remove the field 'text'.
                "field": "text",
                "type": "remove"
            },
            { # Replace the 'array' field value.
                "field": "array",
                "pattern": "[\"a\",\"b\"]",
                "type": "replace"
            }
        ]
    }
    ...
]

Structs

• Transformer