Struct DoraNode

Source

pub struct DoraNode { /* private fields */ }

Expand description

Allows sending outputs and retrieving node information.

The main purpose of this struct is to send outputs via Dora. There are also functions available for retrieving the node configuration.

Implementations§

Source §

impl DoraNode

Source

pub fn init_from_env() -> Result<(Self, EventStream)>

Initiate a node from environment variables set by the Dora daemon or fall back to interactive mode.

This is the recommended initialization function for Dora nodes, which are spawned by Dora daemon instances. The daemon will set a DORA_NODE_CONFIG environment variable to configure the node.

When the node is started manually without the DORA_NODE_CONFIG environment variable set, the initialization will fall back to init_interactive if stdin is a terminal (detected through isatty).

If the DORA_NODE_CONFIG environment variable is not set and DORA_TEST_WITH_INPUTS is set, the node will be initialized in integration test mode. See the integration testing module for details.

This function will also initialize the node in integration test mode when the setup_integration_testing function was called before. This takes precedence over all environment variables.

use dora_node_api::DoraNode;

let (mut node, mut events) = DoraNode::init_from_env().expect("Could not init node.");

Source

pub fn init_from_env_force() -> Result<(Self, EventStream)>

Initialize the node from environment variables set by the Dora daemon; error if not set.

This function behaves the same as init_from_env, but it does not fall back to init_interactive. Instead, an error is returned when the DORA_NODE_CONFIG environment variable is missing.

Source

pub fn init_from_node_id(node_id: NodeId) -> Result<(Self, EventStream)>

Initiate a node from a dataflow id and a node id.

This initialization function should be used for dynamic nodes.

use dora_node_api::DoraNode;
use dora_node_api::dora_core::config::NodeId;

let (mut node, mut events) = DoraNode::init_from_node_id(NodeId::from("plot".to_string())).expect("Could not init node plot");

Source

pub fn init_flexible(node_id: NodeId) -> Result<(Self, EventStream)>

Dynamic initialization function for nodes that are sometimes used as dynamic nodes.

This function first tries initializing the traditional way through init_from_env. If this fails, it falls back to init_from_node_id.

Source

pub fn init_interactive() -> Result<(Self, EventStream)>

Initialize the node in a standalone mode that prompts for inputs on the terminal.

Instead of connecting to a dora daemon, this interactive mode prompts for node inputs on the terminal. In this mode, the node is completely isolated from the dora daemon and other nodes, so it cannot be part of a dataflow.

Note that this function will hang indefinitely if no input is supplied to the interactive prompt. So it should be only used through a terminal.

Because of the above limitations, it is not recommended to use this function directly. Use init_from_env instead, which supports both normal daemon connections and manual interactive runs.

§Example

Run any node that uses init_interactive or init_from_env directly from a terminal. The node will then start in “interactive mode” and prompt you for the next input:

> cargo build -p rust-dataflow-example-node
> target/debug/rust-dataflow-example-node
hello
Starting node in interactive mode as DORA_NODE_CONFIG env variable is not set
Node asks for next input
? Input ID
[empty input ID to stop]

The rust-dataflow-example-node expects a tick input, so let’s set the input ID to tick. Tick messages don’t have any data, so we leave the “Data” empty when prompted:

Node asks for next input
> Input ID tick
> Data
tick 0, sending 0x943ed1be20c711a4
node sends output random with data: PrimitiveArray<UInt64>
[
  10682205980693303716,
]
Node asks for next input
? Input ID
[empty input ID to stop]

We see that both the stdout output of the node and also the output messages that it sends are printed to the terminal. Then we get another prompt for the next input.

If you want to send an input with data, you can either send it as text (for string data) or as a JSON object (for struct, string, or array data). Other data types are not supported currently.

Empty input IDs are interpreted as stop instructions:

> Input ID
given input ID is empty -> stopping
Received stop
Node asks for next input
event channel was stopped -> returning empty event list
node reports EventStreamDropped
node reports closed outputs []
node reports OutputsDone

In addition to the node output, we see log messages for the different events that the node reports. After OutputsDone, the node should exit.

§JSON data

In addition to text input, the Data prompt also supports JSON objects, which will be converted to Apache Arrow struct arrays:

Node asks for next input
> Input ID some_input
> Data { "field_1": 42, "field_2": { "inner": "foo" } }

This JSON data is converted to the following Arrow array:

StructArray
-- validity: [valid, ]
[
  -- child 0: "field_1" (Int64)
     PrimitiveArray<Int64>
     [42,]
  -- child 1: "field_2" (Struct([Field { name: "inner", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]))
     StructArray
     -- validity: [valid,]
     [
       -- child 0: "inner" (Utf8)
       StringArray
       ["foo",]
     ]
]

Source

pub fn init_testing( input: TestingInput, output: TestingOutput, options: TestingOptions, ) -> Result<(Self, EventStream)>

Initializes a node in integration test mode.

No connection to a dora daemon is made in this mode. Instead, inputs are read from the specified TestingInput, and outputs are written to the specified TestingOutput. Additional options for the testing mode can be specified through TestingOptions.

It is recommended to use this function only within test functions.

Source