Firebird/Concepts

Firebird

Overview

pipeline

• A Pipe is a data processing unit, it may have zero or more input ports, and may have zero or more output ports
• A Generator is a Pipe that does not have any input ports
• A Sink is a Pipe that does not have any output ports

wiring

• You can connect an output port to one or more input ports.
• When you emit data via an output port, the data gets sends to all the input port that is connected, and the node owns those input will receive the data.

Models

Node

Represent a node in the pipeline. A node can have bunch of input ports and bunch of output ports. Each port has a unique id within the node.

Properties

id:              The id of the node, it is unique within a pipeline
title:           A human readable name of this node
description:     Detailed description of this node
pipeline:        The pipeline this node belongs to
input_port_ids:  tuple of all input port ids
output_port_ids: tuple of all output port ids
input:           The default input port
output:          The default output port

method: get_port

def get_port(self, id:str) -> Optional["Port"]
# each port has a unique id within the node.
# return a port given port id

# You can also use [] to get port by name
node["foo"]  # same as node.get_port("foo")

method: on_message

on_message(self, port_id:str, data:Any)

Called when this node receives data. port_id is the id of the port which receive the data, data is the JSON payload of the data.

method: >> and <<

# You can use >>, << to connect nodes, here are examples:
nodeA >> nodeB               # connect default output port of nodeA to default input port of node B
nodeA["foo"] >> nodeB        # connect port "foo" of nodeA to default input port of node B
nodeA["foo"] >> nodeB["bar"] # connect port "foo" of nodeA to port "bar" of node B

# You can also use << as a reverse of >>

method: emit

def emit(self, data:Any, port_id:str=DEFAULT_OUTPUT_PORT_ID)

# emit data to it's output port specified by port_id

Generator

Represent a node that does not have input ports, it is derived from Node.

methods: pump

def pump(self)

# derived class must override this method to pump data to the pipeline.

Sink

Represent a node that does not have output ports, it is derived from Node.

Port

Represent an input port or output port of a node.

Properties

type:  Either PortType.INPUT or PortType.OUTPUT, represent it is a input port or output port
id:    ID of the port, it is unique within the node
node:  The node which this port belongs to

method: >> and <<

# connect this port to other port
# see Node document.

method: emit

def emit(self, json_data:Any)
# emit JSON data to this port

Pipeline

Represent a pipeline, which has

• bunch of nodes
• A node's output port can connect to another node's input port

Properties

id:          The id of the pipeline, every pipeline has a unique id.
title:       A human readable short description of this pipeline.
description: Detailed description of this pipeline.
mq:          RabbitMQ instance
nodes:       Containing all node belongs to this pipeline.

method: message_loop

def message_loop(self):

# enters a message loop, allow each node to process data

method: []

# You can use [] to get node by id
pipeline["foo"]  # get node with id "foo" of this pipeline

RabbitMQ

Represent a RabbitMQ connection

Exception Handling

• When you register a pipeline, 3 message queue will be created, they are
• ${pipeline_name}: all the message gets routed here
• ${pipeline_name}-error: when a message failed to process, it will be posted here, it has following fields: error_count, recent_errors: array of {failed_time, error_message}
• ${pipeline_critical}-critical: when a message failed too many times, it will be moved here.