Firebird

Firebird -- Python based Streaming Framework

Introduction

It is a python based streaming processing framework

APIs

Node

Represent a node in the pipeline. A node can have bunch of input ports and bunch of output ports. Each port has a unique name within the node.

Properties:

• name: the name of the node, it is unique within a pipeline
• INPUT: a dictionary like object, allowing you to get input port by name, e.g. node.INPUT["foo"] will return the input port with name "foo"
• input: the default input port, e.g. node.input
• OUTPUT: a dictionary like object, allowing you to get output port by name, e.g. node.OUTPUT["foo"] will return the output port with name "foo"
• output: the default input port, e.g. node.output

Methods:

on_message(self, name:str, data:Any)

This method will be called for this node to handle data point, data is a json object, name represent from which port the data is received.

def connect(self, *dest_ports:Union["Node", "Port"]) -> "Port"

# connect the default output port to a input port of a node
# if dest_port element is a Node, then the default input port is used

def emit(self, data:Any, name:str=DEFAULT_PORT_NAME)

# emit data to it's output port specified by name

Generator

Represent a node that does not have input ports.

Methods:

def pump(self)

# derived class must override this method to pump data to the pipeline.

Sink

Pipeline

RabbitMQ