|
|
| (19 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| = Firebird -- Python based Streaming Framework = | | = [[Firebird/Introduction|Introduction]] = |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| | = [[Firebird/GettingStarted|Getting Started]] = |
| <div class="mw-collapsible-preview">Introduction</div>
| | = [[Firebird/Concepts|Concepts]] = |
| <div class="mw-collapsible-content">
| | = [[Firebird/Command Line Tools|Command Line Tools]] = |
| It is a python based streaming processing framework
| | = [[Firebird/Developer Guidelines|Developer Guidelines]] = |
| </div>
| | = [[Firebird/FAQs|FAQs]] = |
| </div>
| | = [[Firebird/Notes|Notes]] = |
| <p></p>
| |
| | |
| == APIs == | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Node</div>
| |
| <div class="mw-collapsible-content">
| |
| Represent a node in the pipeline. A node can have bunch of input ports and bunch of output ports. Each port has a unique id within the node.
| |
| | |
| Properties:
| |
| * id: the id of the node, it is unique within a pipeline
| |
| * input: the default input port, e.g. node.input
| |
| * output: the default input port, e.g. node.output
| |
| * input_port_ids: tuple of all input port ids
| |
| * output_port_ids: tuple of all output port ids
| |
| | |
| | |
| Methods:
| |
| <pre><nowiki>
| |
| def get_port(self, id:str) -> Optional["Port"]
| |
| | |
| # return a port given id
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| on_message(self, name:str, data:Any)
| |
| | |
| This method will be called for this node to handle data point, data is a json object, name represent from which port the data is received.
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| def connect(self, *dest_ports:Union["Node", "Port"]) -> "Port"
| |
| | |
| # connect the default output port to a input port of a node
| |
| # if dest_port element is a Node, then the default input port is used
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| def emit(self, data:Any, name:str=DEFAULT_PORT_NAME)
| |
| | |
| # emit data to it's output port specified by name
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Generator</div>
| |
| <div class="mw-collapsible-content">
| |
| Represent a node that does not have input ports.
| |
| | |
| Methods:
| |
| <pre><nowiki>
| |
| def pump(self)
| |
| | |
| # derived class must override this method to pump data to the pipeline.
| |
| </nowiki></pre>
| |
| | |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Sink</div>
| |
| <div class="mw-collapsible-content">
| |
| Represent a node that does not have output ports.
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Port</div>
| |
| <div class="mw-collapsible-content">
| |
| Represent a input port or output port
| |
| | |
| Properties:
| |
| * type: either PortType.INPUT or PortType.OUTPUT, represent it is a input port or output port
| |
| * name: name of the port, it is unique within the node
| |
| * owner: the node which this port belongs to
| |
| * connected_ports: other ports connected to this port
| |
| | |
| Methods:
| |
| <pre><nowiki>
| |
| def connect(self, *ports: Union["Port", Node])
| |
| | |
| # connect this port to other port
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| def emit(self, json_data:Any)
| |
| | |
| # emit data to this port
| |
| </nowiki></pre>
| |
| | |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Pipeline</div>
| |
| <div class="mw-collapsible-content">
| |
| Represent a pipeline, which has
| |
| * bunch of nodes
| |
| * A node's output port can connect to another node's input port
| |
| | |
| Properties:
| |
| * mq: message queue
| |
| * node_dict: dictionary, key is node name, value is node
| |
| | |
| Methods:
| |
| <pre><nowiki>
| |
| def message_loop(self):
| |
| | |
| # enters a message loop, allow each node to process data
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| def add_node(self, name:str, node_class, *argc, **kwargs)
| |
| | |
| # add a node to the pipeline, *argc, and **kwargs will be passed to the constructor
| |
| </nowiki></pre>
| |
| | |
| <pre><nowiki>
| |
| def connect(self, *, src_node_name, src_port_name=DEFAULT_PORT_NAME, dst_node_name, dst_port_name=DEFAULT_PORT_NAME)
| |
| | |
| # connect one node's output port to another node's input port
| |
| </nowiki></pre>
| |
| | |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">RabbitMQ</div>
| |
| <div class="mw-collapsible-content">
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| = Command Line Tools =
| |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">pipeline</div>
| |
| <div class="mw-collapsible-content">
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
