Firebird/GettingStarted

Firebird

Preparation

• Assuming you are using a linux based machine. (I am using a Ubuntu 22.04)
• Assuming you can ssh to this machine with a private ssh key
• Create a docker network called "streaming"

docker network create -d bridge streaming

• Create a local directory ~/firebird_test/data, we use it to map some docker volumes.

mkdir -p ~/firebird_test/data

Step 1: checkout the example

git clone https://github.com/stonezhong/firebird.git

# then enters the example home directory
cd firebird/examples/getting-started/

Step 2: customize your ssh config

You need to edit app/ssh_config/config and app/ssh_config/keys/devbox

Step 3: start zookeeper and rabbitmq

docker-compose -f infra/docker-compose.yaml up -d

Step 4: build docker image for your streaming application

# Build docker image for the streaming application
docker-compose -f app/docker-compose.yaml build

# start the console container
docker-compose -f app/docker-compose.yaml up -d

step 5: register your pipeline

# Before we can use any pipeline, we need to register the pipeline with the system
# A pipeline module is a python module which implements function get_pipeline(..) which returns a pipeline
# see app/src/pipe.py, since we use pipe.py, so the pipeline module name is "pipe"
docker exec -it console pipeline register -pmn pipe

step 7: list pipelines

docker exec -it console pipeline list

test:
    module: pipe
    executors: None

• Now you can see pipeline "test" has been registered, the id is returned from app/src/pipe.py

step 8: execute pipeline

# We are executing "test", using docker host "devbox", and let the container name be "foo"
docker exec -it console pipeline execute -pid test -dhn devbox -dcn foo

# once it is done, you can
docker ps # you can see the container with name "foo"

# check the logs
docker logs -f foo

# you can also list pipelines
docker exec -it console pipeline list

test:
    module: pipe
    executors:
        58a6fb38-1bf7-4c77-8f08-377c3162a101:
            docker_host_name      = devbox
            docker_container_name = foo
            worker_count          = 1
            start_time            = 2023-05-04 02:01:27
            pid                   = 1

# You can see one executor has been registered.

step 9: stop an executor

# if we want to stop an executor, we can do
docker exec -it console pipeline stop -pid test -eid 58a6fb38-1bf7-4c77-8f08-377c3162a101
# This command just send a signal to the executor so it will quit after proper cleanup.
# after a while, you can list the pipe again you will see the executor is gone:

docker exec -it console pipeline list
test:
    module: pipe
    executors: None

step 10: web console: create database

# connect to mysql instance (e.g. using MySQL workbench)
CREATE SCHEMA `firebird` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;