Firebird/GettingStarted
From stonehomewiki
Revision as of 02:05, 4 May 2023 by Stonezhong (talk | contribs)
Preparation
- Assuming you are using a linux based machine. (I am using a Ubuntu 22.04)
- Assuming you can ssh to this machine with a private ssh key
- Create a docker network called "streaming"
docker network create -d bridge streaming
Step 1: checkout the example
git clone https://github.com/stonezhong/firebird.git # then enters the example home directory cd firebird/examples/getting-started/
Step 2: customize your ssh config
You need to edit app/ssh_config/config and app/ssh_config/keys/devbox
Step 3: start zookeeper and rabbitmq
docker-compose -f infra/docker-compose.yaml up -d
Step 4: build docker image for your streaming application
# Build docker image for the streaming application docker-compose -f app/docker-compose.yaml build # start the console container docker-compose -f app/docker-compose.yaml up -d
step 5: register your pipeline
# Before we can use any pipeline, we need to register the pipeline with the system # A pipeline module is a python module which implements function get_pipeline(..) which returns a pipeline # see app/src/pipe.py, since we use pipe.py, so the pipeline module name is "pipe" docker exec -it console pipeline register -pmn pipe
step 7: list pipelines
docker exec -it console pipeline list
test:
module: pipe
executors: None
- Now you can see pipeline "test" has been registered, the id is returned from
app/src/pipe.py
step 8: execute pipeline
# We are executing "test", using docker host "devbox", and let the container name be "foo"
docker exec -it console pipeline execute -pid test -dhn devbox -dcn foo
# once it is done, you can
docker ps # you can see the container with name "foo"
# check the logs
docker logs -f foo
# you can also list pipelines
docker exec -it console pipeline list
test:
module: pipe
executors:
58a6fb38-1bf7-4c77-8f08-377c3162a101:
docker_host_name = devbox
docker_container_name = foo
worker_count = 1
start_time = 2023-05-04 02:01:27
pid = 1
# You can see one executor has been registered.
step 9: stop an executor
# if we want to stop an executor, we can do
docker exec -it console pipeline stop -pid test -eid 58a6fb38-1bf7-4c77-8f08-377c3162a101
# This command just send a signal to the executor so it will quit after proper cleanup.
# after a while, you can list the pipe again you will see the executor is gone:
docker exec -it console pipeline list
test:
module: pipe
executors: None