|
|
| Line 1: |
Line 1: |
| <p> [[Firebird]] </p>
| | = Getting Started = |
| | | * [[Firebird/GettingStarted/docker|Using Docker]] |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Preparation</div>
| |
| <div class="mw-collapsible-content">
| |
| * Assuming you are using a linux based machine. (I am using a Ubuntu 22.04) | |
| * Assuming you can ssh to this machine with a private ssh key
| |
| * Create a docker network called "streaming"
| |
| | |
| <pre><nowiki>
| |
| docker network create -d bridge streaming
| |
| </nowiki></pre>
| |
| | |
| * Create a local directory ~/firebird_test/data, we use it to map some docker volumes.
| |
| <pre><nowiki>
| |
| mkdir -p ~/firebird_test/data
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Step 1: checkout the example</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| git clone https://github.com/stonezhong/firebird.git
| |
| | |
| # then enters the example home directory
| |
| cd firebird/examples/getting-started/
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Step 2: customize your ssh config</div>
| |
| <div class="mw-collapsible-content">
| |
| You need to edit app/ssh_config/config and app/ssh_config/keys/devbox
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Step 3: start zookeeper and rabbitmq</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| docker-compose -f infra/docker-compose.yaml up -d
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">Step 4: build docker image for your streaming application</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| # Build docker image for the streaming application
| |
| docker-compose -f app/docker-compose.yaml build
| |
| | |
| # start the console container
| |
| docker-compose -f app/docker-compose.yaml up -d
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 5: register your pipeline</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| # Before we can use any pipeline, we need to register the pipeline with the system
| |
| # A pipeline module is a python module which implements function get_pipeline(..) which returns a pipeline
| |
| # see app/src/pipe.py, since we use pipe.py, so the pipeline module name is "pipe"
| |
| docker exec -it console pipeline register -pmn pipe
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 7: list pipelines</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| docker exec -it console pipeline list
| |
| | |
| test:
| |
| module: pipe
| |
| executors: None
| |
| </nowiki></pre>
| |
| * Now you can see pipeline "test" has been registered, the id is returned from <code>app/src/pipe.py</code>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 8: execute pipeline</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| # We are executing "test", using docker host "devbox", and let the container name be "foo"
| |
| docker exec -it console pipeline execute -pid test -dhn devbox -dcn foo
| |
| | |
| # once it is done, you can
| |
| docker ps # you can see the container with name "foo"
| |
| | |
| # check the logs
| |
| docker logs -f foo
| |
| | |
| # you can also list pipelines
| |
| docker exec -it console pipeline list
| |
| | |
| test:
| |
| module: pipe
| |
| executors:
| |
| 58a6fb38-1bf7-4c77-8f08-377c3162a101:
| |
| docker_host_name = devbox
| |
| docker_container_name = foo
| |
| worker_count = 1
| |
| start_time = 2023-05-04 02:01:27
| |
| pid = 1
| |
| | |
| # You can see one executor has been registered.
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 9: stop an executor</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| # if we want to stop an executor, we can do
| |
| docker exec -it console pipeline stop -pid test -eid 58a6fb38-1bf7-4c77-8f08-377c3162a101
| |
| # This command just send a signal to the executor so it will quit after proper cleanup.
| |
| # after a while, you can list the pipe again you will see the executor is gone:
| |
| | |
| docker exec -it console pipeline list
| |
| test:
| |
| module: pipe
| |
| executors: None
| |
| </nowiki></pre>
| |
| | |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 10: web console: create database</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| # connect to mysql instance (e.g. using MySQL workbench)
| |
| CREATE SCHEMA `firebird` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
| |
| | |
| # give stonezhong full access to this db
| |
| GRANT ALL ON `firebird`.* TO 'stonezhong'@'%';
| |
| FLUSH PRIVILEGES;
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 11: web console: initialize database</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| fbconsole migrate
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed expandable">
| |
| <div class="mw-collapsible-preview">step 12: web console: start web console</div>
| |
| <div class="mw-collapsible-content">
| |
| <pre><nowiki>
| |
| fbconsole runserver 0.0.0.0:8000
| |
| </nowiki></pre>
| |
| </div>
| |
| </div>
| |
| <p></p>
| |
