Firebird/GettingStarted

Firebird

Setup Firebird Kubernete Cluster

Brief

This section talks about how to setup firebird environment. Files references could be found at https://github.com/stonezhong/firebird/tree/master/examples/getting-started

• Assuming you have a kubernete cluster already created. In this example, we are using Kubernete provided by Oracle Cloud Infrastructure (aka OKE).
• Assuming you have a machine with kubectl installed, from which you can control the kubernete cluster

Create kubernete namespace: firebird

• We are going to deploy in this namespace
• namespace.yaml

kubectl apply -f namespace.yaml
# Then check it, you should see firebird as namespace
kubectl get namespaces

# Then you can set firebird as your currrent namespace
kubectl config set-context --current --namespace=firebird

Setup rabbitmq and zookeeper

• rabbitmq.yaml
• zookeeper.yaml

# the rabbitmq.yaml has specified the admin username and password
cd infra
kubectl apply -f rabbitmq.yaml
kubectl apply -f zookeeper.yaml

Setup firebird web console: create database

Although you can create a MySQL instance in your kubernetes cluster, it is easier to create a MySQL instance if you are using a cloud provider, such as AWS, GCP, etc.

# assuming you have a MySQL 8.x installed

# create database user stonezhong
CREATE USER 'stonezhong'@'%' IDENTIFIED WITH mysql_native_password BY 'dbpassword';

# connect to mysql instance (e.g. using MySQL workbench)
CREATE SCHEMA `firebird` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

# give stonezhong full access to this db
GRANT ALL ON `firebird`.* TO 'stonezhong'@'%';

Setup firebird web console: build docker image

cd console
docker build -t firebird-console .
# now push this image to your private repo
docker tag firebird-console iad.ocir.io/idrnu3akjpv5/firebird-console
docker push iad.ocir.io/idrnu3akjpv5/firebird-console

Notes:

• You have config file at firebird_config/config.json, this file contains configurations about mysql, rabbitmq and zookeeper, you may need to modify it in case you have different configuration

Create Secrets

kubectl create secret generic \
    firebird \
    --from-file=app_config.json=secrets/app_config.json \
    --from-file=kube_config=secrets/kube_config \
    --from-file=oci_config=secrets/oci_config \
    --from-file=oci_api_key.pem=secrets/oci_api_key.pem

• secrets/app_config.json, it contains information about zookeeper, rabbitmq and mysql

{
    "zookeeper": {
        "hosts": "zookeeper:2181"
    },
    "rabbitmq": {
        "username": "stonezhong",
        "password": "changeme",
        "host": "rabbitmq",
        "port": 5672,
        "heartbeat": 300
    },
    "mysql": {
        "db_name": "firebird",
        "username": "stonezhong",
        "password": "foobar",
        "server": "10.2.0.29"
    }
}

• secrets/kube_config: this is your kubernete config file, usually you can copy it from ~/.kube/config from machine which you run kubectl
• secrets/oci_config and secrets/oci_api_key.pem are your OCI configs. (this demo is about using firebird on Oracle Cloud Infracture)

Setup firebird web console: iniialize database

kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console:latest -- sh
fbconsole migrate

Deploy firebird web console

cd console
kubectl apply -f console.yaml
# once deployment is done, you can try to access it
curl http://<service-external-ip>/
# and make sure you see the page, you can latter create a ssh tunnel to access the Firebird Web Console.

Deployment Sample Streaming Application

Streaming app: build image

cd app
docker build -t firebird-app1 .
# now push this image to your private repo
docker tag firebird-app1 iad.ocir.io/idrnu3akjpv5/firebird-app1
docker push iad.ocir.io/idrnu3akjpv5/firebird-app1

Streaming app: register your pipeline

# Before we can use any pipeline, we need to register the pipeline with the system
# A pipeline module is a python module which implements function get_pipeline(..) which returns a pipeline
# see app/src/pipe.py, since we use pipe.py, so the pipeline module name is "pipe"

kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-app1 -- sh
pipeline register \
    --pipeline-module-name pipe \
    --pipeline-namespace-name firebird \
    --pipeline-image-name iad.ocir.io/idrnu3akjpv5/firebird-app1

# now you can list pipeline to make sure it is registered
/usr/src/app # pipeline list
test:
    namespace: firebird
    image    : iad.ocir.io/idrnu3akjpv5/firebird-app1
    module   : pipe
    running  : No
    Kubernetes
        618a0014-f9fb-475c-9d3f-840dc80d7821: src
        c3cbcea2-0936-4f80-9635-41fc7d49650e: puller
    executors: None

Streaming app: start pipeline

kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console -- sh
# or
# kubectl exec -it <console_pod_name> -- sh
pipeline start -pid test

Streaming app: stop pipeline

kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console -- sh
# or
# kubectl exec -it <console_pod_name> -- sh
pipeline stop -pid test