Firebird/GettingStarted
From stonehomewiki
Revision as of 09:00, 15 September 2023 by Stonezhong (talk | contribs) (→Setup Firebird Kubernete Cluster)
Setup Firebird Kubernete Cluster
Brief
This section talks about how to setup firebird environment. Files references could be found at https://github.com/stonezhong/firebird/tree/master/examples/getting-started
- Assuming you have a kubernete cluster already created. In this example, we are using Kubernete provided by Oracle Cloud Infrastructure (aka OKE).
- Assuming you have a machine with kubectl installed, from which you can control the kubernete cluster
Create kubernete namespace: firebird
- We are going to deploy in this namespace
- namespace.yaml
kubectl apply -f namespace.yaml # Then check it, you should see firebird as namespace kubectl get namespaces # Then you can set firebird as your currrent namespace kubectl config set-context --current --namespace=firebird
Setup rabbitmq and zookeeper
# the rabbitmq.yaml has specified the admin username and password cd infra kubectl apply -f rabbitmq.yaml kubectl apply -f zookeeper.yaml
Setup firebird web console: create database
Although you can create a MySQL instance in your kubernetes cluster, it is easier to create a MySQL instance if you are using a cloud provider, such as AWS, GCP, etc.
# assuming you have a MySQL 8.x installed # create database user stonezhong CREATE USER 'stonezhong'@'%' IDENTIFIED WITH mysql_native_password BY 'dbpassword'; # connect to mysql instance (e.g. using MySQL workbench) CREATE SCHEMA `firebird` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; # give stonezhong full access to this db GRANT ALL ON `firebird`.* TO 'stonezhong'@'%';
Setup firebird web console: build docker image
cd console docker build -t firebird-console . # now push this image to your private repo docker tag firebird-console iad.ocir.io/idrnu3akjpv5/firebird-console docker push iad.ocir.io/idrnu3akjpv5/firebird-console
Notes:
- You have config file at
firebird_config/config.json, this file contains configurations about mysql, rabbitmq and zookeeper, you may need to modify it in case you have different configuration
Create Secrets
kubectl create secret generic \
firebird \
--from-file=app_config.json=secrets/app_config.json \
--from-file=kube_config=secrets/kube_config \
--from-file=oci_config=secrets/oci_config \
--from-file=oci_api_key.pem=secrets/oci_api_key.pem
secrets/app_config.json, it contains information about zookeeper, rabbitmq and mysql
{
"zookeeper": {
"hosts": "zookeeper:2181"
},
"rabbitmq": {
"username": "stonezhong",
"password": "changeme",
"host": "rabbitmq",
"port": 5672,
"heartbeat": 300
},
"mysql": {
"db_name": "firebird",
"username": "stonezhong",
"password": "foobar",
"server": "10.2.0.29"
}
}
secrets/kube_config: this is your kubernete config file, usually you can copy it from ~/.kube/config from machine which you run kubectlsecrets/oci_configandsecrets/oci_api_key.pemare your OCI configs. (this demo is about using firebird on Oracle Cloud Infracture)
Setup firebird web console: iniialize database
kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console:latest -- sh fbconsole migrate
Deploy firebird web console
cd console kubectl apply -f console.yaml # once deployment is done, you can try to access it curl http://<service-external-ip>/ # and make sure you see the page, you can latter create a ssh tunnel to access the Firebird Web Console.
Deployment Sample Streaming Application
Streaming app: build image
cd app docker build -t firebird-app1 . # now push this image to your private repo docker tag firebird-app1 iad.ocir.io/idrnu3akjpv5/firebird-app1 docker push iad.ocir.io/idrnu3akjpv5/firebird-app1
Streaming app: register your pipeline
# Before we can use any pipeline, we need to register the pipeline with the system
# A pipeline module is a python module which implements function get_pipeline(..) which returns a pipeline
# see app/src/pipe.py, since we use pipe.py, so the pipeline module name is "pipe"
kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-app1 -- sh
pipeline register \
--pipeline-module-name pipe \
--pipeline-namespace-name firebird \
--pipeline-image-name iad.ocir.io/idrnu3akjpv5/firebird-app1
# now you can list pipeline to make sure it is registered
pipeline list
test:
module: pipe
executors: None
Streaming app: start pipeline
kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console -- sh # or # kubectl exec -it <console_pod_name> -- sh pipeline start -pid test
Streaming app: stop pipeline
kubectl run myshell --rm -it --image iad.ocir.io/idrnu3akjpv5/firebird-console -- sh # or # kubectl exec -it <console_pod_name> -- sh pipeline stop -pid test