Revision as of 18:56, 25 November 2025

ETL

ETL Flow

1: User pushes ETL code into ETL Code Repo
2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
- The ETL job is a task within an airflow DAG
3: ETL executor pulls code from ETL Code Repo into loacl disk
4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
5: Thrift Server take the SQL and pass it to Apache Spark to execute

Using MPP Engine

BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
- Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

Using RDBMS

BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
- Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

@@ Line 49: / Line 49: @@
      PlatniumTier --> MPP
      MPP --> JDBC --> BI
+}}
+<br />
+* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
+** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server
+</div>
+</div>
+<p></p>
+<div class="toccolours mw-collapsible mw-collapsed expandable">
+<div class="mw-collapsible-preview">Using RDBMS</div>
+<div class="mw-collapsible-content">
+{{#mermaid:
+graph TD
+    subgraph storage
+        GoldTier[Gold Tier]
+        PlatniumTier[Platnium Tier]
+    end
+    RDBMS[RDBMS#40;Oracle ADW#41;]
+    JDBC[JDBC Interface]
+    BI[BI Tool #40;Power BI#41;]
+    GoldTier --replicate--> RDBMS
+    PlatniumTier --replicate--> RDBMS
+    RDBMS --> JDBC --> BI
 }}
 <br />