Revision as of 18:00, 25 November 2025

ETL

ETL Flow

Diagram	Description
	1 Airflow Scheduler trigger DAG (DAG is generated based on metadata) The ETL job is a task within an airflow DAG 2 ETL executor pulls code from a repo into loacl disk 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server) 4 Thrift Server take the SQL and pass it to Apache Spark to execute

@@ Line 14: / Line 14: @@
      Scheduler[Apache Airflow/Scheduler]
      ETLE[ETL Executor&lt;Airflow Task&gt;]
-     LC[Local Code]
+     LC[Local ETL Code]
+    ER[ETL Code Repo]
      JDBC[JDBC&lt;Thrift Server&gt;]
      Spark[Apache Spark]
@@ Line 21: / Line 22: @@
      ETLE --3 dbt--> JDBC
      JDBC --4--> Spark
+    ER --> LC
 }}
 | style="vertical-align:top;" |