Revision as of 18:02, 25 November 2025

ETL

ETL Flow

Diagram	Description
	1 Airflow Scheduler trigger DAG (DAG is generated based on metadata) The ETL job is a task within an airflow DAG 2 ETL executor pulls code from a repo into loacl disk 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server) 4 Thrift Server take the SQL and pass it to Apache Spark to execute

@@ Line 17: / Line 17: @@
      ER[ETL Code Repo]
      JDBC[JDBC&lt;Thrift Server&gt;]
+    User[User&lt;Data Engineer&gt;]
      Spark[Apache Spark]
      Scheduler --1 trigger--> ETLE
@@ Line 23: / Line 24: @@
      JDBC --4--> Spark
      ER --> LC
+    USER --5 git push-->ER
 }}
 | style="vertical-align:top;" |