Revision as of 17:58, 25 November 2025

ETL

ETL Flow

Diagram	Description
	1 Airflow Scheduler trigger DAG (DAG is generated based on metadata) The ETL job is a task within an airflow DAG 2 ETL executor pulls code from a repo into loacl disk 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server) 4 Thrift Server take the SQL and pass it to Apache Spark to execute

@@ Line 1: / Line 1: @@
 <p>[[dl/home|Data Lake Knowledge Center]]</p>
-= Purpose =
+= ETL =
 <div class="toccolours mw-collapsible mw-collapsed expandable">
 <div class="mw-collapsible-preview">ETL Flow</div>