Revision as of 17:57, 25 November 2025

Purpose

ETL Flow

Diagram	Description
	1 Airflow Scheduler trigger DAG (DAG is generated based on metadata) The ETL job is a task within an airflow DAG 2 ETL executor pulls code from a repo into loacl disk 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server) 4 Thrift Server take the SQL and pass it to Apache Spark to execute

@@ Line 1: / Line 1: @@
 <p>[[dl/home|Data Lake Knowledge Center]]</p>
-= Overview =
+= Purpose =
+<div class="toccolours mw-collapsible mw-collapsed expandable">
+<div class="mw-collapsible-preview">ETL Flow</div>
+<div class="mw-collapsible-content">
 {| class="wikitable grid mono section"
 |-
@@ Line 26: / Line 29: @@
 * 4 Thrift Server take the SQL and pass it to Apache Spark to execute
 |}
+</div>
+</div>
+<p></p>