Dl/Overview

Data Lake Knowledge Center

ETL

ETL Flow

• 1: User pushes ETL code into ETL Code Repo
• 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
  • The ETL job is a task within an airflow DAG
• 3: ETL executor pulls code from ETL Code Repo into loacl disk
• 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
• 5: Thrift Server take the SQL and pass it to Apache Spark to execute

BI Connection

Using MPP Engine

• BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
  • Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server