Dl/DataApplication

Data Lake Knowledge Center

Data Application: for data ingestion

• An application that generates dataframes and update dataset.

Data Application: for data transformation

• Consume one or multiple dataset and generates dataframes and update datasets

Data Pipeline

• Bunch of data application that has dependency
• Runs on a regular basis

It is a good idea to using Apache Airflow to orchestrate data pipeline