stonehomewiki - User contributions [en]

Dl/test

2026-01-05T09:02:26Z

Stonezhong: Created page with "hello"

hello

Dl/Overview

2025-11-25T21:02:24Z

Stonezhong: /* ETL */

<p>[[dl/home|Data Lake Knowledge Center]]</p>
= Data Tiers =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Data Tiers</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.
<hr /><br />

<big><b>Gold Tier</b></big>:<br /><br />
<b>Tables for star schema</b>

* dimention tables and fact tables that forms star schema
* a star schema is designed in such a way that it can answer any question about a business area.
<hr /><br />

<big><b>Platinum Tier</b></big>:<br /><br />
<b>Various query result for specific business questions materized in tables</b>

* Various query result for specific business questions materized in tables
** Queries are generated from star schema from gold tier
* Tables may be replicated to a RDBMS for BI tool to access (sometime you can expose them directly, e.g. Spark Thrift Server)

|}
</div>
</div>
<p></p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor#40;Airflow Task#41;]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC#40;Thrift Server#41;]
User[User#40;Data Engineer#41;]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* Gold Tier data and Platnium Tier data are replicated to RDBMS, such as Oracle DB
* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by RDBMS
* This pattern does not work for very large datalake since Gold Tier and Platnium Tier are too large to be replicated to RDBMS

</div>
</div>
<p></p>

Dl/Overview

2025-11-25T19:02:56Z

Stonezhong: /* Data Tiers */

<p>[[dl/home|Data Lake Knowledge Center]]</p>
= Data Tiers =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Data Tiers</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.
<hr /><br />

<big><b>Gold Tier</b></big>:<br /><br />
<b>Tables for star schema</b>

* dimention tables and fact tables that forms star schema
* a star schema is designed in such a way that it can answer any question about a business area.
<hr /><br />

<big><b>Platinum Tier</b></big>:<br /><br />
<b>Various query result for specific business questions materized in tables</b>

* Various query result for specific business questions materized in tables
** Queries are generated from star schema from gold tier
* Tables may be replicated to a RDBMS for BI tool to access (sometime you can expose them directly, e.g. Spark Thrift Server)

|}
</div>
</div>
<p></p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* Gold Tier data and Platnium Tier data are replicated to RDBMS, such as Oracle DB
* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by RDBMS
* This pattern does not work for very large datalake since Gold Tier and Platnium Tier are too large to be replicated to RDBMS

</div>
</div>
<p></p>

Dl/Overview

2025-11-25T19:00:46Z

Stonezhong: /* Data Tiers */

<p>[[dl/home|Data Lake Knowledge Center]]</p>
= Data Tiers =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Data Tiers</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.
<hr /><br />

<big><b>Gold Tier</b></big>:<br /><br />
<b>Tables for star schema</b>

* dimention tables and fact tables that forms star schema
<hr /><br />

<big><b>Platinum Tier</b></big>:<br /><br />
<b>Various query result for specific business questions materized in tables</b>

* Various query result for specific business questions materized in tables
* Tables may be replicated to a RDBMS for BI tool to access (sometime you can expose them directly, e.g. Spark Thrift Server)

|}
</div>
</div>
<p></p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* Gold Tier data and Platnium Tier data are replicated to RDBMS, such as Oracle DB
* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by RDBMS
* This pattern does not work for very large datalake since Gold Tier and Platnium Tier are too large to be replicated to RDBMS

</div>
</div>
<p></p>

Dl/home

2025-11-25T19:00:30Z

Stonezhong: /* Data Tiers */

Dl/Overview

2025-11-25T19:00:03Z

Stonezhong:

<p>[[dl/home|Data Lake Knowledge Center]]</p>
= Data Tiers =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.
<hr /><br />

<big><b>Gold Tier</b></big>:<br /><br />
<b>Tables for star schema</b>

* dimention tables and fact tables that forms star schema
<hr /><br />

<big><b>Platinum Tier</b></big>:<br /><br />
<b>Various query result for specific business questions materized in tables</b>

* Various query result for specific business questions materized in tables
* Tables may be replicated to a RDBMS for BI tool to access (sometime you can expose them directly, e.g. Spark Thrift Server)

|}
</div>
</div>
<p></p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* Gold Tier data and Platnium Tier data are replicated to RDBMS, such as Oracle DB
* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by RDBMS
* This pattern does not work for very large datalake since Gold Tier and Platnium Tier are too large to be replicated to RDBMS

</div>
</div>
<p></p>

Dl/Overview

2025-11-25T18:58:41Z

Stonezhong: /* BI Connection */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* Gold Tier data and Platnium Tier data are replicated to RDBMS, such as Oracle DB
* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by RDBMS
* This pattern does not work for very large datalake since Gold Tier and Platnium Tier are too large to be replicated to RDBMS

</div>
</div>
<p></p>

Dl/Overview

2025-11-25T18:56:52Z

Stonezhong: /* BI Connection */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using MPP Engine</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
JDBC[JDBC Interface]
MPP[MPP Engine #40;Starburst Trino#41;]
BI[BI Tool #40;Power BI#41;]
GoldTier --> MPP
PlatniumTier --> MPP
MPP --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Using RDBMS</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
subgraph storage
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
end
RDBMS[RDBMS#40;Oracle ADW#41;]
JDBC[JDBC Interface]
BI[BI Tool #40;Power BI#41;]
GoldTier --replicate--> RDBMS
PlatniumTier --replicate--> RDBMS
RDBMS --> JDBC --> BI
}}
<br />

* BI Tool to access Gold Tier data and Platnium Tier data via JDBC interface exposed by a MPP engine
** Why? A MPP Engine provide better interactive SQL query speed than Spark Thrift Server

</div>
</div>
<p></p>

Dl/Overview

2025-11-25T18:55:31Z

Stonezhong: /* BI Connection */

Dl/Overview

2025-11-25T18:54:09Z

Stonezhong: /* BI Connection */

Dl/Overview

2025-11-25T18:53:22Z

Stonezhong: /* BI Connection */

Dl/Overview

2025-11-25T18:43:44Z

Stonezhong: /* ETL */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

= BI Connection =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --2: trigger--> ETLE
ETLE --3: git pull -->LC
ETLE --4: dbt--> JDBC
JDBC --5:--> Spark
ER --> LC
User --1: git push-->ER
}}
<br />

* 1: User pushes ETL code into ETL Code Repo
* 2: Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 3: ETL executor pulls code from ETL Code Repo into loacl disk
* 4: ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 5: Thrift Server take the SQL and pass it to Apache Spark to execute
</div>
</div>
<p></p>

Dl/Overview

2025-11-25T18:05:16Z

Stonezhong: /* ETL */

Dl/Overview

2025-11-25T18:03:04Z

Stonezhong: /* ETL */

Dl/Overview

2025-11-25T18:02:41Z

Stonezhong: /* ETL */

Dl/Overview

2025-11-25T18:02:05Z

Stonezhong: /* ETL */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
User[User<Data Engineer>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --4--> Spark
ER --> LC
USER --5 git push-->ER
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}
</div>
</div>
<p></p>

Dl/Overview

2025-11-25T18:00:14Z

Stonezhong: /* ETL */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local ETL Code]
ER[ETL Code Repo]
JDBC[JDBC<Thrift Server>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --4--> Spark
ER --> LC
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}
</div>
</div>
<p></p>

Dl/Overview

2025-11-25T17:58:06Z

Stonezhong: /* Purpose */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= ETL =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local Code]
JDBC[JDBC<Thrift Server>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --4--> Spark
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}
</div>
</div>
<p></p>

Dl/Overview

2025-11-25T17:57:05Z

Stonezhong: /* Overview */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= Purpose =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">ETL Flow</div>
<div class="mw-collapsible-content">
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local Code]
JDBC[JDBC<Thrift Server>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --4--> Spark
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}
</div>
</div>
<p></p>

Dl/Overview

2025-11-25T17:55:16Z

Stonezhong: /* Overview */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= Overview =
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local Code]
JDBC[JDBC<Thrift Server>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --4--> Spark
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}

Dl/Overview

2025-11-25T17:53:14Z

Stonezhong: /* Overview */

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= Overview =
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|{{#mermaid:
graph TD
Scheduler[Apache Airflow/Scheduler]
ETLE[ETL Executor<Airflow Task>]
LC[Local Code]
JDBC[JDBC<Thrift Server>]
Spark[Apache Spark]
Scheduler --1 trigger--> ETLE
ETLE --2 git pull -->LC
ETLE --3 dbt--> JDBC
JDBC --> Spark
}}
| style="vertical-align:top;" |
* 1 Airflow Scheduler trigger DAG (DAG is generated based on metadata)
** The ETL job is a task within an airflow DAG
* 2 ETL executor pulls code from a repo into loacl disk
* 3 ETL executor uses dbt library to submit job to Apache Spark via JDBC interface (e.g. via Thrift Server)
* 4 Thrift Server take the SQL and pass it to Apache Spark to execute
|}

Dl/Overview

2025-11-25T17:31:50Z

Stonezhong: Created page with "<p>Data Lake Knowledge Center</p> = Overview = {{#mermaid: graph TD Scheduler[Apache Scheduler] ETLE[ETL Executor] CR[Code Repo] Spark[Apache Spark] }}"

<p>[[dl/home|Data Lake Knowledge Center]]</p>

= Overview =
{{#mermaid:
graph TD
Scheduler[Apache Scheduler]
ETLE[ETL Executor]
CR[Code Repo]
Spark[Apache Spark]

}}

Dl/home

2025-11-25T17:22:40Z

Stonezhong: /* Data Tiers */

Dl/Models

2025-11-25T03:15:18Z

Stonezhong:

= Introduction =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Introduction</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

= Models =
== [[dl/Dataset|Dataset]] ==
== [[dl/DataUnit|Data Unit]] ==

== [[dl/DataLocation|Data Location]] ==

== [[dl/DataType|Data Type]] ==

Dl/Models

2025-11-25T03:14:30Z

Stonezhong: /* Overview */

= Introduction =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Introduction</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

= Overview =
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.

<big><b>Gold Tier</b></big>:<br /><br />
<b>Tables for star schema</b>

* dimention tables and fact tables that forms star schema

<big><b>Platinum Tier</b></big>:<br /><br />
<b>Various query result for specific business questions materized in tables</b>

* Various query result for specific business questions materized in tables
* Tables may be replicated to a RDBMS for BI tool to access (sometime you can expose them directly, e.g. Spark Thrift Server)

|}

= Models =
== [[dl/Dataset|Dataset]] ==
== [[dl/DataUnit|Data Unit]] ==

== [[dl/DataLocation|Data Location]] ==

== [[dl/DataType|Data Type]] ==

Dl/Models

2025-11-25T03:09:29Z

Stonezhong: /* Overview */

= Introduction =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Introduction</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

= Overview =
{| class="wikitable grid mono section"
|-
! Diagram
! Description
|-
|
{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}
|
<big><b>Bronze Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
* raw data
* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
* could even be unstructured
* no data quality assurance
<hr /><br />

<big><b>Silver Tier</b></big>:<br /><br />
<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

* Data quality is assurred
* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.

|}

= Models =
== [[dl/Dataset|Dataset]] ==
== [[dl/DataUnit|Data Unit]] ==

== [[dl/DataLocation|Data Location]] ==

== [[dl/DataType|Data Type]] ==

Dl/Models

2025-11-25T02:52:48Z

Stonezhong: /* Overview */

Dl/Models

2025-11-25T02:37:30Z

Stonezhong: /* Introduction */

= Introduction =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Introduction</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

= Overview =
Example:

{{#mermaid:
graph TD
BornzeTier[Bronze Tier]
SilverTier[Silver Tier]
GoldTier[Gold Tier]
PlatniumTier[Platnium Tier]
ExternalData[External Data]

ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
}}

= Models =
== [[dl/Dataset|Dataset]] ==
== [[dl/DataUnit|Data Unit]] ==

== [[dl/DataLocation|Data Location]] ==

== [[dl/DataType|Data Type]] ==

Dl/home

2025-11-25T02:27:47Z

Stonezhong:

About-wiki

2025-11-19T09:53:34Z

Stonezhong: /* Mermaid */

= Wiki Features =
== Global CSS ==
* [[MediaWiki:Common.css|Global CSS]]

== You can use raw HTML tags ==
You can wrap your html content in <nowiki><html>...</html></nowiki>, for example,
<html><div style="background-color:green;">hello</div></html>
source code:
<pre><nowiki><html><div style="background-color:green;">hello</div></html></nowiki></pre>

== You can insert external images ==
=== method 1: use raw img tag ===
for example:

<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>

source code:
<pre><nowiki>
<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>
</nowiki></pre>

=== method 2: use traditional wiki style ===
Example:
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}

source code:
<pre><nowiki>
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}
</nowiki></pre>

== insert youtube video ==
Insert raw html directly. example

<html>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</html>

source code:
<pre><nowiki>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</nowiki></pre>

== expandable ==
Example:

<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>

source:
<pre><nowiki>
<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>
</nowiki></pre>

== Mermaid ==

Example:

{{#mermaid:
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
}}

source:
<pre><nowiki>
{{#mermaid:
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
}}
</nowiki></pre>

About-wiki

2025-11-19T09:52:30Z

Stonezhong: /* test */

= Wiki Features =
== Global CSS ==
* [[MediaWiki:Common.css|Global CSS]]

== You can use raw HTML tags ==
You can wrap your html content in <nowiki><html>...</html></nowiki>, for example,
<html><div style="background-color:green;">hello</div></html>
source code:
<pre><nowiki><html><div style="background-color:green;">hello</div></html></nowiki></pre>

== You can insert external images ==
=== method 1: use raw img tag ===
for example:

<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>

source code:
<pre><nowiki>
<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>
</nowiki></pre>

=== method 2: use traditional wiki style ===
Example:
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}

source code:
<pre><nowiki>
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}
</nowiki></pre>

== insert youtube video ==
Insert raw html directly. example

<html>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</html>

source code:
<pre><nowiki>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</nowiki></pre>

== expandable ==
Example:

<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>

source:
<pre><nowiki>
<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>
</nowiki></pre>

== Mermaid ==

{{#mermaid:
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
}}

About-wiki

2025-11-19T05:30:35Z

Stonezhong: /* test */

= Wiki Features =
== Global CSS ==
* [[MediaWiki:Common.css|Global CSS]]

== You can use raw HTML tags ==
You can wrap your html content in <nowiki><html>...</html></nowiki>, for example,
<html><div style="background-color:green;">hello</div></html>
source code:
<pre><nowiki><html><div style="background-color:green;">hello</div></html></nowiki></pre>

== You can insert external images ==
=== method 1: use raw img tag ===
for example:

<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>

source code:
<pre><nowiki>
<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>
</nowiki></pre>

=== method 2: use traditional wiki style ===
Example:
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}

source code:
<pre><nowiki>
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}
</nowiki></pre>

== insert youtube video ==
Insert raw html directly. example

<html>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</html>

source code:
<pre><nowiki>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</nowiki></pre>

== expandable ==
Example:

<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>

source:
<pre><nowiki>
<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>
</nowiki></pre>

== test ==

{{#mermaid:
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
}}

About-wiki

2025-11-19T05:23:42Z

Stonezhong: /* test */

About-wiki

2025-11-19T05:21:24Z

Stonezhong: /* test */

= Wiki Features =
== Global CSS ==
* [[MediaWiki:Common.css|Global CSS]]

== You can use raw HTML tags ==
You can wrap your html content in <nowiki><html>...</html></nowiki>, for example,
<html><div style="background-color:green;">hello</div></html>
source code:
<pre><nowiki><html><div style="background-color:green;">hello</div></html></nowiki></pre>

== You can insert external images ==
=== method 1: use raw img tag ===
for example:

<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>

source code:
<pre><nowiki>
<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>
</nowiki></pre>

=== method 2: use traditional wiki style ===
Example:
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}

source code:
<pre><nowiki>
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}
</nowiki></pre>

== insert youtube video ==
Insert raw html directly. example

<html>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</html>

source code:
<pre><nowiki>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</nowiki></pre>

== expandable ==
Example:

<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>

source:
<pre><nowiki>
<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>
</nowiki></pre>

== test ==

<mermaid>
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
</mermaid>

About-wiki

2025-11-19T04:54:21Z

Stonezhong: /* expandable */

= Wiki Features =
== Global CSS ==
* [[MediaWiki:Common.css|Global CSS]]

== You can use raw HTML tags ==
You can wrap your html content in <nowiki><html>...</html></nowiki>, for example,
<html><div style="background-color:green;">hello</div></html>
source code:
<pre><nowiki><html><div style="background-color:green;">hello</div></html></nowiki></pre>

== You can insert external images ==
=== method 1: use raw img tag ===
for example:

<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>

source code:
<pre><nowiki>
<html><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg" /></html>
</nowiki></pre>

=== method 2: use traditional wiki style ===
Example:
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}

source code:
<pre><nowiki>
{|
| https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/FullMoon2010.jpg/220px-FullMoon2010.jpg
|-
| This is a picture
|}
</nowiki></pre>

== insert youtube video ==
Insert raw html directly. example

<html>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</html>

source code:
<pre><nowiki>
<iframe width="560" height="315" src="https://www.youtube.com/embed/syp6Lsd8HOo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</nowiki></pre>

== expandable ==
Example:

<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>

source:
<pre><nowiki>
<div class="toccolours mw-collapsible mw-collapsed mono expandable">
<div class="mw-collapsible-preview"></div>
<div class="mw-collapsible-content">
* [https://www.youtube.com/ youtube]
</div>
</div>
<p></p>
</nowiki></pre>

== test ==
<mermaid>
graph TD
A[Start] --> B{Is Mermaid enabled?}
B -->|Yes| C[Render diagram]
B -->|No| D[Show code as text]
</mermaid>

Dl/Best Practices

2024-09-09T09:36:45Z

Stonezhong: /* Platform */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Platform =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Apache Spark</div>
<div class="mw-collapsible-content">
Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
* Scalable
* Well supported (DataBricks is backing up this product)
* Well adopted
* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， [https://www.oracle.com/big-data/data-flow/ OCI dataflow])
* Instead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

</div>
</div>
<p></p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:36:25Z

Stonezhong: /* Platform */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Platform =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Apache Spark</div>
<div class="mw-collapsible-content">
Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
* Scalable
* Well supported (DataBricks is backing up this product)
* Well adopted
* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， [https://www.oracle.com/big-data/data-flow/ OCI dataflow])
* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

</div>
</div>
<p></p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:35:30Z

Stonezhong: /* Platform */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Platform =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Apache Spark</div>
<div class="mw-collapsible-content">
Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
* Scalable
* Well supported (DataBricks is backing up this product)
* Well adopted
* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)
* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

</div>
</div>
<p></p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:35:14Z

Stonezhong: /* Platform */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Platform =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Spark</div>
<div class="mw-collapsible-content">
Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
* Scalable
* Well supported (DataBricks is backing up this product)
* Well adopted
* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)
* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

</div>
</div>
<p></p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:33:25Z

Stonezhong: /* Data Ingestion */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Platform =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Spark</div>
<div class="mw-collapsible-content">
Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
* Scalable
* Well supported (DataBricks is backing up this product)
* Well adopted
* Supported by many cloud providers (AWS EMR, Azure Azure HDInsight, [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)

</div>
</div>
<p></p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:24:41Z

Stonezhong: /* Data Ingestion */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
<div class="mw-collapsible-content">
* Create highly reusable "data connectors" to manage the data ingestio
** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:20:37Z

Stonezhong: /* Data Governance */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
* raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

* raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

* logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also
* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:20:11Z

Stonezhong: /* Data Governance */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Data Ingestion =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Always save a copy of raw data</div>
<div class="mw-collapsible-content">
When you do data ingestion, you want to save the raw data for the following reasons
* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
</div>
</div>
<p></p>

= Data Governance =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Keep good structure of your data</div>
<div class="mw-collapsible-content">
raw, sometime unstructured data:
- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

raw, ingested data
- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

logical data layer
- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

See also [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T09:02:31Z

Stonezhong: /* Data Ingestion */

Dl/Best Practices

2024-09-09T09:00:50Z

Stonezhong: /* Data Ingestion */

Dl/Best Practices

2024-09-09T08:59:48Z

Stonezhong: /* Data Ingestion */

Dl/Best Practices

2024-09-09T08:56:12Z

Stonezhong: /* Best Practices */

Dl/Best Practices

2024-09-09T08:55:07Z

Stonezhong: /* Best Practices */

<p> [[dl/home|Data Lake Knowledge Center]] </p>

= Best Practices =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Definition</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

Dl/Best Practices

2024-09-09T08:54:28Z

Stonezhong: Created page with "= Best Practices = <div class="toccolours mw-collapsible mw-collapsed expandable"> <div class="mw-collapsible-preview">Definition</div> <div class="mw-collapsible-content"> </div> </div> <p></p>"

= Best Practices =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Definition</div>
<div class="mw-collapsible-content">
</div>
</div>
<p></p>

Dl/home

2024-09-09T08:53:42Z

Stonezhong: /* Product Release Checklist */