Dl/Best Practices - Revision history

Stonezhong: /* Platform */

2024-09-09T09:36:45Z

Platform

← Older revision		Revision as of 09:36, 9 September 2024
Line 10:		Line 10:
	* Well adopted		* Well adopted
	* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， [https://www.oracle.com/big-data/data-flow/ OCI dataflow])		* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， [https://www.oracle.com/big-data/data-flow/ OCI dataflow])
	* ~~In stead~~ of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.		* Instead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

	</div>		</div>

Stonezhong: /* Platform */

2024-09-09T09:36:25Z

Platform

← Older revision		Revision as of 09:36, 9 September 2024
Line 9:		Line 9:
	* Well supported (DataBricks is backing up this product)		* Well supported (DataBricks is backing up this product)
	* Well adopted		* Well adopted
	* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， ~~oci~~ dataflow)		* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， [https://www.oracle.com/big-data/data-flow/ OCI dataflow])
	* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.		* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

Stonezhong: /* Platform */

2024-09-09T09:35:30Z

Platform

← Older revision		Revision as of 09:35, 9 September 2024
Line 3:		Line 3:
	= Platform =		= Platform =
	<div class="toccolours mw-collapsible mw-collapsed expandable">		<div class="toccolours mw-collapsible mw-collapsed expandable">
	<div class="mw-collapsible-preview">Spark</div>		<div class="mw-collapsible-preview">Apache Spark</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:		Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:

Stonezhong: /* Platform */

2024-09-09T09:35:14Z

Platform

← Older revision		Revision as of 09:35, 9 September 2024
Line 9:		Line 9:
	* Well supported (DataBricks is backing up this product)		* Well supported (DataBricks is backing up this product)
	* Well adopted		* Well adopted
	* Supported by many cloud providers (AWS EMR, ~~Azure~~ Azure HDInsight, [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)		* Supported by many cloud providers ([https://aws.amazon.com/emr/ AWS EMR], [https://azure.microsoft.com/en-us/products/hdinsight Azure HDInsight] , [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)
			* In stead of building your own data lake, you can use [https://www.databricks.com/ LakeHouse] provided by databricks, they support AWS, Azure and GCP.

	</div>		</div>

Stonezhong: /* Data Ingestion */

2024-09-09T09:33:25Z

Data Ingestion

← Older revision		Revision as of 09:33, 9 September 2024
Line 1:		Line 1:
	<p> [[dl/home\|Data Lake Knowledge Center]] </p>		<p> [[dl/home\|Data Lake Knowledge Center]] </p>

			= Platform =
			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Spark</div>
			<div class="mw-collapsible-content">
			Apache Spark is a good platform for batch based data processing as well as streaming based data processing. Advantage:
			* Scalable
			* Well supported (DataBricks is backing up this product)
			* Well adopted
			* Supported by many cloud providers (AWS EMR, Azure Azure HDInsight, [https://cloud.google.com/dataproc GCP Dataproc]， oci dataflow)

			</div>
			</div>
			<p></p>

	= Data Ingestion =		= Data Ingestion =

Stonezhong: /* Data Ingestion */

2024-09-09T09:24:41Z

Data Ingestion

← Older revision		Revision as of 09:24, 9 September 2024
Line 9:		Line 9:
	* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.		* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
	* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.		* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Use data connectors to manage data ingestions</div>
			<div class="mw-collapsible-content">
			* Create highly reusable "data connectors" to manage the data ingestio
			** An anti pattern is to create too many one time, custome written, poorly documented data ingestion code
	</div>		</div>
	</div>		</div>

Stonezhong: /* Data Governance */

2024-09-09T09:20:37Z

Data Governance

← Older revision		Revision as of 09:20, 9 September 2024
Line 17:		Line 17:
	<div class="mw-collapsible-preview">Keep good structure of your data</div>		<div class="mw-collapsible-preview">Keep good structure of your data</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	raw, sometime unstructured data:		* raw, sometime unstructured data:
	- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.		- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

	raw, ingested data		* raw, ingested data
	- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.		- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

	logical data layer		* logical data layer
	- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)		- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

	See also [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]		See also
			* [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Data Governance */

2024-09-09T09:20:11Z

Data Governance

← Older revision		Revision as of 09:20, 9 September 2024
Line 17:		Line 17:
	<div class="mw-collapsible-preview">Keep good structure of your data</div>		<div class="mw-collapsible-preview">Keep good structure of your data</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
			raw, sometime unstructured data:
			- you stage the raw data (to be ingested) here, sometimes, these data can be unstructured.

			raw, ingested data
			- they are structured, e.g. in parquet format. They captured all the information you interested from raw data. They may orgnized well -- the purpose is to capture all raw information with minimum processing.

			logical data layer
			- well modeled, maybe around a subject model. (a fact table with bunch of dimension tables)

			See also [https://lingarogroup.com/blog/data-lake-architecture Data Lake Architecture: How to Create a Well Designed Data Lake]
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Data Ingestion */

2024-09-09T09:02:31Z

Data Ingestion

← Older revision		Revision as of 09:02, 9 September 2024
Line 9:		Line 9:
	* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.		* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
	* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.		* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
			</div>
			</div>
			<p></p>

			= Data Governance =
			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Keep good structure of your data</div>
			<div class="mw-collapsible-content">
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Data Ingestion */

2024-09-09T09:00:50Z

Data Ingestion

← Older revision		Revision as of 09:00, 9 September 2024
Line 8:		Line 8:
	* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data		* Your ingestion pipeline may have bugs, saving raw data allows you to fix bugs and re-populate the data
	* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.		* Raw data may not meed the data quality and you may ignore it, in case you ignore it, keep the raw data allows you to check what kind of data quality problem they are, and sometimes you can inform the data producer to have it fixed.
			* Raw data is owned by data source team and they have their own retention policy -- raw data is not always accessible afterwards.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>