Dl/Models - Revision history

Stonezhong at 03:15, 25 November 2025

2025-11-25T03:15:18Z

@@ Line 6: / Line 6: @@
 </div>
 <p></p>
 = Models =

2025-11-25T03:14:30Z

Overview

@@ Line 39: / Line 39: @@
 * Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
 * Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.
 |}

2025-11-25T03:09:29Z

Overview

← Older revision		Revision as of 03:09, 25 November 2025
Line 25:		Line 25:
	}}		}}
	\|		\|
	* Bronze Tier:		<big><b>Bronze Tier</b></big>:<br /><br />
	** raw data		<b>The purpose for bronze tier is to store data downloaded from external world into data lake so we can use all sort of tools inside data lake to further process it</b>
	** no uniformed format, could be csv, JSON, binary, anything		* raw data
	** could even be unstructured		* no uniformed format, could be csv, JSON, AVRO, parquet, binary, anything
	** no data quality assurance		* could even be unstructured
	** Usually it is ~~a place for~~ data ingestion application to ~~dump raw~~ data ~~downloaded from external world.~~		* no data quality assurance
	* ~~Silver Tier:~~		<hr /><br />
	** X
			<big><b>Silver Tier</b></big>:<br /><br />
			<b>The purpose for bronze tier is to allow data ingestion application to sanitize data, verify the quality of the data</b>

			* Data quality is assurred
			* Data may not be normalized. One table may use UTC for a timestamp column while aother table may use timestamp without timezone. No stadnardlization for column name.
			* Data format is uniformed, usually it is stored as a format that is best fits the further ETL process, for example parquet.

	\|}		\|}

2025-11-25T02:52:48Z

Overview

← Older revision		Revision as of 02:52, 25 November 2025
Line 8:		Line 8:

	= Overview =		= Overview =
	~~Example:~~		{\| class="wikitable grid mono section"
			\|-
			! Diagram
			! Description
			\|-
			\|
	{{#mermaid:		{{#mermaid:
	graph TD		graph TD
Line 20:		Line 24:
	ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier		ExternalData --ingestion--> BornzeTier --ingestion--> SilverTier --ETL--> GoldTier --ETL--> PlatniumTier
	}}		}}
			\|
			* Bronze Tier:
			** raw data
			** no uniformed format, could be csv, JSON, binary, anything
			** could even be unstructured
			** no data quality assurance
			** Usually it is a place for data ingestion application to dump raw data downloaded from external world.
			* Silver Tier:
			** X
			\|}

	= Models =		= Models =

2025-11-25T02:37:30Z

Introduction

@@ Line 6: / Line 6: @@
 </div>
 <p></p>
 = Models =

2023-08-23T06:22:38Z

Data Unit

2023-08-23T06:21:58Z

Data Object

2023-08-23T06:21:26Z

Data Object

← Older revision		Revision as of 06:21, 23 August 2023
Line 9:		Line 9:
	= Models =		= Models =
	== [[dl/DataObject\|Data Object]] ==		== [[dl/DataObject\|Data Object]] ==
			== [[dl/DataUnit\|Data Unit]] ==

	== [[dl/DataLocation\|Data Location]] ==		== [[dl/DataLocation\|Data Location]] ==

	== [[dl/DataType\|Data Type]] ==		== [[dl/DataType\|Data Type]] ==

2023-08-23T05:37:15Z

Data Location

← Older revision		Revision as of 05:37, 23 August 2023
Line 8:		Line 8:

	= Models =		= Models =
			== [[dl/DataObject\|Data Object]] ==
	== [[dl/DataLocation\|Data Location]] ==		== [[dl/DataLocation\|Data Location]] ==

	== [[dl/DataType\|Data Type]] ==		== [[dl/DataType\|Data Type]] ==

2023-08-23T04:41:58Z

Data Schema

← Older revision		Revision as of 04:41, 23 August 2023
Line 9:		Line 9:
	= Models =		= Models =
	== [[dl/DataLocation\|Data Location]] ==		== [[dl/DataLocation\|Data Location]] ==
	== [[dl/~~DataSchema~~\|Data ~~Schema~~]] ==		== [[dl/DataType\|Data Type]] ==

← Older revision		Revision as of 06:21, 23 August 2023
Line 9:		Line 9:
	= Models =		= Models =
	== [[dl/DataObject\|Data Object]] ==		== [[dl/DataObject\|Data Object]] ==
			== [[dl/DataUnit\|Data Unit]] ==

	== [[dl/DataLocation\|Data Location]] ==		== [[dl/DataLocation\|Data Location]] ==

	== [[dl/DataType\|Data Type]] ==		== [[dl/DataType\|Data Type]] ==