Stonezhong: /* Introduction */

2023-08-23T06:35:36Z

Introduction

← Older revision		Revision as of 06:35, 23 August 2023
Line 11:		Line 11:
	id: UUID		id: UUID
	Primary key		Primary key

			dataset_id: UUID
			A dataset that owns this data unit.

	publisher: str		publisher: str

Stonezhong: Created page with "

Data Lake Knowledge Center | Models

= Introduction =

Definition

A DataUnit object represent the basic unit that carries data. Fields

 id:  UUID     Primary key  publisher: str     Represent user who published this data unit. In case people has question about..."

2023-08-23T06:21:36Z

Created page with "<p> Data Lake Knowledge Center | Models</p> = Introduction = <div class="toccolours mw-collapsible mw-collapsed expandable"> <div class="mw-collapsible-preview">Definition</div> <div class="mw-collapsible-content"> <big><b>A DataUnit object represent the basic unit that carries data.</b></big> <b>Fields</b> <pre><nowiki> id: UUID Primary key publisher: str Represent user who published this data unit. In case people has question about..."

New page

<p> [[dl/home|Data Lake Knowledge Center]] | [[Dl/Models|Models]]</p>

= Introduction =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Definition</div>
<div class="mw-collapsible-content">
<big><b>A DataUnit object represent the basic unit that carries data.</b></big>

<b>Fields</b>
<pre><nowiki>
id: UUID
Primary key

publisher: str
Represent user who published this data unit. In case people has question about this data unit, they should contact the publisher for clarification.

published_time: DateTime
Represent the time this data object is published. A data unit object is only visible and accessible to user once it is published.

locations: List[DataLocation]
Represent a list of locations this data exists. The content of the data should be exactly the same for different locations. For example, one location could be for
ETL pipeline, another location could be for backup/archive. It is also possible the actual payload gets replicated to multiple region with the same content, user
can choose the closest data to read for performance consideration. But nevertheless, the content of the data for different locations should be exactly the same.
Note, every object in DataLake has lables, you can have "purpose" label attached to different location object to distinguish those different purposes.

is_recalled: boolean
If the publisher want to recall the data unit, they can set this flag to True. User should in general ignore the recalled data unit, unless for troubleshooting
purpose.

recall_time: DateTime
If the data unit is recalled, this represent the time the data unit is recalled.

schema: DataType
The schema of the data. Not all data unit object represent structured data, for example, a data unit object could represent a JPEG image. However, when a data unit
represent structured data, the this field describe the schema of the data.

sources: List[DataUnit]
Represent a list of the other data unit that is being used to produce this data unit. This allows us to trace back any data related problem to the root cause.
</nowiki></pre>

<b>Methods</b>
<pre><nowiki>
@property
def is_dirty(self) -> boolean

Returns True if any of the transitive source data unit being recalled. In such case, this data unit should be re-constructed to maintain data accuracy and integrity.

For example, if data unit A is for report A and data unit B is for report B, they both consumes data unit X and Y, and latter X is recalled, and if only A is
re-constructed without B, the dashboard build on top of A will be inconsistent against dashboard build on top of B, and causes lots of customer confusion.
</nowiki></pre>

<b>Examples</b>:
<pre><nowiki>
{
id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
publisher: "stone.zhong@gmail.com",
pubish_time: "2023-08-20 00:00:00",
is_recalled: False,
recall_time: None,
schema_id: "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
source_ids: ["11111111-1111-1111-1111-111111111111" ,"22222222-2222-2222-2222-222222222222"],
location_ids: ["33333333-3333-3333-3333-333333333333", "44444444-4444-4444-4444-444444444444"]
}
</nowiki></pre>

</div>
</div>
<p></p>

= Considerations =
<div class="toccolours mw-collapsible mw-collapsed expandable">
<div class="mw-collapsible-preview">Data Unit is immutable</div>
<div class="mw-collapsible-content">
So when you re-construct a dirty data unit, you should create a new data unit and set the original data unit as "recalled"
</div>
</div>
<p></p>

Dl/DataUnit - Revision history

Stonezhong: /* Introduction */

Stonezhong: Created page with "
Data Lake Knowledge Center | Models
= Introduction =

Definition

A DataUnit object represent the basic unit that carries data. Fields
id: UUID Primary key publisher: str Represent user who published this data unit. In case people has question about..."

Dl/DataUnit - Revision history

Stonezhong: /* Introduction */

Stonezhong: Created page with " Data Lake Knowledge Center | Models = Introduction = Definition A DataUnit object represent the basic unit that carries data. Fields id: UUID Primary key publisher: str Represent user who published this data unit. In case people has question about..."

Stonezhong: Created page with "
Data Lake Knowledge Center | Models
= Introduction =

Definition

A DataUnit object represent the basic unit that carries data. Fields
id: UUID Primary key publisher: str Represent user who published this data unit. In case people has question about..."