Dl/DataType - Revision history

Stonezhong: /* Considerations */

2023-08-23T05:35:47Z

Considerations

← Older revision		Revision as of 05:35, 23 August 2023
Line 97:		Line 97:
	<div class="mw-collapsible-preview">Schema definition can also include constrains</div>		<div class="mw-collapsible-preview">Schema definition can also include constrains</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	Those ~~contains~~ can help us to validate the data in a content ignostic way uniformly.		Those constrains can help us to validate the data in a content ignostic way uniformly.

	For example:		For example:

Stonezhong: /* Considerations */

2023-08-23T05:35:28Z

Considerations

@@ Line 98: / Line 98: @@
 <div class="mw-collapsible-content">
 Those contains can help us to validate the data in a content ignostic way uniformly.
 </div>
 </div>
 <p></p>

Stonezhong: /* Considerations */

2023-08-23T05:33:45Z

Considerations

← Older revision		Revision as of 05:33, 23 August 2023
Line 90:		Line 90:
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	We build full-text index on description field. This allows user to search type or field based on key word shows up in description field, which improves data discovery.		We build full-text index on description field. This allows user to search type or field based on key word shows up in description field, which improves data discovery.
			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Schema definition can also include constrains</div>
			<div class="mw-collapsible-content">
			Those contains can help us to validate the data in a content ignostic way uniformly.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Considerations */

2023-08-23T05:32:02Z

Considerations

← Older revision		Revision as of 05:32, 23 August 2023
Line 82:		Line 82:
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	JSON Schema only support very few primitive types, such as string, number, int, etc. However, many system support more primitive types than JSON, for example, parquet file can have "timestamp" columns, however, "timestamp" is not supported by JSON Schema specifications. We will add more common primitive types to extend JSON schema specification to fits into many of the DataLake use cases.		JSON Schema only support very few primitive types, such as string, number, int, etc. However, many system support more primitive types than JSON, for example, parquet file can have "timestamp" columns, however, "timestamp" is not supported by JSON Schema specifications. We will add more common primitive types to extend JSON schema specification to fits into many of the DataLake use cases.
			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Description field helps to improve data discovery</div>
			<div class="mw-collapsible-content">
			We build full-text index on description field. This allows user to search type or field based on key word shows up in description field, which improves data discovery.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Considerations */

2023-08-23T05:28:16Z

Considerations

← Older revision		Revision as of 05:28, 23 August 2023
Line 74:		Line 74:
	{"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"} references a type with id of "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"		{"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"} references a type with id of "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
	</nowiki></pre>		</nowiki></pre>
			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Possible extension of the JSON schema to support more primitive types</div>
			<div class="mw-collapsible-content">
			JSON Schema only support very few primitive types, such as string, number, int, etc. However, many system support more primitive types than JSON, for example, parquet file can have "timestamp" columns, however, "timestamp" is not supported by JSON Schema specifications. We will add more common primitive types to extend JSON schema specification to fits into many of the DataLake use cases.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Considerations */

2023-08-23T05:24:10Z

Considerations

← Older revision		Revision as of 05:24, 23 August 2023
Line 68:		Line 68:
	<div class="mw-collapsible-preview">Referencing a type</div>		<div class="mw-collapsible-preview">Referencing a type</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	A type can be referenced in the format <code>{"$ref": ""}</~~code~~>		A type can be referenced in the format <code>{"$ref": type_id}</code>

			For example:
			<pre><nowiki>
			{"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"} references a type with id of "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
			</nowiki></pre>
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Considerations */

2023-08-23T05:22:55Z

Considerations

← Older revision		Revision as of 05:22, 23 August 2023
Line 66:		Line 66:
	= Considerations =		= Considerations =
	<div class="toccolours mw-collapsible mw-collapsed expandable">		<div class="toccolours mw-collapsible mw-collapsed expandable">
	<div class="mw-collapsible-preview">~~url should have enough information to locate the data~~</div>		<div class="mw-collapsible-preview">Referencing a type</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	~~url field should have enough information for user to locate~~ the ~~data. For example,~~ <code>"s3:~~//mubucket/stock_quotes/2023-08-20.jsonl~~"</code> ~~is a good url if your datalake only lives one AWS region. If your datalake crosses multiple AWS regions, you should put region ID in the url so you know from which region the bucket belongs to.~~		A type can be referenced in the format <code>{"$ref": ""}</code>
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong: /* Introduction */

2023-08-23T05:21:44Z

Introduction

← Older revision		Revision as of 05:21, 23 August 2023
Line 25:		Line 25:
	<b>Examples</b>:		<b>Examples</b>:
	<pre><nowiki>		<pre><nowiki>
	# This represent a "~~string~~" type		# This represent a "number" type
	{		{
	id: "~~617b2e86~~-~~9698~~-~~4b99~~-~~8956~~-~~57ce99d8de39~~",		id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
	type: "~~string~~"		type: "number",
			description: "Represent a float number"
	}		}

	# This represent a structure which has 2 number member fields for "x" and "y".		# This represent a structure which has 2 number member fields for "x" and "y".
	{		{
	id: "~~617b2e86~~-~~9698~~-~~4b99~~-~~8956~~-~~57ce99d8de39~~",		id: "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
	type: "object",		type: "object",
	properties: {		properties: {
	"x": {		"x": {
	type: "~~number~~",		type: {"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"},
			description: "The x coordinate value, measured in miles"
	},		},
	"y": {		"y": {
	type: "~~number~~",		type: {"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"},
			description: "The y coordinate value, measured in miles"
	}		}
	}		},
			description: "Represent a point on a two dimensional canvas"
	}		}


	# This represent an array of structure which has 2 number member fields for "x" and "y".		# This represent an array of structure which has 2 number member fields for "x" and "y".
	{		{
	id: "~~617b2e86~~-~~9698~~-~~4b99~~-~~8956~~-~~57ce99d8de39~~",		id: "11111111-1111-1111-1111-111111111111",
	type: "array",		type: "array",
	items: {		items: {
	id: "~~617b2e86~~-~~9698~~-~~4b99~~-~~8956~~-~~57ce99d8de3f",~~		"$ref": "#/$defs/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
	~~type: "object",~~
	~~properties: {~~
	~~"x": {~~
	~~type: "number",~~
	},
	~~"y": {~~
	~~type: "number~~",
	}
	}
	}		}
			description: "Represent a series of points on a two dimensional canvas"
	}		}

	</nowiki></pre>		</nowiki></pre>
	</div>		</div>

Stonezhong: /* Introduction */

2023-08-23T05:09:37Z

Introduction

@@ Line 44: / Line 44: @@
+     }
+}
 </nowiki></pre>
 </div>

Stonezhong: /* Introduction */

2023-08-23T05:01:46Z

Introduction

← Older revision		Revision as of 05:01, 23 August 2023
Line 31:		Line 31:
	}		}

	# This represent a structure which has "x" and "y" ~~number~~		# This represent a structure which has 2 number member fields for "x" and "y".
	{		{
	id: "617b2e86-9698-4b99-8956-57ce99d8de39",		id: "617b2e86-9698-4b99-8956-57ce99d8de39",