Dl/DataLocation - Revision history

Stonezhong: /* Considerations */

2023-08-23T04:32:49Z

Considerations

← Older revision		Revision as of 04:32, 23 August 2023
Line 35:		Line 35:
	<div class="mw-collapsible-preview">url should have enough information to locate the data</div>		<div class="mw-collapsible-preview">url should have enough information to locate the data</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	url field should have enough information for user to locate the data. For example, "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a good url if your datalake only lives one AWS region. If your datalake ~~spawns~~ multiple AWS regions, you should put region ID in the url ~~as well~~ so you know from which region the bucket belongs to.		url field should have enough information for user to locate the data. For example, <code>"s3://mubucket/stock_quotes/2023-08-20.jsonl"</code> is a good url if your datalake only lives one AWS region. If your datalake crosses multiple AWS regions, you should put region ID in the url so you know from which region the bucket belongs to.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

2023-08-23T04:30:26Z

Considerations

← Older revision		Revision as of 04:30, 23 August 2023
Line 32:		Line 32:

	= Considerations =		= Considerations =
			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">url should have enough information to locate the data</div>
			<div class="mw-collapsible-content">
			url field should have enough information for user to locate the data. For example, "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a good url if your datalake only lives one AWS region. If your datalake spawns multiple AWS regions, you should put region ID in the url as well so you know from which region the bucket belongs to.
			</div>
			</div>
			<p></p>

2023-08-23T04:27:39Z

Introduction

2023-08-23T04:26:30Z

Introduction

← Older revision		Revision as of 04:26, 23 August 2023
Line 5:		Line 5:
	<div class="mw-collapsible-preview">Definition</div>		<div class="mw-collapsible-preview">Definition</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	<h2>A DataLocation object captures the location of the data and the format of the data.</h2>		<big><b>A DataLocation object captures the location of the data and the format of the data.</b></big>

	<~~big~~>Fields</~~big~~>		<b>Fields</b>
	<pre><nowiki>		<pre><nowiki>
	id: UUID		id: UUID

2023-08-23T04:25:50Z

Introduction

← Older revision		Revision as of 04:25, 23 August 2023
Line 7:		Line 7:
	<h2>A DataLocation object captures the location of the data and the format of the data.</h2>		<h2>A DataLocation object captures the location of the data and the format of the data.</h2>

	<h2>Fields</h2>		<big>Fields</big>
	<pre><nowiki>		<pre><nowiki>
	id: UUID		id: UUID

2023-08-23T04:25:23Z

Introduction

2023-08-23T04:24:42Z

Introduction

← Older revision		Revision as of 04:24, 23 August 2023
Line 17:		Line 17:
	A string represent the format of the data, for example, "JSONL", "CSV", "PARQUET", etc.		A string represent the format of the data, for example, "JSONL", "CSV", "PARQUET", etc.
	</nowiki></pre>		</nowiki></pre>

			Examples:
			<pre><nowiki>
			{
			id: "617b2e86-9698-4b99-8956-57ce99d8de39",
			url: "s3://mubucket/stock_quotes/2023-08-20.jsonl"
			format: "JSONL"
			}
			</nowiki></pre>

	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

2023-08-23T04:22:40Z

Introduction

← Older revision		Revision as of 04:22, 23 August 2023
Line 3:		Line 3:
	= Introduction =		= Introduction =
	<div class="toccolours mw-collapsible mw-collapsed expandable">		<div class="toccolours mw-collapsible mw-collapsed expandable">
	<div class="mw-collapsible-preview"></div>		<div class="mw-collapsible-preview">Definition</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
			<h2>A DataLocation object captures the location of the data and the format of the data.</h2>

			<h2>Fields</h2>
			<pre><nowiki>
			id: UUID
			Primary key
			url: str
			Specifies the location of the data.
			For example: "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a valid url, it represent a data object located in AWS S3, bucket name is mubucket, object key is stock_quotes/2023-08-20.jsonl
			format: str
			A string represent the format of the data, for example, "JSONL", "CSV", "PARQUET", etc.
			</nowiki></pre>
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

2023-08-23T04:12:04Z

← Older revision		Revision as of 04:12, 23 August 2023
Line 1:		Line 1:
	<p> [[Dl/home Data Lake Knowledge Center]] \| [[Dl/Models\|Models]]</p>		<p> [[dl/home\|Data Lake Knowledge Center]] \| [[Dl/Models\|Models]]</p>

	= Introduction =		= Introduction =

2023-08-23T04:11:35Z

← Older revision		Revision as of 04:11, 23 August 2023
Line 1:		Line 1:
	<p> [[Dl Data Lake Knowledge Center]] \| [[Dl/~~models/DataLocation~~]]</p>		<p> [[Dl/home Data Lake Knowledge Center]] \| [[Dl/Models\|Models]]</p>

	= Introduction =		= Introduction =