|
|
| Line 32: |
Line 32: |
|
| |
|
| = Considerations = | | = Considerations = |
| | <div class="toccolours mw-collapsible mw-collapsed expandable"> |
| | <div class="mw-collapsible-preview">url should have enough information to locate the data</div> |
| | <div class="mw-collapsible-content"> |
| | url field should have enough information for user to locate the data. For example, "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a good url if your datalake only lives one AWS region. If your datalake spawns multiple AWS regions, you should put region ID in the url as well so you know from which region the bucket belongs to. |
| | </div> |
| | </div> |
| | <p></p> |
Revision as of 04:30, 23 August 2023
Data Lake Knowledge Center | Models
Introduction
Definition
A DataLocation object captures the location of the data and the format of the data.
Fields
id: UUID
Primary key
url: str
Specifies the location of the data.
For example: "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a valid url, it represent a data object located in AWS S3, bucket name is mubucket, object key is stock_quotes/2023-08-20.jsonl
format: str
A string represent the format of the data, for example, "JSONL", "CSV", "PARQUET", etc.
Examples:
{
id: "617b2e86-9698-4b99-8956-57ce99d8de39",
url: "s3://mubucket/stock_quotes/2023-08-20.jsonl"
format: "JSONL"
}
Considerations
url should have enough information to locate the data
url field should have enough information for user to locate the data. For example, "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a good url if your datalake only lives one AWS region. If your datalake spawns multiple AWS regions, you should put region ID in the url as well so you know from which region the bucket belongs to.
