Dl/DataType

Data Lake Knowledge Center | Models

Introduction

Definition

A DataType object describe the schema of nested data.

We follow the JSON Schema specification to represent types.

Fields

id:  UUID
    Primary key
type: str
    The type of the data. For example: "int", "number", "object".
properties: Optional[dict]
    If type is "object", this field list all properties.
description: Optional[str]
    Human readable document about this type
items: DataType
    If type is "array", this specifies the array element type

Examples:

# This represent a "string" type
{
    id: "617b2e86-9698-4b99-8956-57ce99d8de39",
    type: "string"
}

# This represent a structure which has 2 number member fields for "x" and "y".
{
    id: "617b2e86-9698-4b99-8956-57ce99d8de39",
    type: "object",
    properties: {
        "x": {
            type: "number",
        },
        "y": {
            type: "number",
        }
    }
}

# This represent an array of structure which has 2 number member fields for "x" and "y".
{
    id: "617b2e86-9698-4b99-8956-57ce99d8de39",
    type: "array",
    items: {
        id: "617b2e86-9698-4b99-8956-57ce99d8de3f",
        type: "object",
        properties: {
            "x": {
                type: "number",
            },
            "y": {
                type: "number",
            }
        }
    }
}

Considerations

url should have enough information to locate the data

url field should have enough information for user to locate the data. For example, "s3://mubucket/stock_quotes/2023-08-20.jsonl" is a good url if your datalake only lives one AWS region. If your datalake crosses multiple AWS regions, you should put region ID in the url so you know from which region the bucket belongs to.