Dl/DatasetRepository

Data Lake Knowledge Center

Purpose

Data Management

APIs

• CRUD dataset
• Add, remove dataframe from dataset
• Load dataset

Web UI

• Browse dataset
• CRUD dataset
• Search dataset

Use cases:

• User can see datasets, view author, team, description, schema, publish time, etc
• User can search datasets based on author, team, description, schema, publish time, etc.
• User can create new dataset on Web UI
• A data ingestion app can create dataset via API
• A data ingestion app can add dataframe to a dataset via API

Dataset SLA Management

Lots of datasets has new dataframe with a given frequency, and if the new dataframe not showed up within certain time, we need to be alerted and be aware of it so we can fix the underneath issue, for example, fix a broken data pipeline.

So we allow:

• User can define a frequency for new dataframe for a dataset
• System will monitor the dataset for new dataframe, it new dataframe missed the frequency, it sends an alert.
• User can also see the new dataframe published time so check how often a dataset missed it's fresness SLA.

Dataset Quality Management

• Allow user to define a set of rules to check dataset quality
• Web UI to render dataset quality
• backend daemon to check dataset quality based on rules