Dl/glossary

Data Lake Knowledge Center

Dataframe

A basic data unit that carries data. For example:

• A dataframe in Apache Spark
• A dataframe in Pandas

Dataframe should be immutable once it is published -- the data it carries should never change.

Asset

Represent underneath data a dataframe is loaded from. For example:

• A parquet file in AWS S3 bucket
• A parquet file in HDFS
• A parquet file in local filesystem
• A table in DB (RDBMS)
• A subset of a table in DB (RDBMS) -- via a filter
• A collection in mongodb
• A subset of a collection in mongodb -- via a filter

Asset URI

A URI that uniquely identifies an asset, for example:

• s3://bucket_name/foo.parquet -- represent a parquet file stored in AWS S3
• mysql://myserver/mydb/foo -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
• mysql://myserver/mydb/foo/?batch_id=1& -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1

Dataset

It is a serious of dataframes that has the common schema.

• dataset name is not unique, but name + major_version + minor_version is unique
  • two dataset of the same name, different major_version might have incompatible schema
  • two dataset of the same name, same major_version should have compatible schema, the latter minor_version is just adding new columns.
• Each dataframe of this dataset has a unique key.