Dl/glossary

Data Lake Knowledge Center

Dataframe

A basic data unit that carries data. For example:

• A dataframe in Apache Spark
• A dataframe in Pandas

Dataframe should be immutable once it is published -- the data it carries should never change.

Asset

Represent underneath data a dataframe is loaded from. For example:

• A parquet file in AWS S3 bucket
• A parquet file in HDFS
• A parquet file in local filesystem
• A table in DB (RDBMS)
• A subset of a table in DB (RDBMS) -- via a filter
• A collection in mongodb
• A subset of a collection in mongodb -- via a filter

Asset URI

A URI that uniquely identifies an asset, for example:

• s3://bucket_name/foo.parquet -- represent a parquet file stored in AWS S3
• mysql://myserver/mydb/foo -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
• mysql://myserver/mydb/foo/?batch_id=1& -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1

Dataset

bbb