Dl/glossary: Difference between revisions

Revision as of 23:03, 5 March 2023

Dataframe

A basic data unit that carries data. For example:

Dataframe should be immutable once it is published -- the data it carries should never change.

Asset

Represent underneath data a dataframe is loaded from. For example:

Asset URI

A URI that uniquely identifies an asset, for example:

s3://bucket_name/foo.parquet -- represent a parquet file stored in AWS S3
mysql://myserver/mydb/foo -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
mysql://myserver/mydb/foo</?batch_id=1&code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1


Dataset

bbb

@@ Line 25: / Line 25: @@
 * A collection in mongodb
 * A subset of a collection in mongodb -- via a filter
+</div>
+</div>
+<p></p>
+<p></p>
+<div class="toccolours mw-collapsible mw-collapsed expandable">
+<div class="mw-collapsible-preview">Asset URI</div>
+<div class="mw-collapsible-content">
+A URI that uniquely identifies an asset, for example:
+* <code>s3://bucket_name/foo.parquet</code>       -- represent a parquet file stored in AWS S3
+* <code>mysql://myserver/mydb/foo</code>          -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
+* <code>mysql://myserver/mydb/foo</?batch_id=1&code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1
 </div>
 </div>