Dl/glossary - Revision history

Stonezhong at 00:52, 7 March 2023

2023-03-07T00:52:53Z

← Older revision		Revision as of 00:52, 7 March 2023
Line 67:		Line 67:
	** A data ingestion application is a data application that does not consume any dataset		** A data ingestion application is a data application that does not consume any dataset
	** A data transformation application takes one or more dataset as input.		** A data transformation application takes one or more dataset as input.
			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Data Pipeline</div>
			<div class="mw-collapsible-content">
			* A bunch of data application with dependency defined
			* Runs on a regular basis (has schedule information)
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong at 00:52, 7 March 2023

2023-03-07T00:52:00Z

← Older revision		Revision as of 00:52, 7 March 2023
Line 57:		Line 57:
	* <code>dataframe://stock.msft/1.4/1/2023-01-01</code>: represent a dataframe, it belongs to dataset, dataset name is stock.msft, major_version is 1.4, minor version is 1, and dataframe key is 2023-01-01.		* <code>dataframe://stock.msft/1.4/1/2023-01-01</code>: represent a dataframe, it belongs to dataset, dataset name is stock.msft, major_version is 1.4, minor version is 1, and dataframe key is 2023-01-01.

			</div>
			</div>
			<p></p>

			<div class="toccolours mw-collapsible mw-collapsed expandable">
			<div class="mw-collapsible-preview">Data Application</div>
			<div class="mw-collapsible-content">
			* Data Application consumes 0 or more dataset as input, and generate dataframes and update dataset with new dataframes
			** A data ingestion application is a data application that does not consume any dataset
			** A data transformation application takes one or more dataset as input.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong at 00:09, 6 March 2023

2023-03-06T00:09:17Z

← Older revision		Revision as of 00:09, 6 March 2023
Line 11:		Line 11:
	</div>		</div>
	</div>		</div>
	~~<p></p>~~
	<p></p>		<p></p>

Stonezhong at 23:23, 5 March 2023

2023-03-05T23:23:07Z

← Older revision		Revision as of 23:23, 5 March 2023
Line 51:		Line 51:
	** two dataset of the same name, same major_version should have compatible schema, the latter minor_version is just adding new columns.		** two dataset of the same name, same major_version should have compatible schema, the latter minor_version is just adding new columns.
	* Each dataframe of this dataset has a unique key.		* Each dataframe of this dataset has a unique key.

			Each dataset has a unique URI, for example:
			* <code>dataset://stock.msft/1.4/1</code>: represent a dataset, name is stock.msft, major_version is 1.4, minor version is 1

			Each dataframe of a dataset also have URI, for example:
			* <code>dataframe://stock.msft/1.4/1/2023-01-01</code>: represent a dataframe, it belongs to dataset, dataset name is stock.msft, major_version is 1.4, minor version is 1, and dataframe key is 2023-01-01.

	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong at 23:19, 5 March 2023

2023-03-05T23:19:32Z

← Older revision		Revision as of 23:19, 5 March 2023
Line 34:		Line 34:
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	A URI that uniquely identifies an asset, for example:		A URI that uniquely identifies an asset, for example:
	* <code>s3://bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3		* <code>asset://s3/bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3
	* <code>~~mysql~~://myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo		* <code>asset://mysql/myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
	* <code>~~mysql~~://myserver/mydb/foo/?batch_id=1&</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1		* <code>asset://mysql/myserver/mydb/foo/?batch_id=1&</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1
	</div>		</div>
	</div>		</div>
Line 45:		Line 45:
	<div class="mw-collapsible-preview">Dataset</div>		<div class="mw-collapsible-preview">Dataset</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	It is a ~~serious~~ of dataframes that has the common schema.		It is a set of dataframes that has the common schema.

	* dataset name is not unique, but name + major_version + minor_version is unique		* dataset name is not unique, but name + major_version + minor_version is unique

Stonezhong at 23:11, 5 March 2023

2023-03-05T23:11:15Z

← Older revision		Revision as of 23:11, 5 March 2023
Line 42:		Line 42:
	<p></p>		<p></p>

	<div class="toccolours mw-collapsible mw-collapsed ~~mono~~ expandable">		<div class="toccolours mw-collapsible mw-collapsed expandable">
	<div class="mw-collapsible-preview">Dataset</div>		<div class="mw-collapsible-preview">Dataset</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">

Stonezhong at 23:11, 5 March 2023

2023-03-05T23:11:04Z

← Older revision		Revision as of 23:11, 5 March 2023
Line 45:		Line 45:
	<div class="mw-collapsible-preview">Dataset</div>		<div class="mw-collapsible-preview">Dataset</div>
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	~~bbb~~		It is a serious of dataframes that has the common schema.

			* dataset name is not unique, but name + major_version + minor_version is unique
			** two dataset of the same name, different major_version might have incompatible schema
			** two dataset of the same name, same major_version should have compatible schema, the latter minor_version is just adding new columns.
			* Each dataframe of this dataset has a unique key.
	</div>		</div>
	</div>		</div>
	<p></p>		<p></p>

Stonezhong at 23:04, 5 March 2023

2023-03-05T23:04:15Z

← Older revision		Revision as of 23:04, 5 March 2023
Line 36:		Line 36:
	* <code>s3://bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3		* <code>s3://bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3
	* <code>mysql://myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo		* <code>mysql://myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
	* <code>mysql://myserver/mydb/foo</?batch_id=1&</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1		* <code>mysql://myserver/mydb/foo/?batch_id=1&</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1
	</div>		</div>
	</div>		</div>

Stonezhong at 23:03, 5 March 2023

2023-03-05T23:03:30Z

← Older revision		Revision as of 23:03, 5 March 2023
Line 36:		Line 36:
	* <code>s3://bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3		* <code>s3://bucket_name/foo.parquet</code> -- represent a parquet file stored in AWS S3
	* <code>mysql://myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo		* <code>mysql://myserver/mydb/foo</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo
	* <code>mysql://myserver/mydb/foo</?batch_id=1&code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1		* <code>mysql://myserver/mydb/foo</?batch_id=1&</code> -- represent a table in MySQL, server name is myserver, dbname is mydb, table name is foo, with a filter, which batch_id column need to match 1

	</div>		</div>
	</div>		</div>

Stonezhong at 23:03, 5 March 2023

2023-03-05T23:03:05Z

@@ Line 25: / Line 25: @@
 * A collection in mongodb
 * A subset of a collection in mongodb -- via a filter
 </div>
 </div>