dotnet add package DataWarehouseAutomation --version 1.2.3
NuGet\Install-Package DataWarehouseAutomation -Version 1.2.3
<PackageReference Include="DataWarehouseAutomation" Version="1.2.3" />
paket add DataWarehouseAutomation --version 1.2.3
#r "nuget: DataWarehouseAutomation, 1.2.3"
// Install DataWarehouseAutomation as a Cake Addin #addin nuget:?package=DataWarehouseAutomation&version=1.2.3 // Install DataWarehouseAutomation as a Cake Tool #tool nuget:?package=DataWarehouseAutomation&version=1.2.3
Generation Metadata Schema for Data Warehouse Automation
To provide a collaborative space to discuss an exchange format concerning ETL generation metadata, supporting Data Warehouse Automation. This adapter should contain all metadata necessary to generate the transformation logic for a Data Warehouse solution.
Links / structure
The following directories have been set up:
- Generic interface, containing the Json schema definition.
- Class Library (DataWarehouseAutomation) containing the object model for deserialisation, as well as various utility classes such as validation of files against the Json schema definition.
- Code examples (examples_handlebars), containing C# examples using the generic interface for various purposes.
- Regression test project (test_project)
Across most, if not all, metadata models there is a core set of information that is required for any generation of ETL. If we can separate this from the UI / management of metadata we could have an exchange format that allows anyone to 'plug in' their own desired technology.
As an example 'TEAM' has the intent to separate UI with a view of limiting data entry and validation, but does not focus on SQL generation. Rather, these functions are separated by an adapter that is accessible as Json or database view.
The fundamental requirements of the metadata adapter are:
- Containing all metadata required to generate ETL output. This notably includes:
- source-to-target mappings
- physical model metadata (columns and tables, data types etc.)
- connectivity information, or proxy
- Text-based to support version control
In the Data Warehouse Automation (DWA) domain there are many specialists (i.e. ETL developers, Data Warehouse and Data Architects, BI analysts etc.) who have been, or are working on, proprietary meta models to support forward-engineering of code and designs.
Some of these are built inside existing tools (i.e. ERwin, Powerdesigner) using SDKs or macros. Others use different development frameworks (.net, Java) and most use differently modelled repositories or file formats to persist data on disk.
This is in addition to the many off-the-shelf DWA platforms, each of which has their own repository and format as well.
In the broader sense of meritocracy, it is worth pursuing if a common exchange format for metadata can be defined in a way that any developer can develop to in whatever technology or way their passion drives them.
For any change, create a new branch (no direct commits to master branch).
|.NET||net5.0 net5.0-windows net6.0 net6.0-android net6.0-ios net6.0-maccatalyst net6.0-macos net6.0-tvos net6.0-windows net7.0 net7.0-android net7.0-ios net7.0-maccatalyst net7.0-macos net7.0-tvos net7.0-windows|
|.NET Core||netcoreapp2.0 netcoreapp2.1 netcoreapp2.2 netcoreapp3.0 netcoreapp3.1|
|.NET Standard||netstandard2.0 netstandard2.1|
|.NET Framework||net461 net462 net463 net47 net471 net472 net48|
This package is not used by any NuGet packages.
This package is not used by any popular GitHub repositories.