DataWarehouseAutomation 1.2.3

.NET Standard 2.0
dotnet add package DataWarehouseAutomation --version 1.2.3
NuGet\Install-Package DataWarehouseAutomation -Version 1.2.3
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="DataWarehouseAutomation" Version="1.2.3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add DataWarehouseAutomation --version 1.2.3
#r "nuget: DataWarehouseAutomation, 1.2.3"
#r directive can be used in F# Interactive, C# scripting and .NET Interactive. Copy this into the interactive tool or source code of the script to reference the package.
// Install DataWarehouseAutomation as a Cake Addin
#addin nuget:?package=DataWarehouseAutomation&version=1.2.3

// Install DataWarehouseAutomation as a Cake Tool
#tool nuget:?package=DataWarehouseAutomation&version=1.2.3

Generation Metadata Schema for Data Warehouse Automation

Intent

To provide a collaborative space to discuss an exchange format concerning ETL generation metadata, supporting Data Warehouse Automation. This adapter should contain all metadata necessary to generate the transformation logic for a Data Warehouse solution.

The following directories have been set up:

  • Generic interface, containing the Json schema definition.
  • Class Library (DataWarehouseAutomation) containing the object model for deserialisation, as well as various utility classes such as validation of files against the Json schema definition.
  • Code examples (examples_handlebars), containing C# examples using the generic interface for various purposes.
  • Regression test project (test_project)
Hypothesis

Across most, if not all, metadata models there is a core set of information that is required for any generation of ETL. If we can separate this from the UI / management of metadata we could have an exchange format that allows anyone to 'plug in' their own desired technology.

As an example 'TEAM' has the intent to separate UI with a view of limiting data entry and validation, but does not focus on SQL generation. Rather, these functions are separated by an adapter that is accessible as Json or database view.

Requirements

The fundamental requirements of the metadata adapter are:

  • Containing all metadata required to generate ETL output. This notably includes:
    • source-to-target mappings
    • physical model metadata (columns and tables, data types etc.)
    • connectivity information, or proxy
  • Text-based to support version control
Background

In the Data Warehouse Automation (DWA) domain there are many specialists (i.e. ETL developers, Data Warehouse and Data Architects, BI analysts etc.) who have been, or are working on, proprietary meta models to support forward-engineering of code and designs.

Some of these are built inside existing tools (i.e. ERwin, Powerdesigner) using SDKs or macros. Others use different development frameworks (.net, Java) and most use differently modelled repositories or file formats to persist data on disk.

This is in addition to the many off-the-shelf DWA platforms, each of which has their own repository and format as well.

In the broader sense of meritocracy, it is worth pursuing if a common exchange format for metadata can be defined in a way that any developer can develop to in whatever technology or way their passion drives them.

Working guidelines

For any change, create a new branch (no direct commits to master branch).

Product Versions
.NET net5.0 net5.0-windows net6.0 net6.0-android net6.0-ios net6.0-maccatalyst net6.0-macos net6.0-tvos net6.0-windows net7.0 net7.0-android net7.0-ios net7.0-maccatalyst net7.0-macos net7.0-tvos net7.0-windows
.NET Core netcoreapp2.0 netcoreapp2.1 netcoreapp2.2 netcoreapp3.0 netcoreapp3.1
.NET Standard netstandard2.0 netstandard2.1
.NET Framework net461 net462 net463 net47 net471 net472 net48
MonoAndroid monoandroid
MonoMac monomac
MonoTouch monotouch
Tizen tizen40 tizen60
Xamarin.iOS xamarinios
Xamarin.Mac xamarinmac
Xamarin.TVOS xamarintvos
Xamarin.WatchOS xamarinwatchos
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.2.3 183 10/31/2022
1.2.2 169 10/26/2022
1.2.1 163 10/26/2022
1.2.0 187 10/13/2022