Article 2: The ABCs of XLG

Coordinator
Feb 7, 2009 at 7:10 PM
Edited Mar 14, 2009 at 2:04 AM
Please feel free to contact me anytime:
William M. Rawls
Tallahassee, FL
william.rawls@gmail.com
Yahoo or Skype: willrawls
While I work full time for FBMC, I am happy to do (paid) side work for small to medium sized projects.
For instance, you could pay me to create a new code generation template or give your programmers a personal tour through XLG or maybe just help you generate and use your own data access layers.

NOTE:

These articles assume you already know how to use Visual Studio 2005, SQL Server 20005, and have a firm, deep understanding of xml, xslt 1.0, the Active Record pattern, and some relational database theory / use.
Additionally, one should be very comfortable with XML serialization.

The goals of this article include:
  1. A high level description of XLG,  XLG Pipeline and other high level concepts,
  2. Lightly cover the C# projects that currently make up  XLG,
  3. Discuss basic concepts and techniques used during code generation,
  4. Introduce the XLG Pipeline program,
  5. Lightly document the XLG Pipeline GUI,
  6. Take a quick look into the future of XLG,
  7. Walk through generating a surprisingly robust C# DAL Assembly (Data Access Layer) for AdventureWorks,
  8. Give some code examples of using the generated DAL

1. A high level description of XLG,  XLG Pipeline and other high level concepts:


Terms:
  • XLG is short for XML Library Generator and represents products written to implement my philosophy of code generation, code modification, post gen automation, automated multi environment deployment, and generally using xml whenever possible.
  • XLG Pipeline is the first XLG product: xlgPipeliner.exe
  • A Pipeline is a series of (generally) code generation steps.
    Each user selected step is executed one after another until all steps have successfully performed their work.
    Each step may build upon the last or might perform a  completely independent action.
    Each step can be checked and unchecked allowing some steps to be run on demand.
    All steps in the pipeline are saved to a .xlgs file
    A fellow programmer and life long friend, Lee Saferite, added the idea of extening my code generator to include a pipeline.
  • A .xlgs file contains a <XlgSettings> document serialized to and from xml by the MetX.Data.XlgSettings class.
    The document contains all settings, file names, and the details of each step to execute when required
  • A .xlg.xsl file is simply a normal xsl file who's (text) output happens to be source code.
    There would be one .xlg.xsl file for generating different kinds of code based on the Meta Data Source of each step.
  • A Meta Data Source is the process of "walking" through the structure of some system (such as a database, a list of strings, the contents of a folder, a web service request, etc). A <xlgDoc> xml document is built up from that walk and later transformed against a .xlg.xsl file, thus generating your code.
  • So by walking through a database's tables, fields, indexes, and stored procedure a xml file will be built, saved and then transformed with a .xlg.xsl file perhaps with the intention of generating a sophisticated C# DAL layer, a set of windows forms or web pages, finding "missing" indexes and generating T-SQL to create those indexs (see Uber Index later on).
    The code that actually walks through a database is called a MetX Data Provider.
  • A MetX Data Provider is a separate assembly that both handles all database connection operations, SQL building, and Meta Data Source building. XLG Pipeline supports a SQL Server 2005+, Sybase, and MySql
  • By walking through a folder's hierarchy (files and sub folders and their files sub folders) a <xlgFolder> xml document can be built, written, and transformed with another .xlg.xsl file. Perhaps to create image galleries or bookmark lists.
    The code that actually walks through a file system is called a MetX Gather Provider.
  • A MetX Gather Provider is a separate assembly that handles all non-database (gather style) meta data source operations such as:
    • Walking through a file system,
    • Automating a call to a web page or web service and returning an xml result
    • Executing a batch file or a power shell script and gathering the command's output
  • So basically both data and gather providers build up an xml document. xlgPipeliner then transforms the xml using xsl to create source code, project files and other files... basically everything you need.
  • A Data Glove is a layer of code (specifically a Data Access Layer) that is generated by the "CSharp DAL.xlg.xsl" file. I called it a glove because the code fits your database like a glove.
    Conceptually different kinds of gloves could be created (and probably will) such as a System Glove, Workflow Glove, Code Glove, etc.

Goals:
  • The original idea behind XLG was to turn a single connection string into a complete, compiled .NET 2.0 assembly with as little setup as possible. So one could literally generate a library from scratch and regenerate that library on demand (say when database structure changes).
  • Insure that code generated by a user could be completely replaced with their own (xsl) templates giving ease of use, versatility, and allowing one to generate their code, their way.
  • Maximize customization while minimizing work.
  • Explore generating layers of code
  • Use standards like xml and xsl as much as possible.

... More to follow in this article...
Coordinator
Feb 8, 2009 at 5:30 PM
Edited Feb 8, 2009 at 10:04 PM

2. The C# projects that currently make up  XLG

The XLG Pipeline is currently made up of 8 C# projects. They all exist in the MetX namespace as I thought I was going to name this (eventual) suite of tools after the MetX name. However, when I came up with the XLG concept and later when my good friend Lee added the pipeline concept, it seemed appropriate dub the line of products XLG and the first product as XLG Pipeline.

Eventually I'll re brand the projects all under the XLG namespace, but for now here's how it's laid out as of Gamma 2d ( I know, the names are inconsistent, but the "release early, release often" mentality gripped me. I'll re org/re factor soon):

  1. XLG.Pipeline - This is the XLG Pipeline executable and compiles to XLG.Pipeline.exe
    It supports a primitive command line (will rewrite that one day), supports folder watching with automatic regeneration whenever the selected xsl template changes, and generally facilitates the editing and execution of a set of XLG Settings. Until other XLG products are written, XLG will likely be thought of as actually being this program.
    Namespace: XLG.Pipeline

    Command Line
    Single step only (no pipeline):
        XLG.Pipeline.exe xlgdFile [xslFile] [outputPath] [outputFile]

    xlgdFile = The path and filename of the .xlgd file. If the file doesn't exist, a standard will be used
    xslFile   = The path and filename of the.xlg.xsl XSLT to transform the meta data against
    outputPath = The base path for writing gathered meta data and for writing additional generated files
    outputFile  = The file (usually in the base path) generated code is written to
    NOTE: When you pass only xlgdFile, XLG will infer the other values assuming the .xlg.xsl is in the same folder, that the base path is the same folder as the one containing xlgdFile, and that the target output file is in the same folder but with a .Glove.cs extension.

    (Some planned command lines)

    Pipeline execution (as selected):
        XLG.Pipeline.exe xlgsFile [gen | regen | genall | regenall | Step # | Step Name]
    This would load and execute a .xlgs XLG Pipeline settings file with gen or regen support for current selection, all steps, or a specifically named (or numbered) step.
  2. MetX - This library holds the lion's share of the functionality for XLG and compiles to MetX.dll
    This has grown over the years and contains my personal framework for .net as well as all the code generation and data run time support which was originally based on SubSonic. Eventually, I will break my workhorse library from the XLG specific classes. Look for a SandCastle generated document for to be uploaded soon.


    (Yes, the following assemblies do have ugly names, thank you for noticing)

    ---- [ Data Providers ] ----
  3. MetX.Data.Factory.System_Data_SqlClient
    This houses and isolates the code specific to working with SQL Server 2005+
    Used for generating meta data xml files and actual run time support for the "CSharp DAL.xlg.xsl" template
    Also contains a work in progress gather provider for executing a SQL and returning XML for transformation
  4. MetX.Data.Factory.MySql_Data_MySqlClient
    This houses and isolates the code specific to working with MySql 5+
    Since this assembly contains the MySql library reference, you won't need to have MySql installed unless you're actually generating code against it.
  5. MetX.Data.Factory.Sybase_Data_AseClient
    This houses and isolates the code specific to working with the Sybase.Data.AseClient driver (v 1.1.327.0).
    My current employer requires me to work with this version of the driver, so feel free to upgrade your copy.
    And if you don't use Sybase, don't worry about it. Your code won't have to install or load anythihng Sybase.

    While the naming convention of the data/gather providers is strange, it clears the way for standardized naming when new providers are written. All providers must be in the MetX.Data.Factory" namespace and contain a public class that implements the MetX.Data.IProvide interface AND who's name is the same as the namespace of the data provider one wishes to implement with periods replaced with an underscore. So for the SQL Server driver, the namespace System.Data.SqlClient (this is what will be specified in the <connectionStrings> section of your .config file) lets XLG Pipeline know that when it encounters a step that generates against SQL Server, it needs to instantiate a MetX.Data.Factory.System_Data_SqlClient object cast as (MetX.Data.IProvide).

    ---- [ Gather Providers ] ----
  6. MetX.Data.Factory.MetX_Gather_FileSystem
    Same basic idea here with a near identical calling syntax to the data provider but targetted at taking a supplied folder and returning a <xlgFolder> document for later transformation.
  7. MetX.Data.Factory.MetX_Gather_CommandLine
    This one actually returns a string (the stdout from whatever command line / arguments were passed through Connection Name / Connection String for the step calling this gather provider) and XLG simply writes that string to the output xml file.
    If the actual command you call RETURNS a well formed XML document, then that written output will then be fed into the code generator for transformation by the step's specified .xlg.xsl.
    This is so you can get freedom during the pipelining process and there's a lot that can be done here.
    Perhaps you could call a build, publish and deployment script after generating code.
    Get latest from your source control before generating code and commit generated code back in after compilation
    Sky's the limit on this one, people.
    Finally this makes a great provider to use as a starting point for your own gathers.
  8. MetX.Data.Factory.MetX_Gather_PowerShell
    This is a work in progress and currently is just a copy of the command line gather provider. We don't use PowerShell at my current job so it hasn't gotten any attention. Given that the plumbing is already there, I'd love to see someone give me some working code on this. There's a lot that can be done, and I've been thinking of exposing several commands from XLG Pipeline back into PowerShell itself.
    Maybe something like "xlg-get-xml AdventureWorks"... Again, open to ideas here.
I'd really enjoy seeing and including other ideas for providers, so feel free to drop me a line at william.rawls@gmail.com
Coordinator
Feb 9, 2009 at 12:09 AM

3. Basic concepts and techniques used during code generation

At it's core, XLG Pipeline is a multi-step xml creation, xsl transformation and task automation program.
Initially I have focused it on generating C# class library projects which represent a rather sophisticated data access layer (DAL). Mostly because that's what I've needed most.

A pipeline is a series of steps.
All the steps and their settings are stored in an .xlgs file (A XLG Pipeline settings file).

A step generates a <xlgDoc> document by combining an existing (or standard if blank) .xlgd file with the xml representation of the structure of a database (tables, fields, indexes, and stored procedures currently). Finally that xml document is then transformed with nxslt2.exe causing one or more files to be written as appropriate.

A step could instead perform some action like calling a batch file or executing a program that itself generates an xml file (say by calling a web service).

nxslt2 has an extension class added with a small classof functions I wrote available from urn:

... (work in progress) ...
Coordinator
Feb 17, 2009 at 5:42 PM
Edited Feb 17, 2009 at 8:10 PM

4. Introduction to the XLG Pipeline program

XLG Pipeline loaded with: AdventureWorks.xlgs


The .xlgs file is just a .net xml serialized class: MetX.Data.XlgSettings
Pretty much all XLG classes can be serialized to XML using the exposed ToXml(), FromXml(), SaveFile() and LoadFile() functions.

These settings and meta data sources are what XLG Pipeline is designed to edit.

<XlgSettings>
  DefaultConnectionString : Not currently used
  DefaultProviderName : Not currently used
  <Sources>
    <Source>
BasePath The target path for this step's code generation
Example: C:\data\code\xlg\DAL\AdventureWorks\
ParentNamespace xsl only: The namespace code should generate into
Example: CompanyName.Data
ConnectionName xsl only: The name of the <connectionString> .config entry. Also the sub namespace (ParentNamespace + ConnectioName)
Example AdventureWorks
(So CompanyName.Data.AdventureWorks )
DisplayName The string displayed in the list of meta data sources
Example: SqlClient: AdventureWorks
XlgDocFilename The .xlgd file containing additional xsl and code generation settings for this step
Example: C:\data\code\xlg\DAL\AdventureWorks\AdventureWorks.Glove.xlgd
XslFilename The XSLT file used for transformation of the meta data source
Example: C:\data\code\xlg\Support\CSharp DAL.xlg.xsl
OutputFilename The file written from the XSLT output
Example: C:\data\code\xlg\DAL\AdventureWorks\AdventureWorks.Glove.cs
OutputXml The filename where the XML generated from the meta data source should be written
Example: C:\data\code\xlg\DAL\AdventureWorks\AdventureWorks.Glove.xml
ConnectionString The Connection string to use for code generation (generally high privilege, dba, gen time only connection)
Example: Server=localhost;Database=AdventureWorks;Integrated Security=true;
ProviderName The class providing run time data access.
System.Data.SqlClient
Selected "true" to have this step be checked (selected) in XLG Pipeline. "false" to have that step skipped
RegenerateOnly Generally set to "false (get xml and transform, normal operation)
When set to "true" causes XLG Pipeline to regenerate output from the existing output xml.
    So only the XSL transformation gets done for this step, no XML meta data source walking.
    While set to true, output xml will not be regenerated nor ovewritten.
SqlToXml A SQL statement used as the meta data source and written out as XMLoutput for the step (almost always left blank).
So the contents of output xml will be the result set(s) from this SQL statement (data as meta data).
Coordinator
Feb 17, 2009 at 5:42 PM

5. Light documentation of the XLG Pipeline GUI


Coordinator
Feb 17, 2009 at 5:43 PM

6. The immediate future of XLG


Coordinator
Feb 17, 2009 at 5:43 PM

7. Walk through generating a C# DAL Assembly for AdventureWorks


Coordinator
Feb 17, 2009 at 5:44 PM

8. Some code examples from the generated DAL