AjGenesis, my code generation open source project, use tasks, templates and a free model, to generate text artifacts, usually source code. Most of the examples use models serialized as XML or text files. But you can use anything as the initial input. Even a model could be the output of this process. Month ago, I wrote about generating the model from the database. In my opinion, the database is not the more expressive model to use as starting point, but it is an ubiquitous one (you should admit: I’m improving my English vocabulary…;-)

In my current agile project, the team took the database as the base model. Actually, we are using a more abstract model, but the database was a good asset to use as first model.

So, I back to play with my old example, and improved it. You can download the current example from my Skydrive DatabaseExample01.zip. (the code in flux is in the code repository at Codeplex, under examples\DatabaseExamples).

After download and expand it, you have the folders:

The content of Projects\Northwind\Metadata.xml:

<Metadata>
  <Project>
    <Name>Northwind</Name>
  </Project>
  <Database>
    <Name>Northwind</Name>
    <ConnectionString>server=.\SQLEXPRESS;database=Northwind;Integrated Security=true</ConnectionString>
  </Database>
</Metadata>

It describes the connection string to use to connect to example database. It use Northwind database, in SQL Server (full or express). If you don’t have Northwind, the creation scripts are in Sql folder.

To create the project and entities (the model to generate), run:

MakeModelFromDatabase.cmd

AjGenesis (compiled in Bin folder) begin to work:

The executed commad contains:

Bin\AjGenesis.Console Projects\Northwind\Metadata.xml Tasks\DatabaseProcess.ajg

This command loads Metadata.xml as model in memory, and executes the AjBasic task DatabaseProcess.ajg. This tasks use Information Schema views to obtain information about database structure (using information schema views opens the possibility of database independence).

The task creates Projects\Northwind\Project.xml file:

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<Project>
  <Name>Northwind</Name>
  <Model>
    <Entities>
      <Entity Source="Entities/Customer.xml"/>
      <Entity Source="Entities/Shipper.xml"/>
      <Entity Source="Entities/Supplier.xml"/>
      <Entity Source="Entities/Order.xml"/>
      <Entity Source="Entities/Product.xml"/>
      <Entity Source="Entities/OrderDetail.xml"/>
      <Entity Source="Entities/CustomerCustomerDemo.xml"/>
      <Entity Source="Entities/CustomerDemographic.xml"/>
      <Entity Source="Entities/Region.xml"/>
      <Entity Source="Entities/Territory.xml"/>
      <Entity Source="Entities/EmployeeTerritory.xml"/>
      <Entity Source="Entities/Employee.xml"/>
      <Entity Source="Entities/Category.xml"/>
    </Entities>
  </Model>
</Project>

The project and entities are similar to the ones I used in Application generationg using AjGenesis (but there are not the same). Part of the generated file Projects\Northwind\Entities\Customer.xml:

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<Entity>
  <Name>Customer</Name>
  <Description>Customer</Description>
  <SetName>Customers</Name>
  <Descriptor>Customer</Descriptor>
  <SetDescriptor>Customers</SetDescriptor>
  <SqlCatalog>Northwind</SqlCatalog>
  <SqlSchema>dbo</SqlSchema>
  <SqlName>Customers</SqlName>
  <Properties>
    <Property>
      <Name>CustomerID</Name>
      <Description>CustomerID</Description>
      <SqlName>CustomerID</SqlName>
      <SqlLength>5</SqlLength>
      <IsKey>True</IsKey>
      <SqlType>nchar</SqlType>
      <SystemType>String</SystemType>
      <IsNullable>False</IsNullable>
    </Property>
....
  </Properties>
</Entity>

The main logic resides in Tasks\DatabaseProcess.ajg, written in AjBasic dynamic language, fragment:

cmd = new System.Data.SqlClient.SqlCommand()
cmd.Connection = conn
cmd.CommandText = "select * from Information_Schema.Tables where Table_Type = 'BASE TABLE'"
conn.Open()
PrintLine "Reader"
dr = cmd.ExecuteReader()
Tables = CreateList()
while dr.Read()
  PrintLine "Table " & dr.Item("Table_Name") & ": " & dr.Item("Table_Type")
  Table = CreateObject()
  Table.SqlCatalog = dr.Item("Table_Catalog")
  Table.SqlSchema = dr.Item("Table_Schema")
  Table.SqlName = dr.Item("Table_Name")
  Table.Name = Table.SqlName.Replace(" ","")

  if IsPlural(Table.Name) then
    Table.SetName = Table.Name
    Table.Name = ToSingular(Table.Name)
  else
    Table.SetName = ToPlural(Table.Name)
  end if

  Table.Descriptor = Table.Name
  Table.SetDescriptor = Table.SetName

  Table.Description = Table.Name

  Tables.Add(Table)
end while
dr.Close()

Possible improvements

This examples is a “proof of concept”. To be used in a more general way, it should be improved. Some points to work:

- Use the generated model to generate a working application, scaffolding or not.

- Support of more meta data obtained via Information Schema

- Try another databases, try a real example

- Relation treatment: detecting cascade and other actions

Another posts about initial models for AjGenesis:

Another model for AjGenesis

Textual model for code generation in AjGenesis

AjGenesis: Generating the model from assemblies

AjGenesis: Generating the model from the database

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

I was working on generating C# classes, using as starting point .hbm NHibernate mapping files. As usual, I wrote an example with AjGenesis, my open source code generation engine.

You can download a first example from my Skydrive:

Examples > AjGenesis > NHibernateMappingExample01.zip

(the code is in the trunk, in the current change set, under examples\NHibernateMappinp:

but if you want to go directly to the example, the Skydrive download I mentioned has all you need to run this demo, including AjGenesis trunk code compiled to binaries).

After expanding the file, you have this content:

To create C# classes, execute at command prompt:

GenerateClasses AjFirstExample
GenerateClasses AjTest

To create a .NET project with the .cs and .hbm files, run:

GenerateProject AjFirstExample
GenerateProject AjTest

The generated files are created under Build folder.

The two example projects are AjFirstExample, with two simple plain mappings, and AjTest, with a more interesing mapping, with bags and many to one relations.

Currently, each project is described by a simple Project.xml:

<Project Name="AjTest">
</Project>

This is one of the mapping files in Projects\AjTest\Mappings, Department.hbm:

<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
  assembly="AjTest.Entities"
  namespace="AjTest.Entities"
  >
  <class name="Department" table="departments">
    <id name="Id" column="Id" type="Int32">
      <generator class="native"/>
    </id>
    <property name="Description" type="String"/>
    <bag name="Employees" lazy="true" inverse="true" cascade="all">

      <key column="IdDepartment"/>
      <one-to-many class="AjTest.Entities.Employee, AjTest.Entities"/>
    </bag>
  </class>
</hibernate-mapping>

This is the generated code for this mapping, Department.generated.cs:

using System;
using System.Collections.Generic;
using Iesi.Collections.Generic;
namespace AjTest.Entities
{
  public class Department {
    public int Id { get; set; }
    public string Description { get; set; }
    public IList<Employee> Employees { get; set; }
    public Department()
    {
      this.Employees = new List<Employee>();
    }
  }
}

Lets take a look to generation process. This is GenerateProject.cmd:

@echo off
set ProjectName=%1%
if "%1%"=="" set ProjectName=AjFirstExample
Bin\AjGenesis.Console.exe Projects\%ProjectName%\Project.xml Tasks\AddMappings.ajg Tasks\BuildCSharp.ajg
xcopy Libraries\*.* Build\%ProjectName%\CSharp\Src\Libraries /Y /Q

The main line is the one containing AjGenesis.Console.exe invocation. Project.xml is loaded in memory. The AddMapping.ajg task is loaded and executed, and then, BuildCSharp.ajg task is processed. AddMapping.ajg code (written in AjBasic, the dinamic language currently used by AjGenesis):

' Add mappings from directory if not specified in Project model
Include("Utilities/Utilities.tpl")
if not Project.Mappings then
  Project.Mappings = CreateList()

  di = new System.IO.DirectoryInfo("Projects/${Project.Name}/Mappings")

  for each fi in di.GetFiles("*.hbm.xml")
    filename = fi.Name
    Project.Mappings.Add(filename.Substring(0, filename.Length - 8))
  end for
end if

It adds the name of mapping files contained in the Mapping folder of the project. A more interesing task is GenerateCSharp.ajg. First, it loads the NHibernate library to use its hbm parser:

include "Utilities/Utilities.tpl"
include "Utilities/FileUtilities.tpl"
include "Utilities/TypeUtilities.tpl"
Include("Utilities/NHibernateUtilities.tpl")
include "Templates/CSharp/UtilitiesCs.tpl"
include "Templates/CSharp/CSharpFunctions.tpl"
AssemblyManager.LoadFrom("Libraries/NHibernate.dll")
parser = new NHibernate.Cfg.MappingSchema.MappingDocumentParser()

Then, it creates the solution and project objects:

if not Project.BuildDir then
  Project.BuildDir = "Build/${Project.Name}/CSharp"
end if
message "Creating Directories..."
FileManager.CreateDirectory(Project.BuildDir)
FileManager.CreateDirectory("${Project.BuildDir}/Sql")
FileManager.CreateDirectory("${Project.BuildDir}/Src")
FileManager.CreateDirectory("${Project.BuildDir}/Src/Libraries")
message "Defining Solution and Projects..."
Project.Solution = CreateObject()
Project.Solution.Guid = "FAE04EC0-301F-11D3-BF4B-00C04F79EFBC"
Project.Solution.Projects = CreateList()
message "Defining Entities Project..."
PrjEntities = CreateObject()
PrjEntities.Includes = CreateList()
PrjEntities.Guid = CreateGuid()
PrjEntities.COMGuid = CreateGuid()
Project.Solution.Projects.Add(PrjEntities)
Project.Entities = CreateList()

Then, it iterates on each hbm file, to get information about the entities to generate:

for each MappingName in Project.Mappings
  filename = "Projects/${Project.Name}/Mappings/${MappingName}.hbm.xml"
  mapping = parser.Parse(OpenAsStream(filename))

  for each hbmclass in mapping.Items where IsType(hbmclass, "HbmClass")
    Entity = CreateObject()

    Project.Entities.Add(Entity)

    Entity.ClassName = hbmclass.name
    Entity.Namespace = mapping.namespace

    ' Namespace as default project name for Entities Project
    if not PrjEntities.Name then
      PrjEntities.Name = mapping.namespace
      PrjEntities.Directory = "${Project.BuildDir}/Src/${PrjEntities.Name}"
      FileManager.CreateDirectory(PrjEntities.Directory)
    end if

    Entity.Properties = CreateList()

    if hbmclass.Id then
      Property = CreateObject()
      Property.Name = hbmclass.Id.name
      Property.Type = HbmTypeToCSharp(hbmclass.Id.type1, Entity.Namespace)
      Entity.Properties.Add(Property)
    end if

    for each item in hbmclass.Items
      if IsType(item, "HbmProperty") then
        Property = CreateObject()
        Property.Name = item.name
        Property.Type = HbmTypeToCSharp(item.type1, Entity.Namespace)
        Entity.Properties.Add(Property)
      end if

      if IsType(item, "HbmManyToOne") then
        Property = CreateObject()
        Property.Name = item.name
        Property.Type = HbmTypeToCSharp(item.class, Entity.Namespace)
        Entity.Properties.Add(Property)
      end if
      if IsType(item, "HbmSet") then
        Property = CreateObject()
        Property.Name = item.name
        Property.IsSet = true
        Property.Type = HbmTypeToCSharp(item.Item.class, Entity.Namespace)
        Entity.Properties.Add(Property)
      end if
      if IsType(item, "HbmBag") then
        Property = CreateObject()
        Property.Name = item.name
        Property.IsList = true
        Property.Type = HbmTypeToCSharp(item.Item.class, Entity.Namespace)
        Entity.Properties.Add(Property)
      end if
    end for
  end for
end for

You can extend the capabilities, processing more tags (I should write an example using Meta tags), and detecting more NHibernate mapping idioms. Now, the task generates the code:

for each Entity in Project.Entities
  TransformerManager.Transform("Templates/CSharp/Entity.tpl", "${PrjEntities.Directory}/${Entity.ClassName}.generated.cs", Environment)
  PrjEntities.Includes.Add(CreateFileCs("${Entity.ClassName}.generated"))
end for

The tasks not only generates the .cs files, but it creates a solution and a C# project, copying and embedding the original mapping files:

for each MappingName in Project.Mappings
  filename = "Projects/${Project.Name}/Mappings/${MappingName}.hbm.xml"
  targetfilename = "${PrjEntities.Directory}/${MappingName}.hbm.xml"
  System.IO.File.Copy(filename, targetfilename, true)
  PrjEntities.Includes.Add(CreateFileType(MappingName,"hbm.xml"))
end for
for each CsProject in Project.Solution.Projects where CsProject.ProjectType<>"Web"
  FileManager.CreateDirectory(CsProject.Directory)
  FileManager.CreateDirectory(CsProject.Directory & "/Properties")
  TransformerManager.Transform("Templates/CSharp/CsProject.tpl", "${CsProject.Directory}/${CsProject.Name}.csproj", Environment)
  TransformerManager.Transform("Templates/CSharp/AssemblyInfoCs.tpl", "${CsProject.Directory}/Properties/AssemblyInfo.cs", Environment)
end for
TransformerManager.Transform("Templates/Solution.tpl", "${Project.BuildDir}/Src/${Project.Name}.sln", Environment)

This is the generated solution:

Next steps

I should work in these point:

- Generate a more complete solution (with NHibernate infrastructure, Web Presentation, etc…) as in others AjGenesis examples.

- Support more NHibernate mapping options

- Use meta tags

But now, you are able to play with this example. You can change the templates to generate more artifacts, as Visual Basic .NET source files.

Thanks to @fabiomaulo for pointing me to the NHibernate hbm parser capabilities!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

I created a couple of introductory videos describing dbscript covering topics previously handled on this blog or in the online help:

• Installation
• Database schema import from SQL Server database
• Data Diagram generation (png, Dia)
• Generating HTML documentation

The videos have been created using CamStudio (screen recorder) and VirtualDub (avi editor) and ffmpeg (avi to flv converter). FlowPlayer is embedded by a Joomla plug-in.

The videos can be watched here. dbscript is available for download here.

I have this awesome little ADO.NET code generator called .netSavant.  It analyzes your stored procedures and generates an optimized wrapper class that encapsulates the logic for execution.  After using it myself for quite some time, I thought, oddly, that I could make a little extra cash from the endeavor.  Heh, sadly I was wrong.  Dispite a lot of interest, few people were interested in paying $50 for a visual studio addin that saved a LOT of time and buggy code.  Wierd, but whatever.

So, with mixed feelings I killed the project.  It simply cost too much money to maintain the website and merchant account for a product that only sold a handful of licenses over the 18-24 months.

I’m left wondering what to do with the project.  There are a few options:

1. Create a codeplex project
2. Host the source code myself
3. Release the addin as freeware (no source code)
4. Release the addin as freeware with sponsorship (uhg, gross right?) (no source code)
5. Leave the project inactive and move on to other exciting technologies

I’m wondering if I really want to spend more effort supporting the project as a whole, though with so much time invested, its hard to walk away from it.  Part of me would like to see more people use .netSavant, if only to justify the time I spent creating it.  I’d also like the challenge of updating some of the methodology used to generate the source code, and possibly provide some extensibility points to enhance the analysis features that it supports.

Perhaps some day .netSavant will be resurrected in a better form.  Until that time, thanks for your interest and support.

Joshua

Now that with the release of Android 2.0 we will be targeting different versions for a bit its time to embrace code generation. This is the code generation I am building up:

What you are seeing is my use of Code2Code plugin to generate a base of generators for class templates. You do not have to know that much template engine stuff as it auto generates most o fit right from the java  class file you create.  Than to enable copy the generators you need ot project root and change one line of configuration and you all set to go.

The best kept secret in Redmond, WA are the new T4 Preprocess Text Templates that shipped in Visual Studio 2010 Beta1 and Beta2.  In just a few minutes you’ll be in the know and using them.

Background – Condensed

T4 = text template transformation toolkit

T4 templates have been available since Visual Studio 2005 and are part of DSL (Domain Specific Languages) documentation.

In Visual Studio 2010 the templates are MUCH easier to use because they are preprocessed.  When you save your T4 template, it is compiled.  The T4 template is now “processed” at compile time.  This was not the case in previous releases.

Your compiled template is just another type in an assembly that you can now instantiate and call its single method, TransformText.  The only run-time requirement to use T4 templates is the .NET 2.0 Framework.

Prerequisites

• Visual Studio 2010
• T4 Editor Plug In 
  • Visual Studio 2010 does not provide any editing assistance when editing T4 templates (.tt files).  You can search the Internet for T4 Editor and you’ll find some information and products.
  • I’m currently using the Tangible T4 Editor (free) that you can get on the Visual Studio Gallery here: 
    • http://visualstudiogallery.msdn.microsoft.com/en-us/60297607-5fd4-4da4-97e1-3715e90c1a23
    • The editor is an Alpha version, has a few limitations but I was able to author the below template in a few minutes.

Our Scenario

We have been tasked with creating some code that we can pass data fields to and that code will create a VB.NET property.  This code will be part of a larger program that is fed information and creates class files.

Before T4 Preprocessed templates we had several options available to us.  Write code to spit out the required property, buy a 3rd party product, use CodeDom, etc.

Since we just installed Visual Studio 2010 Beta2, we are going for the productive, inexpensive, simple and out of the box solution, T4 Preprocessed Text Template.

Our Test Bench

This simple application simulates the data harness that will ultimately feed our T4 template and consume its output.  Set a few data fields and press Generate Code.

Code Generation – It’s all about the data

Code generation requires metadata to drive the output.  With the new T4 templates, getting metadata into your T4 template is a simple matter of a partial class.  The members in the partial class are available at design time and run-time.  I know this seems so simple, but T4 did not always work this way.  Now you have full design time strong typing and instant compile time verification of your template and code.

To keep this example as simple as possible, I’ve exposed the metadata directly as properties and passed values for them in the constructor of our Partial Class PropertyTemplate.  For simple scenarios this totally fine.  For a larger or multi-language code generation applications, a better solution is to expose the metadata through an Interface.  The Interface type can contain metadata and helper methods that can be used in your T4 template.

Kathleen Dollard and I have spoken a good bit about this and think using Interfaces for metadata and helper methods makes the most sense.  I’m sure Kathleen will write more on this subject and I know I will.

In the below T4TemplateLibrary project you can see how I’ve set up the project.  I had to turn on “Show All Files” to see the hidden file, “PropertyTemplate.vb” below the “PropertyTemplate.tt” file.

Open the hidden “PropertyTemplate.vb” file up and you will see the type of code you and I used to have to write to generate code.  Now T4 does it for us!

The below partial class is overly simple and exposes metadata that will be consumed by the T4 template.

Namespace My.Templates

  Partial Public Class PropertyTemplate

    Public Property PropertyName As String = String.Empty
    Public Property BackingField As String = String.Empty
    Public Property TypeName As String = String.Empty
    Public Property IsShared As Boolean = False
    Public Property RaisePropertyChangedMethodName As String = String.Empty
    Public Property IsReadOnly As Boolean = False
    Public Property Scope As String = String.Empty
    Public Property IsSetterPrivate As Boolean = False

    Public Sub New(ByVal strScope As String, ByVal bolIsShared As Boolean,
                   ByVal bolIsReadOnly As Boolean, ByVal bolIsSetterPrivate As Boolean,
                   ByVal strPropertyName As String, ByVal strBackingField As String,
                   ByVal strTypeName As String, ByVal strRaisePropertyChangedMethodName As String)
      Me.Scope = strScope
      Me.IsShared = bolIsShared
      Me.IsReadOnly = bolIsReadOnly
      Me.IsSetterPrivate = bolIsSetterPrivate
      Me.PropertyName = strPropertyName
      Me.BackingField = strBackingField
      Me.TypeName = strTypeName
      Me.RaisePropertyChangedMethodName = strRaisePropertyChangedMethodName
    End Sub

  End Class

End Namespace

T4 Template

To add a T4 template to your project, select Add New Item, then type “preprocessed” in the search box.  Select the Preprocess Text Template.

If you used classic ASP, then T4 template editing is a skill you already have.  In classic ASP we used <% and <%=.  In T4 we use <# and <#=.  The reason for this change is so that T4 can be used to generate classic ASP and ASP.NET code.  If you didn’t have the joy of shipping classic ASP web sites, no worries.  15 minutes of trail and error and you’ll be fully qualified.

The below image shows the free Tangible T4 Editor Plug-In IntelliSense in action.  It provides IntelliSense for T4 directives and template specific constructs like <#.  It also colorizes the template code.  Colorizing makes editing the template much easier too.  One current limitation is that its IntelliSense engine does list member properties defined in the partial class for the template.

The below code block is the T4 template.  It has a language directive at the top followed by static text and code blocks.

<#@ template language="VB" #>

<#= Me.Scope #><#= IIF(Me.IsShared, " Shared","") #><#= IIF(Me.IsReadOnly, " ReadOnly", "") #> Property <#= Me.PropertyName #> As <#= Me.TypeName #>
    Get
        Return <#= Me.BackingField #>
    End Get
<# If Not Me.IsReadOnly Then #>
    <#= IIF(Me.IsSetterPrivate, " Private ","") #>Set(ByVal value As <#= Me.TypeName #>)
<# If String.IsNullOrEmpty(Me.RaisePropertyChangedMethodName) Then #>
        <#= Me.BackingField #> = value
<# Else #>
        <#= Me.BackingField #> = value
        <#= String.Format("{0}(""{1}"")", Me.RaisePropertyChangedMethodName, Me.PropertyName) #>
<# End If #>
    End Set
<# End If #>
End Property

<#=  is the same as the ASP Response.Write.  It writes the result of the expression into the template.  For example, if the Scope is Public, <#= Me.Scope #> would yield Public when the template is transformed at run-time.

<# is the beginning of a code block.  Notice how the IsReadOnly property is tested.  Basic on the result, an entire block of code can either run or be by-passed.  This feature is so cool when you bring loops from LINQ queries into the picture.

One thing you’ll notice is that all the code blocks are left justified.  Yes, this makes reading the template a little harder.  However, if you don’t do this, the resulting code will be indented the same amount as the code block indentation.

So what I do when editing my templates is to indent everything to make it easier to edit the template.  When I”m done, I go back and move the code to the left.

Run-Time Rendering of T4 Templates

Dim t As New PropertyTemplate(
    Me.cboScope.SelectedItem.ToString,
    Me.chkIsShared.IsChecked.Value,
    Me.chkIsReadOnly.IsChecked.Value,
    Me.chkIsSetterPrivate.IsChecked.Value,
    Me.txtPropertyName.Text,
    Me.txtBackingField.Text,
    Me.cboTypeName.SelectedItem.ToString,
    Me.cboRaisePropertyChangedMethodName.SelectedItem.ToString)

'How cool is this?  Instantiate, call a single method, get the code!
Dim strResult As String = t.TransformText

Instantiate the template, call the TransformText method.  Not very sexy, but powerful!

Download

Get Your T4 On Demo Application (33KB)

Alternate Download Site

Some corporate firewalls do not allow access to Windows Live Sky Drive. You can download the installer here. Remember to rename the file from .doc to .zip. This is a requirement of WordPress.

Get Your T4 On Demo Application (33KB)

Links

Gareth Jones blog  Gareth works on the DSL Team at Microsoft and works with the Microsoft T4 product.

Kathleen Dollard’s blog Kathleen has been doing code generation for a very long time and is an industry expert in this space.

Kathleen Dollard’s work blog post on why T4 Templates are the best thing since sliced bread.

Oleg Sych very nice blog post on T4 Preprocessed Text Templates

Pablo Galiano T4 Preprocessing Part 1

Pablo Galiano T4 Preprocessing Part 2

Close

There is a fair amount of information on Visual Studio 2005/2008 T4 templates.  When reading these blog posts, magazine articles and MSDN Documentation, keep in mind that the new T4 templates do not have wide spread documentation yet.  Most of what you will find is applicable to editing of the templates and directives used in the templates.  Beyond that, just filter the information and you’ll stay on course.

Visual Studio 2010 T4 Preprocessed Text Templates are a great tool for your toolbox.  Hope to see you use the best kept secret in Redmond.

Have a great day,

Just a grain of sand on the worlds beaches.

A recent question on StackOverflow inspired me to write a utility called oraddlscript:

How to generate scripts for Oracle schema objects, such as tables, procedures, etc.

The answers directed me to the DBMS_METADATA package (9i documentation, 10g documentation). The function DBMS_METADATA.GET_DDL is the functional equivalent of the MSSQL SMO library, which prompted me to adapt my command-line utility SMOscript to Oracle databases. Voilà, oraddlscript.

oraddlscript 0.14.3584.16268 (c) by devio.at 2009

    list and script databases and database objects.

    usage: oraddlscript [options] [command]

    options: (leading '-' or '/')

    -s server       TNS name or host name or host:port
    -svc service    service name (if host name is provided)
    -o owner        owner name

    -u username     username (default: integrated authentication)
    -p password     password (if -u is missing, password for sa)

    -f filename     output to file
    -F directory    output to directory

    -A              current ANSI codepage
    -O              ASCII
    -T              Unicode
    -U              UTF8

    commands:

    l               list objects
    s               script object/s (using dbms_meta.get*ddl)
    -xml            use dbms_meta.get*xml

    list object owners on server (implied by -s)
    list objects in database (implied by -s -o)
    script all objects (implied by -s -o -F/-f)

The command-line arguments are consistent with SMOscript, except for -d (database) which has been replaced by -o (owner name).

The list of objects is retrieved by querying DBA_OBJECTS, ALL_OBJECTS and USER_OBJECTS depending on which of the catalog views is accessible by the user specified by -u.

The package also contains a function GET_XML which is used to retrieve the XML representation of a database object.

The functions of oraddlscript are:

• list usernames of object owners
• list objects of specific owner
• generate CREATE scripts of objects owned by specific user
• generate XML files representing objects owned by specific user
• generate one file per object or single file for all objects

Of course, logging and batch operations work just as previously described for SMOscript.

oraddlscript is available for download here.

The latest version 0.99 of dbscript has been released today providing new functionality and a couple of fixes.

Data diagrams looked a bit distorted if the data model contained circular foreign key constraints. I sketched the problem in my article on cycle detection, and the data diagram now excludes circular foreign keys in the calculation of the tables’ positions.

Comparison results can be restricted to “scopes”, such as new objects only, dropped objects only, etc. This makes it easier to generate schema migration scripts without dropping objects, for example.

Documentation Generators provide a preview to the generated content, and the generated XML now contains the project and project version identifiers to enable linking and referencing in the generator’s output.

Scripting a table in the object’s Generate/Create page now includes all constraints and indexes. (The project version script always included child objects). The same applies to object comparisons of tables, so that changes to indexes etc are easily identifiable.

New Functions

Besides generating .png data diagrams, dbscript now has the capability to generate data diagrams for Dia, an open-source diagrammer. The layout routine is the same as for png’s, but the output is Dia’s native XML format. Generating for Dia means that developers can freely layout and edit the diagram according to their needs, and export it to other formats. I described this feature earlier, and included samples.

Schema comparison is one basic feature of dbscript, and the new version compares multiple versions in one operation. After defining which schema versions to compare, you get a comparison matrix showing the number of differences between any two versions.

If the selected versions are versions of the same schema at different points of time, the comparison timeline shows each object ever changing in any of the versions, along with an indicator of the change.

Within a project, you can define Branches (as known from version control systems) and assign project versions to a branch. This alone would not be too overwhelming, but branches are a precondition of the update notification system, which I will describe in a future post.

The latest version of dbscript is available for download here.

Please leave comments and feedback.

If you use PowerShell to automatically generate code for your project (e.g. during the build process) and you work in TFS-based code, you need to check out existing files before overwriting them. Otherwise the files are read-only and/or not stored in TFS after code generation.

The PowerShell stub script to handle this situation looks like this (assuming the .ps1 file is also inside a TFS directory):

$scriptdir = Split-Path -Path $MyInvocation.MyCommand.Definition -Parent
$basepath = $scriptdir.Substring(0, $scriptdir.Length - "path\from\tfs-base\to\script".Length)

$scriptdir stores the directory name of the currently executed script. If the script file is stored inside your TFS project, you can calculate the file path to checkout from $scriptdir.

Next, we call TFS checkout, generate code, and check in again:

& .\tf-checkout.cmd $basepath
... Code generation is here ...
& .\tf-checkin.cmd $basepath

tf-checkout.cmd needs to set the Visual Studio environment variables (as in Visual Studio Command Prompt) to execute the “tf checkout” command:

@echo off
setlocal
call "c:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat" x86

echo.
echo checking out...

tf checkout %1path\to\file1.cs
tf checkout %1path\to\file2.cs
...

endlocal

tf-checkin.cmd looks similar:

@echo off
setlocal
call "c:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat" x86

echo.
echo checking in...

tf checkin /comment:autogenerated /noprompt %1path\to\file1.cs
tf checkin /comment:autogenerated /noprompt %1path\to\file2.cs

endlocal

In my master we use Alloy (http://alloy.mit.edu/) for formal specifications of software using first-order logic. Alloy is quite a nice and powerful tool, but what in my opinion is missing is that you can reuse the models developed in Alloy in real software projects. Therefore Alloy is at this moment more a tool to develop your logic (and sets) skills and it sure helps you to understand the necessarity of formal specifying software if you wanna validate the quality of it. But: it doesn’t help you in real projects (besides if you call finding solutions to sudoku of river crossing puzzles real projects, well then…).

There are attempts to create this link, namely DynAlloy, VAlloy and there is also a master thesis from a UvA-student implementing a Javacode-generator based on Alloy models. But all these still have drawbacks which won’t be discussed here.

So one of the questions I’ll be working on in the next time is: how can one create this link making Alloy a powerful tool for developing reliable, validated en testable REAL software projects.

If there are already further attempts to do this, please let me know. No one wants to do work which has already be done… and I’m always willing to cooperate with other people.

Applications usually implement some kind of security or grant management which allows a user to perform certain operations or restricts them from invoking them. This post presents a generic way of handling grants in a function hierarchy.

First we need a list of all functions in the application. For illustration, I use a simple shop application with a minimal set of functions. A function can be a general operation, or an operation on a certain database record:

CREATE TABLE #Functions (
	ID	INT NOT NULL PRIMARY KEY,
	Name	NVARCHAR(50),
	ParentFunctionID INT NULL, -- REFERENCES #Functions (ID),
	Parameter1 	NVARCHAR(50),
	-- optionally more parameters
	CargoParameters	NVARCHAR(50),
	ResultType	NVARCHAR(50)
)

SET NOCOUNT ON 

INSERT INTO #Functions (ID, Name, CargoParameters, ResultType)
  VALUES (1, 'CreateProduct', 'ProductDescription pd', 'int')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (10, 'ManageProduct', NULL, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (11, 'EditProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (12, 'RemoveProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name) VALUES (20, 'GoShopping')
INSERT INTO #Functions (ID, Name, ParentFunctionID)
  VALUES (21, 'ListProducts', 20)
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1, CargoParameters, ResultType)
  VALUES (22, 'AddToCart', 20, 'ProductID', 'int Quantity', 'ShoppingResult')

SELECT * FROM #Functions

This results in the following table:

Once we have the complete list of all functions, we can generate code to implement a number of methods for each function:

In this post, I use a class named UserContext which identifies the current user and encapsulates the way the user is identified: whether it is a Guid, an integer, is not relevant for this code.

The usual TSQL cursor magic:

DECLARE @Name NVARCHAR(50), @Parameter1 NVARCHAR(50), @FP_Name NVARCHAR(50),
  @FP_Parameter1 NVARCHAR(50)
DECLARE @CargoParameters NVARCHAR(50), @ResultType NVARCHAR(50)

PRINT 'public partial class FunctionBase
{'

DECLARE c CURSOR FOR
SELECT F.Name, F.Parameter1, FP.Name AS FP_Name, FP.Parameter1 AS FP_Parameter1,
  F.CargoParameters, F.ResultType
FROM #Functions F
LEFT OUTER JOIN #Functions FP ON F.ParentFunctionID = FP.ID
ORDER BY F.ID

OPEN c
FETCH c INTO @Name, @Parameter1, @FP_Name, @FP_Parameter1, @CargoParameters,
  @ResultType

WHILE @@FETCH_STATUS = 0 BEGIN

For each of the methods above, we generate a C# method with the default behavior:

• If the function has a parent, call the parent.
• If the function has no parent, return false.
• The Execute method throws an exception.

  PRINT '	public virtual bool IsPossibleTo' + @Name + '(' +
    CASE WHEN @Parameter1 IS NOT NULL THEN 'int ' + @Parameter1 ELSE '' END + ')'
  PRINT '	{'

  IF @FP_Name IS NULL
    PRINT '		return false;'
  ELSE IF ISNULL(@Parameter1, '') = ISNULL(@FP_Parameter1, '')
    PRINT '		return IsPossibleTo' + @FP_Name + '(' + ISNULL(@Parameter1, '') + ');'
  ELSE IF @FP_Parameter1 IS NULL
    PRINT '		return IsPossibleTo' + @FP_Name + '();'
  ELSE -- no @param, but @fpparam
    PRINT '		return IsPossibleTo' + @FP_Name + '(0);'

  PRINT '	}'

  PRINT '	public virtual bool IsUserAllowedTo' + @Name + '(UserContext u' +
    CASE WHEN @Parameter1 IS NOT NULL THEN ', int ' + @Parameter1 ELSE '' END + ')'
  PRINT '	{'

  IF @FP_Name IS NULL
    PRINT '		return false;'
  ELSE IF ISNULL(@Parameter1, '') = ISNULL(@FP_Parameter1, '')
    PRINT '		return IsUserAllowedTo' + @FP_Name + '(u' +
      CASE WHEN @Parameter1 IS NOT NULL THEN ', ' + @Parameter1 ELSE '' END + ');'
  ELSE IF @FP_Parameter1 IS NULL
    PRINT '		return IsUserAllowedTo' + @FP_Name + '(u);'
  ELSE -- no @param, but @fpparam
    PRINT '		return IsUserAllowedTo' + @FP_Name + '(u, 0);'

  PRINT '	}'

  PRINT '	public virtual ' + ISNULL(@ResultType, 'void') + ' Do' + @Name + '(UserContext u' +
    CASE WHEN @Parameter1 IS NOT NULL THEN ', int ' + @Parameter1 ELSE '' END +
    CASE WHEN @CargoParameters IS NOT NULL THEN ', ' + @CargoParameters ELSE '' END +
    ')'
  PRINT
'	{
		throw new Exception("FunctionBase.Do' + @Name + ' is not implemented");
	}'

End of cursor code:

  FETCH c INTO @Name, @Parameter1, @FP_Name, @FP_Parameter1,
    @CargoParameters, @ResultType

END

CLOSE c
DEALLOCATE c

PRINT '}'

DROP TABLE #Functions

This results in a class FunctionBase which implements security methods for each function, such as:

public partial class FunctionBase
{
  public virtual bool IsPossibleToCreateProduct()
  {
    return false;
  }
  public virtual bool IsUserAllowedToCreateProduct(UserContext u)
  {
    return false;
  }
  public virtual int DoCreateProduct(UserContext u, ProductDescription pd)
  {
    throw new Exception("FunctionBase.DoCreateProduct is not implemented");
  }

Not too impressive at first glance, but look at EditProduct, which is below ManageProduct in the function hierarchy:

  public virtual bool IsPossibleToManageProduct(int ProductID)
  {
    return false;
  }
  public virtual bool IsUserAllowedToManageProduct(UserContext u, int ProductID)
  {
    return false;
  }
  public virtual void DoManageProduct(UserContext u, int ProductID)
  {
    throw new Exception("FunctionBase.DoManageProduct is not implemented");
  }
  public virtual bool IsPossibleToEditProduct(int ProductID)
  {
    return IsPossibleToManageProduct(ProductID);
  }
  public virtual bool IsUserAllowedToEditProduct(UserContext u, int ProductID)
  {
    return IsUserAllowedToManageProduct(u, ProductID);
  }
  public virtual void DoEditProduct(UserContext u, int ProductID)
  {
    throw new Exception("FunctionBase.DoEditProduct is not implemented");
  }

The IsPossibleToEditProduct and IsUserAllowedToEditProduct methods automatically inherit the result from IsPossibleToManageProduct and IsUserAllowedToManageProduct.

What would be the value of a generated function which only throws exceptions, you might ask. Well, this is the point where the actual coding begins:

public partial class Function : FunctionBase
{
  private static Function _instance;
  public static Function Instance
  {
    get
    {
      if (_instance == null)
        _instance = new Function();
      return _instance;
    }
  }

  public override bool IsPossibleToManageProduct(int ProductID)
  {
    // implement your checks here, all functions below ManageProduct inherit this result by default
    return Result;
  }
  public override bool IsUserAllowedToManageProduct(UserContext u, int ProductID)
  {
    // implement your checks here, all functions below ManageProduct inherit this result by default
    return Result;
  }
  public virtual void DoEditProduct(UserContext u, int ProductID)
  {
    // implement your function here
  }
}

And to check, whether an operation is possible and allowed, use:

UserContext uc = RetrieveUserContext();
int id = RetrieveRecordIdentifier();
if (Function.Instance.IsPossibleToEditProduct(id) &&
    Function.Instance.IsUserAllowedToEditProduct(uc, id))
  Function.Instance.DoEditProduct(uc, id);

Note that you can define additional parameters and a result type for the execution method.

The latest version 0.98 of dbscript supports PostgreSQL databases in its documentation generation capabilities.

After importing the database dictionary (via direct connection using ADO.Net and Npgsql) can document a PostgreSQL database in all currently supported documentation format:

MediaWiki

Data Diagram (PNG)

HTML

ScrewTurn wiki

Integration support for PostgreSQL had some consequences: More and more functionality is handled separately for each database engine.

Database import was obviously the first one, since the data access classes (SqlConnection, SqlCommand) in .Net are different for every database library. Same goes for the database dictionary, which is best retrieve from the native system catalogs.

For import and upload, data access classes have been introduced to distinguish the different object types and their properties of each database engine. I mention work on the data access classes in a series of articles already.

In version 0.98, XML generation and object script generation are implemented separately. This results in XSL style sheets being now related to certain a database engine.

For Oracle, XML and object script generation have been updated, and the XSL style sheets have been adjusted to Oracle-specific objects and properties. The results were documented earlier.

The latest version of dbscript is available for download here.

Please leave comments and feedback.

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

Part 4: Generating C# classes

Part 5: Generated C# classes and polymorphism

Part 6: UPDATE record

Our DAL object has two collections, with dictNew holding new values, and dictRecord for previously inserted or retrieved values. To implement an Update() method, the Insert() method has to be fixed to save the inserted values:

   dictRecord = dictNew;
   dictNew = null;

Now we are ready to implement UPDATE. The parameters are collected similarly as described in part 3, which describes the INSERT operation:

private void Update(SqlConnection conn)
{
    if (IdentityColumn != null)
    {
        if (!dictRecord.ContainsKey(IdentityColumn))
            throw new Exception("UPDATE " + this.TableName + " without IDENTITY column set");

        StringBuilder sb = new StringBuilder();
        sb.Append("UPDATE " + TableName + " SET ");

        bool bFirst = true;
        foreach (string s in dictNew.Keys)
        {
            if (!bFirst)
                sb.Append(", ");
            sb.Append(s + " = @" + s);
            bFirst = false;
        }

        sb.Append(" WHERE " + IdentityColumn + " = @" + IdentityColumn);

        SqlCommand cmd = new SqlCommand();
        cmd.Connection = conn;

        foreach (string s in dictNew.Keys)
        {
            if (dictNew[s] == null)
                cmd.Parameters.AddWithValue("@" + s, DBNull.Value);
            else
                cmd.Parameters.AddWithValue("@" + s, dictNew[s]);
        }

        cmd.Parameters.AddWithValue("@" + IdentityColumn, dictRecord[IdentityColumn]);
        cmd.CommandText = sb.ToString();

        foreach (string s in dictNew.Keys)
            if (dictRecord.ContainsKey(s))
                dictRecord[s] = dictNew[s];
            else
                dictRecord.Add(s, dictNew[s]);

        dictNew = null;
    }
    else
        throw new Exception("UPDATE " + this.TableName + " without IDENTITY column is not implemented.");
}

If the table does not have an IDENTITY column, the WHERE condition for the UPDATE is unknown and an exception is raised. If the identity column of the existing record is not set, an exception is raised, too.

http://weblogs.java.net/blog/edburns/archive/2009/09/03/dealing-gracefully-viewexpiredexception-jsf2

http://weblogs.java.net/blog/rogerk/archive/2009/09/04/contexts-and-dependency-injection-jsr-299-and-glassfish

http://martinfowler.com/bliki/FluentInterface.html

http://martinfowler.com/dslwip/

ان كنت قد تجولت في مواقع الانترنت ذات العلاقه بمايكروسوفت ، فأكيد ان صداعا في الرأس بدأ يصيبك من رؤية بعض المصطلحات المعتمدة في الاصدرات القادمة ابرزها T4 ، فعشرات المواقع والمجلات والمدونات تتحدث عن T4 واجزم انك رأيت ذلك المصطلح امامك عشرات المرات ! والسؤال الذي يطرح نفسه : ماذا يقصد بـ T4 ؟

T4 هو رمز الى عبارة Text Templating Transformation Toolkit – اي حرف T اربع مرات بدل TTTT !! – وهي تقنية طورتها مايكروسوفت سابقا وقد قامت حاليا برفع متسوى تطويرها الى مرحلة متقدمة جدا ، هذه التقنية تستخدم في توليد الاكواد Code Generation في مختلف منتجات مايكروسوفت وابرزها بطبيعه الحال Visual Studio . 

محرك T4 موجود اصلا في النسخه 2008 وقادم بشكل قوي جدا في 2010 ، ففي الاصدار الحالي لايوجد اي اداة Tool لاستخدامها في اعادة توليد الاكواد الروتينية ، ولكن في الاصدار 2010 فالوضع مختلف حقا من هذه الناحية ، وبالنسبه للاصدار 2008 فيمكنك استخدام ادوات خارجيه تؤدي هذا الغرض منها اداة Visual T4 Editor و Visual T4 Code Generator وبعضها يتوفر في اصدارات مجانية وهي تتميز بإندماجها بشكل كلي مع Visual Studio .

موقع الاداة Visual T4 هو : http://www.visualt4.com/
وموقع الاداة  Visual T4 Code Generator هو :

http://t4-editor.tangible-engineering.com/T4-Editor-Visual-T4-Editing.html

MDDつながりです。InfoQに次の記事がありました。

1. Role of Code Generation in Java Application Development

Javaの世界を中心にCode Generationを実践する各種ツールを紹介し将来の展望を実践者にインタビューしています。

本ブログではこの方向性が正しくまた避けられないものだと考えています。各種ツールはすべて進化の途上にあり、どれを選ぶかはあるタイミングにおいてもケースバイケースですし、またどれかが急に進化する可能性もあるため、常に広く状況を観察する必要があります。

まだ試された事の無い方は、一度上の記事にあるツール（主なものはカバーされているようです）のデモビデオなどをご覧ください。何か得るものが有るかもしれません。

補足）偶然かもしれませんがSpringSourceのRooの開発者によるRooの説明ビデオがInfoQに掲載されていました。最後の箇所はRoRを意識しているということでしょうか。

1. Extreme Productivity in Application Development with Roo

Today I encountered a problem when using the Oracle JDBC API that only manifested itself in a certain complicated sequence of API invocations. I wanted to understand exactly what the sequence of API invocations was and also be able to extract them, so I could spit them out into a standalone test-case that I could use to debug the problem. Below is a class I whipped up to help automate most of this task.

So now I can do something like:

Invocations t = new Invocations();
PoolDataSource ods= t.trace(PoolDataSourceFactory.getPoolDataSource(),PoolDataSource.class);

ods.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
ods.setURL("jdbc:oracle:thin://localhost:1521/orcl");
ods.setUser("scott");
ods.setPassword("tiger");
Connection conn = ods.getConnection();
CallableStatement stmt = conn.prepareCall("begin dbms_output.put_line('Hello'); end;");
stmt.execute();
stmt.close();
conn.close();
System.out.print(t);

Which produces the following output:

oracle.ucp.jdbc.PoolDataSource v1;
v1.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
v1.setURL("jdbc:oracle:thin://localhost:1521/orcl");
v1.setUser("scott");
v1.setPassword("tiger");
java.sql.Connection v2;
v2 = v1.getConnection();
java.sql.CallableStatement v3;
v3 = v2.prepareCall("begin dbms_output.put_line('Hello'); end;");
boolean v4;
v4 = v3.execute();
v3.close();
v2.close();

Which is a pretty good facsimile of the original code. Note there are obviously limitations to what this kind of approach can achieve:

• Invocations can only handle primitive values and values created within the API itself
• Any non primitive values (including arrays) created outside of the API can only be declared, code to initialize these values will need to be added manually.

The major benefit of the generated code is that it does not have any dependencies other than the API being traced, thus making it easier to extract a stand-alone test case that exhibits the problem. This makes it easier to focus on the problem by eradicating superfluous code. It also very useful when you need a stand-alone test-case for a bug report for a third party API.

The Source Code

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.IdentityHashMap;

public class Invocations {

	@Override
	public String toString() {
		return b.toString();
	}
	public <T> T trace(final Object target, final Class<T> type) {
		return trace(target, type, false);
	}

	private void declare(final Object object, final Class<?> type,
			final boolean force) {
		if (!void.class.equals(type)) {
			if (force
					|| (object != null && !variables.containsKey(object) && !isPrimitive(type))) {
				b.append(typeName(type));
				b.append(" ");
				b.append(register(object));
				b.append(";\n");
			}
		}
	}
	private boolean isPrimitive(final Class<?> type) {
		boolean isPrimitive = type.isPrimitive();
		if (!isPrimitive) {
			for (final Class<?> c : PRIMITIVES) {
				if (c.equals(type)) {
					isPrimitive = true;
					break;
				}
			}
		}
		return isPrimitive;
	}
	private String name(final Object object) {
		String name = "null";
		if (object != null) {
			name = variables.get(object);
			if (name == null) {
				if (isPrimitive(object.getClass())) {
					return render(object);
				} else {
					throw new RuntimeException("Could not find name for: "
							+ object);
				}
			}
		}
		return name;
	}
	private String register(final Object object) {
		String name;
		name = variables.get(object);
		if (name == null) {
			name = "v" + (++var);
			variables.put(object, name);
		}
		return name;
	}
	private String render(final Object object) {
		final Class<?> type = object.getClass();
		if (String.class.equals(type)) {
			return "\"" + object.toString() + "\"";
		} else if (short.class.equals(type) || Short.class.equals(type)) {
			return "(short)" + object.toString();
		} else if (long.class.equals(type) || Long.class.equals(type)) {
			return object.toString() + "L";
		} else if (char.class.equals(type) || Character.class.equals(type)) {
			return "'" + object.toString() + "'";
		} else {
			return object.toString();
		}
	}
	@SuppressWarnings("unchecked")
	private <T> T trace(final Object target, final Class<T> type,
			final boolean force) {
		T result = null;
		if (target != null) {
			if (target.getClass().getInterfaces().length > 0
					&& !isPrimitive(target.getClass())) {
				result = type.cast(Proxy.newProxyInstance(Thread
						.currentThread().getContextClassLoader(), target
						.getClass().getInterfaces(), new Recorder(target)));
			} else {
				result = (T) target;
			}
		}
		declare(result, type, force);
		return result;
	}
	private String typeName(final Class<?> type) {
		if (type.isArray()) {
			return typeName(type.getComponentType()) + "[]";
		} else {
			return type.getName();
		}
	}

	private class Recorder implements InvocationHandler {

		Recorder(final Object target) {
			this.target = target;
		}

		public Object invoke(final Object proxy, final Method method,
				final Object[] args) throws Throwable {
			Object result = null;
			final Class<?> type = method.getReturnType();
			try {
				result = method.invoke(target, args);
			} catch (final InvocationTargetException e) {
				final Throwable t = e.getCause();
				for (final Class<?> ex : method.getExceptionTypes()) {
					if (ex.isAssignableFrom(t.getClass())) {
						throw t;
					}
				}
				throw e;
			}
			result = trace(result, type, true);
			record(result, proxy, method, args);
			return result;
		}

		private void record(final Object result, final Object proxy,
				final Method method, final Object[] args) {
			if (args != null) {
				final Class<?>[] types = method.getParameterTypes();
				for (int i = 0; i < args.length; ++i) {
					declare(args[i], types[i], false);
				}
			}
			if (result != null) {
				b.append(name(result));
				b.append(" = ");
			}
			b.append(name(proxy));
			b.append('.');
			b.append(method.getName());
			b.append('(');
			if (args != null) {
				for (int i = 0; i < args.length; ++i) {
					b.append(name(args[i]));
					if (i < args.length - 1) {
						b.append(',');
					}
				}
			}
			b.append(");\n");
		}

		private final Object target;

	};

	private final StringBuilder b = new StringBuilder();

	private int var = 0;

	private final IdentityHashMap<Object, String> variables = new IdentityHashMap<Object, String>();
	private static final Class<?>[] PRIMITIVES = { String.class, Integer.class,
			Boolean.class, Double.class, Long.class, Short.class, Byte.class,
			Character.class, Float.class };
}

I’ve only used this code with the specific problem I was facing, I’m sure there are issues with it, I haven’t tested it comprehensively, YMMV. For example I know it doesn’t handle arrays very well, particularly those with multiple dimensions.

In November of 2008, I was winding up a project at Insurance Auto Auctions. The project had gone extremely well and I was walking out with a letter from a director on how satisfied they were with my work. Over the course of ten months, I made increment changes to their development practices. They already had a very mature and strong development environment that included all the necessary hardware and software. No architect or developer could ever say that they weren’t given the tools at Insurance Auto Auctions.

After convincing management that we needed a dedicated build server, the first thing I implemented was a set of middle-ware templates that acted as a framework for accessing their databases. Not as robust as nHibernate or any of the common frameworks, but tooled for their environment without all of the extra features that either confuse developers or entangle projects with needless work. For instance, the templates I use do not have relations built-in. You have to wire them up on your own. For most traditional CRUD development projects, you rarely need more than Fowler’s Table Module enterprise design pattern anyway.

The second step was to implement continuous integration using Cruise Control .NET. A fairly common tool in the development world, created by Martin Fowler’s Thoughtworks consulting firm. This gave us immediate feedback on checked in code.

Thirdly, we went round and round about how to build the new system. Did we want to use traditional WebForms? We’re we okay with using the beta of ASP.NET MVC? Did we want to role our own MVP (Model View Presenter) framework or possibly do something else? In the end, we decided the simplest implementation, for various reasons, was to create a home grown MVP framework. Of course MVP is considered obsolete now and we saw what people were recommending, but we moved forward anyway. We felt the ability to cleanly write unit tests was a critical aspect. The downside of course was that we had to rewrite a number of HTML controls that come out of the box in ASP.NET. The out of box controls use ViewState and with ViewState off, we had to go it alone, so to speak. Luckily we had Cliff on the team. Any team that has some has brilliant as Cliff should feel blessed. I’m a pretty smart guy, but more suited to larger problem solving. Cliff has a knack for needling through really fiddly problems with a very elegant solution.

We of course got buy-in to implement unit tests in Visual Studio .NET 2008 Team Test. We installed this on the build server and after every check-in, all tests were executed and the build failed if any tests failed.

The business side worked on user stories, the developers had meetings to gain an understanding of each story, and the rest they say, is history. the development went very smoothly. By implementing all of the code in a single solution and by also implementing a nifty web.config connection string locator, we were able to execute deployments with the built-in Publish feature in Visual Studio. After dealing with complex and error prone deployment processes in the past, the project manager was deleriously happy with the new process.

We deployed the web application to several servers in a farm and after some fairly minor performance tuning, the system was deemed stable and in production.

In the last few weeks of the project, I was also working on securing my next position. This proved to be a very important decision and one that seemed too good to be true. Alas, that old saying is only the tip of the iceberg and will be discussed in Part 2.

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

Part 4: Generating C# classes

Part 5: Generated C# classes and polymorphism

The code presented in the previous articles generates a C# class definition (based on dbscript’s Object table) like this:

public partial class Object : DAL.BaseTable
{
  public Object() { }
  public Object(IDataReader dr) : base(dr) { }
  protected override string TableName { get { return "Object"; } }
  public int OID
  { get { return GetColumnValue<int>(Columns.OID); } }
  public int PV_OID
  {
    get { return GetColumnValue<int>(Columns.PV_OID); }
    set { SetColumnValue(Columns.PV_OID, value); }
  }
  ....
  protected override string IdentityColumn { get { return "OID"; } }
  public static class Columns
  {
    public static string OID = "OID";
    public static string PV_OID = "PV_OID";
    ....
  }
  public static List<Object> GetRecords(IDataReader dr)
  { return GetRecords<Object>(dr); }
  public static List<Object> GetRecords(SqlConnection conn, string sSelect)
  { return GetRecords<Object>(conn, sSelect); }
}

Note that the class is generated as a partial class, so you are free to extend it in your own code.

In the dbscript database, all information about schema objects is stored in the Object table, and of course, not all columns of this table are used by the different object types, such as tables, views, etc. If we take into account different database engines (a concept introduced in dbscript 0.96), then tables, views, etc in MSSQL have different properties than their counterparts in Oracle or PostgreSQL.

Time to create some class which are derived from DAL.BaseTable to inherit the DAL mechanisms, but reference the generated DAL.Object columns:

public class SchemaObject : dbscriptlib.DAL.BaseTable
{
  protected override string TableName { get { return "Object"; } }
  protected override string IdentityColumn { get { return "OID"; } }
  public int OID
  { get { return GetColumnValue<int>(DAL.Object.Columns.OID); } }
  public string OwnerName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.OwnerName); }
    set { SetColumnValue(DAL.Object.Columns.OwnerName, value); }
  }
  public string ID
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.ID); }
    set { SetColumnValue(DAL.Object.Columns.ID, value); }
  }
  ....
}
public class Table : SchemaObject
{
  public string FileGroupName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.FileGroupName); }
    set { SetColumnValue(DAL.Object.Columns.FileGroupName, value); }
  }
  ....
}

Note that the FileGroupName is defined only for tables, rather than for all DAL classes. In Oracle, there is no FileGroupName, but a TablespaceName, which we map to the original column using this definition:

public class OracleTable : SchemaObject
{
  public string TablespaceName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.FileGroupName); }
    set { SetColumnValue(DAL.Object.Columns.FileGroupName, value); }
  }
  ....
}

With this simple mechanism, a sort of polymorphic table can easily be implemented, and only the accessing classes “know” how about the mapping of the table columns.

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

As we have the base class of our DAL almost complete, it’s time to generate the actual table access classes.

CREATE PROCEDURE Generate_DAL AS
PRINT
'using System;
using System.Data;
using System.Data.SqlClient;
using System.Collections.Generic;
/*
 Class Definition for Data Access Layer
 automatically generated by Generate_DAL
 ' + CONVERT(VARCHAR, GETDATE(), 120) + '
*/
namespace DAL
{'
DECLARE @OID INT, @ID VARCHAR(50)
DECLARE cTable CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR

(insert the table SELECT statement here). First we generate the access class constructors:

OPEN cTable
FETCH cTable INTO @OID, @ID
WHILE @@FETCH_STATUS=0 BEGIN
  IF UPPER(@ID) = 'COLUMNS' SET @ID = SUBSTRING(@ID, 1, LEN(@ID)-1)
  PRINT
'    public partial class ' + @ID + ' : DAL.BaseTable
 {
 public ' + @ID + '() { }
 public ' + @ID + '(IDataReader dr)  : base(dr)  { }
 protected override string TableName { get { return "' + @ID + '"; } }'

DECLARE @colID VARCHAR(50), @colDataType VARCHAR(50), @colLength INT
DECLARE @colNullable BIT, @colIdentity BIT, @colComputed BIT
DECLARE @colNetType VARCHAR(50), @colIdentityName VARCHAR(50)
SET @colIdentityName = NULL
DECLARE cCol CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR

(insert the column SELECT statement here). We need to convert SQL data types to C# data types, and translate NULLable to Nullable<>:

OPEN cCol
FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
WHILE @@FETCH_STATUS=0 BEGIN
  IF @colNullable = 1
    SELECT @colNetType = CASE @colDataType
      WHEN 'int' THEN 'int?'
      WHEN 'bit' THEN 'bool?'
      WHEN 'nvarchar' THEN 'string'
      WHEN 'varchar' THEN 'string'
      WHEN 'ntext' THEN 'string'
      WHEN 'text' THEN 'string'
      WHEN 'datetime' THEN 'DateTime?'
      ELSE @colDataType
    END
  ELSE
    SELECT @colNetType = CASE @colDataType
      WHEN 'int' THEN 'int'
      WHEN 'bit' THEN 'bool'
      WHEN 'nvarchar' THEN 'string'
      WHEN 'varchar' THEN 'string'
      WHEN 'ntext' THEN 'string'
      WHEN 'text' THEN 'string'
      WHEN 'datetime' THEN 'DateTime'
      ELSE @colDataType
    END

  IF @colIdentity = 1 SET @colIdentityName = @colID
  PRINT ' public ' + @colNetType + ' ' + @colID + '{
get { return GetColumnValue<' + @colNetType + '>(Columns.' + @colID + '); }'
  IF @colIdentity = 0 AND @colComputed = 0
    PRINT ' set { SetColumnValue(Columns.' + @colID + ', value); }'
  PRINT ' }'
  FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
END
CLOSE cCol

IF @colIdentityName IS NOT NULL
  PRINT '        protected override string IdentityColumn { get { return "' + @colIdentityName + '"; } }'
PRINT '        public static class Columns {'
OPEN cCol
FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
WHILE @@FETCH_STATUS=0 BEGIN
  PRINT    '            public static string ' + @colID + ' = "' + @colID + '";';
  FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
END
CLOSE cCol
DEALLOCATE cCol
PRINT    '        }'
PRINT    '        public List<' + @ID + '> GetRecords(IDataReader dr) {
    return GetRecords<' + @ID + '>(dr);
  }
public List<' + @ID + '> GetRecords(SqlConnection conn, string sSelect) {
    return GetRecords<' + @ID + '>(conn, sSelect);
  }
}'

FETCH cTable INTO @OID, @ID
END
PRINT    '}'

The resulting class definition has typed getter/setter properties for each column of the base table, and a static string array containing the names of the table columns.

Thus changes the data model (column type, column name, dropped table or column) are immediately reflected in the generated code, and automatically raise to compilation warnings or errors in your code which depends on the model.

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

To select multiple records and create their DAL objects, we pass an IDataReader object or a combination of connection and SELECT statement to a static method GetRecords() and receive a typed List<> of records:

protected static List<T> GetRecords<T>(IDataReader dr)
    where T : BaseTable, new()
{
  List<T> li = new List<T>();
  while (dr.Read())
  {
    T record = new T();
    record.GetRecord(dr);
    li.Add(record);
  }
  dr.Close();
  return li;
}

protected static List<T> GetRecords<T>(SqlConnection conn, string sSelect)
    where T : BaseTable, new()
{
  SqlCommand cmd = new SqlCommand(sSelect, conn);
  return GetRecords<T>(cmd.ExecuteReader());
}

The Save() method writes all values of dictNew as INSERT or UPDATE statement:

public void Save(SqlConnection conn)
{
  if (dictNew == null)
    return;

  if (dictRecord == null)
    Insert(conn);
  else
    Update(conn);
}

This is a sample implementation of the Insert() command. Note that in the simplest version, the values of dictNew are INSERTed, and only the IDENTITY column (if defined) is retrieved after the insert.

In more complex scenarios, you would want to retrieve all the values of the newly generated record to handle defaults and computed columns. If the table is lacking an IDENTITY column, the record needs to be selected using the columns of a UNIQUE constraint. These scenarios are not covered right now.

private void Insert(SqlConnection conn)
{
  StringBuilder sb = new StringBuilder();
  sb.Append("INSERT INTO " + TableName + " (");

  bool bFirst = true;
  foreach (string s in dictNew.Keys)
  {
    if (!bFirst)
      sb.Append(", ");
    sb.Append(s);
    bFirst = false;
  }

  sb.Append(") VALUES (");
  bFirst = true;
  foreach (string s in dictNew.Keys)
  {
    if (!bFirst)
      sb.Append(", ");
    sb.Append("@" + s);
    bFirst = false;
  }
  sb.Append(")");

  SqlCommand cmd = new SqlCommand();
  cmd.Connection = conn;

  foreach (string s in dictNew.Keys)
  {
    if (dictNew[s] == null)
      cmd.Parameters.AddWithValue("@" + s, DBNull.Value);
    else
      cmd.Parameters.AddWithValue("@" + s, dictNew[s]);
  }

  if (IdentityColumn != null)
  {
    sb.Append("; SELECT " + IdentityColumn + " FROM " + TableName +
      " WHERE " + IdentityColumn + " = SCOPE_IDENTITY()");
    cmd.CommandText = sb.ToString();
    object oIdentity = cmd.ExecuteScalar();
    dictNew.Add(IdentityColumn, oIdentity);
  }
  else
  {
    cmd.CommandText = sb.ToString();
    cmd.ExecuteNonQuery();
  }
}

The first part of this series dealt with the queries to retrieve the necessary information on tables and columns from the system catalog of a SQL Server 2000/2005 database.

Next we need a base class with minimum functionality to provide table name column names, and SELECT, INSERT and UPDATE statements.

namespace DAL
{
  public abstract class BaseTable
  {
    protected abstract string TableName { get; }
    protected virtual string IdentityColumn { get { return null; } }

    private Dictionary<string, object> dictRecord = null;
    private Dictionary<string, object> dictNew = null;

dictRecord holds the current (old) values as retrieved by SELECT, dictNew contains new values for INSERT or UPDATE.

The parameterless constructor creates a new record, whereas dictRecord is set by a constructor with IDataReader parameter:

protected BaseTable()
{
}

protected BaseTable(IDataReader dr)
{
  GetRecord(dr);
}

private void GetRecord(IDataReader dr)
{
  if (dictRecord == null)
  {
    dictRecord = new Dictionary<string, object>();
    for (int iField = 0; iField < dr.FieldCount; iField++)
      if (dr.IsDBNull(iField))
        dictRecord.Add(dr.GetName(iField), null);
      else
        dictRecord.Add(dr.GetName(iField), dr.GetValue(iField));
  }
}

The generic method GetColumnValue retrieves a (typed) field value from one of the dictionaries, the method SetColumnValue sets a field value in dictNew:

protected CT GetColumnValue<CT>(string sColumnName)
{
  if (dictNew != null && dictNew.ContainsKey(sColumnName))
    return (CT)dictNew[sColumnName];

  if (dictRecord != null && dictRecord.ContainsKey(sColumnName))
    return (CT)dictRecord[sColumnName];

  return default(CT);    
}

protected void SetColumnValue(string sColumnName, object oValue)
{
  if (oValue is string && (oValue as string) == "")
    oValue = null;

  if (dictNew == null)
    dictNew = new Dictionary<string, object>();
  if (dictNew.ContainsKey(sColumnName))
    dictNew[sColumnName] = oValue;
  else
    dictNew.Add(sColumnName, oValue);
}