I apologize for my lack of update – like I noted before, consistency can be a problem of mine. But here I am, so let me share what’s been/is on my mind.

Although I wanted to have a specific theme/focus when I started this blog (e.g. NLP and related things),  it’s already become apparent that my interests and general projects tend to vary just a little too much for that: I’ll work on a Python project for 5 weeks, mess with an Arduino for one, study sound engineering on and off, come back and start on a big Python project, and concurrently start studying game design while still tinkering with my Arduino. It’s all still computer-science related, but the field is huge enough that I can’t really say there is any theme to what I tend to take on other than that, well, they’re computer related. And to complicate things, over the past little bit I’ve been straying further and further away from the natural language-related parts of computer science. This may be partially because I’m not taking any Ling classes right now and all CPSC ones (which isn’t exactly a great sign when it comes to exemplifying any attachment to the field), but I think it may be more so that I’m simply straying more and more to being interested in the general workings of computers and computer applications.

For the time being, then, I’m going to retract my previous statement that this blog serves the purpose to document my journey to being proficient at working with NLP. This blog is simply going to document my little projects and endeavors here and there in my journey to acquire a solid knowledge base in computer science, cool computer-y, techie, physics-y, and electronics-y things in general! I’m sure at some point a general direction for this blog will start to make itself clear, but it is still early. So I guess we’ll just have to see what this blog ends up growing into in the future.

So that aside, I’d like to apologize for the lack of update. First round of midterms at school hit, and I really didn’t do much else but study… and therefore didn’t do anything worth writing about.

Over reading week, however, I did spend a bit of time reading up and working on Java for my software construction course. Having had all my prior programming experience in Python (well, and Racket too I guess) and having never used it’s Object Oriented Programming (OOP) capabilities, the whole OOP mindset has been a little rough to adjust to. Nevertheless, now that I finally understand the difference between static and dynamic, objects and classes, interfaces, abstraction, implementation and so forth… I’m having trouble imagining how I would do certain things if the language were not object-oriented.

For example, after I discovered Java Docs 7 (I was using the old docs… they were horrible. I hope nobody ever makes as dumb a mistake as I did: not making sure the docs they’re using are the latest ones), I finally managed to make my very first GUI “program” (as opposed to CLI):

It’s a simple (slightly more elaborate than the traditional) “Hello World” window with a menu, submenus, and a button… but I’m quite happy with it. I can’t imagine how a programming language could offer a user an elegant yet flexible way to create GUI applications without the concept of objects and classes, though. It’s just so intuitive to take an object and say “add this!” or “set this!”  (e.g. JFrame.setText(“Yo!”))  The whole idea of OOP is just genius.

Of course, thinking about how these different philosophies and paradigms got me thinking about programming language design, and just programming languages in general. It got me thinking about just how fascinating it is that each programming language really does have it’s own outlook, it’s own flavour. And that all these come from the meticulously thought-through decisions the creator of the language made in trying to instill their own beliefs about effective programming into the language.

All that got me thinking that I’d really like to understand this a lot better. And so, I’m toying with the idea of writing a simple compiler and programming language.  I know you’re thinking, “What…? That’s low-level shit for people with actual skills.” But although I know writing a compiler is considered by many programmers to be the ultimate challenge, I don’t think it necessarily HAS to be considered such a daunting task and such an impossible thing to do in every context.

In my case, I simply want to experience some of these decisions that have to be made, and use this project to further my understanding of the structure of certain languages and why their creators decided to structure the language that way. I want to discover more of what I believe, and what my philosophical views on what a “good” language should do. Of course, I’m not going to try to go write the newest C. The language is going to be relatively (okay, very) high-leveled, and will probably be written in Java or Python. And it’ll really only be for me. But that’s all I’ll say about it for now. Who knows, maybe it’ll blossom into something else!

Edit: someone pointed out to me today that what I was kind of  intending to do – write something in Java/Python that will parse a new language into Java/Python then compile accordingly – would really more or less just be a parser using the already-written JVM or Python interpreter. Oops. I’m not looking to simply write a parser (although I was introduced to regular expressions today, and might just write a simple one to mess around with/practice using regexes), so it looks like I’m going to have to do a little more research on what exactly is required for a compiler as opposed to an interpreter, and so forth and so on. Will need to do a bit of mulling over what I’m really after, and do a little more research on what I’m going to get myself into. As always, I spoke a little too much a little too soon. Thanks, John!

In any case, I’ll keep you all posted on that latest project. Until then, you best believe I’ll be reading lots about how to go about it, as well as planning the most kickass syntax you will ever see. As well, I’ll keep working on my GUI skills in Java. My next goal? Learn how to implement Actions so I can actually make a simple, working interactive application.

It’s late, and so I leave you with a song from one of the best games I’ve played in a long time: Braid.

The entire soundtrack is fantastic, and the game itself is just ridiculous. If you haven’t heard of Braid, I highly recommend it as one of the most innovative puzzle games out there. I’d put it up there right alongside Portal. Check it out!

And with that, adieu.

Larissa

Compiling.tv is getting ready for Season 2. We start filming this Saturday March 2nd. We want to hear ideas from our fans on where to place all these pool noodles. Details are all on our official website.

Namely, if we go back to the early days of programming… The first programmable computers would take their input in the form of machine code: each instruction would be encoded as a set of bits, consisting of an opcode (the operation itself, such as ‘add’ or ‘subtract’) and its operands (the data to operate on, which can be a constant, a register, or a memory location for example).

When writing a program, the programmer would have to first line out the program in pseudocode, and then convert it to machine code by hand. In order to make things easier, each opcode would have a ‘mnemonic’: a short name which described the instruction. The programmer could then first write down the program in a listing of mnemonics for each instruction, and then convert each instruction to its machine code representation.

Here is an example of some code written by Steve Wozniak (the co-founder of Apple, and designer of the early Apple computers):

As you can see, he’s written out the code in a number of columns. The first column is the memory address of each instruction, the second column contains the machine code bytes for that instruction, then follows the mnemonic representation of the code (the ‘human readable form’), and finally some comments.

Initially you would have to do this all by hand. The process of converting mnemonics to machine code became known as ‘assembling’ the code, and automated tools were developed for this, which became known as ‘assemblers’. The first assembler was written for the EDSAC computer in 1949, and was called “initial orders”.

The first compiler however, was written by Grace Hopper in 1952, for the A-0 programming language. And that in itself already shows that compiling and assembling are not seen as the same thing. Namely, since we already had assemblers at that time, then why did Grace Hopper bother to coin the term ‘compiler’, rather than just re-using the term ‘assembler’ to describe this new tool? Apparently there is a fundamental difference between the two types of tools.

The main difference between assembly language and other programming languages such as the A-0 language, is that assembly languages are always machine-dependent (after all, the mnemonics are merely a more human-readable form of the instructions that the machine supports, so different machines have different mnemonics), where other programming languages abstract away the physical machine, and work at a higher level. Many early compilers would also compile the higher level source listings into a machine-specific assembly listing, which would then be passed on to an assembler to generate the actual machine code.

Since a compiler works from a source listing at a higher level, it also needs to perform a more complex translation than an assembler. For an assembler there is generally a 1:1 mapping from mnemonics to machine code. There are some exceptions where a single instruction may be encoded in multiple ways, but generally any decisions an assembler needs to make at all during translation are very trivial and unambiguous (for example, picking the shortest encoding for a given instruction). For this reason, writing assembly code is as good as writing machine code by hand, as far as performance and size optimizations go.

Compilers however need to map variables used in the source code to machine-specific registers and memory locations, and try to decide the shortest and/or fastest possible sequence of instructions to translate the code. This causes complicated problems for register allocation and re-use for example. There is no 1:1 translation of high-level keywords and expressions to machine code. There are many possible alternatives, and a compiler will need to do a lot of analysis and use clever heuristics to try and come up with fast code.

Especially in the early days, compilers were rather naïve, and their translations would not come anywhere near assembly code optimized by hand. These days however, compilers have come a long way, and perhaps just as important: so have computers. Where early computers were still designed specifically for handwritten programs, over time more and more people started using compiled languages, and computers were designed more and more to make the job of compilers easier. Compilers generate code in an algorithmic way, and so they would only use a given subset of instructions. RISC CPUs would reduce the instructionset to only include the most-used instructions, and make these instructions run as fast as possible. Which would mean that the job of compilers became easier.

Watch Episode 22  live at Compiling.tv today!

New episodes will be live each week on Thursdays. Season Finale goes live March 14th.

Become a Fan to have free access to Exclusive Content Videos, Fun Facts and lots more!

A great many thanks to everyone who donated during the Indiegogo Campaign. The campaign is now closed, but we continue to accept Open Donations from anyone who wishes to contribute.

Transitive: a <= b and b <= c -> a <= c;
Antisymmetric: a<= b and b <=a -> a = b;

Total order set: every pair is related.

Besides transitive and antisymmetric, it also has comparability: for any a, b, either a <= b or b <= a;

The .Net Architecture (.Net Framework) has the few different very important parts which we will discuss now:-

.Net Framework

CLR:- Common Language Runtime, it is also known as the .Net Runtime. It is a layer above the systems OS which handles the execution of applications. It  has the responsibility of providing the functionality like security, integration, memory management, process management, thread management, releasing unused memory. The advantage of CLR is it creates the whole environment to build the application for the developer that in the past they had to do their own.

MSIL:- Microsoft Intermediate Language, the .Net framework is build to support many languages. So when we uses any .Net compliant language to build the application, the source code is not directly converts to executable binary code, rather than it is converted into a new language code which is known as MSIL. The MSIL is interpreted by the CLR. The advantage of MSIL is it is a platform independent code. When the program executes, this code is gets converted into the binary executable code which is also know is the native code or machine code. Now that’s how the .Net supports the  multiple languages.

Compilers:- Just In Time Compilers (JITers) to execute the new IL code the Common Language Runtime uses the JIT compiler. The result of the JIT is comes in form of .exe or .dll which is used by the machine. But what is the different thing which makes the JIT compilers different from the other traditional compilers is that the JIT compiles the IL into native code whenever the requirement occurs means when the operation regarding that particular code portion is called the IL of that operation is converted or compiled into the machine code at that particular time.

FCL:- Framework Class Library that .Net framework contains the huge amount of classes also know as base classes for standard work. It contains tons of classes to provide the standard functions like I/O, String Manipulation  Security Management, Network Communications, Thread Management, Data Access and many more. These set of classes are managed in Namespaces.

CLS:- Common Language Specification the .Net Framework supports the multiple languages, but how it supports this. what makes a language .Net compliant language. The reason is the CLS every .Net compliant language has to fulfill some specification to qualify for a .Net compliant language.

A CITY PROJECT BATTLE ROYALE

As the keynote speaker at the Urban Sciences Research Coordination Network kickoff last Friday, the City of Chicago’s Brett Goldstein presented a blizzard of exciting city projects at various states of development. One slightly-under-wraps project Goldstein touched upon was the SmartData platform, an ambitious plan to craft a new tool for decision-making and city services out of the abundant raw material of city data. In collaboration with the Computation Institute and the Urban Center for Computation and Data, the city’s Innovation and Technology team hopes to create a tool that will analyze the city’s many large datasets in real time to help the city respond to challenges more quickly and efficiently, while providing frequently updated, useful information to its citizens.

Wednesday, that exciting new effort was announced as a finalist in the Bloomberg Philanthropies Mayors Challenge, a competition among ideas proposed by cities across the United States. As part of the judging, the public is invited to vote for their favorite project among the 20 finalists at the Huffington Post. We’re biased of course, but to help make the case for Chicago’s project, you can read more about the SmartData platform here, or watch a video about the concept featuring Mayor Rahm Emanuel below.

COMPUTER IN A WHITE COAT

Last year at the Research Computing Center opening reception, attendees were given the chance to play Jeopardy against a legendary contestant: Watson. While most famous for once taking down Jeopardy grandmaster Ken Jennings on television, Watson was actually developed by IBM to be a complex problem-solver for any field faced with a tidal wave of data. Medicine is one such field, and in the cover story of The Atlantic this month, Dr. Watson gets the spotlight. Much like Watson drew upon a vast storage of word associations to rapidly locate the right question on Jeopardy, IBM believes that the computer system will be able to quickly scan through an enormous corpus of medical literature and clinical information to find the right diagnosis for a patient, free from the bias or limitations of a regular human physician.

Beyond the details about the cover star, the article is a good overview of how data technology and computation are expected to transform the practice of medicine in the near future. Two main advantages of Watson pointed out by the article — the ability to handle “unstructured data” and the ability to convey uncertainty about a conclusion – reflect broader trends in computational research for medicine and biology, as well as other projects on climate change and the humanities. The article also covers health IT trends such as personalized medicine, using smartphones to monitor/transmit medical data and tele-health interventions.

In the future as the innovators imagine it—“Health 2.0,” as some people have started calling it—you would be in constant contact with the health-care system, although you’d hardly be aware of it. The goal would be to keep you healthy—and any time you were in danger of becoming unhealthy, to ensure you received attention right away. You might wear a bracelet that monitors your blood pressure, or a pedometer that logs movement and exercise. You could opt for a monitoring system that makes sure you take your prescribed medication, at the prescribed intervals. All of these devices would transmit information back to your provider of basic medical care, dumping data directly into an electronic medical record.

OTHER NEWS IN COMPUTATIONAL SCIENCE

Many experts believe that quantum computers will someday become necessary to keep improvements in processor speed along the exponential track of Moore’s Law. But in the meantime, two new papers published in Science this week propose an intermediate step towards quantum computation, demonstrating how a relatively easily-scaled method called photonic boson sampling can be used to solve certain matrix calculations faster than a “classical” computer.

Can big data build a better lie detector? An interesting article at Datanami talks about how fraud prevention services are using data algorithms akin to a souped up version of Facebook’s new graph search to put together the history of a given person…and sniff out if they are up to no good.

Is the human brain computable? That’s going to be one of science’s biggest debates over the next decade as both Europe and the United States plan to launch massive and expensive efforts to simulate and/or map the human brain. But some scientists, including brain-machine interface researcher Miguel Nicolelis, are skeptical that engineering will ever be capable of reproducing the complex abilities and functions of the human brain.

Watch Episode 21  live at Compiling.tv today!

New episodes will be live each week on Thursdays.

Become a Fan to have free access to Exclusive Content Videos, Fun Facts and lots more!

A great many thanks to everyone who donated during the Indiegogo Campaign. The campaign is now closed, but we continue to accept Open Donations from anyone who wishes to contribute.

I’m back to Canada after a two week visit to the Mozilla offices in Mountain View. The visit started with a presentation of Higgs and my PhD research work, which was made available online on Air Mozilla. Then, over the next two weeks, I had several one on one meetings with several members of the JS engine and research teams at Mozilla. The Mozilla people were all extremely friendly and forthcoming. Because of Mozilla’s open culture, I was able to get answers to just about any question I might want to ask about their work. I was also invited to give a talk at Apple and briefly meet with the JavaScriptCore team. This trip was a great opportunity to discuss and refine my research ideas, as well as to gain specific insights into the workings of production JavaScript JIT compilers.

Speaking at Apple

There were also unexpected benefits to my visit. The presentation I gave at Mozilla, which was made available online, garnered some attention on twitter and reddit. As a result, I received invitations to meet from Alex Gaynor of the PyPy project and from Ariya Hidayat of the Esprima parser project (both of which I promptly accepted). I was also invited by Andrei Alexandrescu himself to talk about Higgs and my experience using the D programming language at DConf 2013. Finally, I had the pleasant surprise of getting pull requests on the Higgs github repository.

A taste of America

Being in the Silicon Valley area, I also took the opportunity to visit San Francisco on the weekends. I went to an outdoor pub with some friends and got invited to a barbecue party where I was introduced to the infamous Cards Against Humanity. San Francisco is a charming city with its own highly liberal hippie subculture. It has cute little boutiques and cafés, streetfood, and beautiful parks. Public transportation seems rather good, at least in the city itself (I can’t speak so highly of the suburbs).

San Francisco

The bay area, and San Francisco, are truly different from what I’m used to. It seems everyone is involved with technology, or with a startup, in one way or another. Everyone has a smartphone and Apple TV. People use apps on their phones to summon cabs. The rents are very high and almost everybody has housemates (even couples). Because so many people are in tech, the online communities of the bay area are also more active than elsewhere. Getting to experience Silicon Valley was an interesting experience, and makes me understand why so many tech companies choose to establish themselves in the area: nowhere else are people this excited about tech.

At this point, you might be wondering what I’ve done with my research during this trip, besides talking to many people. For one, I started looking more seriously into the area of trace compilation. The Mozilla people have let me know that they had multiple issues with their TraceMonkey JIT, which eventually led to its retirement. One of these issues was trace explosion: the generation of too many independent traces. Another was constant fallback to the interpreter. What I want to do with Higgs, is implement something closer to region-based compilation, and attempt to solve some of these tracing problems.

In terms of code, I started planning the implementation of a simple backend and tracing JIT for Higgs, so that I can begin experimenting with these concepts. I already made some minor refactorings that will be necessary for this JIT to be possible. I also took the time to fix many (dozens of) bugs that were preventing Higgs from running SunSpider and V8 benchmarks. The good news is that Higgs is now much more standards compliant than before, and even better tested. I updated the lists of supported features and supported benchmarks on the Higgs page to reflect this.

This article is split into 5 posts.

1. Introduction
   • General Approach
   • Generation Model
2. Templating Engines
   • T4
   • StringTemplate
   • DotLiquid
3. Code Generation From Source
   • CodeDom
   • Roslyn
   • Mono
4. Code Generation From Model
   • Emit
   • Expression Trees
   • Mono Cecil
5. Loading
   • Dependency Injection
   • Managed Extensibility Framework
   • Reflection

In-Memory Run-Time Code Generation With .NET – Part 4

Code Generation From Model

Instead of using templaing engines to create source code that need compilation, MSIL can be created using either Reflection Emit or by constructing Linq Expression Trees and compiling those.

Reflection Emit

A more direct approach to generating types to runtime is to directly generate MSIL code. The .NET Framework supplies all nesseccary classes to do so. This approach is very fast, since no source code must be parsed and compiled. On the other hand, the MSIL output must be hand-coded, which is a difficult and error-prone venture. The following class generates the types that were generated by the code templates in the previouse sections. To abbreviate the lengthy code, the generation of properties that return view model lists is skipped.

internal class TypeGenerator
{
    private readonly TypeGenerationModel _model;
    private readonly Dictionary<Type, Type> _viewModelTypes;
    private readonly TypeBuilder _typeBuilder;

    public Type GeneratedType { get; private set; }

    public static Assembly Generate(IEnumerable<TypeGenerationModel> models)
    {
        var appDomain = AppDomain.CurrentDomain;
        var assemblyName = new AssemblyName("dynamicassembly");
        var assemblyBuilder = appDomain.DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.RunAndSave);
        var moduleBuilder = assemblyBuilder.DefineDynamicModule(assemblyName.Name, assemblyName.Name + ".dll");
        var viewModelTypes = new Dictionary<Type, Type>();

        foreach (var model in models.Reverse())
        {
            var generator = new Typeil(moduleBuilder, model, viewModelTypes);
            viewModelTypes.Add(model.ModelType, generator.GeneratedType);
        }

        return assemblyBuilder;
    }

    private TypeGenerator(ModuleBuilder moduleBuilder, TypeGenerationModel model, Dictionary<Type, Type> viewModelTypes)
    {
        var baseType = typeof (ViewModelBase<>).MakeGenericType(model.ModelType);
        _model = model;
        _viewModelTypes = viewModelTypes;
        _typeBuilder = moduleBuilder.DefineType(_model.Namespace + "." + _model.Name, TypeAttributes.Public, baseType);

        GeneratedType = Generate();
    }

    private Type Generate()
    {
        //
        // Create constructor
        //
        var baseConstructor = _typeBuilder.BaseType.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new[] {_model.ModelType}, new ParameterModifier[0]);
        var constructorBuilder = _typeBuilder.DefineConstructor(MethodAttributes.Public, CallingConventions.Standard, new[] {_model.ModelType});
        var il = constructorBuilder.GetILGenerator();

        il.Emit(OpCodes.Ldarg_0);                   // Load the address of "this" onto the stack
        il.Emit(OpCodes.Ldarg_1);                   // Load the first parameter onto the stack
        il.Emit(OpCodes.Call, baseConstructor);     // Call the constructor of the base class
        il.Emit(OpCodes.Ret);                       // Return

        //
        // Create properties
        //
        foreach (var property in _model.Properties)
        {
            CreateProperty(property, _typeBuilder);
        }

        foreach (var property in _model.ViewModelProperties)
        {
            CreateViewModelProperty(property, _typeBuilder);
        }
        
        // In this sample, we wont generate this kind of property.
        //foreach (var property in _model.ViewModelListProperties)
        //{
        //    CreateViewModelListProperty(property, typeBuilder);
        //}

        //
        // Finish up
        //
        return _typeBuilder.CreateType();
    }

    private void CreateProperty(PropertyGenerationModel property, TypeBuilder typeBuilder)
    {
        var propertyType = property.ElementType;
        var propertyBuilder = typeBuilder.DefineProperty(property.Name, PropertyAttributes.None,
                                                         propertyType, null);

        if (property.CanGet)
        {
            var thisModelGetter = typeBuilder.BaseType.GetProperty("Model").GetGetMethod();
            var modelPropertyGetter = _model.ModelType.GetProperty(property.Name).GetGetMethod();

            CreateGetMethod(property, typeBuilder, propertyBuilder, propertyType, il =>
            {
                // return Model.PROPERTY;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Call, thisModelGetter);         // Call "this"."get_Model()"
                il.Emit(OpCodes.Callvirt, modelPropertyGetter); // Call "this"."get_Model()".

                // "get_PROPERTY()"
                il.Emit(OpCodes.Ret);                           // Return
            });
        }

        if (property.CanSet)
        {
            var thisModelGetter = typeBuilder.BaseType.GetProperty("Model").GetGetMethod();
            var modelPropertyGetter = _model.ModelType.GetProperty(property.Name).GetGetMethod();
            var modelPropertySetter = _model.ModelType.GetProperty(property.Name).GetSetMethod();
            var notificationMethod = typeBuilder.BaseType.GetMethod("OnPropertyChanged", BindingFlags.NonPublic | BindingFlags.Instance);

            CreateSetMethod(property, typeBuilder, propertyBuilder, propertyType, il =>
            {
                var labelElse = il.DefineLabel();

                // if(this.Model.PROPERTY == value) return;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Call, thisModelGetter);         // Call "this"."get_Model()"
                il.Emit(OpCodes.Callvirt, modelPropertyGetter); // Call "this"."get_PROPERTY()".

                // "get_PROPERTY()"
                il.Emit(OpCodes.Ldarg_1);                       // Load the parameter onto the stack
                il.Emit(OpCodes.Bne_Un_S, labelElse);           // If the last both values on the stack are not equal, jump to the label
                il.Emit(OpCodes.Ret);                           // Return
                il.MarkLabel(labelElse);

                // this.Model.PROPERTY = value;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Call, thisModelGetter);         // Call "this"."get_Model()"
                il.Emit(OpCodes.Ldarg_1);                       // Load the parameter onto the stack
                il.Emit(OpCodes.Callvirt, modelPropertySetter); // Call "this"."set_PROPERTY()"

                // OnPropertyChanged("PROPERTY");
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldstr, property.Name);          // Load the string onto the stack
                il.Emit(OpCodes.Call, notificationMethod);      // Call OnPropertyChanged()

                il.Emit(OpCodes.Ret);                           // Return
            });
        }
    }

    private void CreateViewModelProperty(PropertyGenerationModel property, TypeBuilder typeBuilder)
    {
        var propertyType = _viewModelTypes[property.ElementType];
        var fieldBuilder = typeBuilder.DefineField("_" + property.Name, propertyType, FieldAttributes.Private);
        var propertyBuilder = typeBuilder.DefineProperty(property.Name, PropertyAttributes.None, propertyType, null);

        if (property.CanGet)
        {
            // Get the view _model constructor that takes only the _model as parameter

            var ctor = propertyType.GetConstructor(new[] {property.ElementType});
            var thisModelGetter = typeBuilder.BaseType.GetProperty("Model").GetGetMethod();
            var modelPropertyGetter = _model.ModelType.GetProperty(property.Name).GetGetMethod();

            CreateGetMethod(property, typeBuilder, propertyBuilder, propertyType, il =>
            {
                var labelElse = il.DefineLabel();

                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldfld, fieldBuilder);           // Get the value of "this"."_field" and push in onto the stack
                                                                                            
                // if(this._field == null)
                il.Emit(OpCodes.Brtrue_S, labelElse);           // If the value on the stack is not null, jump to the label
                // {
                // this._field = new ViewModel(this.Model.Property);
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Call, thisModelGetter);         // Call "this"."get_Model()"
                il.Emit(OpCodes.Callvirt, modelPropertyGetter); // Call "this"."Model"."get_PROPERTY()"
                il.Emit(OpCodes.Newobj, ctor);                  // Create a new instance of the view model and push it onto the stack
                il.Emit(OpCodes.Stfld, fieldBuilder);           // Replace "this"."_field" by the value on the stack
                // }
                il.MarkLabel(labelElse);

                // Return this._field;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldfld, fieldBuilder);           // Get the value of "this"."_field" and push it onto the stack

                il.Emit(OpCodes.Ret);                           // Return
            });
        }

        if (property.CanSet)
        {
            var valueModelGetter = propertyType.BaseType.GetProperty("Model").GetGetMethod();
            var modelPropertySetter = typeBuilder.BaseType.GetProperty("Model").PropertyType.GetProperty(property.Name).GetSetMethod();
            var thisModelGetter = typeBuilder.BaseType.GetProperty("Model").GetGetMethod();
            var notificationMethod = typeBuilder.BaseType.GetMethod("OnPropertyChanged", BindingFlags.NonPublic | BindingFlags.Instance);

            CreateSetMethod(property, typeBuilder, propertyBuilder, propertyType, il =>
            {
                var labelElse = il.DefineLabel();

                // if(this._field == value) return;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldfld, fieldBuilder);           // Get the value of "this"."_field" and push in onto the stack
                il.Emit(OpCodes.Ldarg_1);                       // Load the first parameter onto the stack
                il.Emit(OpCodes.Bne_Un_S, labelElse);           // If the last both values on the stack are not equal, jump to the label
                il.Emit(OpCodes.Ret);                           // Return
                il.MarkLabel(labelElse);

                // this._field = value;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldarg_1);                       // Load the parameter onto the stack
                il.Emit(OpCodes.Stfld, fieldBuilder);           // Replace "this"."_field" by the value on the stack

                // this.Model.PROPERTY = value.Model;
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Call, thisModelGetter);         // Call "this"."get_Model()"
                il.Emit(OpCodes.Ldarg_1);                       // Load the parameter ("value") onto the stack
                il.Emit(OpCodes.Callvirt, valueModelGetter);    // Call "value"."get_Model()"
                il.Emit(OpCodes.Callvirt, modelPropertySetter); // Call "this"."Model"."set_PROPERTY()"

                // OnPropertyChanged("PROPERTY");
                il.Emit(OpCodes.Ldarg_0);                       // Load the address of "this" onto the stack
                il.Emit(OpCodes.Ldstr, property.Name);          // Load the string onto the stack
                il.Emit(OpCodes.Call, notificationMethod);      // Call OnPropertyChanged()

                il.Emit(OpCodes.Ret);                           // Return
            });
        }
    }

    private static void CreateGetMethod(PropertyGenerationModel property, TypeBuilder typeBuilder, PropertyBuilder propertyBuilder, Type propertyType, Action<System.Reflection.Emit.ILil> emit)
    {
        var methodBuilder = typeBuilder.DefineMethod("Get_" + property.Name, MethodAttributes.Public | MethodAttributes.HideBySig, propertyType, Type.EmptyTypes);
        var il = methodBuilder.GetILGenerator();
        emit(il);
        propertyBuilder.SetGetMethod(methodBuilder);
    }

    private static void CreateSetMethod(PropertyGenerationModel property, TypeBuilder typeBuilder, PropertyBuilder propertyBuilder, Type propertyType, Action<System.Reflection.Emit.ILil> emit)
    {
        var methodBuilder = typeBuilder.DefineMethod("Set_" + property.Name, MethodAttributes.Public | MethodAttributes.HideBySig, null, new[] {propertyType});
        var il = methodBuilder.GetILGenerator();
        emit(il);
        propertyBuilder.SetSetMethod(methodBuilder);
    }
}

Expression Trees

A furthur approach to generate code at runtime is the generation of Linq expression trees. Expression trees are not suitable in all cases. For example, with expression trees, the keyword “this” cannot be expressed. Also, constructors cannot be created. These two limitations prohibit using Linq expression trees to generate method bodies as the ones in the previouse section, even if Linq expressions can be compiled into types created via Reflection with the TypeBuilder class.

Dispite the limitation stated bove, Linq expresion trees can be very usefull when the generated code can run in a static methods. Consider the following method. It parses filter descriptions like “FirstName=Munir;LastName=Husseini” and creates expressions that compare the querable item’s properties with the specified values.

public static IQueryable<T> Where<T>(this IQueryable<T> query, string description)
{
    var parameter = Expression.Parameter(typeof (T), "item");
    var comparisons =
        from s in description.Split(new[] {';'}, StringSplitOptions.RemoveEmptyEntries)
        let s2 = s.Split(new[] {'='}, StringSplitOptions.RemoveEmptyEntries)
        let propertyName = s2[0]
        let value = s2[1]
        let property = typeof (T).GetProperty(propertyName)
        select Expression.Equal(
            Expression.Property(parameter, property),
            Expression.Convert(Expression.Constant(value), property.PropertyType));

    var expressionStack = new Stack<Expression>(comparisons);

    while (expressionStack.Count > 1)
    {
        var left = expressionStack.Pop();
        var right = expressionStack.Pop();
        var and = Expression.And(left, right);

        expressionStack.Push(and);
    }

    var filterExpression = expressionStack.Pop();
    var lambda = Expression.Lambda<Func<T, bool>>(filterExpression, parameter);

    return query.Where(lambda);
}

Using the method above, the following two objects query2 and query3 will be equally filtered.

IQueryable<Person> query = ... // Load from database
var query2 = query.Where(p => p.FirstName == "Munir" && p.LastName == "Husseini");
var query3 = query.Where("FirstName=Munir;LastName=Husseini");

This article is split into 5 posts.

1. Introduction
   • General Approach
   • Generation Model
2. Templating Engines
   • T4
   • StringTemplate
   • DotLiquid
3. Code Generation From Source
   • CodeDom
   • Roslyn
   • Mono
4. Code Generation From Model
   • Emit
   • Expression Trees
   • Mono Cecil
5. Loading
   • Dependency Injection
   • Managed Extensibility Framework
   • Reflection

In-Memory Run-Time Code Generation With .NET – Part 3

Code Generation From Source

Once source code is generated, it needs to get compiled into MSIL.

CodeDom

The .NET framework provides compilers for C#, VB and JScript. These are the classes Microsoft.CSharp.CSharpCodeProvider, Microsoft.VisualBasic.VBCodeProvider and Microsoft.JScript.JScriptCodeProvider, repectivly. These classes are thin wrappers around the native compilers csc.exe, vbc.exe and jsc.exe. Since theses compilers operate on files on the file system, the wrapper classes need to save the source files into temporary files to be able to pass them to the compilers. Likewise, the compiled assemblies are (temporarily) saved to disk and then loaded by the wrappers. Because of that, true in-memory assembly generation is not possible with these classes. Also, these classes need full-trust permissions.

public static Assembly Compile(string[] sources, bool isDebug, string tempDir, params AssemblyName[] referencedAssemblies)
{
    var codeProvider = new CSharpCodeProvider(new Dictionary<string, string> {{"CompilerVersion", "v4.0"}});

    var assemblyReferences = new[]
        {
            "System.dll",
            "System.Core.dll",
            "mscorlib.dll"
        }
        .Union(from ass in referencedAssemblies
                select new Uri(ass.CodeBase).LocalPath)
        .Distinct(StringComparer.OrdinalIgnoreCase)
        .ToArray();

    var parameters = new CompilerParameters
        {
            GenerateInMemory = true,
            GenerateExecutable = true,
            IncludeDebugInformation = isDebug,
            OutputAssembly = Path.Combine(tempDir, "generatedassembly.dll")
        };

    parameters.ReferencedAssemblies.AddRange(assemblyReferences);

    if (tempDir.Length > 0) parameters.TempFiles = new TempFileCollection(tempDir);
    parameters.TempFiles.KeepFiles = isDebug;

    var compilerResults = codeProvider.CompileAssemblyFromSource(parameters, sources);

    if (compilerResults.Errors.HasErrors)
    {
        var message = string.Join("\r\n", compilerResults.Errors);
        throw new ApplicationException(message);
    }

    return compilerResults.CompiledAssembly;
}

The compiler option can be set with the class CompilerParameters. Among others, there ist a property called GenerateInMemory. If this is set to true, the compiled assemblies get deleted after being loaded into memory. Using this setting, an in-memory generation can be mimiked. On the other hand, assemblies that are loaded into memory but deleted from disk cannot be referenced by newer compilations. So if for instance different assenmblies must be generated that reference each other, the property GenerateInMemory must be set to false.

Roslyn

“Project Roslyn” is Microsoft’s new approach towards purely managed C#and VB compilers. In contrast to CodeDom, Roslyn does generate assemblies in memory without having to read or wrtie files to disk. Currently being in CTP status, not all functionality is implemented, yet. For example, it cannot reference assemblies that are not located on the files sysetm. So currently, if multiple assemblies that reference each other should be generated, true in-memory generation is not possible, either.

public static Assembly Compile(string[] sources, bool isDebug, string tempDir, params AssemblyName[] referencedAssemblies)
{
    var assemblyFileName = tempDir + "gen" + Guid.NewGuid().ToString().Replace("-", "") + ".dll";
    var assemblyPath = Path.GetFullPath(assemblyFileName);

    var compilation = Compilation.Create(assemblyFileName,
                                            new CompilationOptions(OutputKind.DynamicallyLinkedLibrary))
                                    .AddSyntaxTrees(from source in sources
                                                    select SyntaxTree.ParseCompilationUnit(source))
                                    .AddReferences(from ass in referencedAssemblies
                                                select new AssemblyFileReference(new Uri(ass.CodeBase).LocalPath))
				                              //select new AssemblyObjectReference(Assembly.Load(ass)))
                                    .AddReferences(from ass in new[]
                                                {
                                                    "System",
                                                    "System.Core",
                                                    "mscorlib"
                                                }
                                                select new AssemblyNameReference(ass));

    EmitResult emitResult;

    //using (var stream = new MemoryStream())
    using (var stream = new FileStream(assemblyPath, FileMode.Create, FileAccess.Write))
    {
        emitResult = compilation.Emit(stream);
    }

    if (!emitResult.Success)
    {
        var message = string.Join("\r\n", emitResult.Diagnostics);
        throw new ApplicationException(message);
    }

    return Assembly.LoadFile(assemblyPath);
}

In the code above, there are two commented-out lines. The first on uses the type AssemblyFileReference which is currently not implemented. This type would allow referencing assemblies from memory. The second commented-out line regards where the complied asssembly is written to. Once in-memory references are implemented, complied assemblies can be written into memory and referenced by other assemblies to be generated.

This article is split into 5 posts.

1. Introduction
   • General Approach
   • Generation Model
2. Templating Engines
   • T4
   • StringTemplate
   • DotLiquid
3. Code Generation From Source
   • CodeDom
   • Roslyn
   • Mono
4. Code Generation From Model
   • Emit
   • Expression Trees
   • Mono Cecil
5. Loading
   • Dependency Injection
   • Managed Extensibility Framework
   • Reflection

In-Memory Run-Time Code Generation With .NET – Part 2

Templating Engines

Templating engines allow a very flexible and quite easy, because human-readable definition of the code to be generated. Following, three templating engines are introduced. The main differences between these engines in the context of this article is the level of separation between the WHAT and the HOW.

While T4 or the ASP.NET view engines compile into C#, which bascially allows doing anything C# is capable of inside of the view, StringTemplate brings its own compiler that enforces a strict View-Model-separation. Only read-only on the model or text output operations are allowed inside of the view. DotLiquid does the same, and only processes known and secure model types.

T4

The most common templating engine in the .NET world is probably the T4 Text Template. Since Visual Studio 2010, it comes in two flavors: regular and preprocessed. regular text templates are processed within Visual Studio itself (or a templating host created at runtime and loaded with the template). preprocessed text templates are transformed into C# classes that can be called at runtime for processing. Regular T4 templates need some infrastructure to run, which in turn needs refrences to Visual Studio assemblies. Also, a lot of that infrastructure is practically undocumented, which makes hosting and processing regular T4 templates in custom code not the easiest thing to do. Becasue of that, only preprocessed templates are discussed in this article. Please read my other article about some neet stuff you can do with hosted template processing.

Consider a simple T4 template that is saved in a file called ViewModelTemplate.tt. Visual Studio generates a class named ViewModelTemplate. This generated class has a method called TransformText which renders the template into a string. The generated class ViewModelTemplate can be extended by adding a partial class with the same name. Here, a partial class is added that contains a property to store the generation model:

partial class ViewModelTemplate
{
	public TypeGenerationModel Model { get; set; }
}

Inside the T4 template, the property Model can be accessed for information about the type to generate. The template generates three types of properties: properties that only wrap access to the underlying data model, properties that return other data view models and properties that return lists of data view models. The T4 template may look like this:

<#@ template language="C#" #>
<#@ import namespace="System.Linq" #>
using System.Collections.Generic;
using System.Linq;

namespace <#= Model.Namespace #>
{
  public class <#= Model.Name #> : Demo.ViewModelBase<<#= Model.ModelTypeName #>>
  {
<#foreach(var property in Model.Properties.Where(p => !p.IsViewModel))
  {#>
    public <#= property.TypeName #> <#= property.Name #>
    {
<#  if(property.CanGet) 
    {#> 
      get 
      { 
        return Model.<#= property.Name #>; 
      } 
<#  }        
    if(property.CanSet) 
    {#> 
      set 
      { 
        if(Model.<#= property.Name #> == value) return;
        Model.<#= property.Name #> = value; 
        OnPropertyChanged("<#= property.Name #>");
      } 
<#  }#>      
    }
<#} 
  
  foreach(var property in Model.Properties.Where(p => p.IsViewModel && !p.IsList))
  {#>
    private <#= property.TypeName #> _<#= property.Name #>;
    public <#= property.TypeName #> <#= property.Name #>
    {
<#  if(property.CanGet) 
    {#> 
      get 
      { 
        if(_<#= property.Name #> == null) _<#= property.Name #> = new <#= property.TypeName #>(Model.<#= property.Name #>);
        return _<#= property.Name #>; 
      } 
<#  }        
    if(property.CanSet) 
    {#> 
      set 
      { 
        if(_<#= property.Name #> == value) return;
        _<#= property.Name #> = value; 
        Model.<#= property.Name #> = value.Model; 
        OnPropertyChanged("<#= property.Name #>");
      } 
<#  }#>      
    }
<#} 
  
  foreach(var property in Model.Properties.Where(p => p.IsViewModel && p.IsList && p.CanGet))
  {#>
    private List<<#= property.TypeName #>> _<#= property.Name #>;
    public List<<#= property.TypeName #>> <#= property.Name #>
    {
      get 
      { 
        if(_<#= property.Name #> == null) _<#= property.Name #> = new List<<#= property.TypeName #>>(Model.<#= property.Name #>.Select(m => new <#= property.TypeName #>(m)));
        return _<#= property.Name #>; 
      } 
    }
<#}#>
    public <#= Model.Name #>(<#= Model.ModelTypeName #> model) : base(model)
    {
    }
  }
}

Executing a T4 template is very easy since the template itself is already a .NET class.

public class CodeGenerator
{
    public static string Generate(TypeGenerationModel model)
    {
        var template = new ViewModelTemplate { Model = model };
        return template.TransformText();
    }
}

The main program can call the above class now:

static void Main(string[] args)
{
    var generationModels = CreateGenerationModels(typeof(Person), "Generated");
    var sources = generationModels.Select(T4.CodeGenerator.Generate);

    // ... rest of the work here ...
}

StringTemplate

StringTemplate is a templating engine created by the ANTLR team. It is available for Java, Pyhton and the .NET framework. There are many differences between T4 and StringTemplate, but the main difference that is of matter in this case is that of the strict model-view separation. This means that inside StringTemplate templates, invoking methods or operators is not allowed. Although StringTemplate supports if-conditions, these conditions only match boolean values and the existance of the expression (comparable to JavaScript’s undefined).

To be able to use StringTemplate, the generation model from above must be extended by properties that replace the property filters in the T4 template. So instead of filtering the properties inside the template, the filtering must be applied inside the generation model:

/// <summary>
/// Defines the type that will be generated.
/// </summary>
public class TypeGenerationModel
{
    // ... properties omitted ...

    /// <summary>
    /// Gets the regular properties to generate.
    /// </summary>
    public IEnumerable<PropertyGenerationModel> Properties { get; private set; }

    /// <summary>
    /// Gets the properties to generate that return view model types.
    /// </summary>
    public IEnumerable<PropertyGenerationModel> ViewModelProperties { get; private set; }

    /// <summary>
    /// Gets the properties to generate that return lists of view model types.
    /// </summary>
    public IEnumerable<PropertyGenerationModel> ViewModelListProperties { get; private set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="TypeGenerationModel"/> class.
    /// </summary>
    /// <param name="modelType">Type of the data model for which to generate the view model.</param>
    /// <param name="namespace">The @namespace of the generated view model.</param>
    public TypeGenerationModel(Type modelType, string @namespace)
    {
        Namespace = @namespace;
        ModelType = modelType;
        ModelTypeName = modelType.GetTypeName();
        Name = modelType.Name;

        var allProperties = ModelType
            .GetProperties(BindingFlags.Instance | BindingFlags.Public)
            .Select(p => new PropertyGenerationModel(this, p))
            .Where(p => p.CanGet || p.CanSet)
            .ToList();

        Properties = allProperties.Where(p => !p.IsViewModel);

        ViewModelProperties = allProperties.Where(p => p.IsViewModel && !p.IsList);

        ViewModelListProperties = allProperties.Where(p => p.IsViewModel && p.IsList && p.CanGet)
    }
}

The T4 template from above can quite easily be transformed into StringTemplate syntax:

using System.Collections.Generic;
using System.Linq;

namespace $Model.Namespace$
{
   public class $Model.Name$ : Demo.ViewModelBase<$Model.ModelTypeName$>
   {
   $Model.Properties:
   {
      property|
      public $property.TypeName$ $property.Name$
      {      
      $if(property.CanGet)$
         get 
         { 
            return Model.$property.Name$; 
         \} 
      $endif$
               
      $if(property.CanSet)$
         set 
         { 
            if(Model.$property.Name$ == value) return;
            Model.$property.Name$ = value; 
            OnPropertyChanged("$property.Name$");
         \} 
      $endif$         
      \}
   }$ 
   
   $Model.ViewModelProperties:
   {
      property|
      private $property.TypeName$ _$property.Name$;
      public $property.TypeName$ $property.Name$
      {
      $if(property.CanGet)$
         get 
         { 
            if(_$property.Name$ == null) _$property.Name$ = new $property.TypeName$(Model.$property.Name$);
            return _$property.Name$; 
         \} 
      $endif$
            
      $if(property.CanSet)$
         set 
         { 
            if(_$property.Name$ == value) return;
            _$property.Name$ = value; 
            Model.$property.Name$ = value.Model; 
            OnPropertyChanged("$property.Name$");
         \} 
      $endif$
      \}
   }$
   
   $Model.ViewModelListProperties:
   {
      property|
      private List<$property.TypeName$> _$property.Name$;
      public List<$property.TypeName$> $property.Name$
      {
         get 
         { 
            if(_$property.Name$ == null) _$property.Name$ = new List<$property.TypeName$>(Model.$property.Name$.Select(m => new $property.TypeName$(m)));
            return _$property.Name$; 
         \} 
      \}
   }$
      public $Model.Name$($Model.ModelTypeName$ model) : base(model)
      {
      }
   }
}

A StringTemplate template must be loaded into the StringTemplate engine. While StringTemplate offers functionality for more complex scenarios (e.g. dynamically loading sub templates), here only a basic set of this functionality is required:

public static string Generate(TypeGenerationModel model)
{
    var template = new Antlr4.StringTemplate.Template(Resource.STViewModelTemplate, '$', '$');
    template.Add("Model", model);

    return template.Render();
}

The main program can call the above class now:

static void Main(string[] args)
{
    var generationModels = CreateGenerationModels(typeof(Person), "Generated");
    var sources = generationModels.Select(ST.CodeGenerator.Generate);

    // ... rest of the work here ...
}

DotLiquid

DotLiquid is the new kid on the block. It is a port of Ruby’s Liquid Markup. Like StringTemplate, DotLiquid does not allow calling any methods from inside the view. On the other hand, it does allow comparison operators. The main difference to other templating engines is that only certain types of models get processed. This means that only basic types like strings or integers or types that derive from DotLiquid.Drop are allowed as models. DotLiquid’s syntax is simmilar to that of StringTemplate. The above template looks for DotLiquid like this:

using System.Collections.Generic;
using System.Linq;

namespace {{ Model.namespace }}
{
	public class {{ Model.name }}: Demo.ViewModelBase<{{ Model.model_Type_name }}>
	{
	{% for property in Model.Properties -%}
		public {{ property.type_name }} {{ property.name }}
		{
		{% if property.CanGet %}
			get 
			{ 
				return Model.{{ property.name }}; 
			} 
		{% endif %}
		{% if property.CanSet %}
			set 
			{ 
				if(Model.{{ property.name }} == value) return;
				Model.{{ property.name }} = value; 
				OnPropertyChanged("{{ property.name }}");
			} 
		{% endif %}	
		}
	{% endfor -%}
	
	{% for property in Model.view_model_properties -%}
		private {{ property.type_name }} _{{ property.name }};
		public {{ property.type_name }} {{ property.name }}
		{
		{% if property.CanGet %}
			get 
			{ 
				if(_{{ property.name }} == null) _{{ property.name }} = new {{ property.type_name }}(Model.{{ property.name }});
				return _{{ property.name}}; 
			} 
		{% endif %}
		{% if property.CanSet %}
			set 
			{ 
				if(_{{ property.name }} == value) return;
				_{{ property.name }} = value; 
				Model.{{ property.name }} = value.Model; 
				OnPropertyChanged("{{ property.name }}");
			} 
		{% endif %}
		}
	{% endfor -%}
	
	{% for property in Model.view_model_list_properties -%}
		private List<{{ property.type_name }}> _{{ property.name }};
		public List<{{ property.type_name }}> {{ property.name }}
		{
			get 
			{ 
				if(_{{ property.name }} == null) _{{ property.name }} = new List<{{ property.type_name }}>(Model.{{ property.name }}.Select(m => new {{ property.type_name }}(m)));
				return _{{ property.name }}; 
			} 
		}
	{% endfor -%}

		public {{ Model.name }}({{ Model.model_Type_name }} model) : base(model)
		{
		}
	}
}

The code needed to execute thee template loosk like this:

public class CodeGenerator
{
    public static string Generate(TypeGenerationModel model)
    {
        var template = DotLiquid.Template.Parse(Resource.DotLiquid_ViewModelTemplate);
        return template.Render(DotLiquid.Hash.FromAnonymousObject(new { Model = model }));
    }
}

This article is split into 5 posts.

1. Introduction
   • General Approach
   • Generation Model
2. Templating Engines
   • T4
   • StringTemplate
   • DotLiquid
3. Code Generation From Source
   • CodeDom
   • Roslyn
   • Mono
4. Code Generation From Model
   • Emit
   • Expression Trees
   • Mono Cecil
5. Loading
   • Dependency Injection
   • Managed Extensibility Framework
   • Reflection

In-Memory Run-Time Code Generation With .NET – Part 1

Introduction

In this series, I will show how to generate, compile and load .NET code at runtime using different techniques. Since this series will be somewhat lengthy, I’ll need some time to complete it. In the meanwhile, preliminary result will be published.

For illustration purposes, a simple example is used throughout this article. The scenario of the example is a MVVM application with 100s of models. Many of these models need a corresponding view model that wrapes the models properties into e.g. notifiying properties.
A possible approach to build such an application could be to build a view model base class, that “magically” wrapps around a model class and instantly generates a matching view model with all the models properties. This article demonstrates how to do that.

General Approach

As often in software developement, the recommended approach here is to separate WHAT will be generated from HOW it is generated, i.e. to define a model and a view.
The objective here is to generate .NET assemblies that contain classes that can dynamically be instanciated and used.

A data model that describes the classes contained in such an assembly will be referred to as the generation model.
The object that will generate these classes (using either of the techniques discussed in this article) will be referred to as the generation view.

The models, view models and views of the example application will be referred to as the runtime models, view models and views.

The idea how to create a “magic” view model is the following: at runtime…

1. …the runtime model is analyzed via reflection to determine the properties of the runtime view model to generate. These properties and other configuration parameters make up the generation model
2. Call a generation view with the generation model as a parameter to generate an assembly. The generation view could use one of the techniques discussed in this article:
   1. Generate source code and compile it into MSIL using templating engines or pure string-concatenation, generate source code of any .NET langauge (in this article, C# is used) and compile it into a .NET assembly.
   2. Directly generate MSIL code. The .NET framework allowes to emit MSIL opcodes into dynamic assemblies. This can be done directly using Reflection or by abstraction using Linq Expression Trees.
3. Load the generated types. Once the view model assemblies are generated, the appropriate types must be loaded and instanciated with model instances as constructor parameters.

In short, these steps are:

Data Model → Generation Model → (Source Code →) Dynamic Assembly → Loading.

Generation Model

The generation model will define types and properties to be generated. Therefore, there are two classes that make up the generation model: TypeGenerationModel and PropertyGenerationModel.

/// <summary> 
/// Defines the type that will be generated.
/// </summary>
public class TypeGenerationModel
{
    /// <summary>
    /// Gets the namespace of the generated view model.
    /// </summary>
public string Namespace { get; private set; }

    /// <summary>
    /// Gets the name of the generated view model for which to generate the view model.
    /// </summary>
    public string Name { get; private set; }

    /// <summary>
    /// Gets the type of the data model.
    /// </summary>
    public Type ModelType { get; private set; }

    /// <summary>
    /// Gets the type name of the data model.
    /// </summary>
    public string ModelTypeName { get; private set; }

    /// <summary>
    /// Gets the properties to generate.
    /// </summary>
    public IEnumerable<propertygenerationmodel> Properties { get; private set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="TypeGenerationModel"/> class.
    /// </summary>
    /// <param name="modelType">Type of the data model for which to generate the view model.</param>
    /// <param name="namespace">The @namespace of the generated view model.</param>
    public TypeGenerationModel(Type modelType, string @namespace)
    {
        Namespace = @namespace;
        ModelType = modelType;
        ModelTypeName = modelType.GetTypeName();
        Name = modelType.Name;
        Properties = ModelType
                    .GetProperties(BindingFlags.Instance | BindingFlags.Public)
                    .Select(p => new PropertyGenerationModel(this, p))
                    .Where(p => p.CanGet || p.CanSet)
                    .ToList();
    }
}

/// <summary>
/// Defines a view model's property that will be generated.
/// </summary>
public class PropertyGenerationModel
{

    /// <summary>
    /// Gets the full name of the property type.
    /// </summary>
    public string TypeName { get; private set; }

    /// <summary>
    /// Gets the name of the property.
    /// </summary>
    public string Name { get; private set; }

    /// <summary>
    /// Gets a value indicating whether this property leads to another generation model.
    /// </summary>
    public bool IsGenerationModel { get; private set; }

    /// <summary>
    /// Gets a value indicating whether this instance is a list.
    /// </summary>
    public bool IsList { get; private set; }

    /// <summary>
    /// Gets or sets a value indicating whether this property has a setter.
    /// </summary>
    public bool CanSet { get; private set; }

    /// <summary>
    /// Gets or sets a value indicating whether this property has a getter.
    /// </summary>
    public bool CanGet { get; private set; }

    /// <summary>
    /// Gets or sets the type of the element returned by the property.
    /// </summary>
    public Type ElementType { get; private set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="PropertyGenerationModel"/> class.
    /// </summary>
    /// <param name="parent">The type model that contains the property.</param>
    /// <param name="property">The property info to generate from.</param>
    public PropertyGenerationModel(TypeGenerationModel parent, PropertyInfo property)
    {
        // Let the property of the view model as the property of the model.
        Name = property.Name;

        var getter = property.GetGetMethod();
        CanGet = getter != null && getter.IsPublic;

        var setter = property.GetSetMethod();
        CanSet = setter != null && setter.IsPublic;

        // In this small demo, only support Lists to contain child view models.
        IsList = property.PropertyType.IsGenericType && property.PropertyType.GetGenericTypeDefinition() == typeof(List<>);

        // If this is a list, we'll need type of the elements,
        // not of the property itseld.
        var itemType = IsList
                    ? property.PropertyType.GetGenericArguments().First()
                    : property.PropertyType;

        // Assume that all models are in one assembly and can be
        // identified by just being in that assembly.
        IsViewModel = itemType.Assembly == property.DeclaringType.Assembly;

        ElementType = IsGenerationModel ? itemType : property.PropertyType;

        // Get the type name in a format that can be placed into code
        // (i.e. MyType<T> rather than MyType`1 for generic types)
        TypeName = IsGenerationModel
                    ? parent.Namespace + "." + itemType.Name
                    : property.PropertyType.GetTypeName();
    }
}

Since the data model may reference other data models, the types of these data models need to be collected and added to the generation model as well. The following code creates a generation model from the root type and traverses the proeprties until all data model types are collected:

internal class Program
{
    private static void Main(string[] args)
    {
        var generationModels = CreateGenerationModels(typeof(Person), "Generated");

        // ... rest of the work here ...
    }

    public static IEnumerable<TypeGenerationModel> CreateGenerationModels(Type type, string @namespace, HashSet<Type> processedTypes)
    {
        if (processedTypes == null) processedTypes = new HashSet<Type>();
        processedTypes.Add(type);

        var model = new TypeGenerationModel(type, @namespace);
        yield return model;

        var models = from p in model.Properties
                     where p.IsGenerationModel && !processedTypes.Contains(p.ElementType)
                     from m in CreateGenerationModels(p.ElementType, @namespace, processedTypes)
                     select m;


        foreach (var referencedModel in models)
        {
            yield return referencedModel;
        }
    }
}

The runtime view models (the types that will be generated) will inherit a common base class called ViewModelBase. This base class contains common code needed to do some of the “magic”. For instance, if one of the models properties isn’t a simple type, but another model a collection of other models, then the properties values must be wrapped in a view model as well. The base class contains the methods needed to do this kind of stuff.

The next sections will describe generation views that use this generation model to generate the described types.

public Classname (param list) [throws exception...]

public: Indicates the constructor is available to other classes.

Classname: MUST be the same as the class containing the constructor.

Construction Work (Photo credit: gullevek)

(param list): Can be empty or accept parameters.

• A class can have multiple constructors as long as each has a unique list of parameters.
• One constructor in a class can contain NO params.
• No two constructors in a class can have the same number of params because the compiler will not be able to tell them apart even if the types in the param list are different.

If you create a class without declaring any constructors, Java creates a default constructor with NO parameters and NO statements. It does not do anything other than allow your class to be instantiated.

If you create a class with constructors of your own making, but do NOT create a zero parameter constructor and then try to use the class with a zero paremeter call, you will get a compile error.

[Source: See jaiofd3d p. 246f]

This lady was know as the Queen of software then. Grace Hopper (1906 – 1992) once walked around with a quote: “Nobody believed that I had a running compiler and nobody would touch it. They told me computers could only do arithmetic.” As always with all great people, no one listened.

Visit this link to watch the video: http://www.myvidster.com/video/425708/Grace_Hopper_on_Letterman

In March of last year is when I first wrote about LLVM/Clang being toyed with for Debian. Nearly one year ago it was found that LLVM’s C/C++ compiler was actually building much of the Debian archive while providing better compiler warnings/errors than GCC. The results were more positive than originally anticipated with less than 10% of the packages having problems being built with the alternative open-source compiler.

The Debian developers working on Clang support aren’t doing so because they want to deprecate GCC but rather they want to provide their users with greater options and Clang does have some rather nice error/warnings (although GCC’s reporting has improved recently too), static analysis tools, and other features to benefit developers. Pushing the packages through multiple compilers also generally ensures better code quality too.

In July was another LLVM/Clang update for Debian. It was reiterated the Clang work wasn’t to wage some GPL vs. BSD war in the Debian camp but for technical reasons. With the code at that time, just over 10% of the Debian archive was having problems being built under the latest LLVM/Clang over GCC.

At FOSDEM 2013 last weekend was another presentation by Sylvestre Ledru on building Debian with LLVM/Clang. Embedded below is the WebM video of this presentation. For those just wanting build results and other technical details, visit clang.debian.net.

http://video.fosdem.org/2013/crossdistro/Make_Debian_compiler_agnostic___Building_Debian_with_LLVM_Clang.webm

FreeBSD (Photo credit: Wikipedia)

FreeBSD developers are working on enabling support for the C11 and C++11 programming language standards within their operating system.

Shared during the BSD track at FOSDEM 2013 by David Chisnall was an update on the FreeBSD’s support for C11 and C++11, the latest revisions to the C and C++ programming languages. While most of the work resides within the compilers supporting these latest standards, FreeBSD is looking to take proper use of the new features and functionality. 

During the quick overview for those not up to speed on the latest ISO standards of these languages, C11 introduces features like alignment specifies, multi-threading support as part of the standard, a memory model, various cosmetic improvements to the code, static assertions, and unicode support. C++11 introduces better locale support, atomics/threads/thread-local storage, smart pointers, tuples, and various language changes.

FreeBSD system header files have already been updated to reflect the C++11 changes and their preferred LLVM/Clang compiler already has begun its C11/C++11 support. FreeBSD is just shipping GCC 4.2.1 (the last GPLv2 FSF GCC compiler release) and are deprecating the compiler so they aren’t really concerned about the GCC support for C++11/C11. LLVM/Clang has most pieces in place while the rest should be dropped in soon. There’s also some support via libc++, LLVM’s own more permissively licensed C++ standard library.

In terms of the current status in broad terms for C++11 on FreeBSD, “most of C++11 works.” For the latest C revision on FreeBSD, “most of the interesting bits of C11 work.” However, there is still lots of missing pieces for C11/C++11 on FreeBSD that still need to be implemented. For more specific details about the current state in FreeBSD, see the FOSDEM PDF slides.

THE CI’S MOST FAMOUS CAR WASH OWNER

A recurring theme of Breaking Bad is getting out of difficult situations with science. Yet, you still probably wouldn’t expect to run into a character from the hit TV show on the campus of Argonne National Laboratory or University of Chicago. But if you happen to spot a man who looks just like protagonist Walter White’s former boss at his car wash job, no need for a double take — you’re not losing your mind. Marius Stan, a Computation Institute Senior Fellow and Argonne scientist studying computational chemistry and physics, provides the memorable eyebrows and Romanian curses for the role of Bogdan, a character who has appeared in a handful of episodes of the AMC drug-trade drama.

This week in the Chicago Tribune, reporter Ted Gregory profiled Stan and told the story of how he got involved with the show when he lived in Albuquerque, before moving to Chicago. Stan might humbly list “Breaking Bad, Bogdan” below a computational microscope and a book about modeling and simulation in materials science on his CI web page. But his colleague, CI fellow Andrew Siegel, said most people at Argonne find his moonlighting career “extremely cool.”

“Everybody finds it hilarious and great. In science, you’re so uncool, at least in this country, and the world of acting is so opposite of that. It’s a funny convergence of things.”

THE MATH INSIDE ELECTRICAL OUTLETS

Electrical power grids present a complex and challenging mathematical problem, with questions of how to efficiently produce, store and distribute the energy. To study the mathematics behind tomorrow’s power lines, Argonne National Laboratory and Computation Institute scientists formed the Multifaceted Mathematics for Complex Energy Systems Project (M2ACS), which recently received a $17.5 million grant from the Department of Energy. The project will bring together experts from several different areas of mathematical study, including optimization, dynamical systems, uncertainty quantification, random processes, data analysis, discrete mathematics and linear algebra to find new techniques needed to drive next-generation “smart” power grids and other technologies.

CI senior fellow Mihai Anitescu, who is leading the project, described their approach:

“To address these challenges, we will develop, analyze, and integrate predictive models of system behavior, new optimization-based sampling approaches, and new analysis methods and scalable algorithms that can exploit the rich mathematical structure,” said Mihai Anitescu, Argonne computational mathematician and director of M2ACS. “Our aim is to create the mathematical underpinnings of the future engineering analysis tools that will ensure the efficiency and resiliency of critical energy systems planning and operations.”

OTHER NEWS IN COMPUTATIONAL SCIENCE

Many of the world’s largest cities grew to their current size with a minimum of urban planning, chaotic growth outpacing attempts to create an orderly, efficient layout. But as Elizabeth Leake writes at Datanami, our current age of open data and powerful computation creates new opportunities for effective urban planning in cities new and old. Leake reports from last year’s International Conference on eScience in Chicago, describing the panel led by the CI’s Charlie Catlett on “Designing and operating cities using open data” (also covered on ScaleOut) and the Australia Urban Research Infrastructure Network in Melbourne, Australia. The article includes examples of how the two cities use datasets to track obesity, walkability and food deserts in their respective neighborhoods.

Computation is also making its way into journalism, which is increasingly excited by the promise of data-driven investigations and interactive graphics. For those who couldn’t attend, the 2013 Computation + Journalism conference, held last week at Georgia Tech, was helpfully recapped by Reporters’ Lab.

A new journal on Big Data called, helpfully, Big Data, premiered with its first issue in January with articles on visualization, life sciences data and “thought crime.”

Watch Episode 19  live at Compiling.tv today!

New episodes will be live each week on Thursdays.

Become a Fan to have free access to Exclusive Content Videos, Fun Facts and lots more!

Hurry! Only 8 days left to the fundraising campaign. Visit the Indiegogo Campaign page today to make your donation and claim your perk.