Web applications are nice. They’re useful, they’re cross-platform, users need no installation, no upgrades, no maintenance, not even the computing or storage power to which they are used. As weird as it may sound, I’ve even seen announcements for web applications supposed to run your games on distant high-end computers so that you can actually play on low-end computers. Go web application!

Of course, there are a few downsides to web applications. Firstly, they require a web connexion. Secondly, they are largely composed of plumbing. Finally, ensuring their security is a constant fight.

How many pipes do you need?

If you have ever developed a web application, you know what I mean by plumbing: just writing an online TODO list — the equivalent of maybe twenty minutes of work in Visual Basic, in Python, Java or Objective-C — requires mixing an insane amount of languages, to define your user interface (HTML, JavaScript, CSS), to define your storage (DDL, DML, DCL, plus possibly an ORM language), to get your server and your client to communicate (XML, more JavaScript, PHP or one of its competitors, as well as some HTTP and a little MIME configuration), to launch your application (whichever configuration languages are used by your server). Of course, if your application is a bit more complex and requires something like compatibility with smartphones, or like distant storage, or distributed computing, or backups, or modularity (in the world of the web, it’s called “web services”)… well, you will probably require a few additional languages.

All of this is just plumbing. Only once you have written it can you concentrate on the core of the application. And once the application is written, the pain is just starting, because chances are that your application can be attacked by hijacking the link between your user interface and the core (cross-site scripting) or between the core and the storage (SQL injection) or by keeping the user interface and replacing the application core (man-in-the-middle attacks) or by replacing the user interface by a malicious client or by taking the place of a currently connected user to steal some of its credentials (rebinding) or by taking advantage of low-level bugs (buffer over/underflows), etc.

None of this is a show-stopper, of course — just take a look at the web and you will see thousands of web applications. Just like the complexity of Software Development Kits in the early days of Windows, MacOS or X didn’t stop adventurous hackers from developing desktop applications. But of course, if twenty-five years of desktop application development have taught us one thing, it is that the life of developers can be made easier. Nowadays, a few generations of SDKs later, Windows developers have .Net, C# and Visual Studio, Macintosh developers have Cocoa, Objective-C and XCode, while X-based developers have the libraries of Gnome/KDE, Python and a variety of programming environments. The growing popularity (and libraries) of Haskell, F#, OCaml, Scala and other functional programming languages could mean that one of the next generations of SDKs will increase safety and security.

The web hasn’t quite reached that stage yet. Even the state-of-the-art in web frameworks only provides features slightly more advanced than early Windows/Mac/X SDKs: low-level bindings for low-level mechanisms, designed to ensure low-level properties. Or, rephrased differently, in the current state of web development, GMail, Google Maps or Facebook are still considered complicated applications, although they are conceptually quite simple and should therefore be equally simple to implement.

We can do better. How? By removing the need for plumbing. By providing automated mechanisms for ensuring high-level security properties. By providing language support for common patterns.

Enter OPA

Let me introduce OPA. OPA, or One Pot Application, is a complete development platform for web applications and web services. Development in OPA requires no plumbing. Applications developed with OPA are automatically checked for safety and security before they are executed. Applications developed with OPA are automatically (and provably) immune to cross-site scripting, to SQL injections and to most existing forms of attacks. And OPA provides language support for storage, communication between client and server (Ajax and Comet), concurrency, distribution, mobility, etc.

With OPA, we intend to skip several generations of SDKs and provide right now a high-level and modern programming platform. OPA has been 6 years in the making: 4 years of sketches, mockups and prototypes as part of academic research projects and 2 years of actual implementation at MLstate. A few days ago, OPA has officially entered demonstrable status. Not quite ready for prime time, but definitely usable for development. Do you want to write an online note-taking application? That’s about 20 lines of code, from scratch. A minimal chat? About 30 lines. A multi-channel, distributed chat? About 80 lines. A minesweeper? About 100. We’re using it to develop utilities, content management systems, tools for administrations and games.

Pre-alpha builds of OPA have been distributed to selected partners. A public version will be made available within a few weeks, as well as commercial applications developed with OPA. In the meantime, we are busy improving the syntax, completing the standard library, making error messages intelligible, fixing the bugs and extending the range of safety and security checks.

Interested? Well, few details are public at this time. However, you can take a look at a video recorded during ICFP presenting OPA and MLstate.

Stay tuned.

Like in Ruby, there is no explicit call in a function to return values, the last expression of a function is evaluated and used as its return value.

Consider the following 2 functions:

(defn hello-with-println [name]
    (println "Hello," name))
    

(defn hello-with-str [name]
    (str "Hello, " name))

The first function when evaluated on the REPL (read-eval-print-loop), which is an interactive programming environment, returns the following:

user=> (hello-with-println "punk")
Hello, punk
nil

The second function returns:

user=> (hello-with-str "munk")
"Hello, munk"

At first glance, these two functions seem to be doing almost the same thing – takes in a string input, which is a name, and outputs a string that says Hello to that name. However, hello-with-println returns nil while hello-with-str returns the evaluation of the str function.

This can be proven further with special variables in the REPL. When working within the REPL, the results of evaluating the three most recent expressions are stored in the special variables *1, *2 and *3. If I call hello-with-println twice with the inputs “munk” and “punk” and use the special variables *1 and *2 to get the results of those expressions, I get the following:

user=> (str *1 " and " *2)
" and "

If I do the same with hello-with-str, I get the following output:

user=> (str *1 " and " *2)
"Hello, punk and Hello, munk"

When hello-with-println was used, there were no return values so the special variables were empty unlike in hello-with-str.

I seem to be liking Clojure …

Let me backup a bit, in the last couple of weeks I have been debating between picking up Scala or Clojure (don’t get me wrong Ruby is still my favorite).

I always wanted to pick up a functional programming language so I dabbled a bit with Erlang and Haskell, liked Haskell a lot but without much practice it kind of died (sad times)  and Scala seems too much like Java, yeah I know it seems to have a bigger crowd than Clojure and there are a lot of big names behind it.

Maybe that’s exactly why I choose Clojure (since its the underdog), or cause it is different enough from Java or simply cause it has a better syntax and seems more elegant (apparently Clojure has better integration with Java, don’t quote me on it), anyways I decided to learn Clojure.

Peepcode has a nice screen-cast to get you started off on Clojure. If you are on the Mac there is a nice bundle for TextMate and anywhere else Netbeans with the enclojure plugin seems to be the best.

On a side note it seems more and more that Netbeans has the latest and greatest plugins for everything, then comes IntelliJ and finally eclipse, what’s going on with eclipse ? has it reached its peak and now it will start dropping off ? but on the flip side there seems to be more and more apps built on top of the Eclipse RCP like Xmind, so is Eclipse no longer going to be the leader of the IDE and just become a platform for building RCPs. This of course depends  on what Oracle is going to do with NetBeans, I really hope they give the same amount of love to NetBeans as Sun did.

Ok getting back to Clojure, don’t get your panties in a bunch when you see all those parenthesis, it is just the layout that is shocking, indent it well and it is no more than what you are used to.

Here’s an example
(defn fac
"Returns the factorial of n, which must be a positive integer."
[n]
(if (= n 1)
1
(* n (fac (- n 1)))
)
Is same as
(defn fac [n] (if (= n 1) 1 (* n (fac (- n 1)))))
But the first one is a lot more easier on the eyes (even brain?) than the second one. Most examples that you see look like the second one and it frightens people, don’t let that stop you take my word and go for it.

Clojure seems to be very easy to pick up, things seem very intuitive, like the other day I was wondering, how to return a default value from a map if the key is not found and there is was right there in the api.
(map key default-value)
So simple! I was easily able to extend the examples that came with the peepcode screencast. Anyways I have started on this path, let’s see where it goes.

Update 2009/12/03

- Looked at the Clojure source code, looks squeaky clean, I applied to become a member so that I can expand on the test coverage, hopefully they will accept me.

-It is (load-file “hello.clj”) and not load-file “hello.clj” , I keep forgetting that and after a few mins I realize it.

Functional programming has always been a nightmare for me and it started off in college with a class called 15-212: Principles of Programming, which was taught in SML. Almost everything that was taught in that class went over my head and it’s a miracle that I managed to get a B for it. When I graduated, I swore I never wanted to have anything to do with functional programming ever again, until a few of my coworkers started talking about languages like F#, Scala and Clojure last year. I have tried to avoid it for as long as I can, but it has proven inevitable.

In starting to read and play with Clojure, I have had to learn a lot of basic concepts that I might or might not have known before. I will attempt to highlight those concepts as I stumble upon them and illustrate them with code snippets that I find or ones that I wrote myself. Pardon the juvenile approach, I find I remember things better if I write them down or explain it to someone.

Lexical Scoping – with lexical or static scope, a variable always refers to its top-level environment. You can always determine a value of a variable based on just reading the code, substituting variable bindings as you go. Of course most programming languages work this way, I just never knew there was a term for it.

Here’s some code (taken from this Clojure tutorial by R. Mark Volkmann) that illustrates lexical scoping and the usages of def, let and binding.

(def v 1) ; v is a global binding

(defn f1 []
  (println "f1: v =" v))

(defn f2 []
  (println "f2: before let v =" v)
  (let [v 2]
    (println "f2: in let, v =" v)
    (f1))
  (println "f2: after let v =" v))

(defn f3 []
  (println "f3: before binding v =" v)
  (binding [v 3]
    (println "f3: in binding, v =" v)
    (f1))
  (println "f3: after binding v =" v))

(defn f4 []
 (def v 4))

Calling f2 spits out the following output:

f2: before let v = 1
f2: in let, v = 2
f1: v = 1
f2: after let v = 1

The value of v is 2 only within the let form, and functions that are called within the body cannot see the local bindings of let.

Calling f3:

f3: before binding v = 1
f3: in binding, v = 3
f1: v = 3
f3: after binding v = 1

The binding macro is similar to let, but it creates thread-local values temporarily for existing global bindings. These new values are seen within the form and the functions that are called from within that form.

Calling f4: (println "after calling f4, v =" v):
after calling f4, v = 4

This actually changes the value of the global binding. Now calling f2 and f3 will have the following output:

f2: before let v = 4
f2: in let, v = 2
f1: v = 4
f2: after let v = 4
nil
f3: before binding v = 4
f3: in binding, v = 3
f1: v = 3
f3: after binding v = 4
nil

vLabs: Reactive Extensions for .NET (Rx)

http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx

 

About Rx

Rx is a library for composing asynchronous and event-based programs using observable collections.

The “A” in “AJAX” stand for asynchronous, and indeed modern Web-based and Cloud-based applications are fundamentally asynchronous. In fact, Silverlight bans all blocking networking and threading operations. Asynchronous programming in by no means restricted to Web and Cloud scenarios, however. Traditional desktop applications also have to maintain responsiveness in the face of long latency IO operations and other expensive background tasks.

Another common attribute of interactive applications, whether Web/Cloud or client-based, is that they are event-driven. The user interacts with the application via a GUI that receives event streams asynchronously from the mouse, keyboard, and other inputs.

Rx is a superset of the standard LINQ sequence operators that exposes asynchronous and event-based computations as push-based, observable  collections via the new .NET 4.0 interfaces IObservable<T> and IObserver<T>.  These are the mathematical dual of the familiar IEnumerable<T> and IEnumerator<T> interfaces for pull-based, enumerable collections in the .NET framework.

The IEnumerable<T> and IEnumerator<T> interfaces allow developers to create reusable abstractions to consume and transform values from a wide range of concrete enumerable collections such as arrays, lists, database tables, and XML documents. Similarly, Rx allows programmers to glue together complex event processing and asynchronous computations using LINQ queries over observable collections such as .NET events and APM-based computations, PFx concurrent Task<T>,  the Windows 7 Sensor and Location APIs, SQL StreamInsight temporal event streams , F# first-class events, and async workflows.

Play with Rx, stress it, evaluate it, and tell us what you think.

Wrote some “functions” in Function Programming (FP) the other day. Interesting, to say the least. It was for a class but it was really my first hands on experience with a functional language.

Although there is an interpreter for this language, it isn’t as developed as it could be and it definitely needs a debugger. Anyway, interesting first experience.

Now onto Haskell. Oh fun. At least there’s a huge amount of documentation and books on how to program in Haskell =) It’s always good to have resources. Will write more about it when I know what I’m doing!

As good as Processing is, it’s still java. I’ve slowly been learning clojure, and trying to use it with Processing, via clj-processing. While the clojure docs and the book are both good, I’m still in that flounder-around stage, trying out tiny experiments to understand how things work.

I wanted a function to make saw-step sequences: (1 2 3 2 1 2 3 2 …). You give it the low bound, the high bound, and the step-size. For example:

• (saw-step 1 4 1) gives (1 2 3 4 3 2 1 2 3 4 3 ...)
• (saw-step 0 15 5) gives (0 5 10 15 10 5 0 5 ...)
• (saw-step 0 5 2) gives (0 2 4 5 3 1 0 2 4 5 ...)

I’ve coded this kind of thing up in ruby, javascript, or java a dozen times, though it’s usually inside a loop, not returning a list. Something like this:

var min = 3;
var max = 7;
var step = 2;

var n = min;
while(shouldContinue()) {
    doSomethingWith(n);
    n += step;
    if (n+step > max || n-step < min) {
        step *= -1;  // turn around
    }
}

My first try in clojure took a similar tack. I never got it right, but it was something like this:

(defn saw-step
  [min max step-size]
    (let [step (ref step-size)]
      (iterate
       (fn [x]
         (if (or ( min (- x @step)))
           (dosync (alter step -)))
         (+ @step x))
         min)))

It’s a disaster — the parentheses don’t even match — but you get the gist. I was trying to cram the same imperative algorithm into clojure: keep some mutable state, add step to it, and when you reach the edges, mutate step to change direction. I kept getting bugs where it would go one step beyond the edge, or it would start working its way back from one edge, only to turn around again, and bounce against the edge forever.

I gave up and went to bed, but a few days later, I had better luck.

(defn saw-step
  [min max step]
     (cycle
      (into (vec (range min max step)) ; vec, so (into) adds to end
	    (for [x
		  (iterate #(- % step) max)
		  :while (> x min)] x))))

The first not-entirely-wrong step I made was to try breaking the list of numbers I wanted into two parts: the going-up numbers, and the coming-down numbers. (0 2 4 5 3 1) is just (0 2 4) + (5 3 1). The (range min max step) part gives you the going-ups, and the (for [x (iterate ...) stuff is the going-downs, as a list comprehension.

(One mistake I made was trying (range max min step) for the going-downs, which yields an empty list; another was using (iterate dec max), which never ends, because it keeps decrementing into the negatives. I found my way out with the list comprehension, but I bet there’s a better way.)

Once you have those two lists, you can use into to add each item from the second list to the first, giving you (0 2 4 5 3 1). That goes into cycle for a lazy, infinite sequence.

The solution’s not too bad: a saw-step is a cycle of the going-ups, followed by the going-downs. The code looks about that way.

(It occurred to me after that I could always use a default step of 1, and pipe the result through a scaling map. That would give me the bounds I wanted, with the extra benefit of evenly-spaced steps. Maybe I’ll remove step later.)

Alan Perlis said, “A language that doesn’t affect the way you think about programming, is not worth knowing.” He also said, “The only difference(!) between Shakespeare and you was the size of his idiom list – not the size of his vocabulary.”  Clojure’s changing my mind, a bit at a time.

There are a lot of people @Lab who know things like Erlang, Haskell, Python, R, etc., etc., etc.  I’m actually considering learning Lisp, but I need a project.  I “learned” php for a project over the summer, but I didn’t really like it since its so much like classic ASP (really messy).  Maybe I’ll learn F# instead…  That shouldn’t be such a strech since I built an ML compiler in grad school (F# is based on ML).  I wonder what kind of project I’d be interested in would benefit from a functional language…  I suppose I’ll have to read up on the language and maybe that will give me some ideas.

Its funny, doing WPF, my umpteenth technology, is just not enough.

Places are going fast for this year’s Open Source in Finance eXchange on December 15.  Speaking across six expert talks and two parkbench discussions on the day will be Alan Hardy, Rob Davies, Angelo Corsaro and David Vincent.

This year’s eXchange is bringing together the leading experts in Open Source technology and Financial Services software development with innovators, technologists, architects and developers in the developer community.

The day promises to combine deep technical insight with friendly discussion around creating highly scalable, secure systems and leveraging OS technology within the finance sector.

The programme is structured to encourage discussion and bring together the leading thinkers and experts on the use open source technology in Financial Services with passionate members of the developer community in this sector.

To keep it as informal as possible we are strictly limiting the event to 125 delegates, and it is already proving to be popular.  What’s more, on the day we will be raffling off a ticket to this year’s QCon in London.

Book now — and get your tickets for £65.  Just click the banner at the top of the page.

Here’s a great tutorial on F# written from the perspective of a developer already comfortable with C#. I’m just starting to play around with some functional development, and as this is a great jumping off point for someone that wants a quick introduction to F#, I thought I’d share.

Another project i’m working on is translating Paul Graham’s On Lisp (which is available for free) into Clojure, well some of it anyway.

It’s really fun and interesting to read a book by graham since his essays inspired me to learn Lisp in the first place. As with the project Euler sums, i’ll post more info as I discover it

And it can change the face of the web. Honest.

Stay tuned…

So, as you might already know, Scala is a functional language. This means that it is heavily based on functions (Scala actually mixes Object Orientation and Functional Programming).

Lets take a look on how to use functions in Scala. It is really fun, after you get the feel for it.

First, you must know that in Scala, almost (if not) everything can be used as a function identifier. An example:

1.+(2)

This will  result in, obviously, 3. It is even like Scala had operator overload, except that +, -, / and * are not exactly operators in Scala. They are functions, like everything else. Take a look at the scaladoc for the Int class, and notice all the functions for those operators.

Of course, the code above is really ugly. So you should write it like this:

1 + 2

and the Scala compiler takes care of transforming it to the previous example. And what’s more: this is not any kind of special case. This is valid for any function. So, instead of writing this:

calculator.sum(2,3)

you could write this:

calculator sum(2,3)

Dumb example, but nice anyway, isn’t it?

Something more interesting now. If you are used to Javascript, you are used to sometimes pass functions around to other functions. Scala allows us to do the same thing:

def operationTest(operation: (Float, Float) => Float): Float = {
    operation(5,3)
}

The code above declares a function, operationTest, that receives another function, operation, which receives two Floats and returns a Float as well.

The operationTest function then calls the function it received. The result of operationTest is whatever results from operation – in Scala, the return value of a function is always the result of the last statement in it.

You can exercise operationTest this way:

// returns 8.0
operationTest( (x: Float, y: Float) => x + y )

// returns 2.0
operationTest( (x: Float, y: Float) => x - y )

// returns 1.66666666
operationTest( (x: Float, y: Float) => x / y )

// returns 15.0
operationTest( (x: Float, y: Float) => x * y )

So in the code above, we are declaring anonymous functions and passing them to operationTest, which then executes whatever function it received.

Here, (x: Float, y: Float) is the function header, or declaration, and everything after => is the function body: x + y, x – y, x / y and x * y in these cases. Notice also that we are not declaring the functions’ return types. Scala infers them from the execution of the body, and thus from the type of the parameters.

That’s it for now. Willing to learn something in specific about Scala? Post a comment and let me know! I’m in the quest for mastering this language, so challanges are much appreciated! =)

The video of my talk, Haskell and C++ Template Metaprogramming, is now available; and so are the slides. I pretty much covered the material from my last blog post, but many people (including me) find a video presentation easier to follow.

This is also a plug for the Northwest C++ Users Group that meets in Redmond every third Wednesday of the month. If you live in Seattle or on the east side of Lake Washington, check it out. You won’t be disappointed.

Recently I’ve been looking for a job involving Haskell or functional programming in general. No luck. I know that there are is page in Haskell wiki about that, but it’s almost empty, if I filter out those which doesn’t match my preferences.

First thing is: most companies are from US. Next came UK finance companies (like Barclays or Credit Suisse), which are way too far for me. There are also a couple of links to now-dead startups that meant to use Haskell as a silver bullet. No luck I suppose. The one interesting company listed there is Starling Software. Too bad they are from Japan. Another notable exception is Erlang Training and Consulting Ltd. Sadly, while they do have an office in Kraków, they don’t seem to willing to expand their staff there.

I also tried several job search engines for a FP job in Poland, without luck.

My research seems to agree with a previous work you can read about here.

It is sad that most companies that use FP seem to be either too far away or long dead.

I would really love to see one of those near Wrocław.

I’ve been reading on F# a bit, and came across it’s ability to do tail call recursion. Tail recursion as defined by Wikipedia:

In computer science, tail recursion (or tail-end recursion) is a special case of recursion in which the last operation of the function, the tail call, is a recursive call. Such recursions can be easily transformed to iterations. Replacing recursion with iteration, manually or automatically, can drastically decrease the amount of stack space used and improve efficiency. This technique is commonly used with functional programming languages, where the declarative approach and explicit handling of state promote the use of recursive functions that would otherwise rapidly fill the call stack.

So, I decided to see if I could implement this in C#, and after a little bit of monkeying around I was able to get something working.

First off, I needed two helper classes, the first of which is the Ref<T> class. It makes it possible to keep track of a reference to an object across closures. There is also an implicit conversion back to the type it is referencing for convenience.

Next was the LazyRef<T>, which is the same as Ref<T> but won’t actually compute the value until it is needed.

Alright, now all that remains is the method that does all of the magic, I’ll present it piecewise then all together at the end. First, let’s look at the method’s signature,

It will return a delegate that represents the tail call optimized method, taking a T as the only parameter and returning an R. The only argument is the target method/delegate to be transformed. Note that the target takes a delegate as it’s first argument, this will be the recurse function that the method will call instead of itself when it wishes to proceed. It then takes a T, and returns a LazyRef to an R.

Now for the first part of the method, just some Ref and LazyRef variables, I’ll explain these as they are used.

Next up is setRef, it’s the function that will be the recurse function that is passed in as the first argument when target is called.

setRef first flips the calledRef flag, so that we know that target wants to do one more recursion, then we store off the value that target wanted to pass on, and we return the lazyResult. lazyResult will return currResult’s value when asked what it’s value is.

Next, is the actual function that is returned,

First off, it sets up the current parameter value as t, since it needs to be set for the first iteration. Then it enters the loop and flips the calledRef flag to false, so if target doesn’t call the recurse function the loop will not continue. After that target is called, passing in setRef as the recurse function, and the current parameter value. It’s return is stored as the currentResult which will be lazyResult until target doesn’t call the recurse function, at which point it will return something else. This goes on until target doesn’t call the recurse function and the loop exits. After that we return the value or lazyResult, which is the value or currentResult. Ok, so here is everything all together:

All of the refs end up in closures in either setRef or the return value, so the values they reference can be shared and changed. Having the current result be lazy in itself makes the whole thing possible. If we didn’t have LazyRef target would be expecting a computed value to be returned from recurse which would require actual recursion.

Time for a quick example, this will run until the x overflows:

Feel free to post questions and comments.

If you want to understand C++ template metaprogramming (TMP) you have to know functional programming. Seriously. I want you to think of TMP as maximally obfuscated (subset of) Haskell, and I’ll illustrate this point by point. If you don’t know Haskell, don’t worry, I’ll explain the syntax as I go.

The nice thing about single-paradigm languages like Haskell is that they have very simple syntax (think of Lisp that does everything with just a bunch of parentheses). I will start the Haskell-C++TMP mapping with basics, like functions and recursion, but I’ll try to cover a lot more, including higher-order functions, pattern matching, list comprehension (did you know it was expressible in C++?), and more.

Keep in mind that my Haskell examples are runtime functions operating on runtime data whereas their C++ TMP equivalents are compile-time templates operating mostly on types. Operation on types are essential in providing correct and efficient implementations of parametrized classes and functions.

By necessity the examples are simple, but the same mapping may be applied to much more complex templates from the C++ Standard Library and Boost. As a bonus, I’ll also explain the hot new thing, variadic templates.

Functional Approach to Functions

How do you implement useful functions if you don’t have mutable variables, if statements, or loops? To a C++ programmer that might seem like an impossible task. But that’s the reality of C++ compile-time language that forms the basis of TMP. Functional programming to the rescue!

As a warm-up, let’s see how Haskell implements a simple function, the factorial:

fact 0 = 1
fact n = n * fact (n - 1)

The first line states that the factorial of zero is one. The second line defines factorial for a non-zero argument, n (strictly speaking it only works for positive non-zero arguments). It does it using recursion: factorial of n is equal to n times the factorial of n-1. The recursion stops when n is equal to zero, in which case the first definition kicks in. Notice that the definition of the function fact is split into two sub-definitions. So when you call:

fact 4

the first definition is looked up first and, if it doesn’t match the argument (which it doesn’t), the second one comes into play. This is the simplest case of pattern matching: 4 doesn’t match the “pattern” 0, but it matches the pattern n.

Here’s almost exactly the same code expressed in C++ TMP:

template<int n> struct
fact {
    static const int value = n * fact<n - 1>::value;
};

template<> struct
fact<0> { // specialization for n = 0
    static const int value = 1;
};

You might notice how the horrible syntax of C++ TMP obscures the simplicity and elegance of this code. But once you are equipped with the C++/Haskell decoder ring, things become a lot clearer.

Let’s analyze this code. Just like in Haskell, there are two definitions of fact, except that their order is inverted. This is because C++ requires template specialization to follow the template’s general definition (or declaration, as we’ll see later). The pattern matching of arguments in C++ does not follow the order of declarations but rather is based on “best match”. If you instantiate the template with argument zero:

cout << "Factorial of 0 = " << fact<0>::value << endl;

the second pattern, fact<0>, is a better fit. Otherwise the first one, <int n>, is used.

Notice also the weird syntax for “function call”

fact<n>::value

and for the “return statement”

static const int value = n * fact<n - 1>::value;

This all makes sense if you look at templates as definitions of parameterized types, which was their initial purpose in C++. In that interpretation, we are defining a struct called fact, parameterized by an integer n, whose sole member is a static const integer called value. Moreover, this template is specialized for the case of n equal zero.

Now I want you to forget about what I just said and put on the glasses which make the C++ code look like the corresponding Haskell code.

Here’s another example–this time of a predicate (a function returning a Boolean):

is_zero 0 = True
is_zero x = False

Let’s spice it up a little for C++ and define a predicate on types rather than integers. The following compile-time function returns true only when the type T is a pointer:

template<class T> struct
isPtr {
    static const bool value = false;
};

template<class U> struct
isPtr<U*> {
    static const bool value = true;
};

This time the actual argument to isPtr is first matched to the more specialized pattern, U* and, if it fails, the general pattern is used.

We can add yet another specialization, which will pattern-match a const pointer:

template<class U> struct
isPtr<U * const> {
    static const bool value = true;
};

These types of type predicates may be, for instance, used to select more flexible and efficient implementations of parameterized containers. Think of the differences between a vector of values vs. a vector of pointers.

Lists

The basic data structure in functional languages is the list. Haskell’s lists are introduced using square brackets. For instance, a list of three numbers, 1, 2, 3, looks like:

[1, 2, 3]

List processing in functional languages follows the standard pattern: a list is split into head and tail, an operation is performed on the head, and the tail is processed using recursion. The splitting is done by pattern matching: the Haskell pattern being (head:tail) (to be precise, the colon in parentheses represents the cons operation–the creation of a list by prepending an element to an existing list).

Here’s a simple function, count, that calculates the length of a list:

count [] = 0
count (head:tail) = 1 + count tail

The first pattern, [], matches an empty list; the second a non-empty one. Notice that a function call in Haskell doesn’t use parentheses around arguments, so count tail is interpreted as a call to count with the argument tail.

Before C++0x, TMP was severely crippled by the lack of a list primitive. People used separate definitions for a list of zero, one, two, etc., elements, and even used special macros to define them. This is no longer true in C++0x, thanks to variadic templates and template parameter packs. Here’s our Haskel count translated into C++ TMP:

// Just a declaration
template<class... list> struct
count;

template<> struct
count<> {
    static const int value = 0;
};

template<class head, class... tail> struct
count<head, tail...> {
    static const int value = 1 + count<tail...>::value;
};

First we have a declaration (not a definition) of the template count that takes a variable number of type parameters (the keyword class or typename introduces a type parameter). They are packed into a template parameter pack, list.

Once this general declaration is visible, specializations may follow in any order. I arranged them to follow the Haskell example. The first one matches the empty list and returns zero. The second one uses the pattern, <head, tail…>. This pattern will match any non-empty list and split it into the head and the (possibly empty) tail.

To “call” a variadic template, you initiate it with an arbitrary number of arguments and retrieve its member, value, e.g.,

int n = count<int, char, long>::value; // returns 3

A few words about variadic templates: A variadic template introduces a template parameter pack using the notation class… pack (or int… ipack, etc…). The only thing you may do with a pack is to expand it and pass to another variadic template. The expansion is done by following the name of the pack with three dots, as in tail…. You’ll see more examples later.

Variadic templates have many applications such as type-safe printf, tuples (objects that store an arbitrary number of differently typed arguments), variants, and many more.

Higher-Order Functions and Closures

The real power of functional programming comes from treating functions as first class citizens. It means that you may pass functions to other functions and return functions from functions. Functions operating on functions are called higher-order functions. Surprisingly, it seems like compile-time C++ has better support for higher-order functions than run-time C++.

Let’s start with a Haskell example. I want to define a function that takes two predicate functions and returns another predicate function that combines the two using logical OR. Here it is in Haskell:

or_combinator f1 f2 =
    λ x -> (f1 x) || (f2 x)

The or_combinator returns an anonymous function (the famous “lambda”) that takes one argument, x, calls both f1 and f2 with it, and returns the logical OR of the two results. The return value of or_combinator is this freshly constructed function. I can then call this function with an arbitrary argument. For instance, here I’m checking if 2 is either zero or one (guess what, it isn’t!):

(or_combinator is_zero is_one) 2

I put the parentheses around the function and its arguments for readability, although they are not strictly necessary. 2 is the argument to the function returned by or_combinator.

The lambda that’s returned from or_combinator is actually a closure. It “captures” the two arguments, f1 and f2 passed to or_combinator. They may be used long after the call to or_combinator has returned.

It might take some getting used to it before you are comfortable with functions taking functions and returning functions, but it’s much easier to learn this stuff in Haskell than in the obfuscated C++. Indeed, here’s an almost direct translation of this example:

template<template<class> class f1, template<class> class f2> struct
or_combinator {
    template<class T> struct
    lambda {
        static const bool value = f1<T>::value || f2<T>::value;
    };
};

Since in the metalanguage a function is represented by a template, the template or_combinator takes two such templates as arguments. It “calls” these templates using the standard syntax f<T>::value. Actually, the or_combinator doesn’t call these functions. Instead it defines a new template, which I call lambda, that takes the argument T and calls those functions. This template acts like a closure–it captures the two templates that are the arguments to or_combinator.

Here’s how you may use the or_combinator to combine two tests, isPtr and isConst and apply the result to the type const int:

std::cout
   << "or_combinator<isPtr, isConst>::lambda<const int>::value = "
   << or_combinator<isPtr, isConst>::lambda<const int>::value
   << std::endl;

Such logical combinators are essential for predicate composability.

Higher-Order Functions Operating on Lists

Once you combine higher-order functions with lists you have a powerful functional language at your disposal. Higher-order functions operating on lists look very much like algorithms. Let me show you some classic examples. Here’s the function (or algorithm), all, that returns true if and only if all elements of a list satisfy a given predicate.

all pred [] = True
all pred (head:tail) = (pred head) && (all pred tail)

By now you should be familiar with all the techniques I used here, like pattern matching or list recursion.

Here’s the same code obfuscated by the C++ syntax:

template<template<class> class predicate, class... list> struct
all;

template<template<class> class predicate> struct
all<predicate> {
    static const bool value = true;
};

template<
    template<class> class predicate,
    class head,
    class... tail> struct
all<predicate, head, tail...> {
    static const bool value =
        predicate<head>::value
        && all<predicate, tail...>::value;
};

Except for the initial declaration required by C++ there is a one-to-one paradigm match between the two implementations.

Another useful algorithm, a veritable workhorse of functional programming, is “fold right” (together with it’s dual partner, “fold left”). It folds a list while accumulating the results (that’s why in runtime C++ this algorithm is called “accumulate”). Here’s the Haskell implementation:

foldr f init [] = init
foldr f init (head:tail) =
    f head (foldr f init tail)

Function f, which is the first argument to foldr, takes two arguments, the current element of the list and the accumulated value. Its purpose is to process the element and incorporate the result in the accumulator. The new accumulated value is then returned. It is totally up to the client to decide what kind of processing to perform, how it is accumulated, and what kind of value is used. The second argument, init, is the initial value for the accumulator.

Here’s how it works: The result of foldr is generated by acting with f on the head of the list and whatever has been accumulated by processing the tail of the list. The algorithm recurses until the tail is empty, in which case it returns the initial value. At runtime this type of algorithm would make N recursive calls before starting to pop the stack and accumulate the results.

For instance, foldr may be used to sum the elements of a list (so_far is the accumulator, which is initialized to zero):

add_it elem so_far = elem + so_far
sum_it lst = foldr add_it 0 lst

The accumulator function is add_it. If, instead, I wanted to calculate the product of all elements, I’d use a function mult_it and the starting value of one. You get the idea.

Here’s the same algorithm in C++ TMP:

template<template<class, int> class, int, class...> struct
fold_right;

template<template<class, int> class f, int init> struct
fold_right<f, init> {
    static const int value = init;
};

template<template<class, int> class f, int init, class head, class...tail> struct
fold_right<f, init, head, tail...> {
    static const int value = f<head, fold_right<f, init, tail...>::value>::value;
};

Once you understand the Haskell version, this complex code suddenly becomes transparent (if it doesn’t, try squinting ).

Lists of Numbers

Let’s now switch to integers for a moment. Haskell defines a function sum that adds all elements of a list:

sum [] = 0
sum (head:tail) = head + (sum tail)

We can do the same in C++ TMP (in five times as many lines of code):

template<int...> struct
sum;

template<> struct
sum<> {
    static const int value = 0;
};

template<int i, int... tail> struct
sum<i, tail...> {
    static const int value = i + sum<tail...>::value;
};

List Comprehension

Haskell has one more trick up its sleeve for operating on lists without explicit recursion. It’s called list comprehension. It’s a way of defining new lists based on existing lists. The nomenclature and notation are borrowed from Set Theory, where you often encounter definitions such as: S is a set of elements, where… Let’s look at a simple Haskell example:

[x * x | x <- [3, 4, 5]]

This is a set (list) of elements x * x, where x is from the list [3, 4, 5].

Remember our recursive definition of count? Using list comprehension it’s reduced to a one-liner:

count lst = sum [1 | x <- lst]

Here, we create a list of ones, one for each element of the list. Our result is the sum of those ones. To make this definition more amenable to translation into C++, let’s define an auxiliary function one that, for any argument x, returns 1.

one x = 1

Here’s the modified definition of count:

count lst = sum [one x | x <- lst]

Now we are ready to convert this code to C++ TMP:

template<class T> struct
one {
    static const int value = 1;
};

template<class... lst> struct
count {
    static const int value = sum<one<lst>::value...>::value;
};

Here our list is stored in a template parameter pack, lst. If we wanted to expand this pack, we’d use the notation lst…, but that’s not what’s happening here. The ellipsis appears after the pattern containing the pack:

one<lst>::value...

Compare this with the equivalent Haskell:

[one x | x <- lst]

In C++, when the ellipsis follows a pattern that contains a pack, it’s not the pack that’s expanded, but the whole pattern is repeated for each element of the pack. Here, if our list were <int, char, void*>, the pattern would be expanded to:

<one<int>::value, one<char>::value, one<void*>::value>

The subsequent call to sum would be made with those arguments.

Notice that a different positioning of the ellipsis would result in a completely different expansion. This pattern:

one<lst...>::value

would result in the call to one with the list of types, which would be an error.

Here’s another example of pattern expansion: a function that counts the number of pointers in a list of types:

template<class... lst> struct
countPtrs {
    static const int value = sum<isPtr<lst>::value ...>::value;
};

In this case the pattern is:

isPtr<lst>::value ...

and it expands into a list of Booleans. (I’m taking advantage of the fact that false is zero and true is one, when converted to integers.)

You may find a more complex practical example in the Gregor, Järvi, and Powell paper (see bibliography).

Continuations

List comprehension can be used to define some very useful higher-order functions. One of such functions is map, which takes a list and applies a unary function to each element, resulting in a new list. You might be familiar with the runtime implementation of this algorithm in C++ STL under the name of transform. This is what map looks like in Haskell:

map f lst = [f x | x <- lst]

Here, f is the unary function and lst is the input list. You have to admire the terseness and elegance of this notation.

The first impulse would be to translate it into C++ TMP as:

template<template<class> class f, class... lst> struct
map {
    typedef f<lst>... type;
};

This is surprisingly terse too. The problem is that it doesn’t compile. As far as I know there is no way for a template to “return” a variable list of elements. In my opinion, this is a major language design flaw, but that’s just me.

There are several workarounds, none of them too exciting. One is to define a separate entity called a typelist along the lines of:

template struct
typelist<hd, tl...> {
    typedef hd head;
    typedef typelist<tl...> tail;
};

(As a matter of fact I have implemented typelists and related algorithms both in C++ and D.)

Another approach is to use continuations. Template parameter packs cannot be returned, but they can be passed to variadic templates (after expansion). So how about defining an algorithm like map to take one additional function that would consume the list that is the result of mapping? Such a function is often called a continuation, since it continues the calculation where normally one would return the result. First, let’s do it in Haskell:

map_cont cont f lst = cont [f x | x <- lst]

The function map_cont is just like map except that it takes a continuation, cont, and applies it to the result of mapping. We can test it by defining yet another implementation of count:

count_cont lst = map_cont sum one lst

The continuation here is the function sum that will be applied to the list produced by acting with function one on the list lst. Since this is quite a handful, let me rewrite it in a more familiar notation of runtime C++:

int map_cont(int (*cont)(list), int (*f)(int), list lst) {
    list tmp;
    for (auto it = lst.begin(); it != lst.end(); ++it)
        tmp.push_front(f(*it));
    return cont(tmp);
}

Now for the same thing in compile-time C++:

template<template<class...> class cont,
         template<class> class f,
         class... lst> struct
map_cont {
    static const int value =
        cont<typename f<lst>::type ...>::value;
};

It’s a one-to-one mapping of Haskell code–and it has very little in common with the iterative runtime C++ implementation. Also notice how loose the typing is in the TMP version as compared with the runtime version. The continuation is declared as a variadic template taking types, function f is declared as taking a type, and the list is a variadic list of types. Nothing is said about return types, except for the constraints that f returns a type (because cont consumes a list of types) and cont returns an int. Actually, this last constraint can be relaxed if we turn integers into types–a standard trick (hack?) in TMP:

template<int n> struct
Int {
    static const int value = n;
};

Loose typing–or “kinding,” as it is called for types of types–is an essential part of compile-time programming. In fact the popularity of the above trick shows that C++ kinding might be already too strong.

The D Digression

I’m grateful to Andrei Alexandrescu for reviewing this post. Since he objected to the sentence, “I’m disappointed that the D programming language followed the same path as C++ rather than lead the way,” I feel compelled to support my view with at least one example. Consider various implementations of all.

In Haskell, beside the terse and elegant version I showed before:

all pred [] = True
all pred (head:tail) = (pred head) && (all pred tail)

there is also a slightly more verbose one:

all pred list =
    if null list then True
    else pred (head list) && all pred (tail list)

which translates better into D. The D version (taken from its standard library, Phobos) is not as short as Haskell’s, but follows the same functional paradigm:

template allSatisfy(alias F, T...) {
    static if (T.length == 1)
    {
        alias F!(T[0]) allSatisfy;
    }
    else
    {
        enum bool allSatisfy = F!(T[0]) && allSatisfy!(F, T[1 .. $]);
    }
}

It definitely beats C++, there’s no doubt about it. There are some oddities about it though. Notice that the type tuple, T… gets the standard list treatment, but the split into the head and tail follows the array/slice notation. The head is T[0] and the tail is an array slice, T[1..$]. Instead of using value for the return value, D uses the “eponymous hack,” as I call it. The template “returns” the value using its own name. It’s a hack because it breaks down if you want to define the equivalent of “local variable” inside a template. For instance, the following code doesn’t compile:

template allSatisfy(alias F, T...) {
    static if (T.length == 1)
    {
        alias F!(T[0]) allSatisfy;
    }
    else
    {
        private enum bool tailResult = allSatisfy!(F, T[1..$]);
        enum bool allSatisfy = F!(T[0]) && tailResult;
    }
}

This breaks one of the fundamental property of any language: decomposability. You want to be able to decompose your calculation into smaller chunks and then combine them together. Of course, you may still use decomposition if you don’t use the eponymous hack and just call your return value value. But that means modifying all the calling sites.

By the way, this is how decomposition works in Haskell:

all2 pred [] = True;
all2 pred (head:tail) = (pred head) && tailResult
    where tailResult = all2 pred tail

Anyway, the important point is that there is no reason to force functional programming paradigm at compile-time. The following hypothetical syntax would be much easier for programmers who are used to imperative programming:

template allSatisfy(alias Pred, T...) {
    foreach(t; T)
        if (!Pred!(t))
            return false;
    return true;
}

In fact it’s much closer to the parameterized runtime function proposed by Andrei:

bool all(alias pred, Range)(Range r) {
    foreach (e; r)
        if (!pred(e))
            return false;
    return true;
}

This is why I’m disappointed that the D programming language followed the same path as C++ rather than lead the way.

Conclusions

I have argued that some familiarity with Haskell may be really helpful in understanding and designing templates in C++. You might ask why C++ chose such horrible syntax to do compile-time functional programming. Well, it didn’t. The ability to do compile-time calculations in C++ was discovered rather than built into the language. It was a very fruitful discovery, as the subsequent developments, especially the implementation of the Boost MPL, have shown. However, once the functional paradigm and its weird syntax took root in C++ TMP, it stayed there forever. I’m aware of only one effort to rationalize C++ TMP by Daveed Vandevoorde in Reflective Metaprogramming in C++.

I have tested all code in this blog using the Hugs interpreter for Haskell and the GNU C++ compiler v. 4.4.1 with the special switch -std=c++0x. The names I used in the blog might conflict with the standard definitions. For instance, Hugs defines its own map and foldr.

If you want to learn more about template metaprogramming, I recommend two books:

1. Andrei Alexandrescu, Modern C++ Design
2. David Abrahams and Aleksey Gurtvoy, C++ Template Metaprogramming

The reference I used for variadic templates was the paper by Douglas Gregor, Jaakko Järvi, and Gary Powell

What follows is an account from the author’s experience. Some details have been changed for the usual reasons, and a poor memory has fuzzed out the rest.

UPDATE: it turns out the technique described below is known as debouncing. What an awesome name!

My last job was for a company that made case-management software. With this software, you could track all kinds of data about your case: you could categorize it, sub-categorize it, say where and when it happened, note whether it was still open, track who was responsible…all kinds of stuff, great for data fiends.

One of our customers’ favorite features was the reports, and they were pretty spiffin’. But, not being ones to rest on our laurels, we were adding pivot-table reports, so you could count the number of cases, grouped by any criteria you wanted. The input screen let the customer pick their pivot criterion, and which criteria they wanted as columns, for sub-counts. We struggled to name these UI fields — something lucid, something explanatory, something evocative — something to make them understand what kind of report they were in for. After a while, we decided to just show them: when they changed their pivot criterion, we’d run some javascript to render a skeleton report, same as the real report, but minus the data. It would save them time, and save our servers.

The javascript was pretty involved. It had to generate the same HTML as we did for the pivot table, which meant it had to know (or make up) values for each criterion, like all the case categories and sub-categories, and which sub-categories belonged to which categories. And we let the customers pivot on up to THREE, count ‘em, different criteria. And it had to happen each time the user picked different pivot criteria. It took a few tricks, but we got it working. It ran slowly, maybe a few seconds, but it was quick enough, probably, especially once we threw in a little “please wait” spinner. Then we realized we needed to re-render whenever the window resized.

No biggie, I thought, and I quickly added window.onResize(reportPreview);. It worked great, except it re-rendered the report with every pixel-wide movement of the mouse, as the window was dragged to new widths and heights. Calling a function, one that runs for a few seconds, a hundred times in the time it took to widen the browser an inch, meant a locked browser. It meant “time to get more coffee,” and after, “time to fix the bug.”

I knew we could delay calling reportPreview, but we only wanted to delay it when the window was being resized — when the user changed the columns, there was no reason to wait. I was sure window.setTimeout() would do what we needed, but I didn’t want to muck up reportPreview() with it.

I’d been reading The Little Schemer lately, and noticing some striking similarities between javascript and scheme: first-class functions, and higher-order functions that take functions as arguments, and return other functions as values. It was fun reading, with its strange teacher-student dialog style. The material was better than brainteasers, and I knew it would make me a better programmer down the road, but I didn’t think of it as relevant to the day-job, as…applicable.

Then I realized higher-order functions were a way out of this. I could write a function, delay, that would wrap one long-running, slow-poke function in another function: an intermediary that you could call as many times a second as you wanted, but it would only call the slow-poke once things had settled down. delay would let us keep setTimeout out of reportPreview. Something like this:

function delay(millis, slowPoke) {
    var timeoutId = undefined;

    // This is the intermediary.  Call it lots, it won't hurt.
    return function() {
        if (timeoutId) {  // If we're waiting...
            clearTimeout(timeoutId); // re-start the clock.
        }
        timeoutId = window.setTimeout(slowPoke, millis);
    }
}

The first time you call the intermediary, it tells the window to call slowPoke after a bit, but every time you call it after that, it starts the clock over. It’s like when you’re in a five-minute time-out, and you start acting up after only three, so your mom says “Ok, buster, another five minutes.”

var fiveMinutes = 5 * 60 * 1000;
var screamAndShout = delay(fiveMinutes, function() {
    getBackToPlaying();
});

screamAndShout(); // Aw nuts, I'm in time-out.

// I'll be good for as long as I can, but...
screamAndShout(); // dang, five MORE minutes!

Once delay was in place, running reportPreview when the window was resized was no problem.

function reportPreview() {
    // recursion, DOM manipulation, insanity...
}
columnPicker.onChange(reportPreview);
window.onResize(delay(100, reportPreview));

After testing, we found that delaying it for 100 milliseconds made all the difference in the world.

Do you have a war story? Share it, and start the healing: tell a short one in the comments, or link to a longer one on your own damn blog.

Lately I’ve been teaching myself Clojure, a Lisp dialect on the JVM platform. I still love Erlang and still learning it, but Clojure has a special draw for me being a JVM language and its Lisp roots. I studied Scheme (another Lisp dialect) in my college years and deemed it purely academic. However, Clojure has the potential of changing this and bring the expressiveness of Lisp and the power of functional programming to the Java world.

The best way to learn a language is to use it to solve some non-trivial practical problems. I stumbled this problem on Programming Praxis. It’s about building a Bifid cipher, which I thought could be a good practical problem for me to solve in Clojure.

Polybius Square

Bifid cipher is based on Polybius Square, which looks like this:

1 2 3 4 5
1 B G W K Z
2 Q P N D S
3 I O A X E
4 F C L U M
5 T H Y V R

Every encodable letter is given a vector value, which is the coordinate in this square. For example, X=[3 4], H=[5 2]. The first task would be building the Polybius square in Clojure.

My function polybius-square is going to take 2 parameters: charset is a list of encodable characters, and size is the size of the Polybius square.

(defn polybius-square
  "Polybius square by the given charset and size"
  [charset square-size]
  (map #(vector %1 [%2 %3]) charset
    (for [x (range 1 (+ 1 square-size)), y (range 1 (+ 1 square-size))] x)
    (take (* square-size square-size) (cycle (range 1 (+ 1 square-size))))))

Let’s see how this works. Suppose you’re calling this function with
(polybius-square "ABCDE" 3), charset is bound to “ABCDE” and size is bound to 3. What I want is a seq of vectors in the form of: (\Letter [x y]) where x and y are the coordinates. It’s natural to use map, with the function-to-map taking three parameters and vectorize them. That’s what the anonymous function does:
#(vector %1 [%2 %3])

%1 is the letter, %2 and %3 are the coordinates. I want the coordinate sequence to be like this: [1 1], [1 2], [1 3], [2 1], [2 2], [2 3]. In imperative language, you’d use a nested for loop, but we’ve established that’s ugly. We want the parameters fed to map to be of the following:
"ABCDEFGHJI" (1 1 1 2 2 2 3 3 3) (1 2 3 1 2 3 1 2 3 1 2 3)

To generate the first list, we use list comprehension:
(for [ x (range 1 (+ 1 square-size)), y (range 1 (+ 1 square-size))] x)
It reads: for every x in range 1 to square-size+1 (because “range” is right-exclusive) and for every y in the same range, output the value of x and take the whole output as the list. Because the binding of y comes later, and Lisp evaluates from right to left, the result of this list comprehension is (1 1 1 2 2 2 3 3 3)

Now let’s look at the other list: (1 2 3 1 2 3 1 2 3) – a cyclic list. Fortunately, Clojure has a built-in cycle function, which returns a lazy sequence of cycles of a collection. We want to call it with cycle '(1 2 3), so it’s going to be cycle (range 1 (+ 1 square-size)). Because cycle will be evaluated to an infinite sequence, we have to tell it when to stop.
When you see the square as a flat list, it’s clear that the cycle should stop when it reaches square-size^2, hence (take (* square-size square-size) (cycle (range 1 (+ 1 square-size)))).

The codec maps

After the Polybius square is built, we have the bases for our encryption. However, it’s not user-friendly: it’s a flat list of tuples. We want two maps for fast searching from letters to their encoding forms (letter-2-code) and from transposed vectors to the letters (transposed-2-letter).

(defn- letter-2-code
  "Produce a mapping between letters and their codes"
  [square]
  (letfn [(helper [the-square the-map]
          (if (empty? the-square)
            the-map
            (let [the-item (first the-square)
                  the-key (first the-item)
                  the-number (second the-item)]
              (recur (rest the-square) (assoc the-map the-key the-number)))))]
    (helper square (empty hash-map))))

The strategy here is to walk through the list, add each item to a hash-map. Here we define an inner function, which is basically a recursive helper. The function itself is pretty straightforward. Take note that in Clojure, when you need to do tail recursion, always use recur instead of calling the function by its name. This is because JVM cannot do automatic tail cut optimization (TCO), but Clojure does the optimization through the recur keyword. After the binding, we kick off the recursion by calling it with the initial list.

In the similar fashion, we have:

(defn- transposed-2-letter
  "Produce a mapping between the transposed codes to letter"
  [square]
  (letfn [(helper [the-square the-map]
            (if (empty? the-square)
              the-map
              (let [the-item (first the-square)
                    the-key (second the-item)
                    the-letter (first the-item)]
                (recur (rest the-square) (assoc the-map the-key the-letter)))))]
    (helper square (empty hash-map))))

Encode

Now comes the encode function. It’s important to note that the encoding will be different if a different polybius square is provided. Therefore, both encode and decode functions take the Polybius square as parameter.

During the encoding phase, we first convert every character in the message into a vector, then reorganize the vectors. To better illustrate, we use an example:
Suppose we have the following mapping:
{I [3 3], H [3 2], G [3 1], F [2 3], E [2 2], D [2 1], C [1 3], B [1 2], A [1 1]}
and we want to encode the word “CEDE”
CEDE => ([1 3] [2 2] [2 1] [2 2]), then we want to transpose the “matrix” composed of the above vectors, so it would be like:
([1 2] [2 2] [3 2] [1 2]), and finally looking up every item in the transposed-2-letter mapping
{[3 3] I, [3 2] H, [3 1] G, [2 3] F, [2 2] E, [2 1] D, [1 3] C, [1 2] B, [1 1] A}
=>BEHB

To get a mapping of message to vectors, we again map a function to the list:
(map #(get l2c % %) message) (l2c has been bound)
Then we apply the interleave function to the result, this gives:
(1 2 2 2 3 2 1 2). This is good but we want a list of vectors instead of a flat list. Clojure provides partition function, which does exactly it:
(partition 2 '(1 2 2 2 3 2 1 2))
gives:
([1 2] [2 2] [3 2] [1 2])

Then, we want to lookup every item in the result in the transposed-to-letter map and get a list of mapped characters. To achieve this, again, we use map, providing a different mapping:
(map #(get tr2l % %) ...) (tr2l is bound)
This returns a list of characters. We want a string, so we concatenate the characters by applying str function:
(apply str (map #(get tr2l % %) ...))

Here’s the complete code:

(defn encode
  "Return the message encoded by the specified polybius-square"
  [message polybius-square]
  (let [l2c (letter-2-code polybius-square)
        tr2l (transposed-2-letter polybius-square)]
    (apply str
      (map #(get tr2l % %) (partition 2 (apply interleave (map #(get l2c % %) message)))))))

Here you see the expressiveness of Clojure: all operations are chained together and formed in expression.

The decode function behaves similarly. The only trick is we need to split the translated code list in two. We use partition method again, with size being half of the entire list.

(defn decode
  "Return the decoded message using the specified polybius-square"
  [encoded polybius-square]
  (let [l2c (letter-2-code polybius-square)
        c2l (transposed-2-letter polybius-square)
        codes (apply concat (map #(get l2c % %) encoded))
        len (count codes)
        parts (partition (/ len 2) codes)]
    (apply str (map #(get c2l %1 %1) (map #(vector %1 %2) (first parts) (second parts))))))

Putting it together

Now with all the pieces are in place, we can put it together by writing a simple test:

(defn -main
  "A simple test case entry point"
  []
  (let [square (polybius-square "ABCDEFGHIKLMNOPQRSTUVWXYZ" 5)]
    (do (println (encode "BIFIDCIPHER" square))
      (println (decode (encode "BIFIDCIPHER" square) square)))))

Output:

BGAHFRQTOXW
BIFIDCIPHER


Have the complete source code here at http://github.com/kevinjqiu/bifid-clj, I’m sure there are a lot to be improved of the above implementation of the Bifid cipher. Especially currently I have no elegant way to solve the situation where the letter to be encoded is not in the Polybius square. Also, I’m sure my code isn’t entirely the idiomatic Clojure. If you have any suggestions, don’t hesitate to write a comment.

Although I have been learning Clojure for 3 days, I’m already enjoying its expressiveness and the clean look. As a Java programmer, Clojure also gives me the benefit of full access to the Java land. The potential is endless!

To recurse or not to recurse? This is of course a brilliant idea if you’re in search for a cool and geeky name for your new pet project. But is it also wise to code your-favorite-algorithm ™ in a recursive manner?

Let’s start with the basics, Wikipedia has a good definition available:

Recursion, in mathematics and computer science, is a method of defining functions in which the function being defined is applied within its own definition.

Furthermore, in our first Computer Science semester we where taught the logical equivalence of coding an algorithm in an applicative or imperative way. So what’s the matter then? As usual, problems arise when reality hits (Computer Science) theory hard. Generally spoken, those two methods of encoding an algorithm differ physically. Ok, what’s meant by that? Well, depending on the used language (more to that soon), it is wise to use the former or the latter way. Let’s start out with good old procedural languages like C, C++, Java and all the others that actually matter. Procedural languages use functions to encapsulate sub-problems. For example, when a C++ program is transformed into machine-specific assembly, the compiler enforces certain conventions that ensure that these functions can be called in an uniform way. Executing a function usually means to jump to a different portion of code, do something and to jump back to where you’ve been before to use the computed result. So we have to store that return address. The function concept becomes more powerful if you can send parameters to them, so we have to store those too. Functions tend to use local variables for saving intermediate result and every function call needs a local copy of those variables, so these have to be stored somewhere too. The convention on how to store all this is usually referred to as a stack frame. As the name implies, a certain frame is stored on the stack, a certain memory area that was set up by your kind operating system for your proggy’s process in advance. Let’s shamelessly steal a picture from Wikipedia to visualize this concept:

Note that the actual frame layout differs with the language that you use. For example Pascal-derived languages use a different layout than the C world. It easy to see that this unavoidable overhead gets bigger (relative, not absolute) the shorter your functions are. Recursive algorithms generally tend to use short functions, which is shown exemplarily with this C++ implementation of the Fibonacci numbers:


int fib(unsigned int n)
{
if (n == 0) {
return 0;
} else if (n == 1) {
return 1;
} else {
return fib(n - 1) + fib(n -2);
}
}


This means big relative overhead. Let’s look at the imperative (and optimised) version of the same algorithm:


int fib(unsigned int n)
{
static unsigned int tmp[] = {0, 1};
static std::vector fibs(tmp, tmp + sizeof(tmp) / sizeof(unsigned int));

if (n >= fibs.size()) {
for (unsigned int i = fibs.size() - 1; i < n; i++) {
fibs.push_back(fibs[i] + fibs[i - 1]);
}
}
return fibs[n];
}


One function means one stack frame, easy peasy. So to say, the overall runtime and memory cost of those stack frames increases the more excessive you execute short functions from within functions (hey, that’s recursion!). The imperative version only uses one stack frame, which saves us countless bytes on the stack (speak, main memory) and reduces the risk of stack overflows. Furthermore it saves CPU cycles for those push/pop and store/fetch instructions you don’t have to execute (The inline keyword in C/C++ comes to mind here). Therefore it is always faster to go imperative than recursive in the real world. This concludes to:

NEVER EVER USE RECURSION IN A PROCEDURAL LANGUAGE !

However, the situation looks slightly different if you use a functional language (or even handcraft the Fibonacci algorithm in assembly, but that’s another story). Functional languages often use the tail recursion idiom to allow (or force, depending on personal opinion) the user to encode algorithms recursively. What happens when that get’s translated into assembly? That idiom is simply mappend back to simple loops for the exact same reason (remember, they’re equivalent logically). This post already got longer than I expected, so let’s keep it short. Functional languages allow convenience recursion idioms that get translated back to imperative idioms to run performant on real hardware. Maybe someone more experienced in Haskell can provide more detail how this is done exactly (assembly listings appreciated).

If you want to confirm the above, take the attached little test programs and run them through your favorite profiler and compare jump/call counts and stack changes. You might want to check what time one iteration of the algorithm takes (speak one loop iteration versus one recursive function call). That’s it!

• fib_imperative.cpp
• fib_recursive.cpp