My excellent colleague, Luke Forehand, pointed out to me recently that Scala has structural typing. Specifically, a blog walking through how to build a list that takes objects which have a certain method details the concept and gives a compelling example of when/where to use it.

This is similar to duck typing in that run time examination of an object is performed to see if it complies with the setup contract. However, where as duck typing simply tries to dispatch a message to an object which may fail, structural typing is a bit tighter contract which builds in checks that a certain call exists before it is called. To steal a form of the Scala example used in the linked blog article above, consider the statement:

val myListOfThings = List[ { def f(x:Int):Int } ]

The above says, declare a list that only takes objects that have a function named f that takes an Int and returns an Int. The beauty here is that you do not have to maintain a tree of related interfaces to bind together objects of similar or identical functionality. You can also build APIs that do not require your users to implement an interface of abstract method that, prehaps, is surrounded with loads of other method signatures and method definitions that the user really doesn’t care about.

I’m not looking for every excuse I can find to use this.

Bestimmt kennt jeder einen der Werbespots der Volksbanken-Raiffeisenbanken, in denen das wunderschöne Lied “Hungriges Herz” von Mia (hier als Cover von Scala) im Hintergrund zu hören ist.

Ich muss zugeben, vor allem deswegen, also wegen des Liedes, gefallen mir die Werbespots. Aber viel mehr noch spricht mich der Inhalt an. Das Motto “Jeder Mensch hat etwas, das ihn antreibt” kann sicherlich jeder bejahen, was immer es auch ist, was uns antreibt. Aber allein ein Ziel, auf das es sich lohnt, hinzuarbeiten, macht das Leben doch lebenswert. Wenn man weiß, warum man etwas tut, auch wenn es Zeiten gibt, in denen das Ziel unerreichbar oder sinnlos scheint.

Jetzt sollte ich mal zum Punkt kommen. Auf der Internetseite www.was-uns-antreibt.de gibt es jetzt auch einen Dokumentarfilm, in dem die Personen, die in den Videos kurz zu sehen sind, nochmal zu Wort kommen. Für mich ist das ein toller Film, den es – nebenbei gesagt – auch kostenlos zum Download gibt.

Wer Lust bekommen hat, klickt sich einfach rein, wer noch zweifelt, für den habe ich noch einen Spot:

m0lch

Many applications have the primary concern of storing and retrieving data. The raw data by itself is often not very useful, so an additional process is put into place to turn that data into useful information. Many applications generate these reports from the data quickly at the user’s request through either a narrow SQL select statement, in application data processing, or both. However, in larger applications where the data is too large to handle in memory at a time, processing is too heavy, and report customization is too varied, information generation needs to be pushed to its own scalable system.

A lot of data has various pieces of metadata associated with it naturally. Data such as what user added the record, the date it was added or modified (or both), maybe the size of the record, categories, tags, keywords, or other pieces of associate data that is used to break it up into more manageable chunks. This meta data is useful, but only if we can use it as an aggregate to generate specific information relating to the grouping of items based on this data.

Sometimes generating these more specific reports is as easy as adding additional WHERE or GROUP BY clauses in SQL. However, when more advanced business rules are taking place where there isn’t an easy or succinct way of extracting this information via a query, or if the query returns such a large amount of data as to cause memory issues, a different approach can be taken.

For instance; in an application I am currently working on we need to generate reports based on a table with about 5 million rows. The SQL queries we use can limit the amount of rows returned to perhaps a few hundred thousand for some of our larger reports. However, a lot of the data needs to be processed in application code rather than by the database itself due to some special business rules. Because of this, we end up creating a large number of objects in Java to hold these result rows. If multiple users are generating different reports we might end up holding too many of these objects in memory at a time and receive an OOM error. Also, the processing on this data can be intense enough that if the server is slammed with report requests that the entire system slows down, causing difficulties for people wanting to insert or modify data. This is the case I am in while I contemplate offline report generation.

The basic idea is that the main application should be concerned purely with manipulating the data of the system. That is basic CRUD stuff such as creating new records, updating them, the rare deletions, and showing single records to the user (so they can edit or delete it). We want that part of the application to remain fast, and not be effected by the purely read-only needs imposed by report generation. In order to nullify the impact, we move reporting to its own system that reads from a separate read-only replication of our production database.

When a report request comes in to our application, we send a request to the separate reporting system. This can be done either as a web service or maybe an RPC call. The reporting system uses its read-only copy of the data to generate the report numbers and send it back, causing no speed delay for insertion or regular operation of the main application.

This doesn’t solve our OOM issues however, as many drivers for our database (MySQL) return ResultSet objects with the entire contents of the results which might be too large to fit into memory. However, since we’re using a read-only list anyway we can convert the table or tables we use to process our results into flat files that can be read in on a line by line basis, perform some intermediate result processing, deallocate those lines and work on additional lines. Since our reports are mostly generating statistical data over a large data set, we can process results on that data set in parallel using multiple threads or possibly multiple computers using a Hadoop cluster.

By making report generation asynchronous to our applications general work flow we will free up the processing power and the connection pool that’s used to handle requests by asking users to either poll for when the result is finished or to notify the system when a report is finished and thereby avoid the instances where we use all of our available connections or resources processing reports. There is also the added possibility of continuously generating all possible report data on a separate machine or cluster to decrease access time by increasing storage requirements.

I’m currently researching the use of MapReduce in Hadoop for processing this flat file of all data for reports. In addition, I’m researching a few languages that are reported to be good at concurrent processing so that I can benefit from multiple cores when generating reports from our raw data. My current focus is on Scala, Erlang, and Clojure, but I do not have much to report on those areas yet. If anyone has any information those languages as far as report generation based on a largish data set (currently 5 million, but the rate of growth is fairly alarming), let me know.

The Lift web framework includes basic support for OpenId using the the openid4java library. I wanted to learn a little more about Scala, Lift and OpenId – so what follows are my notes on how to get this running, along with some minor improvements that I have made.

You will want to start by familiarizing yourself with the OpenId chapter in the (rather excellent) Lift book.

Out of the box, the lift-openid support gets  you a vanilla login form where the user can enter their OpenId URL and submit the request.  Something that looks like this:

While this is nice for identity geeks, your grandma isn’t going to be using this anytime soon. The OpenId UI issues are being hotly debated as you read this, and I won’t open that can of worms. Suffice to say that mere mortals will prefer to use a “Nascar” branded interface that let’s them login using their favourite email provider such as Google or Yahoo (The Nascar UI is a reference to the logos that are plastered over the race cars).  Most users won’t have a clue what their OpenId provider (OP) URL is , but they can spot the logo of their email provider a mile away.

Too add a slicker login interface, I grabbed a copy Jarret Vance’s OpenId JQuery Plugin.   Lift uses JQuery by default, so integration of Jarret’s plugin is quite easy.  The plugin consists of provider logos (put these in your /webapp/images directory), a Javascript JQuery extension (jquery-openid.js – put this in your webapp/scripts directory), and some html templates for the login page.

I created a Lift template for the login page and placed it under my webapp/templates-hidden directory. The template is pretty much a copy of the default template, but adds some javascript for the openid integration. In the head tag of the template include the following:

<script id="jquery" src="/classpath/jquery.js" type="text/javascript"/>
 <script id="json" src="/classpath/json.js" type="text/javascript" />
 <script id="openid" src="/scripts/openid-jquery.js"  type="text/javascript"/>
 <script type="text/javascript">

 $(document).ready(function() {
 openid.init('openIdUrl');
 });
 </script>

Note that Lift performs “head” merging – which is to say that any content withing <head> tags in a snippet is merged in with the templates head tag. So the above approach is not strictly necessary.

Now that we have a login template, we can define the login page. Most of the content is copied verbatim from Jaret’s plugin

<lift:surround with="openid" at="content">
 <head>
 <title>
 <lift:loc>Login</lift:loc>
 </title>
 </head>
 <h2>
 <lift:loc locid="Welcome">Login Page</lift:loc>
 </h2>
<p>This site uses <a href="openid.net">OpenId</a> so you don't have to remember yet-another password.
 <br/>Enter your OpenId URL, or click one one of the icons below to sign on with an existing account.
 </p>
 <!-- Simple OpenID Selector -->
 <form action="/openid/login" method="post" id="openid_form">
 <div>
 <ul>
 <li title="OpenID"><img src="/images/openidW.png" alt="icon"/>
 <span><strong>http://{your-openid-url}</strong></span></li>
 <li title="Google">
 <img src="/images/googleW.png" alt="icon"/><span>https://www.google.com/accounts/o8/id</span></li>
 <li title="Yahoo">
 <img src="/images/yahooW.png" alt="icon"/><span>http://yahoo.com/</span></li>
 <li title="AOL screen name">
 <img src="/images/aolW.png" alt="icon"/><span>http://openid.aol.com/<strong>username</strong></span>
 </li>
 </ul>
 </div>
 <fieldset>
 <label for="openid_username">Enter your <span>Provider user name</span></label>
 <div><span></span><input type="text" name="openid_username"/><span></span>
 <input type="submit" value="Login"/></div>
 </fieldset>
 <fieldset>
 <label for="openid_identifier">Enter your <a href="http://openid.net">OpenID</a></label>
 <div><input type="text" name="openid_identifier" id="openid_identifier"/>
 <input type="submit" value="Login"/></div>
 </fieldset>
 </form>
</lift:surround>

For brevity I have removed some of the provider logos from the example. The nice thing about this plugin is that it is very easy to add or remove providers.

This gives us a nice “Nascar” login page that looks like this:

Now this will make both the geeks and grandma happy.

In the next blog entry (when I get around to it…) I will describe how to add support for the OpenId attribute exchange protocol (AX). This enables our application to retrieve additional user attributes from the provider such as the user’s email address, screen name, preferred language, etc..  Stay tuned.

I’ve had the chance over a recent weekend to have a first crack at a web framework called Vaadin.

I was originally browsing for news about the latest release of Google’s GWT framework when I stumbled on a reference to Vaadin, and decided to go take a look. What I found intrigued me, and I decided to take it for a test drive, as I was down sick for a couple of days with a laptop nearby…. My back became annoyed with me, but it was worth it, I think.

First Look
First off, the practicalities: Vaadin is open source, and with a reasonable license, the Apache License. The essential bits of Vaadin are contained in a single JAR, and it’s both Ant and Maven friendly right out of the box.

The next thing that struck me about Vaadin was the documentation. The first unusual thing about it’s documentation was the fact of it’s existence, as open source projects are downright notorious for poor documentation. Vaadin is a pleasant exception, with tons of examples, a well-organized API doc, in the usual JavaDoc format, and even the “Book of Vaadin”, an entire PDF book (also available in hardcopy) that takes you through Vaadin in enough detail to be immediately productive.

Given that surprisingly pleasant start, I dug deeper, creating a little app of my own.

Just the Java, Ma’am
The main thing that kept me interested in Vaadin once I started digging further was that it’s pure Java. Many frameworks talk about writing your UI work in Java, such as Wicket, but there’s still a corresponding template and some wiring that happens to put the code and the template together. Not so with Vaadin.

When they say “just Java”, they mean it – your entire UI layer is coded in Java, plain and simple. No templates, no tag libraries, no Javascript, no ‘nuthin. It’s reminiscent of the Echo framework, except in Vaadin’s case the Javascript library that your code automatically produces is Google’s GWT, instead of Echo’s own Core.JS library.

Unlike GWT, though, the Vaadin approach doesn’t bind you to any specific source code though, it’s just a binary jar you put on your classpath.

The only thing in my sample app, other than 2 Java files, was a web.xml and a css stylesheet, both of which were only a few lines long. And this was no “Hello, World”, either, but a rich AJAX webapp with a tree menu, fancy non-modal “fading” notifications, images, complex layouts, and a form with build-in validation. And it took maybe 4 hours of total work to produce – and that was from a standing start, as I’d never heard of Vaadin before last Thursday. Not bad, not bad at all.

I found I was able to get a very capable little webapp up and running with no need to invent my own components, even though I had trees and sliders and menus and other assorted goodies on the page. It worked in every browser I was able to try it in, which is certainly not the case for my own hand-rolled JavaScript most of the time

I haven’t yet tried creating my own custom components, but it certainly looks straightforward enough.

I did try linking to external resources, and included non-Vaadin pages in my app, with no difficulties, so it appears that Vaadin plays well with others, and can be introduced into an existing project that uses, for instance, a whack of JSP’s that one might want to obsolete.

Webapps
I think Vaadin warrants more exploration, and I intend to put it further through its paces in the next few weeks. It appears extremely well-suited to web applications, as opposed to websites with a tiny bit of dynamic stuff in them.

It offers an interesting alternative to some of the patterns I’ve seen for advanced dynamic webapp development so far.

One approach I’ve seen a lot is to divide the duties of creating an app into the “back end” services and the “UI”. Generally the UI is written in either JavaScript, or uses Flex or some other semi-proprietary approach. The “back end” stuff is frequently written to expose it’s services as REST, then the two are bolted together. The pain point here happens when the two meet, as it’s common and easy to have minor (or major!) misunderstandings between the two teams. This usually results in a lot of to-and-fro to work out the differences before the app comes all the way to life.

The other approach, more common on smaller or resource-strapped teams, is to have the same group responsible for both UI and back-end services. This reduces the thrash in the joints a bit, but doesn’t eliminate it, because the two technologies on the two sides of the app aren’t the same. You can’t test JavaScript the same way you write Java, for instance, and they’re two different languages – one of which (Java) has far better tooling support than the other. IDE support, for instance, is superb for Java, and spotty at best for JavaScript.

With Vaadin, both of these approaches become unnecessary, as its the same technology all the way through (at least, what you write is – technically it’s still using JavaScript, but because that’s generated, I don’t count it).

You get to use all of the tools you know and love for the back-end services to write the code for the UI, which you can then unit and functional test to your heart’s content.

The temptation to mix concerns between UI code and back-end service code must still be resisted, of course, but at least that code isn’t buried somewhere in the middle of a JSP page, ready to leap out and bite you later.

Because you’re using dynamic layouts, the app always fits properly on the screen without any extra work, addressing a pet peeve of mine, the “skinny” webapp, restraining itself to the least common denominator of screen size, thus rendering impotent my nice wide monitors.

Scala
Just because Vaadin is a Java library doesn’t restrict you to using Java to drive it, however. I made another little webapp where the whole UI was defined in Scala, calling the Vaadin APIs, and it worked like a charm. In some ways, Scala is an even better fit for Vaadin than straight Java, I suspect. I haven’t tried any other JVM compatible language, but I see no reason they wouldn’t work equally well.

Deployment and Development Cycle
As I was building the app with Maven, I added a couple of lines to my POM and was able to say “mvn jetty:run” to get my Vaadin app up and running on my local box in a few seconds. My development cycle was only a few seconds between compile and interactive tests, as I was experimenting with the trial-and-error method.

TDD would be not only possible, but easy in this situation.

I successfully deployed my little Vaadin app to ServiceMix, my OSGi container of choice, without a hitch.

Performance appeared excellent overall, although I haven’t formally tested it with a load-testing tool (yet).

Summary
So far, I’m impressed with Vaadin. I’m more impressed with any web framework I’ve worked with in a number of years, in fact. I’m sure there are some warts in there somewhere, but for the benefits it brings to the table, I suspect they’re easily worth it. I think the advantages to teams that already speak fluent Java is hard to overstate, and the productivity to produce good-looking and functioning webapps is quite remarkable.

Over the next few weeks I’ll push it a bit harder with a complex example application, and see how it stacks up against other web app technologies I’ve worked with in a more realistic scenario.

By: Mike Nash

スケーラブルプログラミングp.219から少し。

1「ふるまいが再利用されないような場合」なら、具象クラスにする。

2「複数の無関係なクラスで再利用される可能性がある」なら、トレイトにする。

3「Javaコードを継承できるようにしたい場合」は、抽象クラスを使う。

4「ライブラリー」をコンパイル後の形態で配布したい場合」には、抽象クラスを使った方が良いかもしれない。(外部でトレイトを継承しているクラスを使うと、トレイトのメンバー獲得／喪失時に外部側の再コンパイルが必要になる。)

5「処理効率が非常に重視される場合」には、クラスを使った方が良いかもしれない。(Javaはインタフェースよりクラスの方が速い。トレイトはインタフェースとしてコンパイルされるので、ややオーバーヘッドがかかる)

6「以上を検討してみて、まだどちらが良いかはっきりしない場合」、まずトレイとにしてみるのが良いであろう。

個人的に特に気をつけないとなと思ったのは、4でした。トレイトの使い方によっては
、アプリの組み方ががらっと変わってきそうですね。

Here’s a three-for-one special for you: A post about implementing the Levenshtein string distance algorithm in Scala AND refactoring it from an imperative style to a functional style AND I even throw in a short lesson on memoization. To make sure that our refactoring is correct and preserves the expected behavior, I’ll unit test the code along the way using ScalaTest. ScalaCheck, JUnit, or TestNG would work just as well, but I used ScalaTest.

First Things First

“What, exactly,” some of you may be asking, “is Levenshtein string distance? Some kind of Teutonic tailoring terminology?” Not at all. It’s a way of measuring how alike or different two strings of symbols are. For example, the string “sturgeon” is a lot more similar to “surgeon” than to “urgently”. “Sturgeon” and “surgeon” are only a single letter (t) apart. They have a string distance of 1. “Sturgeon” and “urgently” share some letters, but each has some letters not present in the other. So what’s their string distance? It’s not so obvious now.

String distance is useful. The use that most quickly springs to mind is spelling correction. If I type “computwr” a string distance algoritm could tell us that “computwr” is very close to the dictionary word “computer”. But there’s more to it than that. There are a lot of fuzzy problems in which you want to find which two sets of complex data are most similar. One way to solve a problem like that is to encode it into a string compare using string distance. For example, you could write a program to find pen strokes in an image and encode their shapes (up, curve left, down, angle right, etc) as a string which could be compared to known encodings for handwritten letters. By finding the closest matches you can create a handwriting recognition program. String distance is useful in DNA analysis, of course, recognizing patterns in signals, and a host of other situations.

The Levenshtein string distance algorithm was developed by Vladimir Levenshtein in 1965. It can easily tell us the distance from “sturgeon” to “urgently”. This algorithm breaks down string transformation into three basic operations: adding a character, deleting a character, and replacing a character. Each of these operations is assigned a cost, usually a cost of 1 for each operation. Leaving a character unchanged has a cost of 0. So to go from “sturgeon” to “surgeon” you leave the “s” unchanged for a cost of 0. Then you add a “t” for a cost of 1. All the other letters also remained unchanged, so the total cost is 1, just as we expected.

To change “sturgeon” to “urgently” is harder. They have the same number of letters, so we could just do a replacement on each one for a distance of 8. But is that the shortest distance? What if we try to re-use that “urge” from “sturgeon”? Can we re-use the “n”? Does that help? What about the “t”? We need an algorithm that we can follow.

The Grid

What we need is a way to find the cheapest combination of operations which changes the first string to the second. That’s the Levenshtein algorithm. It works like this. Write the first string vertically from top to bottom. To the right of each letter write 1, 2, 3, etc. Write a 0 above the number 1. Then write the second string horizontally and again add the numbers 1, 2, 3 etc. to the right of the 0. It will look something like this:

    u r g e n t l y
  0 1 2 3 4 5 6 7 8
s 1
t 2
u 3
r 4
g 5
e 6
o 7
n 8

That’s the first step. The grid remains un-filled-in at this point. Can you guess what the grid positions are going to hold? They are going to hold the cost to convert the various prefixes of “sturgeon” to the various prefixes of “urgently”. The position at the intersection of row r4, and the column a2, for example, will contain the cost to convert “stur” to “ur”. We fill in these positions (left-to-right then top to bottom) with the smallest of the following three numbers:

• The number above the current position plus one.
• The number to the left of the current position plus one.
• The above-left number if row letter and column letter are the same, or the above-left number plus one otherwise.

When we finish, the bottom right corner contains the cost to convert the first string to the second.

But Why? (Understanding the Algorithm)

Those are some shockingly simple rules! Let’s examine how they translate into string distance.

First, what are those numbers 0 to n that we write along the top and left? They’re not just indices. The numbers along the top represent the cost to convert the empty string to the various prefixes of “urgently”. The cost is 0 to convert “” to “”, 1 to convert “” to “u”, 2 to convert “” to “ur”, and so on. The numbers on the left are the cost to convert the prefixes of “sturgeon” to the the empty string . “” to “” is 0, “s” to “” is 1, and so forth. These costs are obvious. The only way to convert “” to “urgently” is to add eight letters. There’s nothing to delete, nothing to replace. The only way to convert “sturgeon” to “” is to delete all eight letters.

Each position, as we have established, represents the cost to convert the string of characters down the left side ending in the current row’s character into the the string of characters along the top ending at the current column’s character. Put another way, for any given position let’s call the current row’s letter A, and the current column’s letter B. If we use a colon (:) to indicate string concatenation then the beginning string, the one along the left of the grid, can be written Prefix1:A. So for our example word “sturgeon” if we look at a position in row e6 then Prefix1 is “sturg” and the final letter, which we’re calling A, is “e”.

So, speaking in terms of Prefix1, Prefix2, A, and B, we use the following inputs:

• The cost to change Prefix1 to Prefix2:B (the number above the current position).
• The cost to change Prefix1:A to Prefix2 (the number to the left).
• The cost to change Prefix1 to Prefix2 (the above-left number).

If we know the costs of these three conversions we can find the cost to change Prefix1:A to Prefix2:B using this logic:

• We know that if converting Prefix1 to Prefix2:B has a cost of X, then Prefix1:A can be converted to Prefix2:B for the cost of X plus the cost of deleting A, or X + 1.
• We know that if converting Prefix1:A to Prefix2 has a cost of Y, then Prefix1:A can be converted to Prefix2:B for the cost of Y plus the cost of adding B, or Y + 1.
• We know that if converting Prefix1 to Prefix2 has a cost of Z, then Prefix1:A can be converted to Prefix2:B for the cost of Z plus the cost of replacing A with B, or Z + (0 or 1 depending on whether A = B).

If you understand the above logic, then you understand this really neat algorithm. It’s a quintessential example of dynamic programming. The solution is built up by solving simpler problems. You start with the trivial case of converting to and from the empty string, and then you build up the solution for the prefixes until you have the complete solution.

In the grid’s initial configuration there is only one location for which we know all three costs and that is row s1, column u1. That’s the only space for which all three neighboring values (above, left, and above-left) are populated. After we fill in this one, there are two more spaces available to us. Those are (t2, u1) and (s1, r2). Ordinarily, though, the spaces are populated line by line.

A Simple Example

Let’s do a quick example by hand. Then we’ll take a stab at implementing it in code. What’s the string distance from “hat” to “tape”? First, our empty grid:

    t a p e
  0 1 2 3 4
h 1
a 2
t 3

The first space is row h, column t. The letters are different so our choices are 1 + 1 (above), 1 + 1 (left), or 0 + 1 (above-left). 0 + 1 is the smallest value, so the first space gets populated with 1. The next space has choices 2 + 1 (above), 1 + 1 (left), or 1 + 1 (above-left). 1 + 1 is the lowest, so we fill in the second space with 2. Once we finish the row, we have this:

    t a p e
  0 1 2 3 4
h 1 1 2 3 4
a 2
t 3

The next space is row a, column t. Our choices are 1 + 1 (above), 2 + 1 (left), or 1 + 1 (above-left). 1 + 1 is the smallest value, so the space gets populated with 2. The next space has choices 2 + 1 (above), 2 + 1 (left), or 1 + 0 (above-left). Why 1 + 0? Because the above-left value is 1 and both letters for this space are “a” so we can replace “a” with “a” for free. Go ahead and finish the whole grid. This is the result:

    t a p e
  0 1 2 3 4
h 1 1 2 3 4
a 2 2 1 2 3
t 3 2 2 2 3

The strings “hat” and “tape” have a distance of 3.

Some Code, Finally

As fascinating as Levenshtein distance is, and as much more as there is to say on the topic, the time has come to write some code. Here’s a Scala implementation that is very close to the pencil-and-paper approach that we just performed.

import scala.Math.min
import scala.Math.max

object StringDistance {
  def stringDistance(s1: String, s2: String): Int = {
    def minimum(i1: Int, i2: Int, i3: Int) = min(min(i1, i2), i3)

    val dist = new Array[Array[Int]](s1.length + 1, s2.length + 1)

    for (idx <- 0 to s1.length) dist(idx)(0) = idx
    for (jdx <- 0 to s2.length) dist(0)(jdx) = jdx

    for (idx <- 1 to s1.length; jdx <- 1 to s2.length)
      dist(idx)(jdx) = minimum (
        dist(idx-1)(jdx  ) + 1,
        dist(idx  )(jdx-1) + 1,
        dist(idx-1)(jdx-1) + (if (s1(idx-1) == s2(jdx-1)) 0 else 1)
      )

    dist(s1.length)(s2.length)
  }

Do you see what I mean when I say it’s close to the pencil-and-paper approach? We actually construct a two-dimensional Array to represent the grid we drew earlier. It’s a very literal implementation.

To explain the code briefly, we declare a singleton StringDistance having a single method called stringDistance. Within this method we declare a 3-argument minimum method. (I wonder why there’s no “def min(params: Int*): Int” defined in scala.Math.) Then we create an array called “dist” and populate the top row and leftmost column in lines 8-11. The for loop on line 13 cycles through each array position from left to right then top to bottom, and populates them according to the Levenshtein logic. Finally, once the grid is filled in we return the number in the bottom right position.

The job’s not done yet, of course. We haven’t written our unit tests. Here’s the test I wrote:

import org.scalatest._

class StringDistanceSuite extends FunSuite with PrivateMethodTester {

  test("stringDistance should work on empty strings") {
    assert( StringDistance.stringDistance(   "",    "") == 0 )
    assert( StringDistance.stringDistance(  "a",    "") == 1 )
    assert( StringDistance.stringDistance(   "",   "a") == 1 )
    assert( StringDistance.stringDistance("abc",    "") == 3 )
    assert( StringDistance.stringDistance(   "", "abc") == 3 )
  }

  test("stringDistance should work equal strings") {
    assert( StringDistance.stringDistance(   "",    "") == 0 )
    assert( StringDistance.stringDistance(  "a",   "a") == 0 )
    assert( StringDistance.stringDistance("abc", "abc") == 0 )
  }

  test("stringDistance should work where only inserts are needed") {
    assert( StringDistance.stringDistance(   "",   "a") == 1 )
    assert( StringDistance.stringDistance(  "a",  "ab") == 1 )
    assert( StringDistance.stringDistance(  "b",  "ab") == 1 )
    assert( StringDistance.stringDistance( "ac", "abc") == 1 )
    assert( StringDistance.stringDistance("abcdefg", "xabxcdxxefxgx") == 6 )
  }

  test("stringDistance should work where only deletes are needed") {
    assert( StringDistance.stringDistance(  "a",    "") == 1 )
    assert( StringDistance.stringDistance( "ab",   "a") == 1 )
    assert( StringDistance.stringDistance( "ab",   "b") == 1 )
    assert( StringDistance.stringDistance("abc",  "ac") == 1 )
    assert( StringDistance.stringDistance("xabxcdxxefxgx", "abcdefg") == 6 )
  }

  test("stringDistance should work where only substitutions are needed") {
    assert( StringDistance.stringDistance(  "a",   "b") == 1 )
    assert( StringDistance.stringDistance( "ab",  "ac") == 1 )
    assert( StringDistance.stringDistance( "ac",  "bc") == 1 )
    assert( StringDistance.stringDistance("abc", "axc") == 1 )
    assert( StringDistance.stringDistance("xabxcdxxefxgx", "1ab2cd34ef5g6") == 6 )
  }

  test("stringDistance should work many operations are needed") {
    assert( StringDistance.stringDistance("example", "samples") == 3 )
    assert( StringDistance.stringDistance("sturgeon", "urgently") == 6 )
    assert( StringDistance.stringDistance("levenshtein", "frankenstein") == 6 )
    assert( StringDistance.stringDistance("distance", "difference") == 5 )
    assert( StringDistance.stringDistance("java was neat", "scala is great") == 7 )
  }

}

It tests several special cases as well as the general case. All we have to do is compile our StringDistance object and the StringDistanceSuite unit test, fire up the scala interpreter and run the test:

scala> (new StringDistanceSuite).execute()
Test Starting - StringDistanceSuite: recursiveStringDistance should work on empty strings
Test Succeeded - StringDistanceSuite: recursiveStringDistance should work on empty strings
Test Starting - StringDistanceSuite: recursiveStringDistance should work on equal strings
Test Succeeded - StringDistanceSuite: recursiveStringDistance should work on equal strings
Test Starting - StringDistanceSuite: recursiveStringDistance should work where only inserts are needed
Test Succeeded - StringDistanceSuite: recursiveStringDistance should work where only inserts are needed
Test Starting - StringDistanceSuite: recursiveStringDistance should work where only deletes are needed
Test Succeeded - StringDistanceSuite: recursiveStringDistance should work where only deletes are needed
Test Starting - StringDistanceSuite: recursiveStringDistance should work where only substitutions are needed
Test Succeeded - StringDistanceSuite: recursiveStringDistance should work where only substitutions are needed
Test Starting - StringDistanceSuite: stringDistance should work many operations are needed
Test Succeeded - StringDistanceSuite: stringDistance should work many operations are needed

scala>

Refactoring: Reduced Memory Usage

The code passes all the tests, so let’s take things a step further. One shortcoming of our implementation is that it can require a lot of memory. That array has to be of size (n+1)*(m+1) where n and m are the lengths of the two strings we’re comparing. If you want to apply the method to strings that are a few hundred characters long (or longer) then you’re starting to talk about some serious memory requirements. How can we reduce the amount of memory required? Can you think of a way?

Once we complete one row of the grid, we use it again as an input when we compute the next row. But after that we’re done with it. Why leave it to clutter the heap? Let’s tweak our implementation slightly. Try rewriting the method using only two rows. Fill the first row with the initial 0, 1, 2, etc. Then use one to compute the other over and over. Think about how you would implement this, then have a look at my solution below.

  def stringDistance(s1: String, s2: String): Int = {
    def minimum(i1: Int, i2: Int, i3: Int) = min(min(i1, i2), i3)

    var dist = ( new Array[Int](s1.length + 1),
                 new Array[Int](s1.length + 1) )

    for (idx <- 0 to s1.length) dist._2(idx) = idx

    for (jdx <- 1 to s2.length) {
      val (newDist, oldDist) = dist
      newDist(0) = jdx
      for (idx <- 1 to s1.length) {
        newDist(idx) = minimum (
          oldDist(idx) + 1,
          newDist(idx-1) + 1,
          oldDist(idx-1) + (if (s1(idx-1) == s2(jdx-1)) 0 else 1)
        )
      }
      dist = dist.swap
    }

    dist._2(s1.length)
  }

This one uses a Pair (also called a Tuple2) containing two one-dimensional arrays, instead of a 2*(n+1) array. Pair happens to have the very handy “swap” method which we can use to trade out the rows when we’ve finished one and are ready to compute the next.

This is where our unit tests really show their worth. No need to wonder whether this new code really does work. We just recompile, run the tests again, and we can see that the code still gives us the expected results.

Refactoring: Recursion, Kind Of

What are some other ways we could write this code? Can it be improved? I thought I would try to replace the iteration in the previous implementations with recursion. Here’s what that code looks like:

  def stringDistance(s1: String, s2: String): Int = {
    def newCost(ins: Int, del: Int, subst: Int, c1: Char, c2:Char) =
      Math.min( Math.min( ins+1, del+1 ), subst + (if (c1 == c2) 0 else 1) )

    def getNewCosts(s1: List[Char], c2: Char, delVal: Int, prev: List[Int] ): List[Int] = (s1, prev) match {
      case (c1 :: _ , substVal :: insVal :: _) =>
        delVal :: getNewCosts(s1.tail, c2, newCost(insVal, delVal, substVal, c1, c2), prev.tail)
      case _ => List(delVal)
    }

    def sd(s1: List[Char], s2: List[Char], prev: List[Int]): Int = s2 match {
      case Nil => prev.last
      case _ => sd( s1, s2.tail, getNewCosts(s1, s2.head, prev.head+1, prev) )
    }

    (s1, s2) match {
      case (`s2`, `s1`) => 0
      case (_, "") | ("", _) => max(s1.length, s2.length)
      case _ => sd(s1.toList, s2.toList, (0 to s1.length).toList)
    }
  }

It’s a pretty naïve implementation, actually. It just replaces the the repetition of the two for loops with the repetition of the recursion of the two methods sd and getNewCosts. The sd method is even tail-recursive, allowing scala to optimize it. It does the same basic thing as the for loop version, though. It recurses through the characters of a row in the getNewCosts method, and it recurses through the rows of the grid in the sd method.

It looks more complicated than the previous implementations. It’s harder to read. But it passes our unit tests, so we can be pretty sure it’s correct.

Refactoring: List Methods

After the last implementation, I thought it looked a little sloppy. I wondered whether I could improve the situation by using some of the many useful methods built into scala’s List class. Here is the comparatively brief code that resulted:

  def stringDistance(s1: String, s2: String): Int = {
    def sd(s1: List[Char], s2: List[Char], costs: List[Int]): Int = s2 match {
      case Nil => costs.last
      case c2 :: tail => sd( s1, tail,
          (List(costs.head+1) /: costs.zip(costs.tail).zip(s1))((a,b) => b match {
            case ((rep,ins), chr) => Math.min( Math.min( ins+1, a.head+1 ), rep + (if (chr==c2) 0 else 1) ) :: a
          }).reverse
        )
    }
    sd(s1.toList, s2.toList, (0 to s1.length).toList)
  }

Like I say, it’s brief. Those List methods give you a lot of mileage.

Refactoring: Real Recursion

The more I looked at my previous attempt at a recursive solution, the more I realized how hare-brained it was. It wasn’t real recursion. It was just iteration using the stack. So I went back to the drawing board. If I want to know the final answer, the value in the bottom right position, how do I get it? I apply my three rules to the above, left, and above-left positions. How do I get those positions? Apply the rules again. That’s real recursion. Here’s my first stab at it:

  def stringDistance(s1: String, s2: String): Int = {
    def min(a:Int, b:Int, c:Int) = Math.min( Math.min( a, b ), c)
    def sd(s1: List[Char], s2: List[Char]): Int = (s1, s2) match {
      case (_, Nil) => s1.length
      case (Nil, _) => s2.length
      case (c1::t1, c2::t2)  => min( sd(t1,s2) + 1, sd(s1,t2) + 1,
                                     sd(t1,t2) + (if (c1==c2) 0 else 1) )
    }
    sd( s1.toList, s2.toList )
  }

Now, THAT is a nice looking recursive function. That’s more like it. See the pattern match block? If we try to convert from any string to the empty string or from the empty to a non-empty string then we just use the string length. You see? That takes care of the positions along the top and left of the grid. All the others are determined in the last case. That last case just applies our three rules. That is so short and simple. It’s a thing of beauty.

The only problem? It doesn’t work.

It’s technically correct. It will return correct answers … eventually. Or, rather, I think it will. I can’t be sure because it’s too slow to complete my unit tests! For anything but very short inputs (4 or 5 characters), the function takes a long time to return. Why? Let’s look at how many recursive calls are made for some inputs.

If we pass use strings “a” and “b” we pass by the “(_, Nil)” and “(Nil, _)” cases in the first call to function sd, because both our strings (Lists, actually) are non-empty. This results in three more calls to sd. Each of these three calls includes an empty List of characters, so there is no more recursion. That’s a total of four calls to sd for strings “a” and “b”.

What about “ab” and “xy”? Think about it for a moment? Step through the function in your head. How many calls to sd will there be for “ab” and “xy”?

Have you done it? I count 19. What about “abc” and “xyz”? I’ll save you the trouble. It’s 94. For length 4 strings it’s 481. For length 5 it’s 2524. Length 6 is 13,483. Then 73 thousand, then 400 thousand, then 2 million and so forth. Why so many calls? Each position in the grid is computed using all the positions to the left and all the positions above the current position. So a position in the top left will be computed and recomputed many times.

There is a way to get around this, of course. You probably already have some ideas. We’re going to do something called memoization. When you memoize a function, you make it remember results that it computed previously without having to actually recompute them. I’ll do that by caching results in a map. The map’s key is a Pair of List[Char]s, the inputs to my inner function sd, and its data is an Int, the return type of sd. I will modify sd to first check the map to see if the result for the current parameters has already been cached. If so, we simply return it. If not, we compute the value, cache it, and return it.

  def stringDistance(s1: String, s2: String): Int = {
    val memo = scala.collection.mutable.Map[(List[Char],List[Char]),Int]()
    def min(a:Int, b:Int, c:Int) = Math.min( Math.min( a, b ), c)
    def sd(s1: List[Char], s2: List[Char]): Int = {
      if (memo.contains((s1,s2)) == false)
        memo((s1,s2)) = (s1, s2) match {
          case (_, Nil) => s1.length
          case (Nil, _) => s2.length
          case (c1::t1, c2::t2)  => min( sd(t1,s2) + 1, sd(s1,t2) + 1,
                                         sd(t1,t2) + (if (c1==c2) 0 else 1) )
        }
      memo((s1,s2))
    }

    sd( s1.toList, s2.toList )
  }
}

There. That uglies up my function somewhat, but at least it’s usable now. And it passes my unit tests, so I’m reasonably assured that it’s right.

Memoization only works if you expect the same result for each identical function call. If your function takes input from stdin, for example, you can’t memoize that. Or if it has a random component. Or if, for any other reason, its return value is not always the same for the same inputs. You can memoize functions in different ways. There’s a post on Michid’s Weblog about a more general solution, a memoizing class which wraps existing functions to give you a memoized version.

PHEW!

Ok, I’ve tried to keep my rambling to a minimum in this post but it’s still a doozy. The things I wanted to get across are:

• The usefulness of Levenshtein distance in solving a variety of problems.
• How to understand the Levenshtein distance algorithm and why it works.
• How to use unit tests to improve your code while maintaining some assurance that the new code still has the correct behavior.
• Some of the different ways of implementing Levenshtein.

In the end, I think I like the second implementation (the one that switches out the two rows) and the last implementation the best. The second one seems to have good performance. I did some informal performance tests and it has a good mix of performance and simplicity. The last one, the memoized recursive one, appeals to me because it is in a more functional style and still has respectable performance.

Don’t forget to subscribe to my RSS feed, or follow this blog on Twitter.
Copyright © 2009 Matthew Jason Malone

Following on from my post about Friday Feedback Frenzies, Mark, Ben and I looked at how to work out frenzy pairs. The idea with the frenzies is that you exchange feedback with two people each week, and that you rotate the people that you do this with. When I have organised these previously I have generally just picked people at random. Mark, Ben and I decided to generate a frenzy fixture (if you will).

Mark came up with the idea of using something like a football/tennis game scheduling algorithm. We found a Round Robin cyclic algorithm to get us started. Our first step was to get this algorithm working in Scala:

      val numberOfPlayers = 12
      for (round <- 1 to 11) {
        for (court <- 1 to 6) {
          print((if(court == 1) 1 else (round + court - 2) % (numberOfPlayers - 1) + 2),
                (numberOfPlayers - 1 + round - court) % (numberOfPlayers - 1) + 2)
        }
        println()
      }

The output from this is below. Each row represents a round and each bracketed pair in a row represents a match-up of two players on a court.

(1,2)(3,12)(4,11)(5,10)(6,9)(7,8)
(1,3)(4,2)(5,12)(6,11)(7,10)(8,9)
(1,4)(5,3)(6,2)(7,12)(8,11)(9,10)
(1,5)(6,4)(7,3)(8,2)(9,12)(10,11)
(1,6)(7,5)(8,4)(9,3)(10,2)(11,12)
(1,7)(8,6)(9,5)(10,4)(11,3)(12,2)
(1,8)(9,7)(10,6)(11,5)(12,4)(2,3)
(1,9)(10,8)(11,7)(12,6)(2,5)(3,4)
(1,10)(11,9)(12,8)(2,7)(3,6)(4,5)
(1,11)(12,10)(2,9)(3,8)(4,7)(5,6)
(1,12)(2,11)(3,10)(4,9)(5,8)(6,7)

Next we moved from everything being 1 indexed to being 0 indexed. We also changed the fixed player, who always plays on court 0, to be the last one in the player list. This was done only because it made the loops more readable:

      val numberOfPlayers = 12
      for (round <- 0 to 10) {
        for (court <- 0 to numberOfPlayers/2 - 1) {
          print(((if(court == 0) numberOfPlayers - 1 else (court + round) % (numberOfPlayers - 1))),
                (((numberOfPlayers - 1) - court + round) % (numberOfPlayers - 1)))
        }
        println()
      }

Then the output was:

(11,0)(1,10)(2,9)(3,8)(4,7)(5,6)
(11,1)(2,0)(3,10)(4,9)(5,8)(6,7)
(11,2)(3,1)(4,0)(5,10)(6,9)(7,8)
(11,3)(4,2)(5,1)(6,0)(7,10)(8,9)
(11,4)(5,3)(6,2)(7,1)(8,0)(9,10)
(11,5)(6,4)(7,3)(8,2)(9,1)(10,0)
(11,6)(7,5)(8,4)(9,3)(10,2)(0,1)
(11,7)(8,6)(9,5)(10,4)(0,3)(1,2)
(11,8)(9,7)(10,6)(0,5)(1,4)(2,3)
(11,9)(10,8)(0,7)(1,6)(2,5)(3,4)
(11,10)(0,9)(1,8)(2,7)(3,6)(4,5)

Our next step was to switch domains from sports to feedback frenzies:

val participants = 12
val weeks = 10
for (frenzy <- 0 until weeks) {
  for (pair <- 0 until participants/2) {
    print(((if(pair == 0) participants - 1 else (pair + frenzy) % (participants - 1))),
          (((participants - 1) - pair + frenzy) % (participants - 1)))
  }
  println()
}

Then we changed to creating a collection of all the pairs:

      val participants = 12
      val weeks = 10
      val frenzyPairs = for {
        frenzy <- 0 until weeks
        pair <- 0 until participants/2
      } yield (((if(pair == 0) participants - 1 else (pair + frenzy) % (participants - 1))),
                (((participants - 1) - pair + frenzy) % (participants - 1)))
      frenzyPairs.foreach(print)
    }

And the output…

RangeM((11,0), (1,10), (2,9), (3,8), (4,7), (5,6))
RangeM((11,1), (2,0), (3,10), (4,9), (5,8), (6,7))
RangeM((11,2), (3,1), (4,0), (5,10), (6,9), (7,8))
RangeM((11,3), (4,2), (5,1), (6,0), (7,10), (8,9))
RangeM((11,4), (5,3), (6,2), (7,1), (8,0), (9,10))
RangeM((11,5), (6,4), (7,3), (8,2), (9,1), (10,0))
RangeM((11,6), (7,5), (8,4), (9,3), (10,2), (0,1))
RangeM((11,7), (8,6), (9,5), (10,4), (0,3), (1,2))
RangeM((11,8), (9,7), (10,6), (0,5), (1,4), (2,3))
RangeM((11,9), (10,8), (0,7), (1,6), (2,5), (3,4))

As Ben quite rightly pointed out if we take two rows a time, then each person will exchange feedback with two people in the same frenzy. Our final step was to group each pair of successive rows. To do this we first moved to having two nested for loops. We thought that this would make it easier to do the grouping, but it ended up being redundant. Here our final version of the frenzy fixture generator:

      val participants = 12
      val rounds = 10
      val frenzyPairs = for (frenzy <- 0 until rounds)
                      yield for(pair <- 0 until participants/2)
                            yield (((if(pair == 0) participants - 1 else (pair + frenzy) % (participants - 1))),
                                  (((participants - 1) - pair + frenzy) % (participants - 1)))

      val (evenOnes,oddOnes) = frenzyPairs.toList.zipWithIndex.partition(_._2%2 == 0)
      evenOnes.zip(oddOnes).foreach(println)

Our final output was:

((RangeM((11,0), (1,10), (2,9), (3,8), (4,7), (5,6)),0),(RangeM((11,1), (2,0), (3,10), (4,9), (5,8), (6,7)),1))
((RangeM((11,2), (3,1), (4,0), (5,10), (6,9), (7,8)),2),(RangeM((11,3), (4,2), (5,1), (6,0), (7,10), (8,9)),3))
((RangeM((11,4), (5,3), (6,2), (7,1), (8,0), (9,10)),4),(RangeM((11,5), (6,4), (7,3), (8,2), (9,1), (10,0)),5))
((RangeM((11,6), (7,5), (8,4), (9,3), (10,2), (0,1)),6),(RangeM((11,7), (8,6), (9,5), (10,4), (0,3), (1,2)),7))
((RangeM((11,8), (9,7), (10,6), (0,5), (1,4), (2,3)),8),(RangeM((11,9), (10,8), (0,7), (1,6), (2,5), (3,4)),9))

The end result could be tidied up a little but it does provide us with a frenzy fixture for a given number of participants. Given more time we would have used names for participants rather than just numbers.

Before Sam Francisco three piece The Dodos played, North Carolinian three piece MEGAFAUN played (no reason for the shouty caps other than it makes their name even cooler than it all ready is).

They all had good facial hair and their music was just as great. They were cheerful, endearing, energetic… I would most certainly recommend catching them if you can. They have the harmonies of Fleet Foxes but with some real foot stomping fun. For one song they departed from the stage (all three of them!) and did a little dance. It was brilliant. Check them out here.

Then it was time for The Dodos. I had to miss them at End of the Road due to a clash with Neko Case (who was incredible) and in early September because I was at The Antlers (who were incredible) but I had seen them a year earlier supporting Okkervil River. I really enjoyed this particular show at Scala; Meric Long, the singer, puts so much energy into his performance and drummer Logan Kroeber, was fantastic. There was a especially lovely introduction to Fools using bows against the vibraphone. The crowd loved it and they returned for two encores… the place looked pretty sold out. My only complaint reflects my general feeling about most of the Dodos’ material, sometimes their songs just feel like they go on a bit long without developing. But that’s not to say I think they’re very talented and put on a brilliant live performance. A good band more than worth checking out. They are touring in support of their 2009 release Time To Die. Here is their Myspace.

One of my favourite BDD frameworks is easyb. I really like the dsl for user stories. In a previous post called Easy Requirements by Example I discussed how easy and clear this dsl can be.

For Scala I would also like to have a dsl just like the one of easyb. Writing a dsl in Scala is real easy, not as easy as in Groovy, but easy enough. So I decided to see how fast I could write a story dsl in Scala.

So, I need to be able to write things like the following

   given ("a blogger with a blog") {
    	val blogger = new Blogger("Mr Blog", "mrblog@blogger.com", "passwd")
    	val blog = new Blog("blogTitle", blogger)

		when("mr blog writes a new post") {
			blog addPost ("post Title", "post Text");
		}
		and
		when("mr blog publishes its post") {
		    blog publishPost
		}
		then("the post should be publised for reading") {
			blog.published shouldHave "post Title"
		}
	  }

The words given, when, then, and, shouldHave are part of the dsl.
The story dsl words are defined by a curried function. They have a explaining piece of text as the first argument and a code block as the second argument.

See below the definition of given

  def given(text: String)( codeBlock: => Unit ) = {
	  codeBlock;
  }

Because given is a curried function you can pass in the arguments one at a time. And when you pass in one argument you can choose to use curly braces instead of parentheses. As you can see in the above example, the given function has the rest of the story as the second argument within curly braces. So, the when, then, and form the codeBlock argument.

Here is the definition of the other methods

  def then(text: String)( codeBlock: => Unit ) = {
    codeBlock;
  }

  def when(text: String)( codeBlock: => Unit ) = {
    codeBlock;
  }

  def and(text: String)() = {
  }

… Yep this is really simple!

Because the codeBlock parameters are so-called “By-name-parameters” the codeBlocks of the functions are only evaluated when the given method evaluates it’s codeBlock! Also because closures in Scala see changes made to the free variables outside of the closure itself, you get the sequential execution ordering of the story.

The only thing that is left is the shouldHave word. As you might have guessed this is just a simple implicit conversion. To have free integration with IDE’s and other tools I choose to use JUnit as the underlying framework. So all the dsl vertification words are mapped to JUnit asserts as you can see below.

  implicit def shouldHave(l: List[Any]) = {
	  ShouldHave(l)
  }

case class ShouldHave(i: List[Any]) {
    def shouldHave(j: Any) = {
    	assertEquals(j, i.find( _ == j))
    }
}

You can use the dsl in your JUnit test like this

[/sourcecode]
  @Test
  def bloggerSpecification() = {

    given ("a blogger with a blog") {
      .....
    }
   }
[sourcecode]

You can imagine that you would like to execute a story multiple times with different argument values. Lets say that you would like to execute the story with multiple post titles. A little extension of the dsl made that possible. I introduced the words testcases, addCase and execute_testcases_with_specification.
See below an example

    testcases {
    	var postTitle: String = null;

    	addCase { postTitle = "title 1"	}
    	addCase { postTitle = "title 2"	}

    	execute_testcases_with_specification {
    		given ("a blogger with a blog") {
    			val blogger = new Blogger("Mr Blog", "mrblog@blogger.com", "passwd")
    			val blog = new Blog("blogTitle", blogger)		   

    			when ("mr blog writes a new post") {
    				blog addPost (postTitle, "post Text");
    			}
    			and
    			when ("mr blog publishes its post") {
    				blog publishPost
    			}
    			then ("the post should be publised fo reading") {
    				blog.published exists(_.title == postTitle) shouldBe true
    			}
    		}
    	}
 

What you see is that there are two test cases, each specifying a different value for the postTitle. The story is executed for each test case where the free variables are changed according to the test case at hand.

In order to make this work I stored all the test cases in a list. Then for each test case I first evaluate the test case. This sets the free variables of the story closure. Next I evaluate the story itself.

Again the code is really simple as you can see below.

object TestCase {

  var codeBlocks = List[() => Unit]()
  var scenario: Any = 0;

  def addCase( codeBlock: => Unit) = {
	  codeBlocks = codeBlock _ :: codeBlocks
  }

  def execute_testcases_with_specification( scenario: => Unit) = {
    codeBlocks.foreach( codeBlock => {
      codeBlock();
      scenario;
    })
  }

  def testcases( codeBlock: => Unit) = {
	  codeBlock
  }
}

If you made it this far in the post… what do you think of this approach?

Playing around with Scala, a very nice jvm language!,
found Lift (web framework),
looks simple and funny !

I was looking for ldap authentication with lift,
i wasn’t able to find nothing .

So i started to play with it,
Here, I install an openldap server and some simple scala code to play with it.

• Install openldap and some sample data (using arch linux)

  
  1- sudo pacman -S openldap
  2- slappasswd -h {MD5} -s password (generates ldap password to use it later in config and user ldif file)
     {MD5}X03MO1qnZdYdgyfeuILPmQ==
  

• Generate the ldap structure in initial_structure.ldif file

  
  dn: dc=company,dc=com
  dc: company
  description: LDAP Main object
  objectClass: organization
  objectClass: dcObject
  o: company.com
  
  dn: ou=Users,dc=company,dc=com
  ou: Users
  objectClass: organizationalUnit
  
  dn: ou=Groups,dc=company,dc=com
  ou: Groups
  objectClass: top
  objectClass: organizationalUnit
  
  dn: cn=main_group,ou=Groups,dc=company,dc=com
  gidNumber: 2000
  objectClass: posixGroup
  objectClass: top
  cn: main_group
  
  dn: cn=secondary_group,ou=Groups,dc=company,dc=com
  gidNumber: 2001
  objectClass: posixGroup
  objectClass: top
  cn: secondary_group
  

• Configure ldap server in /etc/openldap/slapd.conf

  
  include         /etc/openldap/schema/core.schema
  include         /etc/openldap/schema/cosine.schema
  include         /etc/openldap/schema/courier.schema
  include         /etc/openldap/schema/inetorgperson.schema
  include         /etc/openldap/schema/nis.schema
  
  allow bind_v2
  password-hash {md5}
  
  pidfile   /var/run/slapd.pid
  argsfile  /var/run/slapd.args
  
  database        bdb
  suffix          "dc=company,dc=com"
  rootdn          "cn=admin,dc=company,dc=com"
  rootpw          {MD5}X03MO1qnZdYdgyfeuILPmQ==
  
  directory       /var/lib/openldap/openldap-data
  index   objectClass     eq
  index   uid     eq
  

• Populate initial ldap structure

  
  sudo /usr/sbin/slapadd -l initial_structure.ldif
  

• Populate user and group data, for example in file users.ldif

  
  dn: uid=sample_user_1,ou=Users,dc=company,dc=com
  objectClass: top
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  objectClass: posixAccount
  objectClass: shadowAccount
  uid: sample_user_1
  cn: Test User
  sn: User
  givenName: Test
  userPassword: {MD5}X03MO1qnZdYdgyfeuILPmQ==
  loginShell: /bin/bash
  uidNumber: 10000
  gidNumber: 2000
  homeDirectory: /home/users/test/
  

• Starts ldap server and add users.ldif

  
  sudo /etc/rc.d/sldap start
  ldapadd -x -D "cn=admin,dc=company,dc=com" -f users.ldif -W
  

• Test ldap server searching for sample_user_1 user

  
  ldapsearch -x -D "cn=admin,dc=company,dc=com" -b "dc=company,dc=com" "(&(uid=sample_user_1))" -W
  

• And now scala code

  
  import java.io.FileInputStream
  import java.util.{Hashtable, Properties}
  
  import javax.naming.Context
  import javax.naming.directory.{BasicAttributes, SearchControls}
  import javax.naming.ldap.{LdapName, InitialLdapContext}
  
  import scala.collection.jcl.{MapWrapper}
  import scala.util.logging.{Logged, ConsoleLogger}
  
  implicit def convert(javaMap: Hashtable[String, String]) = {
      Map.empty ++ new MapWrapper[String, String]() {
          def underlying = javaMap
      }
  } 
  
  type StringMap = Map[String, String]
  
  val DEFAULT_URL = "localhost"
  val DEFAULT_BASE_DN = ""
  val DEFAULT_USER = ""
  val DEFAULT_PASSWORD = ""
  
  object SimpleLDAPSearch {
      lazy val ldap: LDAPSearch = {
          if (properties() == null) {
              val p = new Properties()
              p.load(new FileInputStream(propertiesFile()))
  
              // automatically calls convert(javaMap: Hashtable[String, String])
              properties = () => p.asInstanceOf[StringMap]
          }
  
          new LDAPSearch(properties()) with ConsoleLogger
      }
  
      var properties: () => StringMap = () => null
      var propertiesFile: () => String = {
          () => "DEFAULT_PROPERTIES_FILE.properties"
      }
  }
  
  class LDAPSearch(parameters: StringMap) extends Logged {
      lazy val initialContext = getInitialContext(parameters)
  
      def search(filter: String) : List[String] = {
          log("--> LDAPSearch.search: Searching for '%s'".format(filter))
  
          var list = List[String]()
  
          val ctx = initialContext
  
          if (!ctx.isEmpty) {
              val result = ctx.get.search(parameters.getOrElse("ldap.base", DEFAULT_BASE_DN),
                                          filter,
                                          getSearchControls())
  
              while(result.hasMore()) {
                  var r = result.next()
                  list = list ::: List(r.getName)
              }
          }
  
          return list
      }
  
      def bindUser(dn: String, password: String) : Boolean = {
          log("--> LDAPSearch.bindUser: Try to bind user '%s'".format(dn))
  
          var result = false
  
          try {
              var env = new Hashtable[String, String]()
              env.put(Context.PROVIDER_URL, parameters.getOrElse("ldap.url", DEFAULT_URL))
              env.put(Context.SECURITY_AUTHENTICATION, "simple")
              env.put(Context.SECURITY_PRINCIPAL, dn + "," + parameters.get("ldap.base"))
              env.put(Context.SECURITY_CREDENTIALS, password)
              env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory")
  
              var ctx = Some(new InitialLdapContext(env, null))
  
              result = !ctx.isEmpty
              ctx.get.close
          }
          catch {
              case e: Exception => println(e)
          }
  
          log("--> LDAPSearch.bindUser: Bind successfull ? %s".format(result))
  
          return result
      }
  
      private def getInitialContext(props: StringMap) : Option[InitialLdapContext] = {
  
          log("--> LDAPSearch.getInitialContext: Get initial context from '%s'".format(props.get("ldap.url")))
  
          var env = new Hashtable[String, String]()
          env.put(Context.PROVIDER_URL, props.getOrElse("ldap.url", DEFAULT_URL))
          env.put(Context.SECURITY_AUTHENTICATION, "simple")
          env.put(Context.SECURITY_PRINCIPAL, props.getOrElse("ldap.userName", DEFAULT_USER))
          env.put(Context.SECURITY_CREDENTIALS, props.getOrElse("ldap.password", DEFAULT_PASSWORD))
          env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory")
  
          return Some(new InitialLdapContext(env, null))
      }
  
      private def getSearchControls() : SearchControls = {
          val searchAttributes = new Array[String](1)
          searchAttributes(0) = "cn"
  
          val constraints = new SearchControls()
          constraints.setSearchScope(SearchControls.SUBTREE_SCOPE)
          constraints.setReturningAttributes(searchAttributes)
          return constraints
      }
  }
  
  // SimpleLDAPSearch.propertiesFile = () => "ldap.properties"
  
  SimpleLDAPSearch.properties = () => {
      Map("ldap.url" -> "ldap://localhost",
          "ldap.userName" -> "cn=admin,dc=company,dc=com",
          "ldap.password" -> "password",
          "ldap.base" -> "dc=company,dc=com")
  }
  
  val list1 = SimpleLDAPSearch.ldap.search("(uid=sample_user_1)")
  println(SimpleLDAPSearch.ldap.bindUser(list1(0), "password"))
  

  The code is not exactly perfect,
  but shows how simple can scala be.

Here’s my favourites lines of code :

• Automatically convert types

  
  implicit def convert(javaMap: Hashtable[String, String]) = {
      Map.empty ++ new MapWrapper[String, String]() {
          def underlying = javaMap
      }
  }
  

  To automatically convert a java.util.Hashtable (java.util.Properties) into a scala Map,
  Example :

  
  val hashtable: java.util.Hastable[String, String] = new java.util.Hashtable[String, String]()
  hashtable.put("some_key", "some_value")
  
  val map = hashtable.asInstanceOf[Map[String, String]]
  

• A var that contains a method that returns the ldap properties file

  
  object SimpleLDAPSearch {
      var propertiesFile: () => String = {
          () => "DEFAULT_PROPERTIES_FILE.properties"
      }
  ..
  // The SimpleLDAPSearch singleton propertiesFile method can be override in any moment
  SimpleLDAPSearch.propertiesFile = () => "/tmp/ldap.properties"
  

Sometimes when compiling my Scala code, the compiler output may need a little bit of help in sorting out what’s the real issue.

For example, when I compiled today I received the following error:

[scalac] error: error while loading DefaultHttpClient, class file 'C:\A_OpenSource\WicketNotesApp\lib\httpclient-4.0.jar(org/apache/http/impl/client/DefaultHttpClient.class)' is broken
[scalac] (class net.jcip.annotations.ThreadSafe not found.)

[scalac] error: error while loading AbstractHttpClient, class file 'C:\A_OpenSource\WicketNotesApp\lib\httpclient-4.0.jar(org/apache/http/impl/client/AbstractHttpClient.class)' is broken
[scalac] (class net.jcip.annotations.ThreadSafe not found.)

And my instinct tells me to focus on the jar file or the class file and in this case, “DefaultHttpClient.class” or “AbstractHttpClient.class” or “httpclient-4.0.jar”. There’s noting wrong with your jar file or the Java class. The actual problem is in the next line, namely “class net.jcip.annotations.ThreadSafe not found.” It’s a class-not-found error.

So if you encounter a “broken” error when compiling Scala – the following line after the broken message can help you find the real error. It can save you a whole lot of head-aches and chasing the wrong tail.

If you find this article useful, consider signing up for my RSS feed or Email Newsletter. See links on the right side.

I make no secret of the fact that I have a very strong interest in programming languages. So I was naturally very interested when news of the Go Programming Language hit the intertubes. Go is an interesting language. It pulls together some very powerful features with a familiar, but clean syntax and has lightning fast compile times. It certainly takes a place on my to-learn list along with Haskell and Scala. But even as Go becomes the latest hot piece of language news, it dawned on me that over the past few years we’ve seen a slew of interesting languages offering compelling alternatives to the industry “mainstream”.

I guess it all started with the rise of scripting languages like Python, PHP, Ruby and the poster boy of scripting: Perl. Personally, these languages with their dynamic typing, “batteries included” design and interesting syntax provided a breath of fresh air from the likes of C++ and Java. Not that C++ and Java are necessarily bad languages, but they aren’t the most interesting of modern languages. In the early years of this decade computers were just getting fast enough to write large scale software in scripting languages. Things have changed a lot since then.

Dynamic languages aren’t just reserved for small scripts. Software like Ruby on Rails has proved that you can write really robust back end infrastructure with them. The languages for their part have kept on growing, adding features and making changes that keep them interesting and downright fun to use. Python 3.0 was a brave decision to make a break from backwards compatibility in order to do interesting things and it goes to show that these languages are far from ossifying or degrading.

Then there is JavaScript which was supposed to die a slow death by attrition as web programmers moved to Flash or Silverlight. But we all know that didn’t happen. JavaScript has stayed in the background since the rise of Netscape, but it’s only recently with advances in browser technology and growing standards support that it has really come into its own. I’ve only played with it a little, but it’s a fun little language which makes me feel a lot of the same emotions I felt when discovering Python for the first time. Thanks to efforts like Rhino, you can even use JavaScript on the client side for non-web related programming.

Of course, if you want to do really interesting things with these languages, then performance is not optional. Within the last year or two there’s been a strong push in both academia and industry to find ways to make these languages faster and safer. Google in particular seems to be in the thick of it. Chrome’s V8 JavaScript engine is probably the fastest client side JavaScript environment and their still experimental Unladen Swallow project has already made headway in improving Python performance. V8 has already enabled some amazing projects and I’m waiting to see what Unladen Swallow will do.

While we’re on the topic of performance, mentioning the Java Virtual Machine is  a must. The language itself seems to have fallen from grace lately, but the JVM is home to some of the most powerful compiler technology on the planet. It’s no wonder then that the JVM has become the target for a bunch of interesting languages. There are the ports of popular languages — JRuby, Jython and Rhino. But the more interesting ones are the JVM-centric ones. Scala is really interesting in that it was born of an academic research project but is becoming the strongest contender to Java’s position of premier JVM language. Clojure is another language that I don’t think many people saw coming. It brings the power of LISP to a modern JVM unleashing a wide range of possibilities. It has it’s detractors, but it’s certainly done a fair bit to make Lisp a well known name again.

Academia has always been a hot bed when it comes to language design. It’s produced wonders like Lisp and Prolog and is making waves again with creations like Haskell (whose goal is ostensibly to avoid popularity at all costs) and the ML group of languages. These powerful functional languages with wonderful type inference are a language aficionado’s dream come true in many ways and they still have years of innovation ahead of them.

Almost as a corollary to the theoretically grounded functional languages, systems languages have been getting some love too. D and now Go are both languages that acknowledge that C and C++ have both had their heyday and it’s time to realize that systems programming does not have to be synonymous with bit twiddling. D has gotten some flak recently for not evolving very cleanly over the last few years, but something is better than nothing. Also a real shift towards eliminating manual memory management is a welcome addition.

As someone who intends to seriously study language design and the related concepts in the years to come, it’s a really great time to be in getting involved in learning about languages. At the moment I’m trying to teach myself Common Lisp and I have a Scala book sitting on the shelf too. One fo these days, I plan on sitting down and making a little toy language to get used to the idea of creating a language. Till then, it’s going to be really interesting just watching how things work out in an increasingly multilingual world.

Since right now I’m processing large amount of binary files (300+ files * 4MB each) in Scala and store some of results in a list. Some of typical method may not result good when processing in terms of large lists. So here is the tip I can share with you guys from my trial & error:

Beware RECURSIVE method!

This is such an obvious evil that you may already known this. If you’re working with processing the list result and get stack overflow error then I suspect you’re somehow using recursive method. Avoid that if possible. One method I had discovered to lead to stack overflow is dropRight since I use this method in order to reduce list size. Others also had shown that foldRight could cause this error as well. If you want to know which method may harm you then you may tries using scala interpreter and construct List of integer that the size is larger then 4K elements. However if you REALLY want to know then I suggest you go for source code.

Immutable List should be use cautiously.

Why? Because Immutable is not mutable! That means the list may not change over time, and the only way to add / remove element by using immutable list is to return a new list. If you’re trying to constantly append the processing result into a list by the end of each iteration, then you are going to reassign list every iteration. This could drain your memory real fast. My alternatives is that you may prefer scala.collection.mutable.ListBuffer instead.

Watch out for linear-time method usage while processing lists

Since processing multiple files may yield a list of result for each iteration, which each element had to be stored in another list (with same index). Writing codes like this within main file reading loop:

for ( i <- 0 until loopResult.length ) {
    //Some other process also going on
    processResult(i) += loopResult(i)
}

had cost me extra 60 seconds comparing with this loop when I’m processing files in byte-by-byte fashion, with each loop result had 20 elements. (Be advised that my actual logic is more complex than this but just to show the idea)

for ( elem <- loopResult.zipWithIndex ) {
    //Some other process also going on
    processResult(elem._2) += elem._1
}

And it even could be optimized with using zip command like below. However I haven’t tried it myself so I cannot guarantee this is better.

for ( elem <- loopResult zip processResult ) {
    elem._2 += elem._1
}

Processing “result list” after file reading loop if possible.

It’s reasonable to processing file to get result for each loop and append it into a result list. However if you may require extra processing e.g. sort the list or filter out N least element, then it’s better to process these after you’ve complete reading the file because processing these stuffs will usually lead you to O(n^2) when n = file size. You can even use immutable list and sort method for this case. For my case, I had to filter out the least element to reduce the size of the list from 300K to 40K. It took less than a few seconds when I process this after completely processing whole file, comparing to forever (It tooks so long and I kill its process) when I tries to achieve this within file reading loop.

If you’re interested in processing large files, then don’t forget to read Processing large files with Scala as well. His contribution really saves my ass since I totally have no idea on what to start at all.

However I do believe these might not fit everyone since you usually don’t have to work with large list. Some tips might also slow your coding speed down so please adjust this tips accordingly to your need.

That’s it for what I’ve discovered. You guys can share or even comments about this since I’m just a scala (and functional language) newbie.

Edit: Rewrite in easily-readable style.

Sometimes we need to forward to a servlet or resource that’s outside of Wicket such as a servlet. Here’s how:

1. Modify your web.xml to define your servlet as well as your Wicket application “ignorePaths” init-param:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
 <display-name>Wicket Example</display-name>
 <filter>
 <filter-name>MyApplication</filter-name>
 <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
 <init-param>
 <param-name>applicationClassName</param-name>
 <param-value>com.gu.MyApplication</param-value>
 </init-param>
 <init-param>
 <param-name>wicket.configuration</param-name>
 <param-value>deployment</param-value>
 </init-param>
 <init-param>
 <param-name>ignorePaths</param-name>
 <param-value>images/,s/</param-value>
 </init-param>
 </filter>
 <filter-mapping>
 <filter-name>MyApplication</filter-name>
 <url-pattern>/*</url-pattern>
 </filter-mapping>
 <servlet>
 <servlet-name>SomeServlet</servlet-name>
 <servlet-class>com.gu.SomeServlet</servlet-class>
 </servlet>
 <servlet-mapping>
 <servlet-name>SomeServlet</servlet-name>
 <url-pattern>/s/koolkat.do</url-pattern>
 </servlet-mapping>
</web-app>

The first observation is that Wicket is running as a Filter and not a Servlet. Second, the init-param – “ignorePaths” says that anything in the “/images” and “/s/*” will be ignored by Wicket Filter and passed along downstream.

2. Next in your Wicket Form, within your onSubmit method override you can do the following:

class TestForm extends Form(id){
  @override
  public TestForm(String id, SomeFormModel ghfm)
  {
    //Code removed
  }
  @override
  public onSubmit()
  {
    String url = "/s/koolkat.do"; //some url within the same web-context
    getRequestCycle().setRequestTarget(new RedirectRequestTarget(url));
  }
}

Notice that the url field does not contain the “ContextPath”. Wicket takes care of that in the background. On the other hand, suppose you wanted to redirect to a URL ourside of your web context or to an external site then you would set the variable like this.

String url = "http://mythinkpond.wordpress.com/2009/11/13/forwarding-to-a-url-from-wicket/";  //or
url = "http://myserver:port/othercontext/someurl";

On some rare occasions, you might want to get the servlet Request Dispatcher to forward a request. For that it gets a little tricky.

– In your class you want to grab the “HttpServletResponse”. I’ve been unsuccessful in getting it in the class.

class xyz extends WebPage{
  public xyz()
  {
     HttpServletResponse response = getWebRequestCycle().getWebResponse().getHttpServletResponse();
     //additional code
     xyxForm thisForm = new xyzForm(thisModel, response);
  }
}

class xyzForm extends Form{
  HttpServletResponse response;

  public xyzForm(xyzModel model, HttpServletResponse thisResponse)
  {
     response = thisResponse;
  }

   public onSubmit() {
     HttpServletRequest request = getWebRequest().getHttpServletRequest();
     //additional code here
     getWebRequest().getHttpServletRequest().getRequestDispatcher(url).forward(request, response);
  }
}

Please note that the above code is pseudo-code and not actual code because I was writing it in Scala and not Java. But for the audience, I’m trying to express it in Java so that it can have the greatest impact.

That’s it! Happy Coding!

If you find this article useful, consider signing up for my RSS feed or Email Newsletter. See links on the right side.

Netwerkbijeenkomst: Scala ‘Allochtone Vrouwen aan de Top’

Datum: 10 november 2009
Waar: WTC Rotterdam
Hoeveel mensen: rond de 50
Verhouding man/vrouw: 99% vrouw

Vrouwen met Pit aan het woord
Er waren 4 spreeksters met een enorme kracht en drive om goed te zijn in wat zij doen. Ondanks de barrieres die zij in hun leven hebben gehad zijn zij door gegaan. Zij hebben kansen gezien en gegrepen, omdat zij een duidelijk doel voor ogen hadden en nog steeds hebben.

Wie zijn deze vrouwen?

1. Een 38-jarige vrouw van Turkse afkomst met de naam Famile Fatma Arslan heeft 5 jaar bij Ministerie van Justitie gewerkt. Zij is nu al 7 jaar ondernemer en kan zich een van de beste advocaten van Nederland noemen. Op 4-jarige leeftijd kwam zij met haar ouders naar Nederland. Het zou in eerste instantie voor 2 jaar zijn, maar ze zijn nu na 34 jaar niet meer teruggekeerd naar Turkije. Ze komt uit het oost-Turkije dat wordt gedomineerd door een mannencultuur. Toen zij 11 werd, kreeg zij van de basisschool het advies dat zij de huishoudschool moest gaan doen. Haar vader was hier op tegen. Hij had namelijk van zijn vader, die een welvarende veehouderij had in Turkije, niet de kans gekregen om te gaan studeren. Hij werd na een paar uur school door zijn vader van school gesleurd om alles te leren in het bedrijf, zodat hij het later over kon gaan nemen. Dit zorgde ervoor dat hij beter wilde voor zijn 1e dochter.

Na het schooladvies heeft zij 1 jaar lang thuis gezeten. Ze werd gek. Op jonge leeftijd zorgde zij voor haar moeder en haar twee broertjes. De huishoudschool was zeker overbodig. Zij deed al alles in huis. Ze vroeg aan haar vader om scholing en kreeg als antwoord. Als jij ervoor kan zorgen dat je wordt toegelaten tot de mavo, havo of vwo opleiding dan mag jij van mij naar school. Uiteindelijk kreeg zij op 12-jarige leeftijd een directeur zover die haar 2 maanden lang de tijd gaf om proef te draaien. Gedurende deze 2 maanden mocht ze geen enkele onvoldoende halen. Na de mavo heeft zij ook de havo en het atheneum doorlopen. Ze is rechten gaan studeren. Soliciteerde naar een baan op MBO niveau binnen Ministerie van Justitie. Ging vervolgens met de juiste mensen koffie drinken en kreeg hierdoor een baan die aansloot op haar studie Rechten.

2. Een 40-jarige Hindoestaans-Surinaamse vrouw die op haar 30ste een megadeal sloot als jurist voor een telecom bedrijf. Ongelooflijk. Ze verkocht het bedrijf voor 1.2 miljard euro aan Engelsen. Hierna kwam zij terecht in de directie bij hetzelfde of een ander bedrijf. Na 2 jaar stortte de telecom industrie in. Op haar 33ste ging zij trouwen met een manager die werkzaam was bij hetzelfde telecom bedrijf. Haar man ging voor zichzelf beginnen en zij bleef werken. Op haar 35ste kreeg zij haar eerste kind en op haar 39ste haar tweede kind. Haar verhaal inspireerde mij, omdat zij er nu ontzettend goed uitziet, getrouwd is, twee kinderen heeft en een ambitieuze echtgenoot. Zij blijven elkaar de ruimte geven om te groeien. Prachtig om te horen. Wat ik helemaal prachtig vond om te horen, was dat zij altijd de steun van haar ouders heeft gehad. Toen zij voor de beslissing stond om in een Telecom bedrijf te stappen die over 2 maanden verkocht moest worden, was haar vader degene die haar motiveerde om de stap te wagen. Het was nieuw voor haar, maar ze ging de uitdaging aan.

Ze is rechten gaan studeren, omdat zij vanuit huis uit een sterk rechtvaardigheidsgevoel had ontwikkeld. Haar vader is zeer begaan met de politiek en weet precies wat welke minister doet. Na haar rechtenstudie kreeg zij een baan als manager van grote projecten. Projecten waar miljarden euro’s in omgingen.

3. Een Rusissche vrouw van 25 jaar oud kwam ook aan het woord. Zij gaf heel duidelijk aan dat je elke kans die je ziet, ruikt of hoort moet aangrijpen. Iets wat je interesseert, doe het, geeft het aan, schreeuw het van de daken. Laat je horen. Dat is ook iets wat zij tijdens haar opleiding binnen Defensie heeft moeten leren. Spreken met luide stem zonder microfoon. Zij stond er! Waanzinnig gaaf. Een sterke vrouw die opgegroeid is tusen vrouwen. Tussen haar oma en moeder. Zij spreekt goed Nederlands, maar wel met een licht Russische accent. Haar vriend is net zo ambitieus. In de periode dat haar vriend voor 2 jaar naar Amerika ging voor een vliegopleiding is zij een Master gaan doen. Als vrouw moest zij zich binnen de mannencultuur van Defensie bewijzen. Op haar eerste werkdag kwam zij aanzetten met een enorme grote roze mok met een etiket en de leus: ‘the best man for the job is a woman’. De mok bestaat nog steeds

4. De vierde vrouw is over de 50 en van Surinaamse afkomst. Iemand die nu directeur is van de FeM-Kliniek in Zwolle. Zij had een mega inhoudelijke presentatie over hoe je het begrip ‘allochtone topvrouwen’ ook kunt zien. Haar boodschap is: je moet je niet in een hokje laten stoppen. Je moet gaan voor wat jij wil. Blijf jezelf ontwikkelen en elke tegenslag is een kans. Een kans om een andere weg in te stappen en je te ontwikkelen op deze weg. Geweldig!! Geen woorden voor.

Door Chungmei Cheng van Orchid of Life -LifeCoaching

I had this actually running for some time now: It is a graphical debugger which helps me understand how the 68000 instructions in the Kickstart ROM affect my Amiga emulator’s state. Here is a screen shot:

It is currently mainly a view to the Mahatma68k CPU, the two CIA-8520’s and the custom chip registers. It can execute a defined number of steps and any changes to the chip registers or CPU state will show there. I am currently implementing a function which displays a tooltip for each custom chip register to show a little more information than just its numerical value. The CIA’s and the CPU have only a few registers, so it is still feasible to display the role of each flag in the main view. The sheer amount of custom chip registers however, made it even hard to get them all in one frame.
By the way, compliments to the developers of Scala Swing – these are very nice wrappers ! I started with conventional Java Swing components and then realized that Scala Swing makes programming user interfaces much easier. As a long time user,  Swing does really seem to be all that difficult to me, still I like it when the code looks as tidy as it does when using Scala Swing.

What else is new today ? I have made my second file release to Mahatma68k today. As its main addition, it now comes with the Javadocs the distribution archive and made sure only to have the methods and classes in there that matter to the user of the library.

Author:Massimiliano Dessì

Places are going fast for this year’s Open Source in Finance eXchange on December 15.  Speaking across six expert talks and two parkbench discussions on the day will be Alan Hardy, Rob Davies, Angelo Corsaro and David Vincent.

This year’s eXchange is bringing together the leading experts in Open Source technology and Financial Services software development with innovators, technologists, architects and developers in the developer community.

The day promises to combine deep technical insight with friendly discussion around creating highly scalable, secure systems and leveraging OS technology within the finance sector.

The programme is structured to encourage discussion and bring together the leading thinkers and experts on the use open source technology in Financial Services with passionate members of the developer community in this sector.

To keep it as informal as possible we are strictly limiting the event to 125 delegates, and it is already proving to be popular.  What’s more, on the day we will be raffling off a ticket to this year’s QCon in London.

Book now — and get your tickets for £65.  Just click the banner at the top of the page.

In the same manner as Try Ruby comes Simply Scala – a neat way to check out the programming language in a hassle-free way. Work through the tutorial or pursue undirected learning – there’s a full Scala REPL available to use.

Continuing the Scala Mashup Series, this post is about a quite important, often used, but also controversial pattern: The Template Method.

In general, there are great explanations of the pattern in Wikipedia and of course in Head First Design Patterns.

Criticism

A very good summary of the Template Method pattern especially in comparison to the Strategy pattern is on stackoverflow.com:

The Template pattern does compile-time algorithm selection by subclassing.
The Strategy pattern does run-time algorithm selection by containment.

Example

Imagine you want to arrange a dinner. The entire algorithm for arranging a dinner is predefined:

1. Invite your guests
2. Go shopping
3. Clean your fridge to provide enough space
4. Prepare the meal
5. Server the meal
6. Wash the dishes

Now, some parts of this algorithm (which in its entirety is called arrange) are fixed (invariant parts). Other parts are highly dependant on the meal you want to prepare (variant parts).
In Template Method, the algorithm itself is described in a public accessible method, eg. arrange(), which combines and calls the variant and invariant methods. The invariant methods are implemented in the abstract super class. The variant parts are implemented in the concrete classes, eg. BratwurstDinner, which implements the concrete way to do shopping for the BratwurstDinner, as well as the preparation of the Bratwurst itself.
The important part is the compile-time binding – the implementing, concrete classes cannot change their implementation during runtime.
This makes the Template Method almost ideal for framework creation, when you want to provide abstract algorithms with certain concrete implementations (think of abstract database layers, with an algorithm which cares for opening & establishing connections, handles exceptions and cleans up nicely after doing … the variant part, which is implemented in the concrete classes).

Here is the dinner example using the Template Method pattern

Here is the dinner example using the Strategy pattern

There are two clever blog posts, one about disadvantages of the Template Method pattern, and one mentioning a possible alternative in languages with first class functions like Scala.

Language comparison: Java, Scala, Spring

We will happily leave out the Java example, since it is just too obvious. In the Wikipedia article a sample is given, together with the class diagram it should be pretty obvious how Template Method is implemented in Java. The only thing to keep in mind is to set the super class abstract, as well as the variant methods – which are then implemented (using @Override) in the concrete classes.

Scala

In Scala there are is the intuitive way of implemeting the Template Method, analogous to Java:

abstract class DinnerEvent {

  def arrange: Unit = {
    makeGuestList
    doShopping
    cleanFridge
    prepareMeal
    serveMeal
    washDishes
  }

  def prepareMeal: Unit
  def doShopping: Unit

  def makeGuestList: Unit = {
    println("making guest list...")
  }

  def cleanFridge: Unit = {
    println("cleaning fridge...")
  }

  def serveMeal: Unit = {
    println("serving meal...")
  }

  def washDishes: Unit = {
    println("washing dishes...")
  }

}

trait TurkeyDinner {

  def doShopping: Unit =  {
    println("...buying turkey")
  }

  def prepareMeal: Unit =  {
    println("...preparing turkey")
  }

}

object TemplateMethod {

  def main(args:Array[String]) = {
    val dinner = new DinnerEvent with TurkeyDinner
    dinner.arrange
  }

}

However, in languages with first class functions, it is too tempting to use the Strategy pattern:

class DinnerEventWithStrategy(shoppingStrategy: => Unit, preparingStrategy: => Unit) {

  def arrange: Unit = {
    makeGuestList
    shoppingStrategy
    cleanFridge
    preparingStrategy
    serveMeal
    washDishes
  }

  def makeGuestList: Unit = {
    println("making guest list...")
  }

  def cleanFridge: Unit = {
    println("cleaning fridge...")
  }

  def serveMeal: Unit = {
    println("serving meal...")
  }

  def washDishes: Unit = {
    println("washing dishes...")
  }

}

object Closure {

  def turkeyShoppingStrategy: Unit = {
    println ("...shopping turkey strategy")
  }

  def turkeyPreparationStrategy:Unit = {
    println ("...preparing turkey strategy")
  }

  def main(args:Array[String]) = {
    val dinner = new DinnerEventWithStrategy(turkeyShoppingStrategy, turkeyPreparationStrategy)
    dinner.arrange
  }

}

So, you should choose wisely, both patterns definitely have their scope of application – just keep in mind that Strategy also uses composition over inheritance, which is always a good advise.

…and what about Spring?

This is interesting, Spring uses the Template Method a lot – and many think it is one of their greatest strengths.
But wait … the Template Method pattern? Not really. The Spring Framework templates in fact are not implementing the Template Method pattern, even if it feels this way.
Let’s examine a small usage example of Spring’s JdbcTemplate:

package com.commons.sample.dao;

import java.util.List;

import org.springframework.jdbc.core.simple.ParameterizedBeanPropertyRowMapper;
import org.springframework.jdbc.core.simple.SimpleJdbcDaoSupport;

public class JdbcSimpleDao extends SimpleJdbcDaoSupport implements SimpleDao {

	final static String single	= "select id, name from test where id = ?";
	final static String list 	= "select id, name from test"; 

	public Simple getSimpleById(int id) {
		return getSimpleJdbcTemplate().queryForObject(single,
				ParameterizedBeanPropertyRowMapper.newInstance(Simple.class), id);
	}

	public List<Simple> getSimples() {
		return getSimpleJdbcTemplate().query(list,
				ParameterizedBeanPropertyRowMapper.newInstance(Simple.class));
	}

}

What is happening here? Even though the class extends SimpleJdbcDaoSupport, it would not have to.
The Method query(…) takes the SQL and a ParameterizedBeanPropertyRowMapper.newInstance(Simple.class), which defines a row mapper that creates an object of class Simple – this certainly describes a strategy. So, even the so-called Spring Templates actually uses the Strategy pattern.

Summary

The Template Method is a powerful pattern. Use it if you create a framework, since you can inject certain functionality in the implementing classes at compile time, which let the user of your framework extend it easily, but in a controlled and “traceable” way. For Java, it is a natural way to enable users of your classes to extends the functionality at certain points.
In languages with first-order functions (or closures), like Scala, it might feel more natural to use the Strategy pattern instead. However, keep in mind that the patterns are not completely identical.

備忘です。

• ()Unit … Javaのvoidに相当
• Javaっぽくて括弧は必須:
  NG: if i < 10
  OK: if ( i < 10 )
• valは再代入不可、varは再代入可
• 「=>」と「<-」
• 「:::」と「::」。なお、末尾がコロンの場合は、右被演算子
• 「Nil」は空リスト
• タプル。異なる型をもてる。taple._1, taple._2, … 1から始まる。

まだ、traitにはたどり着きません。。。先は長そう。

次は同じものを表しています。

// — imperative style 命令型スタイル —
var i = 0
while (i < args.length ) {
println( args(i) )
i += 1
}

// — functional style 関数型スタイル —
args.foreach(( arg: String ) => plintln(arg))

関数型に慣れていくには、varを使わずvalだけ使っていくこと、とありました。