Web applications are nice. They’re useful, they’re cross-platform, users need no installation, no upgrades, no maintenance, not even the computing or storage power to which they are used. As weird as it may sound, I’ve even seen announcements for web applications supposed to run your games on distant high-end computers so that you can actually play on low-end computers. Go web application!

Of course, there are a few downsides to web applications. Firstly, they require a web connexion. Secondly, they are largely composed of plumbing. Finally, ensuring their security is a constant fight.

How many pipes do you need?

If you have ever developed a web application, you know what I mean by plumbing: just writing an online TODO list — the equivalent of maybe twenty minutes of work in Visual Basic, in Python, Java or Objective-C — requires mixing an insane amount of languages, to define your user interface (HTML, JavaScript, CSS), to define your storage (DDL, DML, DCL, plus possibly an ORM language), to get your server and your client to communicate (XML, more JavaScript, PHP or one of its competitors, as well as some HTTP and a little MIME configuration), to launch your application (whichever configuration languages are used by your server). Of course, if your application is a bit more complex and requires something like compatibility with smartphones, or like distant storage, or distributed computing, or backups, or modularity (in the world of the web, it’s called “web services”)… well, you will probably require a few additional languages.

All of this is just plumbing. Only once you have written it can you concentrate on the core of the application. And once the application is written, the pain is just starting, because chances are that your application can be attacked by hijacking the link between your user interface and the core (cross-site scripting) or between the core and the storage (SQL injection) or by keeping the user interface and replacing the application core (man-in-the-middle attacks) or by replacing the user interface by a malicious client or by taking the place of a currently connected user to steal some of its credentials (rebinding) or by taking advantage of low-level bugs (buffer over/underflows), etc.

None of this is a show-stopper, of course — just take a look at the web and you will see thousands of web applications. Just like the complexity of Software Development Kits in the early days of Windows, MacOS or X didn’t stop adventurous hackers from developing desktop applications. But of course, if twenty-five years of desktop application development have taught us one thing, it is that the life of developers can be made easier. Nowadays, a few generations of SDKs later, Windows developers have .Net, C# and Visual Studio, Macintosh developers have Cocoa, Objective-C and XCode, while X-based developers have the libraries of Gnome/KDE, Python and a variety of programming environments. The growing popularity (and libraries) of Haskell, F#, OCaml, Scala and other functional programming languages could mean that one of the next generations of SDKs will increase safety and security.

The web hasn’t quite reached that stage yet. Even the state-of-the-art in web frameworks only provides features slightly more advanced than early Windows/Mac/X SDKs: low-level bindings for low-level mechanisms, designed to ensure low-level properties. Or, rephrased differently, in the current state of web development, GMail, Google Maps or Facebook are still considered complicated applications, although they are conceptually quite simple and should therefore be equally simple to implement.

We can do better. How? By removing the need for plumbing. By providing automated mechanisms for ensuring high-level security properties. By providing language support for common patterns.

Enter OPA

Let me introduce OPA. OPA, or One Pot Application, is a complete development platform for web applications and web services. Development in OPA requires no plumbing. Applications developed with OPA are automatically checked for safety and security before they are executed. Applications developed with OPA are automatically (and provably) immune to cross-site scripting, to SQL injections and to most existing forms of attacks. And OPA provides language support for storage, communication between client and server (Ajax and Comet), concurrency, distribution, mobility, etc.

With OPA, we intend to skip several generations of SDKs and provide right now a high-level and modern programming platform. OPA has been 6 years in the making: 4 years of sketches, mockups and prototypes as part of academic research projects and 2 years of actual implementation at MLstate. A few days ago, OPA has officially entered demonstrable status. Not quite ready for prime time, but definitely usable for development. Do you want to write an online note-taking application? That’s about 20 lines of code, from scratch. A minimal chat? About 30 lines. A multi-channel, distributed chat? About 80 lines. A minesweeper? About 100. We’re using it to develop utilities, content management systems, tools for administrations and games.

Pre-alpha builds of OPA have been distributed to selected partners. A public version will be made available within a few weeks, as well as commercial applications developed with OPA. In the meantime, we are busy improving the syntax, completing the standard library, making error messages intelligible, fixing the bugs and extending the range of safety and security checks.

Interested? Well, few details are public at this time. However, you can take a look at a video recorded during ICFP presenting OPA and MLstate.

Stay tuned.

I’m switching gears in this post to discuss my experience with installing and integrating Apache ActiveMQ’s C++ client library with Objective-C. ActiveMQ is an open-source message broker, for Windows and UNIX, from the Apache Software Foundation.

Topics covered:

• Intro to ActiveMQ
• Installing ActiveMQ C++ Client Libraries
• Integrating ActiveMQ C++ with Objective-C

In previous posts, I cover iPhone development topics for a prototype I’m working on. Yet, my iPhone app is only one client in a larger software as a service (SaaS) prototype. The plan is to have the SaaS system support clients across other popular mobile platforms and desktop operating systems such as Mac OS X and Windows.

In this heterogeneous computing environment, some of my components need to reliably send and receive messages. So, adding a message broker to the mix makes sense. However, for a prototype, buying a commercial product such as IBM’s WebSphere MQ is out-of-the-question. Consequently, I chose Apache ActiveMQ.

Apache ActiveMQ is an open source message broker which fully implements the Java Message Service 1.1 (JMS). It provides “Enterprise Features” like clustering, multiple message stores, and availability to use any database as a JMS persistence provider besides VM, cache, and journal persistency.

Apart from Java, ActiveMQ can be also used from .NET, C/C++ or Delphi or from scripting languages like Perl, Python, PHP and Ruby via various “Cross Language Clients” together with connecting to many protocols and platforms.

Apache ActiveMQ. (2009, November 26). In Wikipedia, The Free Encyclopedia. Retrieved 23:00, November 26, 2009, from http://en.wikipedia.org/w/index.php?title=Apache_ActiveMQ&oldid=328100430

Installing ActiveMQ is easy. Writing Java clients is straightforward. However, I have a native Mac OS X client (written in Objective-C) that needs to process messages. Fortunately, ActiveMQ has a C++ client library or CMS, which stands for C++ Messaging Service; and since Xcode supports Objective C++ (mixing Objective-C and C++) I have a solution for integrating my client with ActiveMQ.

Installing CMS on a UNIX-based system is not as easy as installing the ActiveMQ server. There are some hoops to jump through.

1. Download CMS here.
2. Follow the README instructions and note that the package is dependent on number of tools and libraries:
   • autoconf version >= 2.61
   • automake version >= 1.10
   • libtool version >= 1.5.24
   • APR version >= 1.3
   • APR-Util version >= 1.3
   • CPPUnit version >= 1.10.2*
   • libuuid version >= ?*

On Mac OS X 10.6.x, I already had autoconf, automake, and libtool. But, I had to install the latter four packages before installing CMS. Each installation follows the same set of standard command lines, check the requisite README for variations:

• ./configure
• make
• make [test or check]
• make install

After installing everything, the CMS libraries will be in usr/local/lib and ready to use in Xcode.

Moving to client development, I had the simple CMS consumer example running straightaway. However, once I began to mix in my Objective-C code, my build was wrought with pre-complier errors of the form: expected ‘;’, ‘,’ or ‘)’ before ‘>’ token.

Answers to this issue on the web were illusive. But, I knew the root cause was the inclusion of <Foundation/Foundation.h> with my C++. Fiddling with build settings was futile. My workaround, remove all references to Foundation.h and explicitly add only the headers the implementation needs. For example: <Foundation/NSString.h>, <Foundation/NSDate.h>, <Foundation/NSDateFormatter.h>… Works like a charm, but tedious. I wish Xcode had a command analogous to Organize Imports in Eclipse.

In any case, my ActiveMQ Objective C++ client is now working nicely. I hope this info helps some other Xcoder out there.

In my previous post I showed how I created a Mandelbrot fractal renderer in Clojure, however its performance left a lot to be desired. Optimising code is actually a very good way to learn more about a new language as it teaches you what is going on under the hood. In this post, I will go through the steps I took to drastically improve the application’s performance.

Inspecting the Code

The first step when optimising code (after not optimising of course) is to inspect the code’s logic and find quick fixes to any obvious potential performance areas. The logic of the Mandelbrot code however is fairly easy to reason about as we have built it up from the mathematical formula. Also, looking for obvious performance issues is difficult in a language you are trying to learn, so it’s time to break out the tools.

Profiling

First of all, let’s see how long it takes to render our fractal with the current implementation:

user=> (time (my-mandelbrot))
"Elapsed time: 29340.692742 msecs"
nil

Note, I have undone the premature optimisation I made last time and made the rendering single threaded by replacing pmap with map. This is important when profiling as it ensures we can properly determine which parts of the code are taking the longest. On a quad core cpu, I found that using pmap gave only a 2x speed increase which shows there is quite a lot of overhead in Clojure when processing on multiple threads.

Now at this stage, we have no idea what is taking so long so let’s run the app through a profiler. As Clojure is hosted on the JVM, it allows us to take advantage of many tools that would normally be used for Java applications. I attached the VisualVM profiling tool (which now comes with Java 6) to my application and then called my-mandelbrot again from the REPL. This is what I got after running for a few minutes:

What strikes you when digging through a Clojure call tree is just how deep it can go. It can also be difficult to relate the Java method calls we see back to the original Clojure code. Here I have expanded the calls made within the render function. You can see the two main consumers of CPU time are calls to count and vec. This is surprising as if you have a look at the original render function code, there’s not a call to either the count or vec functions to be seen:

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (.setPixel wr x y
                (int-array (coord-colour
                  (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                  max-iters)))))
           (get-pixels width height))))

In fact, count is called from within the coord-colour function:

(defn coord-colour
  "Returns a colour for which to draw the given coordinate. If the coordinate
  is within the mandelbrot set, black is returned. Otherwise, a colour within
  a gradient is given based on the number of iterations of the mandelbrot set
  that have been evaluated."
  [[xcoord ycoord] max-iters]
  (let [num-iters (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord))))]
    (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters))))

The call to vec is deeper still within the calls to iter-colour and grad-colour:

(defn grad-colour
  "Returns the colour that is the given fraction of the way between
  the first and second colours given. Returns as a vector of three
  integers between 0 and 255."
  [colA colB frac]
  (vec (map #(+ (* frac (- %2 %1)) %1) colA colB)))

(defn iter-colour
  "Returns the colour needed to paint a point with the given number
  of iterations"
  [num-iters max-iters]
  (grad-colour *grad-colour-a* *grad-colour-b*
    (/ (double num-iters) max-iters)))

The reason we see the count and vec calls at the level of the render function is that Clojure (or perhaps the hotspot) appears to inline certain function calls into the calling function. You can see further evidence of this in the above screenshot if you have a look at the divide, add and multiply methods. These are the operations performed by the get-coord function:

(defn get-coord
  "Returns the coordinates of the given pixel in the complex plane"
  [x y xstart ystart xsize ysize width height]
  [(+ xstart (* (/ x width) xsize))
   (+ ystart (* (/ y height) ysize))])

You should also see that at the time of the snapshot I took above, the number of invocations of each of the mathematical operations was around 67k which is twice the number of invocations of the count function which makes sense as you can see they are called once for each of the x and y coordinates.

Let’s dig deeper into the count function.

As you expand the call tree, and if you ignore the clojure.lang methods, you’ll see calls to take, take-while, iterate and finally mandelformula. Here’s another mystery of the Clojure call stack: we have a call to iterate before the call to mandelformula and yet in our code we can see iterate is called within the mandelformula function.

(defn mandelformula [xcoord ycoord]
  "Returns an infinite sequence of vectors containing the values of
  successive iterations of the mandelbrot formula, given a point on
  the complex plane."
  (iterate
    #(vec (map + (mandel %) [xcoord ycoord]))
    [xcoord ycoord]))

As far as I can tell, the reason for this is that mandelformula returns a lazy sequence which is not actually evaluated until it is needed. So iterate is used to get the next value in the sequence, while mandelformula actually does the job of calculating what the next value should be.

Drilling down into the mandelformula function we open up a whole can of method calls and it gets suddenly very difficult to map back to the original Clojure code. However we can see there seem to be a large number of sequences created and calls to methods in the clojure.lang.Numbers class taking Objects as their parameters. If we could get the formula to only use primitive types, this could potentially remove a lot of object creation and garbage collection. However, before we start re-working the code, let’s look at another profiling technique available to us in Clojure.

Using Clojure Contrib’s Profile Macros

Stuart Sierra has made a vast number of contributions to Clojure including the profile library which provides two macros prof and profile to measure the execution time of any block of code. Clojure’s macro system makes what would normally be a cumbersome task in other languages extremely easy. Let’s put profiling in our coord-colour function:

(defn coord-colour
  [[xcoord ycoord] max-iters]
  (let [num-iters (prof :mandel (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord)))))]
    (prof :colour (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters)))))

We simply wrap the block of code we want to profile around a call to the prof macro and specify a keyword to identify that block of code. Calling the profile macro will then summarise the time taken to evaluate each call to those prof blocks and print a bunch of statistics.

user=> (time (profile (my-mandelbrot)))
  Name      mean       min       max     count       sum
colour      4875      1117   1091200    300000  1462510112
mandel     71729      2514 100182286    300000 21518722246
"Elapsed time: 31456.673404 msecs"

Here we can see that the total time taken by the mandel section of the coord-colour function is around 21 seconds (the values above are in nanoseconds) which gives it 68% of the total running time. The time taken to calculate the gradient colour is about 5% of the total running time. These are quite different statistics from what we saw with VisualVM which gave figures of 60% for the count function and 20% for the calls to vec. The discrepancy is most likely due to the fact that the program did not run to completion when I profiled it with VisualVM. The time taken by the count function depends very much on the number of iterations it executes and during the first 25% of the render, most of its calls finish within 4-5 iterations.

In any case, now that we know the source of the slow performance, let’s turn our attention to optimisation.

Back to Primitives

If we want to eliminate boxed numbers while calculating each term of the Mandelbrot formula, then unfortunately that means we will also have to stop being lazy. It’s not possible to create a lazy sequence of primitives (just yet). This means we will need to calculate all the terms of the Mandelbrot formula in a single function before returning the number of iterations taken.

(defn mandelformula [x0 y0 max-iters]
  "Applies the mandelbrot formula until max-iters iterations
  are reached, or the magnitude of Z exceeds 2"
  (let [x0 (double x0)
        y0 (double y0)
        max-iters (int max-iters)]
    (loop [x (double x0)
           y (double y0)
           n (int 0)]
      (if (== n max-iters)
        n
        (let [mag (+ (* x x) (* y y))]
          (if (>= mag (double 4))
            n
            (let [new-x (+ x0 (- (* x x) (* y y)))
                  new-y (+ y0 (* (double 2) (* x y)))]
              (recur new-x new-y (inc n)))))))))

Here is the newly improved function. Note how I explicitly coerce the function parameters to doubles using a let. The intermediate values for each iteration are now defined in a loop and again explicitly coerced. It can be tricky in Clojure to force primitives – there’s no static type checking as in Java of course – but Clojure will throw an exception if you try to pass non-primitive values to a loop that uses coerced parameters. This helps detect the times when Clojure magically autoboxes your primitives behind your back. (One place where I tripped up was thinking that mathematical operators that take multiple arguments would return a primitive when passed primitives, in fact, (* (int 1) (int 2) (int 3)) will return an Integer whereas (* (int 1) (int 2)) returns an int. This is why the calculation of new-y above involves two calls to *).

In the above function, I’m taking care of calculating all the terms of the mandelbrot formula up to the point where z > 2 or max-iters is reached. This is a bit more than the original mandelformula function so the coord-colour function simplifies to:

(defn coord-colour
  "Returns a colour for which to draw the given coordinate. If the coordinate
  is within the mandelbrot set, black is returned. Otherwise, a colour within
  a gradient is given based on the number of iterations of the mandelbrot set
  that have been evaluated."
  [[xcoord ycoord] max-iters]
  (let [num-iters (mandelformula xcoord ycoord max-iters)]
    (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters)
      )))

Let’s see what kind of performance we get now.

user=> (time (profile (my-mandelbrot)))
  Name      mean       min       max     count       sum
colour      5179      1117   1593220    300000  1553876060
mandel      1787      1396    137448    300000  536248035
"Elapsed time: 10177.274766 msecs"

Wow, that’s a 40x speed improvement for the mandel part of our code. This shows how significant keeping object allocations to a minimum can be. However, the two parts of the code we’re profiling only make up about 2 seconds out of a total of 10 seconds of runtime. Let’s move our profiling to the render function:

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (prof :setPixel (.setPixel wr x y
                (prof :int-array (int-array
                  (prof :coord-colour (coord-colour
                    (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                    max-iters))))))))
           (get-pixels width height))))

And run the render again:

user=> (time (profile (my-mandelbrot)))
        Name      mean       min       max     count       sum
coord-colour      6632      3073   1690997    300000  1989613029
   int-array     11402      7263   1706641    300000  3420617714
    setPixel     30688     21790   2191620    300000  9206612546
"Elapsed time: 11283.777407 msecs"

You can see that setPixel is most costly function, but this is not really a surprise given the other two calls are nested within it. However, even taking the runtime of the nested calls into account, the .setPixel call is taking nearly 6 out of the total 11 seconds (the reason we have an extra 1s of runtime after this run is most likely due to the overhead of profiling another block of code).

Removing Reflection

We can see that .setPixel is called on a binding called wr but Clojure (nor us) has no idea what type wr might be. In fact Clojure has to use reflection to determine which method .setPixel refers to. Let’s turn on reflection warning and load the program again:

Reflection warning, D:\dev\clojure\...\mandelbrot.clj:87 - call to setPixel can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:87 - call to setPixel can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:99 - call to drawImage can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:105 - call to setPreferredSize can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:106 - call to add can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:117 - reference to field getRaster can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:122 - reference to field repaint can't be resolved.
1:1 user=>

We can see that with reflection warning turned on, Clojure complains that it cannot resolve the call to setPixel. We can fix this of course with a type hint, however we must also remember to coerce the parameters into the types expected by the setPixel method:

(defn render [xstart ystart xsize ysize width height max-iters #^WritableRaster wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (prof :setPixel (.setPixel wr (int x) (int y)
                (prof :int-array (int-array
                  (prof :coord-colour (coord-colour
                    (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                    max-iters))))))))
           (get-pixels width height))))

Running the program again we notice the reflection warning has gone (the others remain but we don’t care about them as their calls are not made within a loop), and we get a fairly significant speed up:

user=> (time (profile (my-mandelbrot)))
        Name      mean       min       max     count       sum
coord-colour      6770      3072  108868865    300000  2031060305
   int-array     12263      6704  221714492    300000  3679041832
    setPixel     16247     10057  221746898    300000  4874399625
"Elapsed time: 6739.067116 msecs"

This now means that the setPixel, int-array and coord-colour sections now take 17%, 25% and 30% of the total elapsed time respectively. These proportions seem pretty even so it’s possible that our profiling representing a significant portion of that elapsed time. Let’s run again, this time without making the call to the profile macro:

user=> (time (my-mandelbrot))
"Elapsed time: 1668.624085 msecs"

That’s quite a big drop and it shows how significant the overhead of profiling can be but I’m hoping that this can be reduced with optimisations to the profiling macros themselves in the future.

We saw earlier how the section of code for calculating a colour gradient was taking about 75% of the time of the mandelformula function. It’s possible this function could be sped up with more primitive type arithmetic but let’s try a different, more functional method of optimisation.

Memoization

The iter-colour function takes a number between 1 and max-iter and returns a corresponding colour. The fact that there is only a small, fixed number of possible inputs to the function and the output is always the same for a given input, makes it a perfect candidate for memoization. We can simply redefine the function as a memoized version of itself:

user=> (def iter-colour (memoize iter-colour ))
#'user/iter-colour
user=> (time (my-mandelbrot))
"Elapsed time: 954.501759 msecs"

We have now managed to improve overall performance by over 30 times. It’s possible we could repeat these techniques such as inlining the get-pixels function or making the get-coord function use primitives to get a further performance boost. However, as cgrand says, an optimisation job is never done and for now I believe the performance is “good enough”.

The full optimised code can be found on github. Please feel free to submit your own performance optimisations to improve the code even further

Conclusion

We’ve seen how to use profiling tools to peer beneath Clojure’s dynamic lispy surface and used this knowledge to dramatically improve the performance of our fractal renderer by taking advantage of function inlining, primitive coersions, elimination of reflection and memoization. Hopefully these techniques will be useful to you when improving the performance of your own Clojure programs!

References

Inspiration on how to write a Mandelbrot fractal renderer the ‘right’ way: http://briancarper.net/tag/mandelbrot
Performance tips on Clojure: http://devlog.bigmonachus.org/2009/03/performance-tips-for-clojure.html

Every programmer is familiar with the command Print “Hello World” or some variation depending upon language used (this one being Python).
When we step back to think about what that single line of code represents, in terms of all the processing that a computer undertakes, from interpreting the command to instructing the right peripherals to result in “Hello World” appearing [typically] on the computer screen, we grasp in that instance the power of abstraction.

IT is all about abstraction, with many advances ultimately being about introducing higher levels of abstraction or offering a new perspective on an existing level of abstraction.

As we consider cloud computing, abstractions abound, whether we think of it in terms of infrastructure, platforms or applications we are seeing entities that we readily recognize being abstracted as a set of services.

More exciting is the change in context we can expect as our knowledge around cloud computing matures. Specifically in the realm of PaaS, I see us moving from a container-centric perspective to an application-centric perspective. Currently the container is king, we think about setting up app servers to host our applications, with clouds the app will eventually become king, were we think of running an application in the cloud, and focus more on how we want the app to behave, rather than the topology of components (web servers, app servers, databases etc) required to enable the application to behave the way we want. In this near future that is all abstracted by the cloud.
While we will see and already seeing this future materialize in public cloud offerings, the impact on enterprise data centers that will host private clouds, and the approach enterprises will take to developing and running applications will be much more profound.

I’ve seen the future and the forecast is cloudy.

Google recently announced their new computer language “Go”, which I will admit repulsed me on first impression (you write a new language and THAT is what code looks like?). Perhaps it is more efficient to type and compile, but code is read more than it is written. It is not sexy like Python, but it IS new programming language so I felt obligated to suppress my gag reflex and learn more (http://golang.org/). After reading the tutorials, FAQs, and watching the tech talk I began to understand the the “why” of Go (concurrency + fast compiles), which reduced my “eww” reaction.

Interesting features

• It is a compile time error import a package that you don’t use (builds are fast and minimal)
• All needed packages must be explicitly imported (dependencies are explicit)
• goroutines/channels make writing code that executes concurrently easy
• garbage collected (because multi-threaded memory management is hard)
• Did I mention fast compiles? (perhaps I can forgive some of the syntax if that is what it takes)

I am very intrigued by goroutines and channels since I have not worked with any languages that have features like these (although Go is not the first language to have these features). I hope to write my first Go program that uses these features soon.

I’m spending a lot more time teaching other people to code than I am actually coding at the moment.

Whilst I like teaching, this is not at all in line with how I want to be spending my time.

I TA a mandatory computer course in the management school, and I have a friend taking another mandatory programming course for Engineers. The course I TA teaches VB, and the Engineering course teaches C.

These students aren’t stupid, the students learning VB only have to ask you whether you use VB (of course not, does any self-respecting programmer?) and they know they aren’t learning anything that can’t be done using the built in functions in Excel. The students learning C notice that their TA’s (and the prof) keep accidentally giving them Java code. According to O’Reilly, Java has been overtaking C++ as a teaching language for over a decade. Why are these courses so behind?

I think this is a missed opportunity. Students taking Computer Science have signed up for a degree of torment-by-compiler. These students have not. Instead of a course that they hate why not take the opportunity to create a course that excites them about the potential of programming? Instead of alienating them about the whole concept you motivate them with the reasons and real world examples for why programming is a useful skill to have?

For instance: mashup creation, simple scripting, what an API is (and how to make calls to it), web apps.

Yes, it requires more creativity in course creation. Yes, it may require the people teaching these courses and the TAs to learn more skills. Maybe before it didn’t make a big difference to send a bunch of students into different fields with a hatred for programming. In the new reality though, an understanding of programming and why it is useful would potentially give them a big head start.

What do you think?

Despite all the talk of Web 2.0 and the shiny multicore machines with their gigabytes of RAM and billions of cycles per second, I sometimes can’t help feel that we are still very much in the dark ages of computing. This time around my dark gloomy feelings have been brought about by this message to a mailing list which in turn was sparked off by the announcement of the Go Programming Language. As a computer user and a programmer I feel that the actual use of computers is far below their potential.

As the years go by, it seems like we keep on piling layer on top of layer while the results aren’t proportional to what we have to learn to get things done. Now, I’m not proposing that we all start writing down-to-the-metal code or force everyone to become a programmer, but things are starting to look like a mess. Web programming is an interesting development, but it adds yet another layer on top of the existing kernel, operating system, libraries and GUI toolkits. Add to that the fact all browsers are still a bit different from each other and you can start to understand why I’ve yet to make a serious foray into web programming.

But even without the web and the many formats and barely interoperating systems out there, there’s enough on the desktop to get you depressed. Start with the fact that there are currently three major operating systems out there and if you want to write a program that runs on all three of them, you don’t have an easy task. You either embrace three different toolkits and programming methodologies and maintain 3 very different codebases, or you use something like Java which works on all three, but screams non-native on each one. Even though there are languages like Python that run on all of three, it really puts me off that there is still no top-notch multiplatform GUI library. wxWidgets tries pretty hard, but if you look at the screenshots you can pretty easily that they don’t look quite right. It’s not very surprising that lots of smart developers are flocking to the web, where things in comparison are a lot smoother.

There is also the fact that programming languages, like all other pieces of major software, suck more than others. I still stand by what I said in my last post, that it’s an exciting time for language enthusiasts, but I also feel that there are some lessons we really need to learn. I’m starting to have concerns that there may not be any true general purpose language, simply because there are so many different types of problems to be solved. I think we need to start creating broader categories: a set of systems languages similar to C going in the direction of D and Go. A set of hyper-optimizing VM-based languages designed for long-running, parallel server applications (the current JVM is a good example). A set of languages for writing end-user apps that are significantly high-level, but are still compiled to pretty fast native code (maybe not C or even optimized VM fast, but better than todays Python or Ruby). I’m thinking Python in its Unladen Swallow incarnation might fill this gap.

As a programmer, the state of tools that we have to use is really quite depressing. Tools like Emacs and Vi are powerful and all, but let’s face it: we could really be having much more powerful IDE technology. We should be having full blown incremental compilation with autocompletion and support for rendering documentation for every major language out there. We should also have seamless version control with granularity down to the undo level. Every change I make should be saved and I should be able to visually browse all these changes, see what they are and restore to an older state (or commit them if I want to). We have the raw computing power needed to do all this, but yet we remain stuck doing mostly batch-style edit-compile-debug cycles and mucking around in plain text. Eclipse with its incremental compiler makes things much easier, but there’s so much more we could be using our machines for.

As a user, what irritates me is the amount of manual labor we still have to do on a daily basis. We still have to carefully name and place files so that we can file them later. I have to manually hit the save button (see version control bit above). Even with the Internet collaboration is a mess with most people throwing around emails with increasingly larger attachments. Add to that the fact that most email clients are pretty dumb pieces of software. Google Wave is a step in the right direction, if enough people get around to actually using it (and if it can integrate to some extent at least with the desktop). Also I think the web and the desktop need to be brought closer together. Ideally I would be able to sit down on any computer with a live Internet connection and have my full custom work environment (or at least the most important parts of it).

I’m fully aware that none of the things I’ve mentioned are trivial. In fact, they’re probably very hard projects that will take expert teams a good few years to complete. One day I would like to seriously work on some of the programmer-related issues, especially the IDE part. I love Emacs, but there are some parts of Eclipse I really like too. For the time being I’m going to have to make do with what I have, but I’ll be sure to keep an eye for interesting things and movements in the right direction.

Vamos continuar com as minhas divagações sobre herança prototípica.

Continuarei citando mais vantagens e desvantagens entre a herança prototípica em relação a baseada em classes. Num outro post, quando sobrar um tempinho, vou falar sobre as maneiras de simular a flexibilidade da herança prototípica em Ruby por meio do uso de módulos como mixins, metaprogramação e do método Class#new que aceita um bloco como parâmetro.

Vantagens

Algumas vantagens que não foram citadas no primeiro post:

• Relacionamentos simplificados entre objetos: para entender como os objetos se relacionam em um sistema baseado em classes devemos saber que podem existir dois tipo de relações entre eles. A primeira, citada no paper como ‘is_a’, indica se um objeto é instância de uma classe. A segunda, ‘kind_of’, indica se a classe objeto é subclasse de outra. Quando se tem herança prototípica, a única relação entre dois objetos é ‘inherits_from’ que indica se ele foi clonado de outro.
• Criação por cópia: instanciar um objeto a partir de uma classe é como construir uma casa a partir de uma planta. Clonar um objeto a partir de outro seria então uma metáfora mais simples para instanciação.
• Exemplos de módulos pré-existentes: Protótipos são mais contretos do que classes por que eles são modelos existentes com comportamento definido, que pode ser observado. Eles são exemplos do comportamento que será clonado por outros objetos. Classes são descrições do que os objetos instanciados por elas virão a ser. Há uma diferença entre ser um exemplo que se pode interagir e uma descrição estática.

Desvantagens

As principais desvantagens são relacionadas à organização do código. Como saber se um objeto só existe para ser protótipo de outros? Esse problema é um sistema grande pode dar uma baita dor de cabeça. Uma saída, ainda que bem precária (na minha humilde opinião para isto pode ser usar convenções para nomes de variáveis.

Chris Hundhausen has a really important paper in the latest issue of ACM TOCHI: Can direct manipulation lower the barriers to computer programming and promote transfer of training?.

We’ve known for a couple decades now that programmers read and understand visual programs no better than textual programs — Thomas Green, Marian Petre, and Tom Moher settled that question a long time ago.  However, everybody experiences that starting with a visual programming language is easier than a textual language.  But does it transfer?  If you want students to eventually program in text, does starting out with Alice or Squeak or Etoys hurt? Given Chris found: “We found that the direct manipulation interface promoted significantly better initial programming outcomes, positive transfer to the textual interface, and significant differences in programming processes. Our results show that direct manipulation interfaces can provide novices with a ‘way in’ to traditional textual programming.”  I think that this is big news for computing educators.

Here you are now a new open source programming language called Vala that has been invented by GNOME organization to be one of the best choices for GNOME developers so here you are a list of features that I have discovered in Vala :

• Vala is an object oriented language that mostly influenced by C# and Java.
• Vala works perfectly in a cross-platform environments such as ( Linux , Unix , Mac OS X , Microsoft Windows and more ).
• Vala has been designed to allow access to existing C libraries and especially built for GObject-based libraries.
• Vala has a convenient library binding process.
• Vala has a fabulous speed because the compiler produces C source and header files from the source code files as if you’ve written your library or application directly in C !
• Vala gives the developers freedom to use built in assisted memory management or their own memory management techniques.
• Vala supports UTF-8 encoding.
• Vala supports portability to implement new open source technologies.

I make no secret of the fact that I have a very strong interest in programming languages. So I was naturally very interested when news of the Go Programming Language hit the intertubes. Go is an interesting language. It pulls together some very powerful features with a familiar, but clean syntax and has lightning fast compile times. It certainly takes a place on my to-learn list along with Haskell and Scala. But even as Go becomes the latest hot piece of language news, it dawned on me that over the past few years we’ve seen a slew of interesting languages offering compelling alternatives to the industry “mainstream”.

I guess it all started with the rise of scripting languages like Python, PHP, Ruby and the poster boy of scripting: Perl. Personally, these languages with their dynamic typing, “batteries included” design and interesting syntax provided a breath of fresh air from the likes of C++ and Java. Not that C++ and Java are necessarily bad languages, but they aren’t the most interesting of modern languages. In the early years of this decade computers were just getting fast enough to write large scale software in scripting languages. Things have changed a lot since then.

Dynamic languages aren’t just reserved for small scripts. Software like Ruby on Rails has proved that you can write really robust back end infrastructure with them. The languages for their part have kept on growing, adding features and making changes that keep them interesting and downright fun to use. Python 3.0 was a brave decision to make a break from backwards compatibility in order to do interesting things and it goes to show that these languages are far from ossifying or degrading.

Then there is JavaScript which was supposed to die a slow death by attrition as web programmers moved to Flash or Silverlight. But we all know that didn’t happen. JavaScript has stayed in the background since the rise of Netscape, but it’s only recently with advances in browser technology and growing standards support that it has really come into its own. I’ve only played with it a little, but it’s a fun little language which makes me feel a lot of the same emotions I felt when discovering Python for the first time. Thanks to efforts like Rhino, you can even use JavaScript on the client side for non-web related programming.

Of course, if you want to do really interesting things with these languages, then performance is not optional. Within the last year or two there’s been a strong push in both academia and industry to find ways to make these languages faster and safer. Google in particular seems to be in the thick of it. Chrome’s V8 JavaScript engine is probably the fastest client side JavaScript environment and their still experimental Unladen Swallow project has already made headway in improving Python performance. V8 has already enabled some amazing projects and I’m waiting to see what Unladen Swallow will do.

While we’re on the topic of performance, mentioning the Java Virtual Machine is  a must. The language itself seems to have fallen from grace lately, but the JVM is home to some of the most powerful compiler technology on the planet. It’s no wonder then that the JVM has become the target for a bunch of interesting languages. There are the ports of popular languages — JRuby, Jython and Rhino. But the more interesting ones are the JVM-centric ones. Scala is really interesting in that it was born of an academic research project but is becoming the strongest contender to Java’s position of premier JVM language. Clojure is another language that I don’t think many people saw coming. It brings the power of LISP to a modern JVM unleashing a wide range of possibilities. It has it’s detractors, but it’s certainly done a fair bit to make Lisp a well known name again.

Academia has always been a hot bed when it comes to language design. It’s produced wonders like Lisp and Prolog and is making waves again with creations like Haskell (whose goal is ostensibly to avoid popularity at all costs) and the ML group of languages. These powerful functional languages with wonderful type inference are a language aficionado’s dream come true in many ways and they still have years of innovation ahead of them.

Almost as a corollary to the theoretically grounded functional languages, systems languages have been getting some love too. D and now Go are both languages that acknowledge that C and C++ have both had their heyday and it’s time to realize that systems programming does not have to be synonymous with bit twiddling. D has gotten some flak recently for not evolving very cleanly over the last few years, but something is better than nothing. Also a real shift towards eliminating manual memory management is a welcome addition.

As someone who intends to seriously study language design and the related concepts in the years to come, it’s a really great time to be in getting involved in learning about languages. At the moment I’m trying to teach myself Common Lisp and I have a Scala book sitting on the shelf too. One fo these days, I plan on sitting down and making a little toy language to get used to the idea of creating a language. Till then, it’s going to be really interesting just watching how things work out in an increasingly multilingual world.

I’ve made significant progress on my prototype iPhone app since my last post. I’m continuing to learn something new during each coding session. This post contains my learning nuggets and a few code snippets that I hope you’ll find useful.

Table View Development

I’m developing a productivity app, so it makes heavy use of UIKit’s UITableView classes. Here’s a few key things to remember as you develop your UITableViewController.

1. Set a table cell’s text elements from your model in thecellForRowAtIndexPath method
2. Set a cell’s row height by implementing the heightForRowAtIndexPath method
3. Change the appearance of a cell’s label text (textLabel and detailedTextLabel) by setting the corresponding property.

cell.textLabel.font = [UIFont systemFontOfSize:[UIFont labelFontSize]];
cell.textLabel.textColor = [UIColor grayColor];

4. Respond to row selections in the didSelectRowAtIndexPath method. The code for inclusive and exclusive selection is in Apple’s iPhone Table View Programming guide here.

Manipulating Dates

Displaying dates requires the use of the NSDateFormatter class.

1. Avoid learning the hard way, case matters in your date format strings, e.g., yyyy-MM-dd. Here’s a good post that covers the problem.
2. Computing dates (or what Apple refers to as calendrical calculations) is a bit too verbose from me. Here’s how to calculate the top of the hour.

NSDate *today = [NSDate date];
NSCalendar *gregorian = [[NSCalendar alloc] initWithCalendarIdentifier:NSGregorianCalendar];

NSDateComponents *minutesComponent = [gregorian components:NSMinuteCalendarUnit fromDate:today];
NSDateComponents *secondsComponent = [gregorian components:NSSecondCalendarUnit fromDate:today];

NSDateComponents *componentsToSubtract = [[NSDateComponents alloc] init];
[componentsToSubtract setMinute:0 - [minutesComponent minute] ];
[componentsToSubtract setSecond:0 - [secondsComponent second] ];

NSDate *topOftheHour = [gregorian dateByAddingComponents:componentsToSubtract toDate:today options:0];

Internet Access

For applications accessing internet resources, here’s a few tidbits:

• To indicate network activity, it’s easy to add spinner or UIActivityIndicatorView to a view. However, if the main thread is too busy, it will not appear.
• The result of a URL connection (or NSURLConnection) is a binary NSData object, convert it to a string by invoking initWithData in NSString.
• Parsing JSON objects is a breeze with the JSON Framework for Objective-C. By the way, for server-side Java, check out Flexjson, works like a charm.

Parsing Newlines

Is parsing newlines as simple as searching for “\n”? No, take a gander at this code from Apple’s documentation:

- (NSMutableArray *) parseRawString: (NSString *) rawString  {
	int unsigned length = [rawString length];
	int unsigned paraStart = 0, paraEnd = 0, contentsEnd = 0;
	NSMutableArray *array = [NSMutableArray array];
	NSRange currentRange;
	while (paraEnd < length) {
	      [rawString getParagraphStart:&paraStart end:&paraEnd contentsEnd:&contentsEnd forRange:NSMakeRange(paraEnd, 0)];
	      currentRange = NSMakeRange(paraStart, contentsEnd - paraStart);
	      [array addObject:[rawString substringWithRange:currentRange]];
	}
	return array;
}

Lastly, Google search and stackoverflow are your friends. If you get stuck, Google it, you’ll most likely find the answer in stackoverflow or the iPhone Dev SDK forum. (However, always read Apple’s iPhone programming guides first).

Well,that’s all I have for now, happy programming!

Clojure is a functional language that brings dynamic types and lisp syntax to the JVM. It has a number of features that make it interesting which will be the subject of another post but for now, let’s dive right into some code.

The Mandelbrot Set

The Mandelbrot set is an example of a fractal which is basically something that can be described using a simple formula and yet exhibit fascinating levels of detail and beauty. I thought it would be a good application to get me started learning Clojure.

The Mandelbrot formula is:

Z = Z² + C

Where Z is a complex number and C is a constant. To find out whether or not C is in the set, we apply the formula iteratively until the magnitude of Z becomes larger than 2, or we reach a certain number of maximum iterations.

Most 2D fractals are described mathematically using complex numbers which are just like regular (real) numbers but with an added 2nd dimension. They can be represented by two regular numbers so another way to describe the formula above might be:

[x, y] = [x, y]² + [a, b]

Where [a, b] is equivalent to C and represents the coordinates on the complex plane we’re interested in. By applying the formula we can find out whether or not C is in the Mandelbrot set. If it is, we draw it with a black pixel, otherwise with some other colour depending the number of iterations taken by the formula.

Defining a Functional Mandelbrot Set

Let’s try to write this in Clojure code. First of all, how do you square a complex number? First let’s represent Z as a vector of regular numbers [x, y]. To square them we just apply the formula:

[x, y]² = [x² - y², 2xy]

In Clojure, it looks like this:

(defn mandel [[x y]]
  [(- (* x x) (* y y)) (* 2 x y)])

But this represents just half of the Mandelbrot formula, we need to add that constant C and also somehow call our mandel function iteratively. Normally, in an imperative language, this would be easy. We just assign C to our starting value, then calculate Z in a for loop. In Clojure, it’s a bit different:

(defn mandelformula [xcoord ycoord]
  (iterate
    #(vec (map + (mandel %) [xcoord ycoord]))
    [xcoord ycoord]))

This function uses the iterate function to create successive calls to the mandel function passing our initial starting value [xcoord, ycoord] which represents C in the original formula. The mandel function is actually wrapped in an anonymous function which adds the return value to the initial value hence giving us the Z² + C. The reason we have to use map is that we cannot directly add the two vectors together. The individual x and y values must be added separately and then constructed back into a vector with the vec function.

(defn mag [[x y]]
 (+ (* x x) (* y y)))

(take 50 (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord)))

(defn coord-colour
  [[xcoord ycoord] max-iters]
  (let [num-iters (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord))))]
    (if (= max-iters num-iters)
    *set-colour*
    (iter-colour num-iters max-iters))))

;; Colours used to draw the set
(def *grad-colour-a* [255, 255, 0]) ;yellow
(def *grad-colour-b* [0, 0, 255]) ;blue
(def *set-colour* [0, 0, 0]) ;black

(defn grad-colour
  "Returns the colour that is the given fraction of the way between
  the first and second colours given. Returns as a vector of three
  integers between 0 and 255."
  [colA colB frac]
  (vec (map #(+ (* frac (- %2 %1)) %1) colA colB)))

(defn iter-colour
  "Returns the colour needed to paint a point with the given number
  of iterations"
  [num-iters max-iters]
  (grad-colour *grad-colour-a* *grad-colour-b*
    (/ (double num-iters) max-iters)))

(defn get-coord
  "Returns the coordinates of the given pixel in the complex plane"
  [x y xstart ystart xsize ysize width height]
  [(+ xstart (* (/ x width) xsize))
  (+ ystart (* (/ y height) ysize))])

(defn get-pixels [width height]
  (for [y (range height) x (range width)]
    [x y]))

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (pmap (fn [pixel]
            (let [[x y] pixel]
              (.setPixel wr x y
                (int-array (coord-colour
                  (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                  max-iters)))))
          (get-pixels width height))))

(defn get-img [width height]
  (BufferedImage. width height (BufferedImage/TYPE_INT_RGB)))

(defn get-panel [width height img]
  (proxy [JPanel] [] (paint [g] (.drawImage g img 0 0 (Color/red) nil))))

(defn construct-frame [width height panel]
  "Creates and displays a JFrame of the given dimensions with
  the panel added to it"
  (let [frame (JFrame.)]
    (.setPreferredSize panel (Dimension. width height))
    (doto frame
      (.add panel)
      .pack
      (.setLocationRelativeTo nil)
      .show)))

(defn mandelbrot [xstart ystart xsize ysize width height max-iters]
  "Returns a function to render the mandelbrot set with the
  given parameters on a frame"
  (let [img (get-img width height)
        panel (get-panel width height img)
        wr (.getRaster img)]
    (construct-frame width height panel)
    (fn []
      (do
        (render xstart ystart xsize ysize width height max-iters wr)
        (.repaint panel)))))

(def my-mandelbrot (mandelbrot -2 -1.25 3 2.5 1200 1000 50))

(future (my-mandelbrot))