This post will demonstrate how to build a simple Clojure Web application, with Compojure, that will use Incanter to generate a sample from a normal distribution with parameters (size, mean, and standard deviation) provided by the user, and then return a PNG image of the histogram of the data.

This example is simple, but it demonstrates how to dynamically generate an Incanter chart and pass it directly to the Web client as a PNG image.

For more information on the Compojure Web framework, visit its Github repository, its Google group, its WikiBooks site, and this quick tutorial.

I will use the Leiningen build tool to download dependencies, compile the application, and package it up as jar file. For more information on installing and using Leiningen, see my previous post.

Let’s start off by creating a project directory, I’ll call it incanter-webapp, and give it a src subdirectory. Then create the following Leiningen project.clj file in the project directory.

(defproject incanter-webapp "0.1.0"
  :description "A simple Incanter web-app."
  :dependencies [[incanter "0.9.0"]
                 [org.clojars.ato/compojure "0.3.1"]]
  :main simple_web_app)

This project file includes two dependencies, Incanter and Compojure, and indicates that the -main function for the project is in the file simple_web_app.clj in the src subdirectory of the project.

Next create the file called simple_web_app.clj in the src subdirectory.

(ns simple-web-app
  (:gen-class)
  (:use [compojure]
        [compojure.http response]
        [incanter core stats charts])
  (:import (java.io ByteArrayOutputStream
                    ByteArrayInputStream)))

Note that the namespace for this file is simple-web-app with hyphens instead of the underscores used in the file name or in the :main line of the project.clj file.

In Compojure, you need to define routes, which in this case associate the function sample-form with “/”, and the function gen-samp-hist-png with “/sample-normal”.

(defroutes webservice
  (GET "/"
    sample-form)
  (GET "/sample-normal"
    (gen-samp-hist-png request
                       (params :size)
                       (params :mean)
                       (params :sd))))

The function gen-samp-hist-png takes four arguments, a request object, the sample size, the population mean, and the population standard deviation, and then updates the Compojure response object to include an Incanter histogram of a sample from a normal distribution with the given parameters.

Now define the gen-samp-hist-png function; start by converting the string-arguments, passed in from the params object, into numbers.

(defn gen-samp-hist-png
  [request size-str mean-str sd-str]
    (let [size (if (nil? size-str)
                 1000
                 (Integer/parseInt size-str))
          m (if (nil? mean-str)
              0
              (Double/parseDouble mean-str))
          s (if (nil? sd-str)
              1
              (Double/parseDouble sd-str)))

Each argument has a default value, 1000 for size, 0 for mean, and 1 for the standard deviation (sd).

Next generate the sample from the normal distribution using Incanter's sample-normal function with the given size, mean, and standard deviation, and then create a histogram of the resulting data.

          samp (sample-normal size
                              :mean m
                              :sd s)
          chart (histogram
                  samp
                  :title "Normal Sample"
                   -label (str "sample-size = " size
                                ", mean = " m
                                ", sd = " s))

I've used the x-label option of the histogram function to customize the x-axis label so that it includes the given sample size, mean, and standard deviation.

Next, we need to convert the histogram into a byte array, and feed that into a ByteArrayInputStream that can be passed to Compojure's update-response function, along with a header that sets the Content-Type to "image/png".

        out-stream (ByteArrayOutputStream.)
        in-stream (do
                    (save chart out-stream)
                    (ByteArrayInputStream.
                      (.toByteArray out-stream)))
        header {:status 200
                :headers {"Content-Type" "image/png"}}]
    (update-response request
                     header
                     in-stream)))

The above code is the key to sending a dynamically generated chart directly to the client. In addition to a filename, an OutputStream can be passed to Incanter’s save function; in this case, we use a ByteArrayOutputStream, which is then converted to a byte array that is used to initialize the ByteArrayInputStream that is passed to update-response.

We can create a Web form for submitting requests to /sample-normal. The following function is a helper function that creates a simple html page skeleton.

(defn html-doc
  [title & body]
  (html
    (doctype :html4)
    [:html
      [:head
        [:title title]]
      [:body
       [:div
	[:h2
	 [:a {:href "/"}
          "Generate a normal sample"]]]
        body]]))

Now define the sample-form function, which will generate a Web form using html-doc. This function was associated with “/” earlier using the defroutes macro.

(def sample-form
  (html-doc "sample-normal histogram"
    (form-to [:get "/sample-normal"]
      "sample size: " (text-field {:size 4} :size)
      "mean: " (text-field {:size 4} :mean)
      "sd: " (text-field {:size 4} :sd)
      (submit-button "view"))))

Now, we need to create a -main function that will be called when the incanter-webapp.jar is executed. This function will call Compojure’s run-server function, starting the built-in Jetty server on port 8080.

(defn -main [& args]
  (run-server {:port 8080}
    "/*" (servlet webservice)))

Now we can use Leiningen to download and install all the necessary dependencies, including Incanter and Compojure, and then build and package the app.

$ lein deps
$ lein compile
$ lein uberjar

Finally, to start the server, run

$ java -jar incanter-webapp.jar 

and visit http://localhost:8080 to submit a request, or go directly to the sample-normal app by constructing a URL like the following:

http://localhost:8080/sample-normal?size=500&mean=100&sd=10

The complete code for this example can be found here, and the Leiningen project file can be found here.

This might be proof that I’m crazy:

I’m working on a project for my NLP class that involves generating a semantic vector space for Japanese text, and I decided that this might be a good time to learn one of the LISP dialects.  I’ve been looking at Clojure for a while now, but I hadn’t taken the time to learn it before.  I must say, I’m quite impressed so far.  The fact that reading a Japanese XML document into a data structure “just works” without any tweaking is pretty nice.  I’m still coming to grips with functional programming, but I’m liking it so far.  And the best thing as far as I’m concerned is that Clojure code can easily be made massively parallel thanks to all of its data types being immutable (ala Erlang).  That will help me out a lot when I need to run my code on multi-core / multi-processor machines (which I often have to do because of the sheer amount of data being crunched.)

In my previous post I showed how I created a Mandelbrot fractal renderer in Clojure, however its performance left a lot to be desired. Optimising code is actually a very good way to learn more about a new language as it teaches you what is going on under the hood. In this post, I will go through the steps I took to drastically improve the application’s performance.

Inspecting the Code

The first step when optimising code (after not optimising of course) is to inspect the code’s logic and find quick fixes to any obvious potential performance areas. The logic of the Mandelbrot code however is fairly easy to reason about as we have built it up from the mathematical formula. Also, looking for obvious performance issues is difficult in a language you are trying to learn, so it’s time to break out the tools.

Profiling

First of all, let’s see how long it takes to render our fractal with the current implementation:

user=> (time (my-mandelbrot))
"Elapsed time: 29340.692742 msecs"
nil

Note, I have undone the premature optimisation I made last time and made the rendering single threaded by replacing pmap with map. This is important when profiling as it ensures we can properly determine which parts of the code are taking the longest. On a quad core cpu, I found that using pmap gave only a 2x speed increase which shows there is quite a lot of overhead in Clojure when processing on multiple threads.

Now at this stage, we have no idea what is taking so long so let’s run the app through a profiler. As Clojure is hosted on the JVM, it allows us to take advantage of many tools that would normally be used for Java applications. I attached the VisualVM profiling tool (which now comes with Java 6) to my application and then called my-mandelbrot again from the REPL. This is what I got after running for a few minutes:

What strikes you when digging through a Clojure call tree is just how deep it can go. It can also be difficult to relate the Java method calls we see back to the original Clojure code. Here I have expanded the calls made within the render function. You can see the two main consumers of CPU time are calls to count and vec. This is surprising as if you have a look at the original render function code, there’s not a call to either the count or vec functions to be seen:

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (.setPixel wr x y
                (int-array (coord-colour
                  (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                  max-iters)))))
           (get-pixels width height))))

In fact, count is called from within the coord-colour function:

(defn coord-colour
  "Returns a colour for which to draw the given coordinate. If the coordinate
  is within the mandelbrot set, black is returned. Otherwise, a colour within
  a gradient is given based on the number of iterations of the mandelbrot set
  that have been evaluated."
  [[xcoord ycoord] max-iters]
  (let [num-iters (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord))))]
    (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters))))

The call to vec is deeper still within the calls to iter-colour and grad-colour:

(defn grad-colour
  "Returns the colour that is the given fraction of the way between
  the first and second colours given. Returns as a vector of three
  integers between 0 and 255."
  [colA colB frac]
  (vec (map #(+ (* frac (- %2 %1)) %1) colA colB)))

(defn iter-colour
  "Returns the colour needed to paint a point with the given number
  of iterations"
  [num-iters max-iters]
  (grad-colour *grad-colour-a* *grad-colour-b*
    (/ (double num-iters) max-iters)))

The reason we see the count and vec calls at the level of the render function is that Clojure (or perhaps the hotspot) appears to inline certain function calls into the calling function. You can see further evidence of this in the above screenshot if you have a look at the divide, add and multiply methods. These are the operations performed by the get-coord function:

(defn get-coord
  "Returns the coordinates of the given pixel in the complex plane"
  [x y xstart ystart xsize ysize width height]
  [(+ xstart (* (/ x width) xsize))
   (+ ystart (* (/ y height) ysize))])

You should also see that at the time of the snapshot I took above, the number of invocations of each of the mathematical operations was around 67k which is twice the number of invocations of the count function which makes sense as you can see they are called once for each of the x and y coordinates.

Let’s dig deeper into the count function.

As you expand the call tree, and if you ignore the clojure.lang methods, you’ll see calls to take, take-while, iterate and finally mandelformula. Here’s another mystery of the Clojure call stack: we have a call to iterate before the call to mandelformula and yet in our code we can see iterate is called within the mandelformula function.

(defn mandelformula [xcoord ycoord]
  "Returns an infinite sequence of vectors containing the values of
  successive iterations of the mandelbrot formula, given a point on
  the complex plane."
  (iterate
    #(vec (map + (mandel %) [xcoord ycoord]))
    [xcoord ycoord]))

As far as I can tell, the reason for this is that mandelformula returns a lazy sequence which is not actually evaluated until it is needed. So iterate is used to get the next value in the sequence, while mandelformula actually does the job of calculating what the next value should be.

Drilling down into the mandelformula function we open up a whole can of method calls and it gets suddenly very difficult to map back to the original Clojure code. However we can see there seem to be a large number of sequences created and calls to methods in the clojure.lang.Numbers class taking Objects as their parameters. If we could get the formula to only use primitive types, this could potentially remove a lot of object creation and garbage collection. However, before we start re-working the code, let’s look at another profiling technique available to us in Clojure.

Using Clojure Contrib’s Profile Macros

Stuart Sierra has made a vast number of contributions to Clojure including the profile library which provides two macros prof and profile to measure the execution time of any block of code. Clojure’s macro system makes what would normally be a cumbersome task in other languages extremely easy. Let’s put profiling in our coord-colour function:

(defn coord-colour
  [[xcoord ycoord] max-iters]
  (let [num-iters (prof :mandel (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord)))))]
    (prof :colour (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters)))))

We simply wrap the block of code we want to profile around a call to the prof macro and specify a keyword to identify that block of code. Calling the profile macro will then summarise the time taken to evaluate each call to those prof blocks and print a bunch of statistics.

user=> (time (profile (my-mandelbrot)))
  Name      mean       min       max     count       sum
colour      4875      1117   1091200    300000  1462510112
mandel     71729      2514 100182286    300000 21518722246
"Elapsed time: 31456.673404 msecs"

Here we can see that the total time taken by the mandel section of the coord-colour function is around 21 seconds (the values above are in nanoseconds) which gives it 68% of the total running time. The time taken to calculate the gradient colour is about 5% of the total running time. These are quite different statistics from what we saw with VisualVM which gave figures of 60% for the count function and 20% for the calls to vec. The discrepancy is most likely due to the fact that the program did not run to completion when I profiled it with VisualVM. The time taken by the count function depends very much on the number of iterations it executes and during the first 25% of the render, most of its calls finish within 4-5 iterations.

In any case, now that we know the source of the slow performance, let’s turn our attention to optimisation.

Back to Primitives

If we want to eliminate boxed numbers while calculating each term of the Mandelbrot formula, then unfortunately that means we will also have to stop being lazy. It’s not possible to create a lazy sequence of primitives (just yet). This means we will need to calculate all the terms of the Mandelbrot formula in a single function before returning the number of iterations taken.

(defn mandelformula [x0 y0 max-iters]
  "Applies the mandelbrot formula until max-iters iterations
  are reached, or the magnitude of Z exceeds 2"
  (let [x0 (double x0)
        y0 (double y0)
        max-iters (int max-iters)]
    (loop [x (double x0)
           y (double y0)
           n (int 0)]
      (if (== n max-iters)
        n
        (let [mag (+ (* x x) (* y y))]
          (if (>= mag (double 4))
            n
            (let [new-x (+ x0 (- (* x x) (* y y)))
                  new-y (+ y0 (* (double 2) (* x y)))]
              (recur new-x new-y (inc n)))))))))

Here is the newly improved function. Note how I explicitly coerce the function parameters to doubles using a let. The intermediate values for each iteration are now defined in a loop and again explicitly coerced. It can be tricky in Clojure to force primitives – there’s no static type checking as in Java of course – but Clojure will throw an exception if you try to pass non-primitive values to a loop that uses coerced parameters. This helps detect the times when Clojure magically autoboxes your primitives behind your back. (One place where I tripped up was thinking that mathematical operators that take multiple arguments would return a primitive when passed primitives, in fact, (* (int 1) (int 2) (int 3)) will return an Integer whereas (* (int 1) (int 2)) returns an int. This is why the calculation of new-y above involves two calls to *).

In the above function, I’m taking care of calculating all the terms of the mandelbrot formula up to the point where z > 2 or max-iters is reached. This is a bit more than the original mandelformula function so the coord-colour function simplifies to:

(defn coord-colour
  "Returns a colour for which to draw the given coordinate. If the coordinate
  is within the mandelbrot set, black is returned. Otherwise, a colour within
  a gradient is given based on the number of iterations of the mandelbrot set
  that have been evaluated."
  [[xcoord ycoord] max-iters]
  (let [num-iters (mandelformula xcoord ycoord max-iters)]
    (if (= max-iters num-iters)
      *set-colour*
      (iter-colour num-iters max-iters)
      )))

Let’s see what kind of performance we get now.

user=> (time (profile (my-mandelbrot)))
  Name      mean       min       max     count       sum
colour      5179      1117   1593220    300000  1553876060
mandel      1787      1396    137448    300000  536248035
"Elapsed time: 10177.274766 msecs"

Wow, that’s a 40x speed improvement for the mandel part of our code. This shows how significant keeping object allocations to a minimum can be. However, the two parts of the code we’re profiling only make up about 2 seconds out of a total of 10 seconds of runtime. Let’s move our profiling to the render function:

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (prof :setPixel (.setPixel wr x y
                (prof :int-array (int-array
                  (prof :coord-colour (coord-colour
                    (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                    max-iters))))))))
           (get-pixels width height))))

And run the render again:

user=> (time (profile (my-mandelbrot)))
        Name      mean       min       max     count       sum
coord-colour      6632      3073   1690997    300000  1989613029
   int-array     11402      7263   1706641    300000  3420617714
    setPixel     30688     21790   2191620    300000  9206612546
"Elapsed time: 11283.777407 msecs"

You can see that setPixel is most costly function, but this is not really a surprise given the other two calls are nested within it. However, even taking the runtime of the nested calls into account, the .setPixel call is taking nearly 6 out of the total 11 seconds (the reason we have an extra 1s of runtime after this run is most likely due to the overhead of profiling another block of code).

Removing Reflection

We can see that .setPixel is called on a binding called wr but Clojure (nor us) has no idea what type wr might be. In fact Clojure has to use reflection to determine which method .setPixel refers to. Let’s turn on reflection warning and load the program again:

Reflection warning, D:\dev\clojure\...\mandelbrot.clj:87 - call to setPixel can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:87 - call to setPixel can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:99 - call to drawImage can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:105 - call to setPreferredSize can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:106 - call to add can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:117 - reference to field getRaster can't be resolved.
Reflection warning, D:\dev\clojure\...\mandelbrot.clj:122 - reference to field repaint can't be resolved.
1:1 user=>

We can see that with reflection warning turned on, Clojure complains that it cannot resolve the call to setPixel. We can fix this of course with a type hint, however we must also remember to coerce the parameters into the types expected by the setPixel method:

(defn render [xstart ystart xsize ysize width height max-iters #^WritableRaster wr]
  (dorun
    (map (fn [pixel]
            (let [[x y] pixel]
              (prof :setPixel (.setPixel wr (int x) (int y)
                (prof :int-array (int-array
                  (prof :coord-colour (coord-colour
                    (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                    max-iters))))))))
           (get-pixels width height))))

Running the program again we notice the reflection warning has gone (the others remain but we don’t care about them as their calls are not made within a loop), and we get a fairly significant speed up:

user=> (time (profile (my-mandelbrot)))
        Name      mean       min       max     count       sum
coord-colour      6770      3072  108868865    300000  2031060305
   int-array     12263      6704  221714492    300000  3679041832
    setPixel     16247     10057  221746898    300000  4874399625
"Elapsed time: 6739.067116 msecs"

This now means that the setPixel, int-array and coord-colour sections now take 17%, 25% and 30% of the total elapsed time respectively. These proportions seem pretty even so it’s possible that our profiling representing a significant portion of that elapsed time. Let’s run again, this time without making the call to the profile macro:

user=> (time (my-mandelbrot))
"Elapsed time: 1668.624085 msecs"

That’s quite a big drop and it shows how significant the overhead of profiling can be but I’m hoping that this can be reduced with optimisations to the profiling macros themselves in the future.

We saw earlier how the section of code for calculating a colour gradient was taking about 75% of the time of the mandelformula function. It’s possible this function could be sped up with more primitive type arithmetic but let’s try a different, more functional method of optimisation.

Memoization

The iter-colour function takes a number between 1 and max-iter and returns a corresponding colour. The fact that there is only a small, fixed number of possible inputs to the function and the output is always the same for a given input, makes it a perfect candidate for memoization. We can simply redefine the function as a memoized version of itself:

user=> (def iter-colour (memoize iter-colour ))
#'user/iter-colour
user=> (time (my-mandelbrot))
"Elapsed time: 954.501759 msecs"

We have now managed to improve overall performance by over 30 times. It’s possible we could repeat these techniques such as inlining the get-pixels function or making the get-coord function use primitives to get a further performance boost. However, as cgrand says, an optimisation job is never done and for now I believe the performance is “good enough”.

The full optimised code can be found on github. Please feel free to submit your own performance optimisations to improve the code even further

Conclusion

We’ve seen how to use profiling tools to peer beneath Clojure’s dynamic lispy surface and used this knowledge to dramatically improve the performance of our fractal renderer by taking advantage of function inlining, primitive coersions, elimination of reflection and memoization. Hopefully these techniques will be useful to you when improving the performance of your own Clojure programs!

References

Inspiration on how to write a Mandelbrot fractal renderer the ‘right’ way: http://briancarper.net/tag/mandelbrot
Performance tips on Clojure: http://devlog.bigmonachus.org/2009/03/performance-tips-for-clojure.html

Many applications have the primary concern of storing and retrieving data. The raw data by itself is often not very useful, so an additional process is put into place to turn that data into useful information. Many applications generate these reports from the data quickly at the user’s request through either a narrow SQL select statement, in application data processing, or both. However, in larger applications where the data is too large to handle in memory at a time, processing is too heavy, and report customization is too varied, information generation needs to be pushed to its own scalable system.

A lot of data has various pieces of metadata associated with it naturally. Data such as what user added the record, the date it was added or modified (or both), maybe the size of the record, categories, tags, keywords, or other pieces of associate data that is used to break it up into more manageable chunks. This meta data is useful, but only if we can use it as an aggregate to generate specific information relating to the grouping of items based on this data.

Sometimes generating these more specific reports is as easy as adding additional WHERE or GROUP BY clauses in SQL. However, when more advanced business rules are taking place where there isn’t an easy or succinct way of extracting this information via a query, or if the query returns such a large amount of data as to cause memory issues, a different approach can be taken.

For instance; in an application I am currently working on we need to generate reports based on a table with about 5 million rows. The SQL queries we use can limit the amount of rows returned to perhaps a few hundred thousand for some of our larger reports. However, a lot of the data needs to be processed in application code rather than by the database itself due to some special business rules. Because of this, we end up creating a large number of objects in Java to hold these result rows. If multiple users are generating different reports we might end up holding too many of these objects in memory at a time and receive an OOM error. Also, the processing on this data can be intense enough that if the server is slammed with report requests that the entire system slows down, causing difficulties for people wanting to insert or modify data. This is the case I am in while I contemplate offline report generation.

The basic idea is that the main application should be concerned purely with manipulating the data of the system. That is basic CRUD stuff such as creating new records, updating them, the rare deletions, and showing single records to the user (so they can edit or delete it). We want that part of the application to remain fast, and not be effected by the purely read-only needs imposed by report generation. In order to nullify the impact, we move reporting to its own system that reads from a separate read-only replication of our production database.

When a report request comes in to our application, we send a request to the separate reporting system. This can be done either as a web service or maybe an RPC call. The reporting system uses its read-only copy of the data to generate the report numbers and send it back, causing no speed delay for insertion or regular operation of the main application.

This doesn’t solve our OOM issues however, as many drivers for our database (MySQL) return ResultSet objects with the entire contents of the results which might be too large to fit into memory. However, since we’re using a read-only list anyway we can convert the table or tables we use to process our results into flat files that can be read in on a line by line basis, perform some intermediate result processing, deallocate those lines and work on additional lines. Since our reports are mostly generating statistical data over a large data set, we can process results on that data set in parallel using multiple threads or possibly multiple computers using a Hadoop cluster.

By making report generation asynchronous to our applications general work flow we will free up the processing power and the connection pool that’s used to handle requests by asking users to either poll for when the result is finished or to notify the system when a report is finished and thereby avoid the instances where we use all of our available connections or resources processing reports. There is also the added possibility of continuously generating all possible report data on a separate machine or cluster to decrease access time by increasing storage requirements.

I’m currently researching the use of MapReduce in Hadoop for processing this flat file of all data for reports. In addition, I’m researching a few languages that are reported to be good at concurrent processing so that I can benefit from multiple cores when generating reports from our raw data. My current focus is on Scala, Erlang, and Clojure, but I do not have much to report on those areas yet. If anyone has any information those languages as far as report generation based on a largish data set (currently 5 million, but the rate of growth is fairly alarming), let me know.

My previous post on the state of affairs with Eclipse garnered some interest. I’ve continued to explore the editor scene. Back in the very late 90’s and early 00’s I really started playing around with linux, at the time there was emacs and vim. Interestingly that hasn’t really changed. Back then I chose the way to be VI / VIM. Until getting into the world of lisp I hadn’t looked back. Clojure has me pretty deeply interested in Emacs.

Emacs has shown me a different way of looking at editors. It is so highly scriptable / customizable that I am slightly peeved it took me so long to start experimenting. Sure VI has a bunch of cool features, it is powerful, etc, but compared to Emacs it is like night vs day. That is not to say that Emacs is “better” than VIM – but it is a very different beast.

Indentation is a problem though. Over the last few weeks I threw myself head first into Emacs, I’m not good at learning any other way, and now I’m coming up against barriers in my work flow. I prefer the VIM way of letting you control your indentation. Emacs, thus far, makes too many assumptions about how you want your code to look, for LISP it nails it – but for java, javascript, html, css, etc it really doesn’t need to do anything more than highlight my text in the appropriate manner.

Going forth I think most programming takes place in VIM, lispy stuff takes place in emacs. The limited capacity of my brain will just have to delete everything to do with Eclipse / Netbeans / IntelliJ. Only really refactoring loses out here.

Stuart Halloway’s Programming Clojure book is the only game in town for learning Clojure (at least until Luke VanderHart’s Practical Clojure is released), and Clojure Box is the easiest way to get Clojure running on Windows with a good IDE, if you don’t already have cygwin installed, but they don’t quite play nicely out of the box.

Craig McDaniel has some generally useful code for Emacs/Clojure, and that’s almost enough to get things working. With the following changes incorporated, all of the examples work for me:

(defun reset-swank ()
  "Because changing swank-clojure-extra-classpaths is not enough to force a new instance of slime to use it."
  (interactive)
  (setq slime-lisp-implementations
        (assq-delete-all 'clojure slime-lisp-implementations))
  (add-to-list 'slime-lisp-implementations
               `(clojure ,(swank-clojure-cmd) :init swank-clojure-init) t))

(defun run-slime (dir)
  (interactive "Project directory: ")
  (cd dir)
  (when (file-directory-p "~/.clojure") (directory-files "~/.clojure" t ".jar$"))
    (make-directory "classes"))
  (setq swank-clojure-extra-classpaths '("src" "classes" "lib/*" ""))
  (reset-swank)
  (slime))

Like in Ruby, there is no explicit call in a function to return values, the last expression of a function is evaluated and used as its return value.

Consider the following 2 functions:

(defn hello-with-println [name]
    (println "Hello," name))
    

(defn hello-with-str [name]
    (str "Hello, " name))

The first function when evaluated on the REPL (read-eval-print-loop), which is an interactive programming environment, returns the following:

user=> (hello-with-println "punk")
Hello, punk
nil

The second function returns:

user=> (hello-with-str "munk")
"Hello, munk"

At first glance, these two functions seem to be doing almost the same thing – takes in a string input, which is a name, and outputs a string that says Hello to that name. However, hello-with-println returns nil while hello-with-str returns the evaluation of the str function.

This can be proven further with special variables in the REPL. When working within the REPL, the results of evaluating the three most recent expressions are stored in the special variables *1, *2 and *3. If I call hello-with-println twice with the inputs “munk” and “punk” and use the special variables *1 and *2 to get the results of those expressions, I get the following:

user=> (str *1 " and " *2)
" and "

If I do the same with hello-with-str, I get the following output:

user=> (str *1 " and " *2)
"Hello, punk and Hello, munk"

When hello-with-println was used, there were no return values so the special variables were empty unlike in hello-with-str.

Comparing computer languages can usually be interesting, sometimes informative, often amusing and in lots of cases just damn annoying!

Still this recent Hacker News post about Python vs Clojure did catch my eye.

Despite the fact the author of the blog post had pulled the Python code from answers to Project Euler the pythonista’s quite correctly cried foul because it wasn’t well written idiomatic python example and so was an unfair comparison.

There’s a lot of code out there posted on those intertubes over the years. However languages and practises evolve but unfortunately intertube posts can remain set in stone! Perl is often weighed down by its past history on the intertubes. One good way forward is just to produce more Modern Perl posts.

Anyway i digress, so moving on lets take a look at the Euler 4 – Finding Palindroms. This was the Clojure example that was put forward in the blog post:

(reduce max
    (filter #(let [s (str %)]
               (= (seq s) (reverse s)))
            (for [x (range 100 1000)
                  y (range 100 1000)]
              (* x y))))

Now in reply to this a much more idiomatic python example was provided:

print max(s for s in (x * y
    for x in range(111, 1000)
    for y in range(x, 1000))
    if list(str(s)) == list(reversed(str(s))))

This uses Python’s list comprehension. Definitely a very powerful construct. However i do find i get lost very quickly when trying to follow long nested comprehension’s

In this same Python post there is also mention of a twitter post by Yukihiro Matz, the creator of Ruby. Matz provided the following Ruby example:

p [*100..1000].product([*100..1000]).map{|x,y| x*y}.select{|s|s=s.to_s; s==s.reverse}.max

Despite being formatted for twitter (ie. one line) i found it easy to follow through the logic.

Ruby & Python are not the only languages that can produce an elegant solution! So not to be out done here is a Perl example. In fact its the exact same code repeated three times but formatted differently:

use Modern::Perl;
use List::Util q(max);

# twitter one liner
say max grep { $_ eq reverse $_ } map { my $x = $_; map { $x * $_ } 100..1000 } 100..1000;

# again in lispy style format
say
  max
    grep { $_ eq reverse $_ }
      map {
        my $x = $_;
        map { $x * $_ } 100..1000;
      } 100..1000;

# and finally in my preferred more perlish alignment
say max
    grep { $_ eq reverse $_ }
    map {
        my $x = $_;
        map { $x * $_ } 100..1000;
    } 100..1000;

The perl code speaks for itself.. its wonderfully clear and succinct. Also its fast, each version runs in under a second on my machine!

/I3az/

I seem to be liking Clojure …

Let me backup a bit, in the last couple of weeks I have been debating between picking up Scala or Clojure (don’t get me wrong Ruby is still my favorite).

I always wanted to pick up a functional programming language so I dabbled a bit with Erlang and Haskell, liked Haskell a lot but without much practice it kind of died (sad times)  and Scala seems too much like Java, yeah I know it seems to have a bigger crowd than Clojure and there are a lot of big names behind it.

Maybe that’s exactly why I choose Clojure (since its the underdog), or cause it is different enough from Java or simply cause it has a better syntax and seems more elegant (apparently Clojure has better integration with Java, don’t quote me on it), anyways I decided to learn Clojure.

Peepcode has a nice screen-cast to get you started off on Clojure. If you are on the Mac there is a nice bundle for TextMate and anywhere else Netbeans with the enclojure plugin seems to be the best.

On a side note it seems more and more that Netbeans has the latest and greatest plugins for everything, then comes IntelliJ and finally eclipse, what’s going on with eclipse ? has it reached its peak and now it will start dropping off ? but on the flip side there seems to be more and more apps built on top of the Eclipse RCP like Xmind, so is Eclipse no longer going to be the leader of the IDE and just become a platform for building RCPs. This of course depends  on what Oracle is going to do with NetBeans, I really hope they give the same amount of love to NetBeans as Sun did.

Ok getting back to Clojure, don’t get your panties in a bunch when you see all those parenthesis, it is just the layout that is shocking, indent it well and it is no more than what you are used to.

Here’s an example

(defn fac
“Returns the factorial of n, which must be a positive integer.”
[n]
(if (= n 1)
1
(* n (fac (- n 1)))
)

Is same as

(defn fac [n] (if (= n 1) 1 (* n (fac (- n 1)))))

But the first one is a lot more easier on the eyes (even brain?) than the second one. Most examples that you see look like the second one and it frightens people, don’t let that stop you take my word and go for it.

Clojure seems to be very easy to pick up, things seem very intuitive, like the other day I was wondering, how to return a default value from a map if the key is not found and there is was right there in the api.

(map key default-value)

So simple! I was easily able to extend the examples that came with the peepcode screencast. Anyways I have started on this path, let’s see where it goes.

Functional programming has always been a nightmare for me and it started off in college with a class called 15-212: Principles of Programming, which was taught in SML. Almost everything that was taught in that class went over my head and it’s a miracle that I managed to get a B for it. When I graduated, I swore I never wanted to have anything to do with functional programming ever again, until a few of my coworkers started talking about languages like F#, Scala and Clojure last year. I have tried to avoid it for as long as I can, but it has proven inevitable.

In starting to read and play with Clojure, I have had to learn a lot of basic concepts that I might or might not have known before. I will attempt to highlight those concepts as I stumble upon them and illustrate them with code snippets that I find or ones that I wrote myself. Pardon the juvenile approach, I find I remember things better if I write them down or explain it to someone.

Lexical Scoping – with lexical or static scope, a variable always refers to its top-level environment. You can always determine a value of a variable based on just reading the code, substituting variable bindings as you go. Of course most programming languages work this way, I just never knew there was a term for it.

Here’s some code (taken from this Clojure tutorial by R. Mark Volkmann) that illustrates lexical scoping and the usages of def, let and binding.

(def v 1) ; v is a global binding

(defn f1 []
  (println "f1: v =" v))

(defn f2 []
  (println "f2: before let v =" v)
  (let [v 2]
    (println "f2: in let, v =" v)
    (f1))
  (println "f2: after let v =" v))

(defn f3 []
  (println "f3: before binding v =" v)
  (binding [v 3]
    (println "f3: in binding, v =" v)
    (f1))
  (println "f3: after binding v =" v))

(defn f4 []
 (def v 4))

Calling f2 spits out the following output:

f2: before let v = 1
f2: in let, v = 2
f1: v = 1
f2: after let v = 1

The value of v is 2 only within the let form, and functions that are called within the body cannot see the local bindings of let.

Calling f3:

f3: before binding v = 1
f3: in binding, v = 3
f1: v = 3
f3: after binding v = 1

The binding macro is similar to let, but it creates thread-local values temporarily for existing global bindings. These new values are seen within the form and the functions that are called from within that form.

Calling f4: (println "after calling f4, v =" v):
after calling f4, v = 4

This actually changes the value of the global binding. Now calling f2 and f3 will have the following output:

f2: before let v = 4
f2: in let, v = 2
f1: v = 4
f2: after let v = 4
nil
f3: before binding v = 4
f3: in binding, v = 3
f1: v = 3
f3: after binding v = 4
nil

Clojure is a functional language that brings dynamic types and lisp syntax to the JVM. It has a number of features that make it interesting which will be the subject of another post but for now, let’s dive right into some code.

The Mandelbrot Set

The Mandelbrot set is an example of a fractal which is basically something that can be described using a simple formula and yet exhibit fascinating levels of detail and beauty. I thought it would be a good application to get me started learning Clojure.

The Mandelbrot formula is:

Z = Z² + C

Where Z is a complex number and C is a constant. To find out whether or not C is in the set, we apply the formula iteratively until the magnitude of Z becomes larger than 2, or we reach a certain number of maximum iterations.

Most 2D fractals are described mathematically using complex numbers which are just like regular (real) numbers but with an added 2nd dimension. They can be represented by two regular numbers so another way to describe the formula above might be:

[x, y] = [x, y]² + [a, b]

Where [a, b] is equivalent to C and represents the coordinates on the complex plane we’re interested in. By applying the formula we can find out whether or not C is in the Mandelbrot set. If it is, we draw it with a black pixel, otherwise with some other colour depending the number of iterations taken by the formula.

Defining a Functional Mandelbrot Set

Let’s try to write this in Clojure code. First of all, how do you square a complex number? First let’s represent Z as a vector of regular numbers [x, y]. To square them we just apply the formula:

[x, y]² = [x² - y², 2xy]

In Clojure, it looks like this:

(defn mandel [[x y]]
  [(- (* x x) (* y y)) (* 2 x y)])

But this represents just half of the Mandelbrot formula, we need to add that constant C and also somehow call our mandel function iteratively. Normally, in an imperative language, this would be easy. We just assign C to our starting value, then calculate Z in a for loop. In Clojure, it’s a bit different:

(defn mandelformula [xcoord ycoord]
  (iterate
    #(vec (map + (mandel %) [xcoord ycoord]))
    [xcoord ycoord]))

This function uses the iterate function to create successive calls to the mandel function passing our initial starting value [xcoord, ycoord] which represents C in the original formula. The mandel function is actually wrapped in an anonymous function which adds the return value to the initial value hence giving us the Z² + C. The reason we have to use map is that we cannot directly add the two vectors together. The individual x and y values must be added separately and then constructed back into a vector with the vec function.

(defn mag [[x y]]
 (+ (* x x) (* y y)))

(take 50 (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord)))

(defn coord-colour
  [[xcoord ycoord] max-iters]
  (let [num-iters (count (take max-iters (take-while #(<= (mag %) 4) (mandelformula xcoord ycoord))))]
    (if (= max-iters num-iters)
    *set-colour*
    (iter-colour num-iters max-iters))))

;; Colours used to draw the set
(def *grad-colour-a* [255, 255, 0]) ;yellow
(def *grad-colour-b* [0, 0, 255]) ;blue
(def *set-colour* [0, 0, 0]) ;black

(defn grad-colour
  "Returns the colour that is the given fraction of the way between
  the first and second colours given. Returns as a vector of three
  integers between 0 and 255."
  [colA colB frac]
  (vec (map #(+ (* frac (- %2 %1)) %1) colA colB)))

(defn iter-colour
  "Returns the colour needed to paint a point with the given number
  of iterations"
  [num-iters max-iters]
  (grad-colour *grad-colour-a* *grad-colour-b*
    (/ (double num-iters) max-iters)))

(defn get-coord
  "Returns the coordinates of the given pixel in the complex plane"
  [x y xstart ystart xsize ysize width height]
  [(+ xstart (* (/ x width) xsize))
  (+ ystart (* (/ y height) ysize))])

(defn get-pixels [width height]
  (for [y (range height) x (range width)]
    [x y]))

(defn render [xstart ystart xsize ysize width height max-iters wr]
  (dorun
    (pmap (fn [pixel]
            (let [[x y] pixel]
              (.setPixel wr x y
                (int-array (coord-colour
                  (get-coord (double x) (double y) xstart ystart xsize ysize width height)
                  max-iters)))))
          (get-pixels width height))))

(defn get-img [width height]
  (BufferedImage. width height (BufferedImage/TYPE_INT_RGB)))

(defn get-panel [width height img]
  (proxy [JPanel] [] (paint [g] (.drawImage g img 0 0 (Color/red) nil))))

(defn construct-frame [width height panel]
  "Creates and displays a JFrame of the given dimensions with
  the panel added to it"
  (let [frame (JFrame.)]
    (.setPreferredSize panel (Dimension. width height))
    (doto frame
      (.add panel)
      .pack
      (.setLocationRelativeTo nil)
      .show)))

(defn mandelbrot [xstart ystart xsize ysize width height max-iters]
  "Returns a function to render the mandelbrot set with the
  given parameters on a frame"
  (let [img (get-img width height)
        panel (get-panel width height img)
        wr (.getRaster img)]
    (construct-frame width height panel)
    (fn []
      (do
        (render xstart ystart xsize ysize width height max-iters wr)
        (.repaint panel)))))

(def my-mandelbrot (mandelbrot -2 -1.25 3 2.5 1200 1000 50))

(future (my-mandelbrot))