QUnit-CLI riding a Dog

Today I wanted to put together a QUnit-CLI example that leveraged Java 6′s included scripting features.  Seeing as the Java 6 JDK includes a recent version of Rhino as its primary JavaScript engine, I thought this would be a piece of cake.  Wrong.

To the javax.script package’s credit, Creating a new scripting engine and evaluating some script code is dead simple.  Example below from Oracle’s own pages…

import javax.script.*;

public class EvalFile {
  public static void main(String[] args) throws Exception {
    // create a script engine manager
    ScriptEngineManager factory = new ScriptEngineManager();

    // create JavaScript engine
    ScriptEngine engine = factory.getEngineByName("JavaScript");

    // evaluate JavaScript code from given file - specified by first argument
    engine.eval(new java.io.FileReader(args[0]));
  }
}

The trouble came into play when I ran QUnit-CLI ‘s Rhino based suite.js file.   Ka-blew-ey!

Exception in thread "main" javax.script.ScriptException:
    sun.org.mozilla.javascript.internal.EcmaError: 
    ReferenceError: "load" is not defined. (#1) in  at line number 1
  at com.sun.script.javascript.RhinoScriptEngine.eval(RhinoScriptEngine.java:110)
  at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:232)
  at Java6RhinoRunner.load(Java6RhinoRunner.java:29)
  at Java6RhinoRunner.main(Java6RhinoRunner.java:14)

It turns out that scripts running from in a Rhino Shell environment have access to extra functions that are not directly provided when executing embedded from Java.  The top-level “load” function which is used to load additional JavaScript files from the Rhino Shell is not directly available when running from the scripting engine.

It turns out, I am not the only one with this problem.  My solution was to bind an additional object with a Java-based “load” function and add a new top-level “load” function to the JavaScript scope that invokes the Java code.

import javax.script.*;
import java.io.*;

public class Java6RhinoRunner {
  public static void main(String[] args) throws ScriptException {
    new Java6RhinoRunner().load(args[0]);
  }

  private final ScriptEngine engine;

  public Java6RhinoRunner() throws ScriptException {
    ScriptEngineManager factory = new ScriptEngineManager();
    this.engine = factory.getEngineByName("JavaScript");

    this.engine.put("Java6RhinoRunner", this);
    this.engine.eval("function load(filename) { Java6RhinoRunner.load(filename); }");
  }

  public void load(String filename) throws ScriptException {
    try {
      this.engine.eval(new FileReader(filename));
    }
    catch(FileNotFoundException e) {
      throw new RuntimeException("Error loading javascript file: " + filename, e);
    }
  }
}

To make matters worse, from the Rhino Shell, the “print” function will output the given text to standard output with a newline character.  As far as I can tell there isn’t a non-newline print command available on the Shell.  Annoyingly, when running from Java both print and println are available and tied to their common java behaviors.  This means that my suite.js code which uses “print” needs to use “println” when running from Java.  My first thought was to override print to execute println from my Java runner, but it looks like these basic top-level functions can’ be redefined from JavaScript.

this.engine.eval("function print(message) { println(message); }");

(no errors at runtime, but didn’t work)

The solution was to use add a level of indirection to my suite.js

var out = (typeof println !== "undefined") ? println : print;
out("FAIL - " + name); // used to be a call to print directly

Now we have suit.js running from Rhino embedded within the Java 6 JDK … but things are perfect.  The current code throws “Inappropriate array length” errors for a few QUnit tests.  I will be looking into these next.

Previously I talked about wanting to run QUnit outside the browser and about some issues I ran into.  Finally, I have QUnit running from the command line: QUnit-CLI

After a good deal of hacking and a push from jzaefferer, I gotten the example code in QUnit-CLI to run using Rhino and no browser in sight.  This isn’t a complete substitute for in-browser testing, but makes integration with build servers and faster feedback possible.

/projects/qunit-cli/js suite.js
PASS - Adding numbers
FAIL - Subtracting numbers
PASS|OK|subtraction function exists|
FAIL|EQ|Intended bug!!!|Expected: 0, Actual: -2
PASS - module without setup/teardown (default)
PASS - expect in testPASS - expect in test
PASS - module with setup
PASS - module with setup/teardown
PASS - module without setup/teardown
PASS - scope check
PASS - scope check
PASS - modify testEnvironment
PASS - testEnvironment reset for next test
PASS - scope check
PASS - modify testEnvironment
PASS - testEnvironment reset for next test
PASS - makeurl working
PASS - makeurl working with settings from testEnvironment
PASS - each test can extend the module testEnvironment
PASS - jsDump output
PASS - raises
PASS - mod2
PASS - reset runs assertions
PASS - reset runs assertions2
----------------------------------------
PASS: 22  FAIL: 1  TOTAL: 23
Finished in 0.161 seconds.
----------------------------------------

The first hurdle was adding guards around all QUnit.js’s references to setTimeout, setInterval, and other browser/document specific objects.  In addition I extended the test.js browser checks to include all of the asynchronous tests and fixture tests.  Finally I cleaned up a bit of the jsDump code to work better with varying object call chains.  My alterations can be found on my fork here.

The second hurdle was getting QUnit-CLI using my modified version of QUnit.js and adjusting how Rhino errors are handled.  Adding a QUnit submodule to the QUnit-CLI git repository easily fixed the first (I previously posted my notes on git submodules and fixed branches).  QUnit.js’s borrowed jsDump code is used to “pretty-print” objects in test messages.  jzaefferer ran into an issue when running QUnit’s own tests through QUnit-CLI resulting in the cryptic error:

js: "../qunit/qunit.js", line 1021: Java class "[B" has no public instance field or method named "setInterval".
at ../qunit/qunit.js:1021
at ../qunit/qunit.js:1002
at ../qunit/qunit.js:1085
at ../qunit/qunit.js:1085
at ../qunit/qunit.js:1085
at ../qunit/qunit.js:110
at ../qunit/qunit.js:712 (process)
at ../qunit/qunit.js:304
at suite.js:84

It turns out that errors objects (e.g. ReferenceError) throw in Rhino include an additional property of rhinoException which points to the underlying Java exception that was actually thrown.  The error we saw is generated when the jsDump code walks the error object tree down to a byte array off of the exception.  Property requests executed against this byte array throw the Java error above, even if they are done part of a typeof check, e.g.

var property_exists = (typeof obj.property !== 'undefined');

Once I figured this out, I wrapped the object parser inside QUnit.jsDump to properly pretty-print error objects and delegate to the original code for any other type of object.

...
var current_object_parser = QUnit.jsDump.parsers.object;
QUnit.jsDump.setParser('object', function(obj) {
  if(typeof obj.rhinoException !== 'undefined') {
    return obj.name + " { message: '" + obj.message + "', fileName: '" + obj.fileName + "', lineNumber: " + obj.lineNumber + " }";
  }
  else {
    return current_object_parser(obj);
  }
});
...

With these changes we have a decent command line executable test suite runner for QUnit.  With a bit more work QUnit-CLI will hopefully be able to print Ant/JUnit style XML output and/or include stack traces when errors bubble out of test code.

Foremost was the fact that the QUnit.log callback passed back HTML in the message parameter.  I wasn’t the only one to catch onto this issue (GitHub issue 32).  While I was still formulating a plan of attack to refactor out all of the browser oriented code, Jörn Zaefferer was putting a much simpler fix into place.  His commit added an additional details object to all QUnit.log calls which will contains the original assertion message w/o HTML and possibly the raw expected and actual values (not yet documented on the main page).  Problem solved!

Or so it seemed.

As I tried to hack my example CLI test runner to use the quick fix I ran into several issues.

Core QUnit logic still includes browser assumptions

Even with changes to wrap browser calls and guard against assuming certain browser objects existing, qunit.js is full of browser code.  It would be great if the core unit testing code could be separated by the code necessary to execute properly within a browser page and separate from how test results should be displayed on an HTML page.  If these three responsibilities were found in 3 different objects, it would be simple to replace one or more with ones that fit the scenario much more closely without needing to resort to hacks or breaking backward compatibility.

Single Responsibility Principle

Lifecycle callbacks break down outside of a browser

QUnit has some really nice lifecycle callbacks which anyone needing to integrate a testing tool can use.  They include callbacks for starting and finishing each test, individual assertions, and the whole test suite.  The first thing I wanted to add was reporting of the total number of passing and failing tests along with execution time when the tests were all done.  This looked like a simple job for QUnit.begin and QUnit.done.

It turns out that QUnit.begin won’t get called unless there is a window “load” event …. which doesn’t happen in Rhino …. so that is out.  To make matters worse, QUnit.done is getting called twice! For each test!  This means that my “final” stats are spammed with each test run.  With help of Rhino’s debugger app, I saw that the culprit were the successive “done” calls near the end of the “test” function.  Not sure how to fix that yet.

“Truthiness” in JavaScript is a double edged sword

Most of the time it is great not having to worry about if a value is really “False” or if it is undefined or some other “falsey” value.  Only one problem, zero (0) is falsey too.  Going back to my C days this isn’t too big of a deal (and can be used in clever ways).  However, if you are checking for the existence of properties of an object by doing a boolean expression … don’t.  Sure, undefined is falsey and if an object doesn’t have a property it will return undefined … but what if the value of that proeperty really IS undefined or in my case zero.  No good.

var foo = {cat: "dog", count: 0 }
!!foo.cat   // true
!!foo.bar   // false (doesn't exist, returns undefined)
!!foo.count // false (0 is falsey)

** The double bang (!!boolean) convention is a convienent way for converting between a falsey or truthy value to the explicit values TRUE and FALSE

Unit test frameworks should standardize the order of arguments!

This is a personal pet peeve.  I wish all unit testing framework would standardize whether the EXPECTED value or the ACTUAL value should go first on equals assertions.  JUnit puts expected first.  TestNG puts it second.  QUnit puts it second.  This is damn confusing if you have to switch between these tools frequently.

I moved my current code to a new GitHub repo -> QUnit-CLI.  As I learn more and find a better solution, I will keep this repo updated.  Currently the code outputs one line for each passing test and a more detailed set of lines for each failing test (including individual assertion results).  Because of the QUnit.done problem above, the “final” test suite results are shown twice for each test (making them not very “final”) which shows the total test results and execution time. [Edited to have correct link]

~~~~~~~~~~

Side Note: As an end goal, I would like to build a CLI for QUnit that will output Ant+JUnit style XML output which would make integrating these test results with other tools a piece of cake.  I CAN’T FIND THE XML DOCUMENTED ANYWHERE!  Lots of people have anecdotal evidence of how the XML will be produced but no one seems to have a DTD or XSD that is “official”.  If anyone knows of good documentation of the XML reports Ant generates for JUnit runs please let me know.  Thanks.

~~~~~~~~~~

λ This post has nothing to do with Lisp or Clojure λ

• it has tests
• code and tests are in source control
• tests are running on the continuous integration (CI) build

Sound crazy?  If so, that is my kind of crazy.

Lately I have been hacking around a lot with JavaScript (HTML5, NodeJS, etc…) and was asked by another team if I had any suggestions on testing JavaScript and how they could integrate it into their build.  Most of my experience testing JavaScript has been done by executing user acceptance tests written with tools like Selenium and Cucumber (load a real browser and click/act like a real user).  Unfortunately these tests are slow and brittle compared to our unit tests.  More and more of today’s dynamic web apps have large amounts of business logic client side that doesn’t have a direct dependency on the browser.  What I want are FAST tests that I can run with every single change, before every single commit, and headless on our build server.  They might not catch everything, but they will catch a lot long before the UATs are finished.

Goal : Run JavaScript unit tests as part of our automated build

The first hurdle was finding a way to run the code headless outside of a browser.  Several of today’s embedded JavaScript interpreters are available as separate projects but for simplicity I like Rhino.  Rhino is an interpreter written entirely in Java and maintained by the Mozilla Foundation.  Running out of a single .jar file and with a nice GUI debugger, Rhino is a good place to start running JavaScript from the command line.  While not fully CommonJS compliant yet, Rhino offers a lot of nice native functions for loading other files, interacting with standard io, etc.  Also, you can easily embed Rhino in Java apps (it is now included in Java 6) and extended with Java code.

My JS testing framework of choice lately has been JSpec which I really like: nice GUI output, tons of assertions/matchers, async support, Rhino integration, and really nice r-spec-esque grammar.  Unfortunately JSpec works best on a Mac and the team in question needs to support Windows/*nix mainly.

Enter QUnit: simple, fast, and used to test JQuery.  Only one problem, QUnit is designed to run in a browser and gives all of its output in the form of DOM manipulations.  Hope was almost lost when I found a tweet by John Resig himself that suggested that QUnit could be made to work with a command line JavaScript interpreter.  Sadly despite many claims of this functionality, I couldn’t find a single good tutorial which showed me how nor a developer leveraging this approach.  A bit of hacking, a lucky catch while reading the Env.js tutorial, and twada’s qunit-tap project came together in this simple solution:

Step 1 : Write a test (myLibTest.js)

Create a simple test file for a even simpler library function.

test("Adding numbers works", function() {
    expect(3);
    ok(newAddition, "function exists");
    equals(4, newAddition(2, 2), "2 + 2 = 4");
    equals(100, newAddition(100, 0), "zero is zero");}
);

Step 2 : Create a test suite (suite.js)

This code could be included in the test file, but I like to keep them separate so that myLibTest.js could be included into a HTML page for running QUnit in normal browser mode without making any changes.

This file contains all of the Rhino specific commands as well as some formatting changes for QUnit.  After loading QUnit we need to a do a bit of house work to setup QUnit to run on the command line and override the log callback to print our test results out to standard out.  QUnit offers a number of callbacks which can be overridden to integrate w/ other testing tools (find them on the QUnit home page under “Integration into Browser Automation Tools”) Finally load our library and our tests. [EDIT: ADD]

load("../qunit/qunit/qunit.js");

QUnit.init();
QUnit.config.blocking = false;
QUnit.config.autorun = true;
QUnit.config.updateRate = 0;
QUnit.log = function(result, message) {
    print(result ? 'PASS' : 'FAIL', message);
};

load("myLib.js");
load("myLibTest.js");

Step 3 : Write the code (myLib.js)

function newAddition(x, y) {
    return x + y;
}

Step 4 : Run it

Running Rhino is piece of cake but I am VERY lazy so I created a couple of aliases to make things dead simple. (taken from the Env.js tutorial, jsd runs the debugger)

export RHINO_HOME="~/development/rhino1_7R2"
alias js="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.shell.Main -opt -1"
alias jsd="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.debugger.Main"

Once the alias are created, simple run the suite

> js suite.js
> PASS function exists
> PASS <span>2 + 2 = 4</span>, expected: <span>4</span>
> PASS <span>zero is zero</span>, expected: <span>100</span>
> 

Step 5 : Profit!

There you go, command line output of your JavaScript unit tests.  Now we can test our “pure” JavaScript which doesn’t rely on the DOM.  Using a tool like Env.js this is also possible and will be discussed in a future post. [EDIT: ADD]

Step 5+ : Notice the problem -> HTML output on the command line

You may have noticed that my output messages include HTML markup.  Sadly QUnit still has assumptions that it is running in a browser and/or reporting to an HTML file.  Over the next couple weeks I am going to work on refactoring out the output formatting code from the core unit testing logic and hopefully build out separate HTML, human readable command line, and Ant/JUnit XML output formatters so that integrating QUnit into your build process and Ant tasks is a piece of cake.

Track my progress on my GitHub fork and on the main QUnit project issue report.

[EDIT] I have posted about some of my progress and setbacks here.