Ivan

Simple Jester

At CITCON I wanted to demo Jester in the session on mutation testing. I tried to run Jester on Nat Pryce and Steve Freeman's JMock2 code but couldn't get it to work in the time I had available, so I gave up and wrote some simple code specifically for the demo.

As a result of this bad experience, I thought I'd make Jester much simpler to use. I've called the result "simple-jester" and the first release of it is available from the Jester sourceforge site.

An example of using simple-jester

This article describes an example of how to run simple-jester, by showing how to run it on JMock2. I have chosen JMock2 because I know that it'll be well written and well tested (because I know Nat Pryce and Steve Freeman), it's not too small and not too big, the build is fast, it wasn't written with Jester in mind, and I didn't write it.

This article describes what I did - on windows, on a machine which I've already got Java, ant, Python and subversion installed on. This is what really happened ... warts and all, so I hope the hints and tips that arise from the problems described here will help you get simple-jester working for yourself.

A note on the style of this article

Sorry if you find the writing style odd (even by my standards) - it's a mix of the instructions that I thought I should write, what is happening as I'm trying to follow them and then instructions to fix the mistakes I made with the earlier instructions. I've written it like this because in practice you will end up going through something similar to get Jester to work - the mistakes weren't deliberate and are instructional.

Download and install Jester

• Download simple-jester-1.0.zip to c:\temp
• unzip - it gets unzipped into C:\temp\simple-jester-1.0

Check that everything is OK by executing, in a command prompt:

• cd C:\temp\simple-jester-1.0
• setcp.bat
• test.bat

Then:

• wait about a minute for Jester to finish it's run
• open "jester.html" to see that Jester has done something.

I've just realised that setcp.bat added JUnit 3.8 to the classpath - that might not be good for running the JMock2 build. Jester doesn't need JUnit on it's classpath to run. So:

• close this command prompt

Get JMock source code

• open a new command prompt
• cd c:\temp
• svn co http://svn.codehaus.org/jmock/trunk/jmock2/
• cd c:\temp\jmock2
• execute "ant"

Unfortunately, the build fails:

run.tests:
    [mkdir] Created dir: C:\temp\jmock2\build\reports\tests
    [junit] Testsuite: org.jmock.example.announcer.AnnouncerTests
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.031 sec

    [junit] Testsuite: org.jmock.example.qcon.DJTests
    [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0 sec

    [junit] Testcase: warning(junit.framework.TestSuite$1):     FAILED
    [junit] No tests found in org.jmock.example.qcon.DJTests
    [junit] junit.framework.AssertionFailedError: No tests found in org.jmock.ex
ample.qcon.DJTests



BUILD FAILED
C:\temp\jmock2\build.xml:208: Tests failed

Sorry Nat and Steve. Maybe I've got something wrong with the environment. I don't have time to investigate. Almost certainly something I've got wrong and I'm not worried anyway, because the fix is simple and I don't expect the side-effects of the fix to be too bad.

so:

• delete C:\temp\jmock2\test\org\jmock\example\qcon\DJTests.java
• execute "ant"
• BUILD SUCCESSFUL - on my machine it took 14 seconds

Run Jester on JMock2

• add Jester to the classpath by executing
  "set classpath=C:\temp\simple-jester-1.0\simple-jester.jar;C:\temp\simple-jester-1.0"
• execute "java jester.TestTester ant src"

Unfortunately, this fails:

Jesting failed!
jester.SourceChangeException: couldn't run tests
        at jester.RealTestRunner.testsRunWithoutFailures(RealTestRunner.java:26)

        at jester.TestTester.run(TestTester.java:144)
        at jester.TestTester.doMain(TestTester.java:127)
        at jester.TestTester.main(TestTester.java:81)
Caused by: java.io.IOException: CreateProcess: ant error=2
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
        at java.lang.ProcessImpl.start(ProcessImpl.java:30)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
        at jester.Util.runCommand(Util.java:10)
        at jester.RealTestRunner.testsRunWithoutFailures(RealTestRunner.java:21)

        ... 3 more

Doh! I made the mistake that I now remember documenting in the README.html for Jester. You have to tell Jester to run ant.bat on windows, rather than just ant.

To prove this, execute "java jester.RealTestRunner ant" which runs the code that Jester will run for executing your build command. This produces the same output as above. If you now execute "java jester.RealTestRunner ant.bat" instead, then the result is:

C:\temp\jmock2>java jester.RealTestRunner ant.bat
true

which means that this build passes for Jester.

So - trying again:

• execute "java jester.TestTester ant.bat src"
• the "progress" dialog opens, and after 30 seconds or so, you should see something happening in the "progress" dialog
• lots of stuff gets printed out in the command prompt - it's ugly and alarming. Just ignore it.
• now go and make a cup of tea; this could take some time.

I feel happy that I've been able to get Jester to work in a very short time. This would have been enough for CITCON - I could have run it on a small subset of the source rather than all the source and it would have all been done, set up and executed, in less than 30 minutes. Here I'm being more ambitious by trying to run Jester on all of the JMock2 source.

It goes horribly wrong

I've gone back to the machine - the progress bar showing not much progress. Not much seems to be happening, but after waiting about 30 seconds I see some more stuff in the "progress" dialog and return to the important task of drinking the tea that I've just made.

I go back to the machine much later - it really does look stuck now. One of the cores is maxed out running a java process, and the "progress" dialog has shown: "src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java - changed source on line 54 (char index=2029) from if ( to if (false &&" for several minutes.

So:

• kill jester (ctrl-c)
• execute "ant" and the result is:

BUILD FAILED
C:\temp\jmock2\build.xml:15: Unable to delete file C:\temp\jmock2\build\jmock-2-
SNAPSHOT\jmock-2-SNAPSHOT.jar

I open the task manager and find a java process that is hanging around that I wouldn't expect so:

• kill any stray java processes
• delete the offending jmock-2-SNAPSHOT.jar file.

I suspect a problem with Jester - if a mutation causes the build not to terminate, then Jester times out that run of the build and keeps going. In this case, it looks like that's what has happened, and there is a java process hanging around that is keeping a lock on the jmock-2-SNAPSHOT.jar file which the build needs to delete in order to continue, so now it's hung.

One of the least endearing features of Jester is that if the whole run isn't successful then you can't easily get anything out of Jester - i.e. mostly either the complete run of Jester works or doesn't work, if it crashes or is killed before it's finished then you don't get much useful out of it. I might fix this in the future. In the meantime, it's safest to run Jester on subsets of your code base (e.g. at most a package) at a time. But I won't follow my own advice here because I'm determined to get Jester to work for all the source files of JMock2 in one run.

To try to get Jester to work for JMock2, I'll get Jester to ignore the part of the source file where it made the mutation that caused the hang problem.

• revert changes to "SynchronousScheduledExecutor.java" by executing: svn revert src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java
• add "//stopJesting" as line 58 (if you have a look at the original source you'll see that's immediately before the mutation that was described in the "progress" dialog).
• add "//resumeJesting" as line 60
• to check that there are no other problems for Jester with this file, run Jester only on this file rather than on everything, i.e. execute:
  "java jester.TestTester ant.bat src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java"
• Jester will complete mutation testing of this (takes about a minute)
• execute "java jester.TestTester ant.bat src"
• some time later, a lot later, Jester will finish. Yipee. Here's the last couple of lines of output:

43 mutations survived out of 284 changes. Score = 85
took 42 minutes

I'd expect that having deleted one of the tests (which might not have been the best way to get the build to pass) will have slightly worsened the results. Nevertheless, these results indeed show that the tests for JMock2 are impressively good. Jester often gives lower "scores" than code coverage tools. I'd be interested to know what the results are for a code coverage tool on the same code but my time has run out for this and my tea has gone cold.

However - I still have time for one thing, which is possibly the most important output from Jester:

• execute "python c:\temp\simple-jester-1.0\makeWebView.py"

This produces web pages that show the modifications that Jester can make where the tests still pass. I'll send these to Nat (here's an example chosen because it's short, which I have hand edited to add his license in for publication). These files are the most interesting output from Jester.

The future of Jester

I'm pleased with both the code and usage simplifications to Jester in "simple-jester". The downside is that it's now even slower than before; but that doesn't worry me unduly because at least now I and others will be able to get it to work for many more projects. There are still some areas of improvement (e.g. if your build calls ant multiple times then Jester can think the build was successful even if it wasn't) - but generally it's much improved thanks to some judicious deletion of code.

If someone wants to pay me to do it, I'd be interested in making Jester faster by having it clustered on multiple machines. There are alternative approaches - e.g. jumble which uses byte code mutation (it's Java specific) which is much faster than Jester's approach.

I'm not planning on spending lots more time on Jester (unless someone pays me to do it) because I'm now happy that Jester is both simplified and improved (as simple-jester) such that it's much more usable, as I hope this article shows with it's "warts and all" commentary, because the warts really weren't very large.