Ivan

Simple Jester

At CITCON I wanted to demo Jester in the session on mutation testing. I tried to run Jester on Nat Pryce and Steve Freeman's JMock2 code but couldn't get it to work in the time I had available, so I gave up and wrote some simple code specifically for the demo.

As a result of this bad experience, I thought I'd make Jester much simpler to use. I've called the result "simple-jester" and the first release of it is available from the Jester sourceforge site.

An example of using simple-jester

This article describes an example of how to run simple-jester, by showing how to run it on JMock2. I have chosen JMock2 because I know that it'll be well written and well tested (because I know Nat Pryce and Steve Freeman), it's not too small and not too big, the build is fast, it wasn't written with Jester in mind, and I didn't write it.

This article describes what I did - on windows, on a machine which I've already got Java, ant, Python and subversion installed on. This is what really happened ... warts and all, so I hope the hints and tips that arise from the problems described here will help you get simple-jester working for yourself.

A note on the style of this article

Sorry if you find the writing style odd (even by my standards) - it's a mix of the instructions that I thought I should write, what is happening as I'm trying to follow them and then instructions to fix the mistakes I made with the earlier instructions. I've written it like this because in practice you will end up going through something similar to get Jester to work - the mistakes weren't deliberate and are instructional.

Download and install Jester

• Download simple-jester-1.0.zip to c:\temp
• unzip - it gets unzipped into C:\temp\simple-jester-1.0

Check that everything is OK by executing, in a command prompt:

• cd C:\temp\simple-jester-1.0
• setcp.bat
• test.bat

Then:

• wait about a minute for Jester to finish it's run
• open "jester.html" to see that Jester has done something.

I've just realised that setcp.bat added JUnit 3.8 to the classpath - that might not be good for running the JMock2 build. Jester doesn't need JUnit on it's classpath to run. So:

• close this command prompt

Get JMock source code

• open a new command prompt
• cd c:\temp
• svn co http://svn.codehaus.org/jmock/trunk/jmock2/
• cd c:\temp\jmock2
• execute "ant"

Unfortunately, the build fails:

run.tests:
    [mkdir] Created dir: C:\temp\jmock2\build\reports\tests
    [junit] Testsuite: org.jmock.example.announcer.AnnouncerTests
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.031 sec

    [junit] Testsuite: org.jmock.example.qcon.DJTests
    [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0 sec

    [junit] Testcase: warning(junit.framework.TestSuite$1):     FAILED
    [junit] No tests found in org.jmock.example.qcon.DJTests
    [junit] junit.framework.AssertionFailedError: No tests found in org.jmock.ex
ample.qcon.DJTests



BUILD FAILED
C:\temp\jmock2\build.xml:208: Tests failed

Sorry Nat and Steve. Maybe I've got something wrong with the environment. I don't have time to investigate. Almost certainly something I've got wrong and I'm not worried anyway, because the fix is simple and I don't expect the side-effects of the fix to be too bad.

so:

• delete C:\temp\jmock2\test\org\jmock\example\qcon\DJTests.java
• execute "ant"
• BUILD SUCCESSFUL - on my machine it took 14 seconds

Run Jester on JMock2

• add Jester to the classpath by executing
  "set classpath=C:\temp\simple-jester-1.0\simple-jester.jar;C:\temp\simple-jester-1.0"
• execute "java jester.TestTester ant src"

Unfortunately, this fails:

Jesting failed!
jester.SourceChangeException: couldn't run tests
        at jester.RealTestRunner.testsRunWithoutFailures(RealTestRunner.java:26)

        at jester.TestTester.run(TestTester.java:144)
        at jester.TestTester.doMain(TestTester.java:127)
        at jester.TestTester.main(TestTester.java:81)
Caused by: java.io.IOException: CreateProcess: ant error=2
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
        at java.lang.ProcessImpl.start(ProcessImpl.java:30)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
        at jester.Util.runCommand(Util.java:10)
        at jester.RealTestRunner.testsRunWithoutFailures(RealTestRunner.java:21)

        ... 3 more

Doh! I made the mistake that I now remember documenting in the README.html for Jester. You have to tell Jester to run ant.bat on windows, rather than just ant.

To prove this, execute "java jester.RealTestRunner ant" which runs the code that Jester will run for executing your build command. This produces the same output as above. If you now execute "java jester.RealTestRunner ant.bat" instead, then the result is:

C:\temp\jmock2>java jester.RealTestRunner ant.bat
true

which means that this build passes for Jester.

So - trying again:

• execute "java jester.TestTester ant.bat src"
• the "progress" dialog opens, and after 30 seconds or so, you should see something happening in the "progress" dialog
• lots of stuff gets printed out in the command prompt - it's ugly and alarming. Just ignore it.
• now go and make a cup of tea; this could take some time.

I feel happy that I've been able to get Jester to work in a very short time. This would have been enough for CITCON - I could have run it on a small subset of the source rather than all the source and it would have all been done, set up and executed, in less than 30 minutes. Here I'm being more ambitious by trying to run Jester on all of the JMock2 source.

It goes horribly wrong

I've gone back to the machine - the progress bar showing not much progress. Not much seems to be happening, but after waiting about 30 seconds I see some more stuff in the "progress" dialog and return to the important task of drinking the tea that I've just made.

I go back to the machine much later - it really does look stuck now. One of the cores is maxed out running a java process, and the "progress" dialog has shown: "src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java - changed source on line 54 (char index=2029) from if ( to if (false &&" for several minutes.

So:

• kill jester (ctrl-c)
• execute "ant" and the result is:

BUILD FAILED
C:\temp\jmock2\build.xml:15: Unable to delete file C:\temp\jmock2\build\jmock-2-
SNAPSHOT\jmock-2-SNAPSHOT.jar

I open the task manager and find a java process that is hanging around that I wouldn't expect so:

• kill any stray java processes
• delete the offending jmock-2-SNAPSHOT.jar file.

I suspect a problem with Jester - if a mutation causes the build not to terminate, then Jester times out that run of the build and keeps going. In this case, it looks like that's what has happened, and there is a java process hanging around that is keeping a lock on the jmock-2-SNAPSHOT.jar file which the build needs to delete in order to continue, so now it's hung.

One of the least endearing features of Jester is that if the whole run isn't successful then you can't easily get anything out of Jester - i.e. mostly either the complete run of Jester works or doesn't work, if it crashes or is killed before it's finished then you don't get much useful out of it. I might fix this in the future. In the meantime, it's safest to run Jester on subsets of your code base (e.g. at most a package) at a time. But I won't follow my own advice here because I'm determined to get Jester to work for all the source files of JMock2 in one run.

To try to get Jester to work for JMock2, I'll get Jester to ignore the part of the source file where it made the mutation that caused the hang problem.

• revert changes to "SynchronousScheduledExecutor.java" by executing: svn revert src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java
• add "//stopJesting" as line 58 (if you have a look at the original source you'll see that's immediately before the mutation that was described in the "progress" dialog).
• add "//resumeJesting" as line 60
• to check that there are no other problems for Jester with this file, run Jester only on this file rather than on everything, i.e. execute:
  "java jester.TestTester ant.bat src\org\jmock\lib\concurrent\SynchronousScheduledExecutor.java"
• Jester will complete mutation testing of this (takes about a minute)
• execute "java jester.TestTester ant.bat src"
• some time later, a lot later, Jester will finish. Yipee. Here's the last couple of lines of output:

43 mutations survived out of 284 changes. Score = 85
took 42 minutes

I'd expect that having deleted one of the tests (which might not have been the best way to get the build to pass) will have slightly worsened the results. Nevertheless, these results indeed show that the tests for JMock2 are impressively good. Jester often gives lower "scores" than code coverage tools. I'd be interested to know what the results are for a code coverage tool on the same code but my time has run out for this and my tea has gone cold.

However - I still have time for one thing, which is possibly the most important output from Jester:

• execute "python c:\temp\simple-jester-1.0\makeWebView.py"

This produces web pages that show the modifications that Jester can make where the tests still pass. I'll send these to Nat (here's an example chosen because it's short, which I have hand edited to add his license in for publication). These files are the most interesting output from Jester.

The future of Jester

I'm pleased with both the code and usage simplifications to Jester in "simple-jester". The downside is that it's now even slower than before; but that doesn't worry me unduly because at least now I and others will be able to get it to work for many more projects. There are still some areas of improvement (e.g. if your build calls ant multiple times then Jester can think the build was successful even if it wasn't) - but generally it's much improved thanks to some judicious deletion of code.

If someone wants to pay me to do it, I'd be interested in making Jester faster by having it clustered on multiple machines. There are alternative approaches - e.g. jumble which uses byte code mutation (it's Java specific) which is much faster than Jester's approach.

I'm not planning on spending lots more time on Jester (unless someone pays me to do it) because I'm now happy that Jester is both simplified and improved (as simple-jester) such that it's much more usable, as I hope this article shows with it's "warts and all" commentary, because the warts really weren't very large.

build-o-matic gets more advanced

If you have a long enough build and large enough team, then you can quite often have the situation where there are multiple commits between continuous integration server builds. In these circumstances, if the build is broken then you don't know which commit broke the build.

Keith Braithwaite told me that in one team he worked in, someone would manually perform a binary search between the multiple commits to find out which commit broke the build. At a Google event, I was talking to someone about build-o-matic and he said that the feature he'd like for a continuous integration server was to do a binary search to find out which commit broke the build.

blame-o-matic

Build-o-matic is the only continuous integration server that I know of which automatically performs a binary search if the build is broken and there are multiple commits between builds (I call this feature "blame-o-matic"). The binary search is between the last known passing build and the first known broken build. Build-o-matic only performs one step of the binary search at a time - if someone commits while it's running one of these binary search builds, then it'll pick up those changes and run the build. If the build is still broken and there are no other commits, then build-o-matic will do the next binary search, etc.

The fine line between stupid and clever.

The binary search feature is clever but not always ideal. If the build is broken, sometimes it's obvious from the build output (which build-o-matic usefully shows and saves for you). For example, sometimes someone will realise straight away that they forgot to check a file in. If the build can be fixed straight away by (for example) checking a file in then it could be that build-o-matic has set off a blame-o-matic build, so you have to wait for that build to finish before your commit gets picked up and the build gets fixed.

Turning the dials up to eleven.

Having been using the "blame-o-matic" feature of build-o-matic for a while, I've now also added a feature called "filling in the gaps". If there are no incoming changes for build-o-matic to run the build for, and it isn't doing a binary search, then build-o-matic runs the build for the most recent revision that it hasn't run the build for. There are cases where doing a binary search doesn't work - if the build is broken, fixed then broken again within the revisions that are checked in at the same time, then the only way to be sure which revision was the most recent that broke the build, you have to build every revision.

Another benefit of running the build for every revision is that if your build includes any statistics (e.g. the build time, amount of sloppiness, etc) then you get the statistics for every revision.

Smell the glove

You might expect that the "blame-o-matic" and "filling in the gaps" features would result in the build-o-matic machine running the build all the time when you want it to be ready to run the build on a new commit. In practice, it works out well. If the build is fast enough, then it is rare that there are multiple commits between builds and the "blame-o-matic" and "filling in the gaps" features will not run very often. If the build is slow, then there is a very large chance that someone has checked in between builds, in which case, this will take precendence over the "blame-o-matic" or "filling in the gaps" builds. If the build is slow and people are careful not to check in if the build is broken, unless they are fixing the build, then the "blame-o-matic" can help find which commit caused the problem in parallel with people looking to see if they can work out what broke the build. The "filling in the gaps" feature can work over night while no one is checking code in, ready to show you the next day the full history of which builds broke and which passed.

An alternative - killing blame-o-matic builds

A possible improvement to the blame-o-matic feature would be to kill a blame-o-matic build if build-o-matic detects that someone has committed while a blame-o-matic build is running. This would probably be simple enough for build-o-matic to implement, but I didn't implement this because it means you would have to make sure that your build can be killed at any point without leaving the environment in which the build runs in a state in which the build cannot be run again. One common, but not necessarily good, feature of many builds I've seen is for acceptance tests to run against a database, which has to be partially set up and torn down rather than set up from scratch, due to the length of time it would take to do it fully. I didn't want build-o-matic to impose "interuptability" onto the builds it runs yet - maybe a future version will - maybe a future version will support both interruptable and non interruptable builds ... I don't know yet.

Another alternative - only check in if the build passes

I understand that Team City has a feature where you can submit some code for building, and then it is checked in if the build passes (that's my understanding having talked to someone who has used it - but I haven't used it myself). Using that feature as the only way to check code in would mean that the build would never be broken. I didn't implement this for a couple of reasons. It's harder to implement, and I'm not sure I'm keen on it - I want code integrated as soon as possible - a delay in integrating code due to waiting for a build on a continuous integration server doesn't seem ideal to me. I reserve the right to change my mind.

Try build-o-matic yourself

build-o-matic is open source - try it yourself. It's somewhat limited at the moment - it only supports subversion, and ant (or maven) "out of the box". I'd like to add support for perforce some time soon, but have no plans to support any VCS system that I don't want to use myself.

Url logger server - simple logging from multiple servers and languages.

Ever spent ten minutes or more trying to find where some logging is going to? Ever wanted a simple way to log from multiple processes or servers? Today my pair and I were having a bad logging day, and having had bad logging days before, I thought I'd do something about it. I wrote a simple http server (based on a previous article) that logs whatever you hit the web server with (here I'm calling it the "url logger server").

Url logger server

The url logger server makes logging for debugging purposes simple. It's not meant for "proper" logging, but as a simple and quick way to add logging for debugging for things where it's difficult to get access to files (or just to work out where they end up!) or where there are a variety of places you want to put logging (e.g. several different servers, processes or code in multiple languages). Please let me know if there's something that already does this. My pair and I found the url logger server useful for our purposes, so I've published it here for others (with permission from my lovely client).

Example use of the url logger server

This is how you log to the url logger server from Java. Most languages can do the equivalent quite easily. In this example I'm running the url logger server on the same machine that this code is running on, but there's no need - it could be anywhere on the network.

	private void someMethod() {
		log("this is a log message");
	}

	private void log(String message) {
		try {
			new URL("http://localhost:8010/log/"+URLEncoder.encode(message)).getContent();
		} catch (MalformedURLException e) {
		} catch (IOException e) {
		}
	}

Viewing the log

In the example above, a browser pointed at "http://localhost:8010/" will show:

this is a log message

Hitting "http://localhost:8010/clear" will clear the log.

The url logger server code

This can be run by jython or python.

import BaseHTTPServer, urllib, os.path

class DBHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    def append(self, message):
        f=open('log','a')
        f.write(urllib.unquote(message).replace('+',' ')+'\n')
        f.close()
        
    def clear(self):
        open('log','w').close()
        
    def read(self):
        if not os.path.exists('log'):
            return ""
        return open('log').read()
        
    def do_GET(self):
        path = urllib.unquote(self.path[1:]).split('/')
        
        self.send_response(200)
        self.send_header("Content-type", "text/html") 
        self.end_headers() 
        
        if path[0] == 'log':
            self.append('/'.join(path[1:]))
        else:
            if path[0] == 'clear':
                self.clear()
            else:
                html = "<html><head></head><body><pre>"
                html = html + self.read()
                html = html + "< /pre></body></html>"
                self.wfile.write(html)

server_address = ('', 8010)
httpd = BaseHTTPServer.HTTPServer(server_address, DBHandler)
httpd.serve_forever()

build-o-matic and the visibility of the build results

Following on from a previous article about build-o-matic and having talked to Andy Pols at XTC about it, Andy commented:

Can you put some screen shots of your build with the images of the people who committed. I remember you told me you used southpark charactures.

Can you blog about how this changed the teams attitude to the build. I think it's a facinating and important discovery.

Making the build web page interesting

Someone (probably James Gellately-Smith) suggested using South Park Studio to create pictures of everyone on the team for use on the planning board to put next to stories to show who is working on which story. We each made an image of someone else on the team, these were then printed in color, cut out and laminated, with velcro stuck on the back so they could be put on the planning board, and moved as appropriate. They were a bit hit with the team.

I made build-o-matic scrape the commit messages to find developers' names or initials and match them to these images and put them in the build results page. The results can be seen below:

(this is based on a real build-o-matic page from my previous client (included here with permission from the team) - the names and check in messages have been altered to protect the innocent).

The effect of the build-o-matic web page

The effect of having the images in the build page took me by suprise. When people first saw the page, they laughed, and it certainly got everyone's attention. People started taking more notice of the build. Chris Clarke modified build-o-matic to make the page background red if the build was broken. People took even more notice - their pictures would be on a red build page if they broke the build. People took fixing the build more seriously than before.

Also, build-o-matic shows who checked in as soon as a modification is detected and the build started. This meant that people who had just checked in would notice if their pictures appeared on the web page and know that build-o-matic was running the build with their modifications.

Having the pictures on the build page also gives a more immediately noticable indication of who has been checking in. A quick scroll down the page shows who is working by themselves, who is checking in frequently and who isn't. This information is available in other ways, but the build-o-matic web page is immediate and easy to view.

More Monkey

I've written a couple more Eclipse Monkey scripts that I hope might be useful to others. They aren't meant as examples of anything clever using Monkey - for that, you'd be better looking elsewhere.

There are some bits of Java code editing typing that I see often, so, although they don't take long, I've written a couple of monkey scripts to help.

To install the scripts, install eclipse monkey, for each script, copy the script (including the "---" lines at the beginning and end), then from the "Monkey" menu in eclipse, select "Paste new script".

Convert method parameter declaration into method call parameter list

Converts:

String string, int integer, int[] nums, List list

into:

string, integer, nums, list

The script:

--- Came wiffling through the eclipsey wood ---
/*
 * Menu: Strips types out of parameter list
 Converts:
 	String foo, int x
 into: 
 	foo, x
 * Kudos: Ivan Moore
 * License: EPL
 */

function main() {
	systemClipboard = Packages.java.awt.Toolkit.getDefaultToolkit().getSystemClipboard();
	contents = systemClipboard.getContents(null);
	msg = contents.getTransferData(Packages.java.awt.datatransfer.DataFlavor.stringFlavor);
	parts = msg.split(',');
	result = "";
	for(partNumber in parts){
		part = parts[partNumber].trim();
		if(partNumber > 0){
			result += ", ";
		}
		result += part.split(" ")[1]
	}
	newContents  = new Packages.java.awt.datatransfer.StringSelection(result);
	systemClipboard.setContents(newContents, newContents);
}
--- And burbled as it ran! ---

Convert method parameter declaration into local variable declarations.

Converts:

String string, int integer, int[] nums, List list

into:

		String string = null;
		int integer = -1;
		int[] nums = null;
		List list = null;

The script:

--- Came wiffling through the eclipsey wood ---
/*
 * Menu: Paramater list INTO declarations
 Converts:
 	String foo, int x
 into: 
 	String foo = null;
	int x = -1;
 * Kudos: Ivan Moore
 * License: EPL
 */

function main() {
	systemClipboard = Packages.java.awt.Toolkit.getDefaultToolkit().getSystemClipboard();
	contents = systemClipboard.getContents(null);
	msg = contents.getTransferData(Packages.java.awt.datatransfer.DataFlavor.stringFlavor);
	parts = msg.split(',');
	declarations = "";
	for(partNumber in parts){
		if(partNumber > 0){
			declarations += "\t\t";
		}
		part = parts[partNumber].trim();
		declarations += part + " = " + suitableValue(part) + ";\n";
	}
	newContents  = new Packages.java.awt.datatransfer.StringSelection(declarations);
	systemClipboard.setContents(newContents, newContents);
}

function suitableValue(part){
	type = part.split(" ")[0];
	if(type == "long") return "-1L";
	if(type == "boolean") return "false";
	if(type == "int" || type == "short" || type == "byte") return "-1";
	if(type == "double") return "-1.0";
	if(type == "float") return "-1.0F";
	return "null";
}
--- And burbled as it ran! ---

How to run the scripts

Once monkey and the scripts are installed:

• Copy the method parameters (e.g. String string, int integer, int[] nums, List list).
• Select the appropriate monkey menu item (e.g. "Paramater list INTO declarations")
• Paste, and you get the results as described above

What I'd like

Can you associate a keyboard shortcut to each monkey script? That would be really useful. The point of any script as tiny as these shown is to eliminate the jarring flow-destroying few seconds of manual editing. Therefore, to be of most use, they might be better available from a keyboard shortcut. Maybe.

If you know of a better way of doing these, then please comment.

Jar file viewing

I've come across a couple of teams where at least one person is missing an easy to use jar viewing tool.

For one of these people, working on windows, he renames somefile.jar to somefile.zip and uses the built-in compressed folder viewer. Someone else, working on a mac, extracts the jar somewhere to see the contents.

Existing solutions

I used to use WinZip for viewing jars, but the nagging to buy a license has driven me to look for an open source alternative - I now use 7-zip.

Writing a zip/jar viewer

I don't know why there aren't more zip/jar viewing tools available - it seems such a simple thing that I thought I'd have a go a writing one in a few evenings and seeing how it went.

Tech choices

To get something written quickly, I've chosen Python as the language, wxPython for the GUI, Boa Constructor as an IDE with GUI builder (for wxPython GUIs), and py2exe to produce a windows executable so that whatever is produced can be easily used from Windows (via "open with" or by association with a "file type").

Experiences from this mission

I tried Boa Constructor a few years ago. It's improved a lot - it's now really very good. It has code completion :-) If you want to put together a windows GUI quickly, it's free and works.

wxPython comes with demos - I found them great for working out how to do stuff. However, getting the (very simple) layout to work took much longer than I had expected.

I found it difficult to implement something satisfactory for "opening" a selected file from the zip/jar - ideas in a comment please.

py2exe is great - really simple to use. Unfortunately, the executables and associated files produced are often quite large - as in this case.

The finished product - simple and not great

It's available from my often unavailable server under an MIT style license. To produce an executable version of the zip/jar viewer, install python, wxPython and py2exe and run "python setup.py py2exe" - that'll produce a folder called "dist" that contains the executable "ZippApp.exe" and associated files. If you copy those files to another machine, the executable can then be run without having to install python, wxPython or anything else.

To specify the zip/jar to view it supports:
- command line argument - to make the executable easy to use from Windows, e.g. "open with"
- drag-and-drop a file onto it
- typing the file path and pressing enter in text input

Double clicking on a file in the list executes "start" with the file name - a simple way to get an aappropriate application opened. This is currently implemented rather badly though - I couldn't work out how to do it better in the limited time I've spent on it. The problems with the current implementation are: sometimes you want to choose something other than what "start" will do to your file, and the current implementation leaves temp files lying around even after you've exited the application, for which I can only appologize as that's really quite horrible.

This code is around 150 lines (including code generated by Boa Constructor and the py2exe script) - quite short, so please have a read and tell me of improvements.

I haven't added any "extract" or "add" behaviour - I've just done what I wanted in order to see what it takes to write a jar/zip viewer. The missing features are "just work" and not so interesting from a learning point of view. If you add them, please post a link to your improved version.

I hope this little example is enough of a taster to encourage you to try writing a desktop application using the technologies featured.

Comments

From Mark Jenner:

Hi Ivan,

I just came across this on the Server side and thought it might help you with your Jar viewing post. Haven't tried it myself, but it sounds interesting, especially for digging into the gubbins of a and ear file.

http://www.theserverside.com/news/thread.tss?thread_id=40514

Eclipse Monkey

Eclipse Monkey, from Bjorn Freeman-Benson and Ward Cunningham, lets you script the Eclipse IDE.

An Example - cut and paste of multi-line strings in Java.

In Eclipse (version 3.1.2 - maybe this is fixed in other versions) when you paste multiple lines into a java file, it doesn't automatically convert the string into multiple lines for you. This came up when pairing yesterday, so I've written the following script, which converts whatever text is in your copy buffer into a multi-line string ready for pasting into java:

--- Came wiffling through the eclipsey wood ---
/*
 * Menu: Convert multi-line string
 * Kudos: Ivan Moore
 * License: EPL
 */
 
function main() {
	systemClipboard = Packages.java.awt.Toolkit.getDefaultToolkit().getSystemClipboard();
	contents = systemClipboard.getContents(null);
	msg = contents.getTransferData(Packages.java.awt.datatransfer.DataFlavor.stringFlavor);
	msg = msg.replace('\\','\\\\');
	msg = msg.replace('"','\\"');
	msg = msg.replace('\n','\\n" +\n\t"');
	newContents  = new Packages.java.awt.datatransfer.StringSelection('"' + msg + '"');
	systemClipboard.setContents(newContents, newContents);
}
--- And burbled as it ran! ---

To install this script, install eclipse monkey, copy the script above (including the "---" lines at the beginning and end), then from the "Monkey" menu in eclipse, select "Paste new script".

To run the script, copy some text, and then from the "Monkey" menu in eclipse select "Convert multi-line string". Then paste, and admire the multi-line goodness.

If you know of a better way of doing this in Eclipse (don't say "IntelliJ IDEA"), then please comment - although that wasn't the only reason for writing this script - it was also educational to write a Monkey script with a definite goal rather than just hacking around, which is all I've done with Monkey before yesterday.

Thanks to Monkey, it's much easier to extend Eclipse than it used to be; you don't have to write a plug-in for simple things like this example. I'll be using it more.

CSV - more useful than your average file format.

Graphs are great, but sometimes you want to do analysis on some data. Excel and similar tools can be very powerful - many of my "customers" (in the XP sense of the term) in recent years have been Excel super-power-mega users.

I've often seen developers put too much effort into trying to format output for their customer. It can be quicker for the developers and more useful for the customer to provide a csv file and let the customer decide how they want to see the data (of course, suggesting this to the customer and doing this if they agree, not just unilaterally deciding that's what your going to do). This can also apply to data used only by developers.

Here's a little python program which takes the input to Graphviz used in my previous article, and produces a csv file containing the number of "incoming" and "outgoing" dependencies as shown by the graph in that article.

The code was saved as "graph2csv.py" and executed by running "python graph2csv.py < graph.txt > graph.csv", and here's the raw output
and here it is in Excel:

import sys
incoming = {}
outgoing = {}
nodes = set()
lines = sys.stdin.readlines()[2:-2]
for line in lines:
    source, destination = line.split(" -> ")
    destination = destination[:-2]
    incoming.setdefault(destination,[]).append(source)
    outgoing.setdefault(source,[]).append(destination)
    nodes.add(source)
    nodes.add(destination)
def num(dictionary, node):
    if not dictionary.has_key(node):
        return "0"
    return str(len(dictionary[node]))
print "incoming,node name,outgoing"
for node in nodes:
    print num(incoming,node)+","+node+","+num(outgoing,node)

Graphviz - a tool for the postmodern programmer

Graphviz is a great tool for the postmodern programmer. It's got a simple to integrate interface of text files and command line execution. Bjorn Freeman-Benson is visiting at the moment. He suggested we demonstrate graphviz using the dependencies between eclipse plugins. So here's the code that reads the eclipse manifest files and the resulting graph. This graph shows the dependencies between a significant subset of the standard eclipse 3.1 plugins. Bjorn and I have paired on this to bring you "postable" code that doesn't embarrass us too much.

import sys, time
from os import listdir, system
from os.path import join, isdir, exists
import os.path
from zipfile import is_zipfile, ZipFile

class DependencyGrapher:
    def run(self, pluginDirectory):
        files = [join(pluginDirectory, file) for file in listdir(pluginDirectory)]
        graphText = self.process_files(files)
        
        f=open("graph.txt","w")
        graphText = 'digraph G {\n "' +\
            "generated on " + time.asctime() + '";\n' +\
            graphText + "\n}"
        f.write(graphText)
        f.close()
        system("dot -Tpng -ograph.png graph.txt")
        
    def process_files(self, files):
        result = ""
        for file in files:
            if isdir(file):
                result += self.process_plugin_directory(file)
            if is_zipfile(file):
                result += self.process_plugin_jar(file)
        return result

    def process_plugin_directory(self, directory):
        propertiesFileName = join(directory,"META-INF","MANIFEST.MF")
        if exists(propertiesFileName):
                return self.process_properties(self.graphviz_friendly_name_from_plugin_filename(directory), open(propertiesFileName).read())
        return ""
                    
    def process_plugin_jar(self, jarName):
        try:
            jar = ZipFile(jarName)
            propertiesFile = jar.read("META-INF/MANIFEST.MF")
            return self.process_properties(self.graphviz_friendly_name_from_plugin_filename(jarName), propertiesFile)
        finally:
            jar.close()
        
    def process_properties(self, pluginName, properties):
        result = ""
        start = properties.find("Require-Bundle:")
        if start != -1:
                lines = [line.strip() for line in properties[start+len("Require-Bundle:"):].split('\n')]
                for line in [line for line in lines if len(line) > 0]:
                        lastLine = not line.endswith(',')
                        if not lastLine:
                                line = line[:-1]
                        parts = line.split(';')
                        line = parts[0]
                        result += self.graphviz_friendly_name(pluginName) + " -> " + self.graphviz_friendly_name(line) + ";\n"
                        if lastLine:
                                break
        return result
        
    def graphviz_friendly_name(self, line):
        return line.replace('.','_')
            
    def graphviz_friendly_name_from_plugin_filename(self, fileName):
        return self.graphviz_friendly_name(os.path.split(fileName)[-1].split('_')[0])

if __name__ == "__main__":
    if len(sys.argv) != 2:
            print "need plugin directory location"
            print 'e.g. python dependencies.py "C:\Program Files\eclipse\plugins"'
            sys.exit(1)
            
    pluginDirectory = sys.argv[1]
    DependencyGrapher().run(pluginDirectory)

40 line Java web app

I discovered something yesterday that is just so cool; it's cooler than cool; it's ice cold. Those clever Jython folks implemented the python module BaseHTTPServer - I didn't realise this the last time I looked at Jython (which I thought was pretty clever just as a scripting language for Java).

A 40 line java web app¹

Sometimes it's useful to get data out of a database and put it on a web page. (If you don't know what I'm talking about - that's what Enterprise IT is). Sometimes it's useful to do this just for development, independent of the actual production code. For example: for tools, visualisation, testing, system debugging, setting up test data etc.

¹In Ruby on Rails or Django (see A comparison of Django with Rails) it would be less, but that's not what this article is about.

Using Jython's BaseHTTPServer you can write handy little Web Apps in hardly any code, really quickly, still using your Java Enterprise Heavy Duty IT code if you need.

Here's an example (which I hope will also be useful to someone out there). I've got Jython 2.1 installed, MySql running, and the MySql JDBC driver on the classpath².

This creates a web app that shows you the contents of any table in the database. Run this code (having configured database, user, password to suit), open a web browser on "http://localhost:8000/table/foo" to see the contents of table "foo". Note - this has no protection against sql injection attacks so be careful where you run this. This code is provided under the GNU Lesser General Public License and has NO WARRANTY.

import BaseHTTPServer, urllib, java.lang, java.sql

java.lang.Class.forName('com.mysql.jdbc.Driver')
url = 'jdbc:mysql://localhost:3306/database'

class DBHandler(BaseHTTPServer.BaseHTTPRequestHandler):
	def do_GET(self):
		path = urllib.unquote(self.path[1:]).split('/')
		
		self.send_response(200)
		self.send_header("Content-type", "text/html") 
		self.end_headers() 
		
		if path[0] != 'table' or len(path) != 2:
			self.wfile.write("don't understand "+str(path))
		else:		
			con = java.sql.DriverManager.getConnection(url,'user','password')
			s = con.createStatement()
			table = path[1]			
			r = s.executeQuery("select * from "+table)
			
			html = java.lang.StringBuffer("<html><head></head><body><table border=1>")

			numCols = r.getMetaData().getColumnCount()
			while(r.next()):
                                values = [str(r.getString(i)) for i in range(1, numCols+1)]
				html.append("<tr><td>")				
				html.append("</td><td>".join(values))
				html.append("</td></tr>")

			r.close()
			s.close()
			con.close()
			html.append("</table></body></html>")
			self.wfile.write(html.toString())

server_address = ('', 8000)
httpd = BaseHTTPServer.HTTPServer(server_address, DBHandler)
httpd.serve_forever()

An example of it's use, (having done the Django tutorial in case you recognise the table name or data - although this has got nothing to do with Django as such), http://localhost:8000/table/polls_polls shows me:

1	1	Not much	3
2	1	The sky	1

Adding column names is left as an exercise for the interested reader.

²Note that jython didn't like it when the classpath contained a directory "mysql-connector-java-3.1.11" - it didn't like the mix of "-" and ".". I don't know if this is a known issue - I'll look into it.

What's the Big Deal?

Someone read this and asked what the big deal was. He mostly uses Microsoft technologies rather than Java; I've added this section to answer the question.

In Java, you'd usually have to have an application server installed, configured and running, and write a servlet or equivalent and compile/deploy into the application server.

Using the code shown here, you're not having to do any of this. The code shown is all you need - it's the whole server thing - the whole database access thing too - no separate application server or configuration or deployment. Just run that code - it starts it's own server almost instantly. Just starting up some application servers takes a minute. Installing multiple application servers so you can have a lighter weight one for tools is an option, but configuring them can be lots of hassle. There are exceptions, like Orion, which is relatively straightforward, but even in that, it's more hassle than copying and pasting the code I've shown and editing to suit. At some places, there would be techno-political issues of installing another application server. Installing Jython for scripting tools is less likely to be objected too - in that sense, in some cases, it could be considered partially a technical solution to a techno-political problem.

If you were using .NET I guess you'd usually have IIS already installed. In many ways, this is an example of the classic Microsoft vs Java technical differences. Microsoft stuff often being very simple for simple things but eventually getting horrible for difficult things (for example, being difficult to script IIS configuration) and Java sometimes being less simple for simple things, but with more things configurable and capable of doing more difficult things. (For some things Java is also very simple for simple things - but probably less so, on average, than Microsoft equivalents, but I probably shouldn't have even started along that line as I'll probably get lots of "my technologies better than yours" comments).

Final words

I was so impressed to find that BaseHTTPServer is implemented for Jython. It makes it really easy to create a HTTPServer.

Just to clarify, in case you haven't used Jython - you can import any of your existing Java code and use it in a jython program like the example shown.

The example code isn't how I'd normally look at the contents of a table in a database; I'd usually use something like SQuirreL - this is just an example that I hope you find useful as a starting point for implementing other useful things.

Pexpect - replaces me with small script.

Pair-programming with one of my wonderful team colleagues last week, we needed to automate an SSH thing.

The problem

SSH doesn't allow the password to be specified on the command line. Usually you'd set up SSH so you don't need to type your passwords). Due to one thing and another, we couldn't do this.

The solution

We found Pexpect (a pure Python version of expect). Pexpect makes it easy to automate things that require user interaction (like typing in a password when prompted). As Python was available on the relevant machine, we tried it out - the example using ssh worked first time (here included under the appropriate license):

#!/usr/bin/env python '''This runs "ls -l" on a remote host using SSH. At the prompts enter hostname, user, and password. ''' import pexpect import getpass host = raw_input('Hostname: ') user = raw_input('User: ') password = getpass.getpass('Password: ') child = pexpect.spawn("ssh -l %s %s /bin/ls -l"%(user, host)) child.expect('password:') child.sendline(password) child.expect(pexpect.EOF) print child.before

Conclusion

Cool! Another useful addition to the post-modern toolbox.

Find Bugs - OOPSLA 2004

I enjoyed the talk on Find Bugs at OOPSLA 2004. Find Bugs is a useful, practical and pragmatic tool produced by David Hovemeyer, Bill Pugh and others at the University of Maryland.

It is much like a style checker but with rules that find things that are probably bugs. It's a very pragmatic, and simple, approach. It is somewhat reminiscent of SmallLint (for Smalltalk). The bug patterns that Find Bugs looks for are based on bugs that the Find Bugs developers have found in real code. In their talk, they presented figures of using Find Bugs on the JDK1.5 code base - it identified many potential bugs.

I tried Find Bugs on one of my open source projects, Jester. Mostly, the things it found weren't things that worry me too much - it identified my sloppy coding where streams won't be closed if an exception is thrown, which I'll probably fix some time. What pleased me most was that it found a couple of fields that I can delete (not bugs), and I do love deleting code.

Nester - a C# port of Jester

I recently found out that Nester has been released. It's a C# port of Jester (one of my first open source projects).

Jester is a mutation testing tool. It tests test quality by seeing if mutations to your source code cause tests to fail or not. If no tests fail as a result of changing some code then that code might not be tested. It's different to code coverage. All code coverage can tell you is whether your code is executed by the tests; it says nothing about the quality of the tests. You could write tests with no assertions that had a 100% code coverage measure.

I'm thrilled that someone has done this port; I feel it's a milestone for any open source project that someone ports it.

There are lots of code coverage tools but very few mutation testing tools. I'm not entirely sure why this is. Jester is pretty difficult to use, slow to run and the results aren't always very easy to interpret, but apart from that ...

OT2004 : Delegation in Java

The session Delegation in Java - presented by Erik Groeneveld and Willem van den Ende was really interesting and thought-provoking - it's about "true delegation" rather than what most people these days think of as delegation (which should possibly be better called "forwarding"). True delegation is where methods can be delegated by one object to another but the receiver (i.e. this) stays bound to the object doing the delegating rather than the object being delegated to.
The consequences of this are that using delegation allows you to do things like multiple implementation inheritance and dynamic inheritance (changing what class an object "inherits" from at run time - very good for implementing the State Pattern).

The presenters have written an open source library for delegation in Java called Delegator which they introduced in this session, and we did some hands-on coding with it during the session. Unfortunately, "true delegation" in Java is rather difficult and doesn't come out very neatly, but it's an interesting concept to get your head around. If you want to try a language that does it really well, have a look at Self