Ivan

CITCON 2007

I attended CITCON (Continuous Integration and Testing CONference) in Brussels last weekend - and just like last year, it was great! It's an expertly facilitated open space conference by Jeffrey Fredrick and Paul Julius, through their Open Information Foundation. I think I enjoyed it even more this year than last. It really is an exellent conference. I'm already looking forward to the next one.

Mutation testing - Jumble and Jester

Squirrel proposed a session on mutation testing; he wanted to demo Jumble and suggested that I should say something about Jester. It turned out to be a very popular session - it's still an idea that people are interested in.

Mutation testing

In a nutshell:
* Make some change to your code
* Run your tests
* If the tests pass then either you are missing a test or you have unnecessary code.

Jester difficult to use

It's a long time since I've tried to use Jester - I haven't worked on it for a long time - I checked with sourceforge and the last release was February 26, 2005. I spent an hour or so before the session, preparing a demo. It was quite horrible trying to get it to work.

New version of Jester

After finding out the level of interest, and being reminded of the difficulty getting Jester to work, I'm intending to release a new version of Jester - probably a fork because I've handed over project admin to Elliotte Rusty Harold and what I've got in mind is a simplification that has some disadvantages as well as some advantages. It'll be simpler code, simpler to use, but even less efficient. For me, making it simple to use is more important than the performance.

Jumble and others

Since Jester was released, there have been other implementations of the same sort of thing. Squirrel demoed Jumble, which he was keen on. An ex-ThoughtWorks colleague of mine, Stacy Curl, wrote "ajester" which, like Jumble, used byte code mutation rather than source mutation - but I don't know if he ever released it - I can't find a release package. Michael Nyika wrote Grester - a Maven2 Plugin for Jester. There's also a dot net version of Jester, called Nester. Another mutation testing tool is muJava which apparently now has an eclipse plugin. There is an eclipse plugin for Jester, but I'm afraid it never quite got properly released (sorry).

CruiseControl - still the daddy

CITCON is about continuous integration as well as testing. It's very clear from CITCON that cruisecontrol is still by far the most used continuous integration server. There are dozens of other continous integration servers available but none of them have much "market share". I was suprised by how few people had even heard of TeamCity because it seems to be becoming trendy with the London XP crowd.

The second most used continous integration server by CITCON attendees appeared to be Hudson demoed by Eric Lefevre which looks nice enough but still doesn't do what I want.

The only continous integration server that does what I want is build-o-matic - because I wrote most of it! I did a demo which I think went OK - I think the feature people like the most is putting the pictures on the results page. Nat Pryce wrote a plugin called team-piazza that does something similar for TeamCity.

More on the latest developments of build-o-matic in a future post.

No bug reports != no bugs

It's difficult getting feedback about software.

Do people log bugs?

Mostly, you get no feedback. Sometimes people log bugs, but not very often. With open source software, or web sites providing some service, if it doesn't work straight away (and very simply), mostly people will ignore it and try the next thing that looks like it might do what they want.

If an open source project has no bugs logged against it, chances are that it's very little used (possibly because it's totally broken) rather than because it is so high quality that it has no bugs. When people do log bugs, it's against software that they want to improve, not software that they can't be bothered with. In fact, I'd say it's generally a good sign if a project has several bugs logged against it and a bad sign if it has none. And yes, I have at least one open source project in the no bugs category (and I think it's good despite what I've just written)!

From "oops" to "ah ha"

I remember a few years ago, I released a new version of Jester (as a zip file). It was downloaded a couple of dozen times before someone (thanks - sorry I've forgotten who it was) told me that they couldn't unzip the file. It turned out that it had got corrupted on upload to sourceforge - I hadn't expected that. Of course, since then, I download and check that the zip wasn't corrupted (a bit of a pain, and I've not had a file corruption problem since, but I guess these things happen once in a while).

Your responsibility to kittens

In his keynote speach at last years ACCU conference, Mark Shuttleworth (Ubuntu founder) said something along the lines of "every time you fail to log a bug, a kitten dies". I like the sentiment. Please do log a bug, even if it seems so obvious that you think that surely the creator of the software (or person running a website) knows about it.

Knives behind your back

In the case of my open source projects, sometimes I'll google to see if someone has said anything about any of them. Sometimes I'll find an article slagging off one of my beautiful creations by someone who hasn't thought to tell me their criticisms or log any bugs.

Ignoring feedback

When using the O2 website I tried to do a search and got the following very obvious error:

Try it yourself. Simply enter anything in the search text box and press the search button. Following my own advice, I sent o2 an email about this (8th September). I tried again tonight (30th September) and got the same error. Never mind. I'll continue to try to save the kittens.

And another thing

There is a whole other article I could write about how to write bug reports ...