If you have a long enough build and large enough team, then you can quite often have the situation where there are multiple commits between continuous integration server builds. In these circumstances, if the build is broken then you don't know which commit broke the build.
Keith Braithwaite told me that in one team he worked in, someone would manually perform a binary search between the multiple commits to find out which commit broke the build. At a Google event, I was talking to someone about build-o-matic and he said that the feature he'd like for a continuous integration server was to do a binary search to find out which commit broke the build.
Build-o-matic is the only continuous integration server that I know of which automatically performs a binary search if the build is broken and there are multiple commits between builds (I call this feature "blame-o-matic"). The binary search is between the last known passing build and the first known broken build. Build-o-matic only performs one step of the binary search at a time - if someone commits while it's running one of these binary search builds, then it'll pick up those changes and run the build. If the build is still broken and there are no other commits, then build-o-matic will do the next binary search, etc.
The binary search feature is clever but not always ideal. If the build is broken, sometimes it's obvious from the build output (which build-o-matic usefully shows and saves for you). For example, sometimes someone will realise straight away that they forgot to check a file in. If the build can be fixed straight away by (for example) checking a file in then it could be that build-o-matic has set off a blame-o-matic build, so you have to wait for that build to finish before your commit gets picked up and the build gets fixed.
Having been using the "blame-o-matic" feature of build-o-matic for a while, I've now also added a feature called "filling in the gaps". If there are no incoming changes for build-o-matic to run the build for, and it isn't doing a binary search, then build-o-matic runs the build for the most recent revision that it hasn't run the build for. There are cases where doing a binary search doesn't work - if the build is broken, fixed then broken again within the revisions that are checked in at the same time, then the only way to be sure which revision was the most recent that broke the build, you have to build every revision.
Another benefit of running the build for every revision is that if your build includes any statistics (e.g. the build time, amount of sloppiness, etc) then you get the statistics for every revision.
You might expect that the "blame-o-matic" and "filling in the gaps" features would result in the build-o-matic machine running the build all the time when you want it to be ready to run the build on a new commit. In practice, it works out well. If the build is fast enough, then it is rare that there are multiple commits between builds and the "blame-o-matic" and "filling in the gaps" features will not run very often. If the build is slow, then there is a very large chance that someone has checked in between builds, in which case, this will take precendence over the "blame-o-matic" or "filling in the gaps" builds. If the build is slow and people are careful not to check in if the build is broken, unless they are fixing the build, then the "blame-o-matic" can help find which commit caused the problem in parallel with people looking to see if they can work out what broke the build. The "filling in the gaps" feature can work over night while no one is checking code in, ready to show you the next day the full history of which builds broke and which passed.
A possible improvement to the blame-o-matic feature would be to kill a blame-o-matic build if build-o-matic detects that someone has committed while a blame-o-matic build is running. This would probably be simple enough for build-o-matic to implement, but I didn't implement this because it means you would have to make sure that your build can be killed at any point without leaving the environment in which the build runs in a state in which the build cannot be run again. One common, but not necessarily good, feature of many builds I've seen is for acceptance tests to run against a database, which has to be partially set up and torn down rather than set up from scratch, due to the length of time it would take to do it fully. I didn't want build-o-matic to impose "interuptability" onto the builds it runs yet - maybe a future version will - maybe a future version will support both interruptable and non interruptable builds ... I don't know yet.
I understand that Team City has a feature where you can submit some code for building, and then it is checked in if the build passes (that's my understanding having talked to someone who has used it - but I haven't used it myself). Using that feature as the only way to check code in would mean that the build would never be broken. I didn't implement this for a couple of reasons. It's harder to implement, and I'm not sure I'm keen on it - I want code integrated as soon as possible - a delay in integrating code due to waiting for a build on a continuous integration server doesn't seem ideal to me. I reserve the right to change my mind.
build-o-matic is open source - try it yourself. It's somewhat limited at the moment - it only supports subversion, and ant (or maven) "out of the box". I'd like to add support for perforce some time soon, but have no plans to support any VCS system that I don't want to use myself.