Ian Hugo was at the first conference on Software Engineering (the term was invented for the conference!) in 1968. He gave a talk reflecting on that conference (and the 1969 follow-up) and looking to the future.
He advised not to push any technique beyond what it's good for; a particularly pertintent comment for an XP conference. XP is great for many projects but not for all. The White Book ("eXtreme Programming explained", by Kent Beck) says what sort of projects XP is suitable for, and it's worth remembering Hugo's advice.
Hugo said that in 1968 there was a "fashion" for throwing large numbers of people at problems; many recognised that this didn't work, and this was written about by Fred Brooks in The Mythical Man-Month (1975), a classic that is still very relevant today. Some problems in software development, that we still see today, have been known about for over 30 years.
Hugo mentioned an interesting experiment (sorry, I don't have a reference for it) where a team produced a Fortran compiler several times. First time, they had a 50% overrun compared to their estimates. On their second attempt they still had a 50% overrun compared to their new estimates, but produced the compiler in 50% of the original time. On their third attempt, they had a 20% overrun and used 50% of the time for the second attempt (I think these were the figures; my notes are a bit rubbish - sorry if they are wrong, but they were something similar). Therefore, overall, although the development team got faster on subsequent similar projects, their estimation, although it improved, was always over-optimistic. This matches my experience too, which is why I like the eXtreme Planning approach of XP; it features feedback so that estimates only have to be consistent rather than accurate.
Hugo mentioned one of my favourite truisms, that has been known since 1968 but still many people don't seem to understand (otherwise, why would waterfall projects still be numerous?). The existence of a software system changes what the customer wants from that software system.
One of the most suprising comments by Hugo was that in 1968 it was recognised that you couldn't separate design from construction of software. My impression of "software engineering" from my University days was that "software engineering" assumed that you could make that separation, so to hear that at the very first conference on the subject it was recognised that you couldn't was interesting. Talking to Hugo afterwards, he clarified that the hardware of 1968 was extremely limited by todays standards; performance was a major issue. Design could not be abstracted away from the actual implementation because performance constraints had to be taken into account for the implementation.
Hugo said that although hardware and software technology have changed enourmously since 1968, people haven't; developers can still only write the same amount of code as they could 30 years ago. The difference is in the level of the code and the amount of libraries etc that can now be used.
Posted by ivan at June 14, 2004 11:54 AMIan Hugo sent me this email and has given permission for it to be included here as it's very relevant to this entry (Ivan).
Ivan,
I've now had time to think some more about the point you raised on
why the 1968 conference thought that design and construction
could not be separated. I gave you one example but think that,
more generally, it was the realisation that designs (at the time)
were never sufficiently detailed to be explicit on all points of
construction. Decisions that had to be made at the construction
stage could, in effect, send the designers "back to the drawing
board". If design is really to be separated from construction, then
there should be no construction desicions that can modify the
design. At the time, the construction phase effectively had as one
of its tasks to elucidate the design and, in practice, revise the
design through a process of iteration. That strikes me as a very
similar process to the one you adopt in XP (if I'm allowed to call
that a "process"!)
I think it's probably a very small step from there to your current
perception of best practice being to design only the next small part
of the software that is to be constructed. After Garmisch, there
was a movement to find design languages, which never produced
anything very useful until UML came along. A reason for that could
be that design langauges are inherently of limited use, if design
and construction should properly go hand in hand.
It further occurs to me that scope has a role here. If we dispense
with large groups of developers and large software entities having to
be considered all at once, then the need for wide-scale
communication disappears. A small team approaching a task of
fairly easily intellectually manageable scope can work effectively
with a shared mental model, which disposes of the need for much
in the way of design documentation. This would tend to prove your
case.
However, as always, I am suspicious of any precept that is
supposed to apply to all cases. I mentioned the requirement for
very high security as one case that might need more initial design
work. I mentioned this because security, if it is to be tight, must
be seamless and is typically most vulnerable where exceptions or
inconsistencies occur. How can inconsistencies be avoided if
there is no overall plan?
The answer to my question may lie in the fact that this is properly
a system rather than an application problem, since I doubt whether
a truly secure application can be built on an insecure system. And
I think the major relevance of XP is to application rather than
system programming.
I had a further thought that came to me from observing the XP
"game". One of the teams was being criticised for not interrogating
the "customer" sufficiently over the requirements. This seems to
me to impose the prime responsibility for clarity in the
requirements on the developers. If this inference is correct, then I
think it is an important (and positive) point. However, maybe it
shouldn't be made too explicit; it probably has implications in
contract law!
Anyway, thank you for giving me something serious to think about.
I now think that perhaps that design and construction are even
more tightly linked than I thought before. Whereas, before, I
thought they couldn't be separated for essentially reasons of
practicality, I am now considering whether there is a more innate
reason.
Cheers,
Ian Hugo