Thursday, October 07, 2004

Waterfall Method: A Colossal Blunder!

Larman, Craig. Agile and Iterative Development. Addison Wesley, 2003.

I have received many requests for documentation of project failures caused by the waterfall method and the history of the many disasters introduced by accident when the Department of Defense standardized on a method that was unproven on large projects and essentially, a blunder by a consultant who had little experience with real software development.

The DOD has long since abandoned the waterfall method, and the consultant has recanted, but the waterfall approach persists as an urban myth in many software development organizations. Craig Larman details and documents this historical tragedy in Chapter Six of his book which compares the two leading agile methods, SCRUM and XP, along with RUP and Evo, an early iterative method.

This five star book is certainly required reading for anyone interested in Scrum, particularly those who are Certified ScrumMasters.


vijay said...

Scrum seems to be hot sir and no doubt about that but my concern is not the fact that agile analysis is In but the poin you made in this blog about waterfall method...

I have to say i disagree big time, i have had some exposure to SDLC and have met and talked to a lot of people including software developers, Project Mannagers, Business Analysts, Team Leaders, Testers and i never heard such harsh words about Waterfall..

most people did mention that though waterfall is certainly not followed as it was invented, but in some way or the other it is used and used very widely, i cant quote any one on this but most people said they might not be willing to admit they work on waterfall BUT they do use it flexibly rather then the strict guidelines mentioned in the methodology..

As they teach waterfall methods in schools / colleges to the beginners & novices still.. i really have strong feeling that waterfall might not be a blunder if used according to one's requirements..

i am sure you have had some personal experiences in regards to the same, but any how calling the whole method a "Colossal Blunder" is'nt justified.

Anyhow i afree there is wide crticsim of the waterfall method but dont we all use it all the time in some way or the other is the point i wana make...

Jeff Sutherland said...

The Waterfall process is a "colossal" blunder because it has cost 100s of billions of dollars of failed projects in the U.S. alone. Capers Jones noted 63% failure rates in projects over 1M lines of code in 1993. By the late 1990's, military analysts were documenting a 75% failure rate on billions of dollars worth of projects. In the U.K. the failure rate was 87%.

A failure rate this high would kill the U.S. space program but it seems unable to eliminate the waterfall mentality from professional software development. This is one of the great psychological aberrations of our time.

The high risk of the waterfall process and the fact that it was a blunder is confirmed by the person who wrote the intial waterfall standard into DOD specifications. Craig Larman has thoroughly documented this in the posting above.

Schools and universities are teaching bad business practices. They are often 20 years behind the times. I've been on advisory committees to universities and continually pointed out that the computer science they were teaching was obsolete and not used in high technology companies. I remember at Brown University after one of my presentations, a Ph.D. candidate jumped up and said everything he had learned in 4 years was wasted. This is the sad situation we find in even the best universities.

That said, there are some successful companies that always deliver on time and on budget using the waterfall method. Most of them are moving to Scrum because cycle time is too long with the waterfall approach and the product they deliver does not meet customer needs as requirements have changed during the development process.

So even if you like the waterfall process and are successful with it, in todays global economy you cannot use a process that takes twice as long as Scrum and costs twice and much and delivers software than no longer meets customer needs. This is what most of the biggest and best companies in the world tell me.

vijay said...

I do understand your point of view, but what i had to suggest was'nt that waterfall model is good, but that it is considered deafult model by many in the industry and also if one uses waterfall model flexibly rather then the rigidity of the typical model it can reap benefits rather then being a disaster..

That is some point of view which seems to be completely missing from your obsevation of the waterfall method..

Also i would like to add,as comapred to your knowldege and experience my words or deeds dont carry any value, i have just passed out from school and am very much a novice, but i do try to learn and learn quickly by seeking all kinds of opinions and learning about technologies which are relevent or which have had some kind of influence on SDLC primarly.. my technical knowldge is pretty much low as my background is'nt technical but from the limited amount of resources and knowldege i have i just wanted to put my point forward and make sure i can propose a theory which seems to be working for some people , even if they are not the best in the business, they are acheving their tasks by using this obselete method according to their convenience.

to sum it up, am sure nothing beats agile, XP is something which probably every one would be using in foreseen future and going through the SCRUM stuff it seems you guys are on the right track... cheers

有祥 said...

Every software development methodology or approach has advantage and shorcomming.
According my understanding, the each sprint in Scrum is a samll waterfall.
The following steps will be launched in each sprint:
Planning->User scenario--> High level design-->Coding-->Testing-->Milestone driver.
It's a small waterfall, isn't it?

Maybe we could say, the Scrum process bases on the Waterfall process..

Jeff Sutherland said...

Running a miniWaterfall in a Scrum iteration is called WaterScrum and viewed as an error in implementation. A typical Scrum with have Product Backlog "ready" in the form of an Agile specification, just enough to describe the user experience. At my company all design and technical specification is produced inside the Sprint with acceptance test cases created first, maybe some test driven development and coding, which will then influence design. Testing does not come at the end but the beginning. There are many other differences from WaterScrum too numerous to mention here.

Let me reiterate, for projects over $3M-$5M, the Waterfall has an 85% failure rate. For those projects that are successful, an average of 65% of the software is never used. The Waterfall is a collosal blunder. The most successful Waterfall company I have worked with had a 100% Waterfall project success rate with on time, on features, and on budget. This led to a 100% failure rate in customer acceptance because the customer's business had changed or because the customer did not understand the requirements.