Although there are many reasons for Waterfall Failure or Waterfall ‘Leaks’, here are some common causes:
Requirements & Scope Control
Requirements tend to be unclear, lack agreement, lack priority, can be contradictory, ambiguous, and imprecise. With that said, Waterfall assumes a detailed set of requirements will be 100% accurate at the beginning and locked for the duration of the project. Changes to requirements are bad and Waterfall tries to minimize once the plan is created.
Planning (schedules & budgets)
Schedules and budgets can be based on insufficient data, missing items, insufficient details, and poor estimates. Waterfall assumes the initial estimates are locked and accurate. Frequently dates and budget are decided well before even project engineers had a chance to provide their own input on estimates. These same team members are then forced into long spanning timelines they don’t agree with. This can also lead to overly cautious PMs who end up in ‘analysis paralyses’ during the planning phase in an attempt to perfect estimates which can delay the start of the project.
Another common reason for failure is poor communication and stakeholder engagement. This also causes lack of clarity and trust. Waterfall assumes the PM should plan all the work or tasks (i.e. planning from the center) and the project team must execute the tasks they are assigned. There is no sense of ownership from the team’s perspective.
Customer Value (and quality):
Typically, Waterfall does not provide any value back to the customer until the very end after all other phases. All requirements are locked together as they journey through all the waterfall phases only to realize that what they wanted a year ago is already obsolete and needs changing. Waiting to do the testing until the end of the project risks finding out a major issue right before deployment which will cause the project dates to slide and be over budget. A common waterfall solution to this is to reduce testing effort to ‘keep the dates’ and turn over a poor quality system into production.
Scrum is an agile process that allows us to focus on delivering the highest business value in the shortest time. It allows us to rapidly and repeatedly inspect actual working software (every two weeks to one month). The business sets the priorities. Teams self-organize to determine the best way to deliver the highest priority features. The customer can see real working software and decide to release it as is or continue to enhance it for another sprint.
In 1993 Jeff Sutherland created the scrum process and borrowed the term “scrum” from an analogy put forth in a 1986 study by Takeuchi and Nonaka, published in the Harvard Business Review. In that study, Takeuchi and Nonaka compare high-performing, cross-functional teams to the scrum formation used by Rugby teams. In 2001, the Agile Manifesto was signed by 17 developers at the Snowbird Resort in Utah and the agile movement was born. The manifesto has four main principles:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
As you can see, the agile fundamentals on the left are in contrast from the typical waterfall ideas on the right.
In our first article let’s discuss how Scrum tackles the above mentioned Requirements and Scope Control issues by creating a Product Owner role, using the Product Backlog and creating User Stories.
The Product Owner is a key role in the Scrum framework. The core responsibilities of the Product Owner are to represent the customer and stakeholder requirements to the project team, and to respond to the questions from the team. They must keep the product backlog current and prioritized. To maintain the product backlog, they must communicate regularly with customers, stakeholders, and the team. Product Owner should meet with stakeholders at least every one to two weeks. The order in which user stories are implemented will affect the payback period and the amount of work that the team must perform. The team will help the Product Owner understand how the sequence of user stories affects the work.
Product Owners reduce the need for detailed specifications by being available to team members when they have questions about how to implement functionality. To keep the team moving, they must have these answers or be able to find them very quickly (in a few hours). If they are unavailable, team results will suffer.
The Product Owner will also work with both the scrum team and the stakeholders to build acceptance tests. Acceptance Tests help determine whether a particular user story is completed and ready for the sprint review.
The product backlog is a list of user stories managed by the Product Owner that describe what the customer needs and values are. The user stories are prioritized by business value and risk, and the team estimates user stories in abstract units that are called story points.
As the scrum team makes progress the team will add detailed information to the user stories, break them down into smaller stories, prioritize and estimate them, and finally, implement them and deliver the results to your customers.
Over the course of the project, user stories can be added to the product backlog and user stories may be removed. In addition, the priority of some user stories will change, and some user stories may be implemented earlier or later than originally expected.
User Stories are functional requirements broken into stories – user does x to accomplish y (or something of business value) and written by the Product Owners and scrum team. The INVEST mnemonic was created by Bill Wake as a reminder of the characteristics of a good quality user story.
|I||Independent||The user story should be self-contained, in a way that there is no inherent dependency on another user story.|
|N||Negotiable||User stories, up until they are part of an iteration, can always be changed and rewritten.|
|V||Valuable||A user story must deliver value to the end user.|
|E||Estimatable||You must always be able to estimate the size of a user story.|
|S||Scalable (small sized)||User stories should not be so big as to become impossible to plan/task/prioritize with a certain level of certainty.|
|T||Testable||The user story or its related description must provide the necessary information to make test development possible.|
When the scrum team creates a user story, they make sure that it represents customer value by answering the following questions:
- Who the user is
- What the user needs to do
- Why the user needs to do that
A familiar User Story format was created by Mike Cohn who suggests the following format:
“As a <user>, I need to <action> in order to <reason>”.
The user stories should be defined in a way that allows them to be implemented in any order and small enough to be implemented in the sprint. This allows them to be easily moved around in the backlog. They are also just detailed enough for the scrum team to describe and estimate the work that is required to implement the story.
By using Product Owner, Product Backlog, and User Stories, Scrum allows for better control of business requirements and scope change. The Product Owner becomes the requirement custodian – the one bridge between the stakeholders that need the change and the project team that builds it. As the Agile Manifesto dictates, the Product Owner plays a big part in the ‘Individuals and Interactions’ as opposed to processes. Consider the Product Owner as the Captain of the ship. He or she steers the ship (or scrum team) through the project journey to its final destination.
The Product Backlog contains the User Stories that are managed by the Product Owner. This removes uncertainty from the team on what to work on – the Product Owner clearly has the priority laid out! The team works on those User Stories until completion at the end of the Sprint (typically 2 weeks). The scrum team develops the User Stories through the normal development phases of analysis, coding, testing. It also encourages frequent changes in business requirements and allows the team to adjust with little impact to cost or schedule – there is no scope creep.
Well-written user stories allow the scrum team to accurately estimate the effort needed. User Stories also allow the project team to focus on a smaller portion of the overall scope. This focus therefore creates better quality code. All this promotes better project team moral since the team has ownership of the estimates and tasks. They know what needs to get done in the next two weeks without getting overwhelmed by the entire project scope.
Requirements and scope control are one of the most important items to consider when projects start up. History will tell that this Waterfall ‘leak’ has sunk so many other projects into the abyss! Scrum may not be the perfect fit for your project and it is not the silver bullet to fix all projects; however Scrum as a project methodology has the tools to stop these common project failures. Stay tuned for future articles that detail out how Scrum fills the other leaks in the Waterfall Methodology.