Understanding schedule and scope in Agile projects.

Here is an example of something that the software world needs to improve.

In his blog post from January 4th, James shore talks about software project estimation. He describes a waterfall project with 4 milestones and asks the reader to guess how much that software will be late based on the information he gives. Here's the question (check the answer in Jame's blog):

Bob and Jane are leading a project that started January 1st, 2008. The deadline is December 31st, 2008. There are four milestones, three months apart: one each at the end of March, June, September, and December. Each milestone requires an internal release, so each milestone does a good job of exercising the team's ability to create a shippable product.

The team doesn't meet their March deadline until the end of June. In other words, they're three months late. Assuming the team size and makeup doesn't change and the requirements don't change, what's the most reasonable thing Bob and Jane could tell their boss about their schedule?

1. The project will ship on time, in December, as planned.
2. The project will ship three months late, at the end of March 2009.
3. The project will ship six months late, at the end of June 2009.
4. The project will ship a year late, at the end of December 2009.
5. The project will never ship at all and should be cancelled.

James goes on to explain why he chooses option 4 as the correct answer and how he is surprised everytime people select answers 1 or 2.

I agree with James when he states that options 1 and 2 are not possible or reasonable answers. The team is late, there's no chance that they will ever recover the time and complete what they had planned to complete.

Why?, you may ask (good question btw). The reason is that empirical evidence shows that when you start a software project, that is the moment in time when you know the least about that project. It stands to reason that as you go in the project your knowledge of the domain, features, requirements, environment grows. With that knowledge grows also the scope of the software project.

More knowledge = larger scope

"More knowledge = larger scope" is a truism anchored in experience over many projects, and something on which you can count 100% unless your project is small enough to be done in a few hours and you have worked in the same environment and domain before.

Scope grows, not out of "feature creeping" phenomenon but because we honestly don't know in the start of the project as much as we do when we are in the middle of it or in the end.

Complexity and paying for knowledge

Within this truism lies one of the core reasons for the success of Agile development methods. In Agile software development methods there is a concept of spike (or paying upfront for knowledge).

An Agile team, recognizing that a certain feature is complex or a certain domain area is not clear enough, will invest time into learning more about that area. The way that team does it is to decide how much time they want to invest (time boxed schedule) and prototyping a possible solution/implementation as they go along. I call this "paying for knowledge". The team takes time out of the other tasks to learn more about a certain area (paying for knowledge) and therefore avoiding surprises later on when they no longer can afford investing in learning.

The time box ensures that no more time is spent than what the team has agreed to invest and provides a good tool to avoid being late.
The prototype provides a simple tool to learn the domain or requirements.

It is important to acknowledge that the prototype developed in a short time box like this is a throw away, but it's usefulness lives on in the form of deeper knowledge about the problem to be solved.

The spikes like this can be of different natures:

• they can be technical, where the prototype can be code, or implementation;
  
• they can be about requirements, where the prototype can be a more detailed description of certain requirements;
  
• they can be about user interaction, where the prototype could be a paper or flash-like prototype;
  
• they can be about processes or ways of working, in which the prototype could be a full sprint/iteration or a new continuous integration system.
  

There are other types of spikes and the concept should be used whenever necessary to avoid being surprised too late in the project and having to make hasty decisions.

Why waterfall does not work for predetermined schedules

The point that James makes in his blog (independently of the answer he chooses) is quite important, if you try to spec a project in advance (including scope, time and resources) you will fail, and the result can be catastrophic if not stopped in time.

Waterfall software development does not work because it does not allow for "answering to surprises". You can not plan spikes as the ones described above before you actually start the project. You just don't know enough.

And that (not knowing enough) is the reason why waterfall projects will always be late and over budget. You cannot plan for what you don't know.

Agile software development methods recognize that failure and create a set of tools to deal with that. Here are some of the tools and their expected outcomes:

• On site customer (XP), to provide quick answers and clarifications about the details of the product so that the team does not get stuck with unclear requirements;
  
• Time boxed iterations with reflection workshops, to provide a clear time limit for the team to accomplish their planned goals and also a moment to reflect and learn from what they did during that time;
  
• Continuous requirements management through the planning days and continuous backlog management, this avoids big scope surprises late in the project by forcing the stakeholders to review the requirements on an ongoing basis;
  
• Team alignment through team design sessions, this increases the level of knowledge about the problem and solution in the whole team as opposed to having only one person create the architecture and design in isolation and then having to transfer that knowledge to the coders. NOTE: because design is not just the document, but the document + the mental model of the designer it is impossible to transfer completelythe design from one person to the other - I've commented on this recently in my twitter stream.
  

If you are aware of other techniques that are specific or a requirement for Agile teams list them below in the comments section.