User Stories are representations of a conversation, not a list of requirements!

Great post by Martin Fowler about a hidden aspect of the User Story flow within a team. 

Many times I see teams that expect the Product Owner to behave just like a Requirements manager did in the past. Come up with the content, put it in the tool and stay out of the way! That's wrong!

In this article Fowler calls our attention to a key aspect of Agile SW development: the fact that it is a constantly ongoing conversation between the team (that implements) and the person/people that hold the product's vision (typically: product owner):

This notion of product owners coming up with DecreedStories is a profound misunderstanding of the way agile development should work. When we were brainstorming names at Snowbird, I remember Kent suggesting "conversational". This emphasized the fact that the heart of our thinking was of an on-going conversation between customers and developers about how a development project should proceed.

Photo credit: jakuza @ flick

Agile scales, Waterfall doesn't! The myth is busted!

I have, for many years, been working in software development and for about 6 years with Agile Software Development adoption. First as a project manager and later driving the adoption of Agile in the organizations where I've worked.

One clear characteristic has become clear to me over time. You may not hear about it much today, but some years ago the biggest fault people would point to Agile methods was that they did not scale! (despite some work by Alistair Cockburn to prove them wrong).

Today, it is quite clear that Agile does scale, and scales to very large organization like the one where I'm working right now. This organization is adopting Agile at a scale that is probably unprecedented, a colossal scale.

So, the cat is out of the bag. Agile does scale! But during my years in this industry I've started to note another trend: Waterfall-like processes do not scale. Many of the problems that we are now solving with Agile are problems that were originally caused by the ideas developed for larger projects using the waterfall model of software development (I include the V-model in this group).

Take an example: Requirements Engineering. In the waterfall-like models the requirements were collected upfront with considerable efforts to try and be as exhaustive as possible (everybody remembers Boehm's cost model for late-found requirements). This approach basically created a huge inventory of requirements (overproduction anyone?) that would then require large groups of people to manage (in large organization, remember this post is about scaling).

These large groups of requirements would not deliver any value to the end-customers, and in fact many of them would never be implemented because others would be found later. This inevitable late discovery of requirements in turn created even more requirements that needed more people to manage them.

The end result was that large software organizations ended up with large requirements-related departments (analysts, user-specialists, user-research, product management, product concepts, etc.) and relatively small engineering departments. It is no wonder then that companies that followed that model would end up slow and unadapted to the fast changing world of software development (remember IBM?).

At the end, the conclusion can only be that Waterfall does not scale! But Agile does!

I'll be exploring this and other topics regarding Agile at scale in my talk at Agile Eastern Europe that will take place on September 18-19 in Kiev.

If you are there or near by don't miss the conference that promises to have extremely interesting talks!



Photo credit: "It's this big" by bobu @ flickr

Another post in favor of another User Story format

Some time ago I wrote a post supporting the idea of a new template for the User Story. In that post I tried to explain why we should bring the "value" (because) clause to the fore, to make it the most important clause of the User Story template.

Then John Arrowwood commented on that post and made a reasonable argument to stick to the "old" (or traditional) format for User Stories.

This is a second post in defense of the new template, where I try to explain why the "value" clause in the user story should be brought to the fore, to be the most important clause in the User Story. Here's the template I propose:

• In order to "benefit/why/value"
  
• As a "user role"
  
• I want "feature/functionality"
  

Although I don't see any logical flaw in John's argument I don't agree with it. The reason is that in reality I don't see that people understand the ultimate goal of the User Story which is to justify every functionality in terms of what value it brings to the user.

No matter how much we stress the need to have the "value" properly described I don't see anyone (literally) using it. Normally the User Story writer will not understand the real "value" for the user before I do the 5 why analysis on the "feature" so that the person understands the deep reason why that "feature" is valuable to the user.

I also see that people don't spend anytime writing the "value" part of the clause, they tend to use banalities and obvious things. Writing an obvious "value"-clause delivers zero value when communicating the user story. Remember that it is the "unexpected" and "non-obvious" that actually delivers value, because it contains information that was not obvious to the reader.

Here's an example: everybody knows that as a user of Yahoo! e-mail client I want to save emails sent and received because I may want to return to some business-critical information later on, what not everybody knows is that as a Gmail user I prefer not to have a folder structure to archive old e-mails because it's easier to search than to create a folder system for archiving.

If you take as value the "archiving" then you will create a stupid folder structure for people to be able to "store" old e-mails. However if you take the "finding easily a relevant e-mail" as the value people are looking for, you may start to think twice and ask yourself if all users (all over the globe) are really librarians that think in archive-system terms. Guess not.

When you write a User Story start from the value. Always. No Exception. Ever. Start by asking yourself what does the user/customer really want to do. I mean really! (hint: seldom customers are happy with just using your product, they normally want to accomplish something!)

The value part of the User Story should be the non-obvious part of that value, the obvious explanation for a User Story does not deliver any information to the person downstream that will read the story and have to implement it.

I don't expect everybody to understand this, then again I don't expect all companies to create great products.

Learning to be better is not mandatory. Then again, neither is survival...

Nokia goes agile. Will the rest follow?

Ari Jaaksi from maemo.org is blogging about Scrum in Nokia's Linux/Open source projects.
Also, Juha-Markus Aalto (of Object development fame) presented what I would call the "Nokia Agile Requirements Model" (and what Dean Leffingwell calls "Lean, Scalable Requirements Model") at the Object days in Tampere. See the slides here(PDF alert).

With all of this I can only say that Nokia is an exciting place to work at if you are interested in the improvement of our industry!

Now, if only the other Finnish software organization would start following the lead... Yes, you know I mean also the Banks, the Post office, the Tax office and the agonizing slow adopters that are the big consulting firms.

If others start to follow, Finland can become the most exciting place for a software person to work, even better than Silicon Valley!

The value Black Swan (or the killer improvement in software development)

Thinking back to how we value our work, we must recognize that, in software, quantity is not value!

The number of things we do in a sprint does not vary too much, we can consider it a Gaussian (assuming correct and consistent measurement) -- or in Black Swan (TBS) parlance: velocity is a variable from Mediocristan. However, the value of each item does not follow a Gaussian -- in Black Swan parlance value per item is a variable from Extremistan. It is conceivable to think that we can have one item that accounts for 90% of the value we deliver, in fact that is most often the case, but even knowing this we may underestimate (or lack understanding of) the real extreme value of an item.

What does that mean in practice? How does this affect our planning or item selection? Can we in anyway predict the value? (not, according to TBS).... The payoff of a "large value" item would however be much bigger! (altavista vs Google)

In my view, given the current knowledge I have of the software development process/world, I'd say that this assertion means that the most value in a software development process can be obtained by concentrating on selecting those extreme value items that can make the software a success. Current science and practice on this, however, seems to lack any repeatability or reliability when it comes to reliably selecting the "killer" features... Do you have any links to papers/articles about value focus in the requirements selection process? link them up in the comments section.

PS: if you don't know what a Black Swan is, you better read this, or the book.

For a better format for User Stories

Karl follows up on a post by Liz Keogh about the fact that User Stories need to have a different format. The original format was:

• As a <customer role>
• I want <functionality>
• because <customer value explanation>.

Before explaining the new format let me dissect a little bit this old format. Why is it wrong? Clearly the most important thing we want to achieve with the User Story format is "focus on the customer", which for us means focus on the value that customer needs to achieve with the functionality we are developing. So the question is: why is the most important thing the last in the current used user story format?

When teaching User Stories I've started to teach a "thinking model" (credit to Pekka Usva for pushing me to do this). It goes like this: whenever you start writing a user story always, always start by writing the goal (because...) part first. Never, ever write the functionality (I want...) first.

That's why Karl's post clicked: it makes perfect sense to change the format of the User Story. Therefore I hereby join my (blog) voice to his and Liz's and suggest to the community: let's start writing User Stories with the following format:

• In order to <value to achieve>
• as a <customer role>
• I want <some functionality>

What do you think?

End the never ending design discussions: Timeboxed estimation for User Stories, a pattern in Agile adoption

In this post, Jason explains one common problem with the estimation and planning sessions in Agile teams. The problem he describes however, is quite common and has lead to implementation of the Separate Design and Requirements anti-pattern in the traditional software development world.
So I thought I'd write a pattern about a simple and common solution that I've seen and applied to solve that problem.

Timeboxed User Story estimation

Problem:

In the first part of the planning meeting the Scrum team and the Product Owner will review the top User Stories in the backlog to make sure the team and the Product Owner share the same understanding of each User Story.
In the second part of the planning meeting the team will do a short design discussion for each of the items. The discussion should be around what needs to get done for each User Story in order to collect Meaningful Tasks and make the work clear to the whole team.
When the team gets to the definition of meaningful tasks they will get stuck in design discussions often spending too much time in the first one or two stories and then not having a complete decomposition of the other User Stories that would organize the sprint for the team.
Some consequences of this problem are:

• missed tasks in the sprint backlog that end up not being done;
• mis-understanding of the User Story design that leads to inconsistent design decisions;
• unclear definition of the design for a User Story that leads to inconsistent and sometime incompatible architectural patterns in components that need to interact (especially when in the presence of Component Teams);
• Cowboy Coding.
  

Context:

This pattern is applicable in the context of the planning meeting, but can be used also when reviewing a Product Backlog before the planning meeting (see Prepared Product Backlog).
The same pattern can be used to estimate any attribute of a task, such as effort, risk or business value.
In order for this pattern to be applied the team needs to agree to use the same technique. Sometimes some of the team members will refuse to use this practice by stating that it is not "serious" enough or challenging it's credibility. Alternatively people can say that with this method we cannot get "good" design, however it is important to point out that the aim of the planning meeting is not to set the "final" design, but rather the first iteration of the design and especially to achieve consensus with the team on the design patterns and architectural approaches selected. The design, as well as the product, should evolve during the sprint.
In my experience this technique can be successfully used in teams of up to 10 people during the planning meeting.

Solution

The solution is to time-box the estimation for every single User Story, not just for the meeting. Note that the team needs to agree how much time they will spend for each item they accepted for the sprint that is starting. We have used 5 minute time boxes for development stories and 2 minutes for management-related stories.
At the start of the meeting the team agrees on the size of the time box in such a way that it allows to cover all the stories needed within the length of the meeting. For example, if you have 10 stories to cover and 60 minutes meeting that means that after 5 minutes you need to move to the next story (that will give you a 10 minute buffer).
The process for each story is:

1. The facilitator reads the story from the product backlog
   
2. The teams ask questions or define tasks by talking about the design of the implementation
• Example questions are: what modules will this story touch?; do we need help from any other team?; do we already have module X in our version control?; do we have the design for the UI?; will we need to manually test this feature or is it enough to have the automated user acceptance tests?; etc.
  
3. While the team discusses the needed design changes/additions the facilitator writes down the clear tasks (example: refactor class X, change the algorithm for Y, design a view for Z, get support from the DB admin for the DB upgrade in the test environment, etc.)
• For some tasks to be clear, the facilitator may need to ask clarifying questions. See step 2 for examples.
  
4. At 4 minutes (if the timebox is 5 minutes), the facilitator will request everybody to estimate the story (using the Planning Poker method for example).
5. If there is no consensus on the estimation, the discussion continues
• At this time the facilitator can focus the discussion by asking the highest and lowest estimate person: "Why do you think that is the case?"
6. At 5 minutes the facilitator will ask everybody to estimate the story and note down the most popular estimate value
7. The facilitator closes the estimation for this story. Return to step 1.

After the meeting the team has a preliminary list of tasks (that will be improved during the sprint) and a consensus on some basic, yet critical design/architectural decisions.
Even if this method seems "minimalist", our experience is that it works well-enough even for 4 week sprints. Although, due to the size of the tasks it is easier to apply to 2 week sprints.

Resulting context

• The first few times the team will try this technique their discussion may be very superficial, especially if the team has never had design discussions in short spurts before. This is to be expected and the experienced ScrumMaster will keep her ears open for symptoms of this during the daily meetings and call for additional design discussions during the sprint, when needed.
  
• The first few times the team may feel that the process is "artificially" strict. This is a common critique. However the team should stick to the practice for at least 4 sprints before changing it or "customizing" it. Our experience is that people will adapt to the method and, after some sprints, start producing high-quality high-level design already during the short discussions in the planning meeting.
  
• If the first part of the planning meeting was not enough to clarify the aim of each of the stories to be estimated the team may have to discuss the stories from the requirements perspective before they can have a design conversation. In this case the facilitator should stop the discussion and defer the estimation of that task to when the Product Owner has explained well enough the story (this is the Unclear Stories anti-pattern).
  

See Also (patterns and anti-patterns not defined yet):

• User Stories for communication across functions
  
• Meaningful tasks
  
• Component teams (anti-pattern)
• Cowboy coding (anti-pattern)
• Unclear stories (anti-pattern)
• Prepared product backlog
  

Resources

A good practice for the actual estimation is the "planning poker game". Here are some resources:

• A timer for planning poker
• Planning poker explained
• A web-based planning poker for distributed teams
  

Separation of design and requirements may have destroyed more software projects than waterfall...

The people that convinced us to never mix design with requirements may have wrecked more software projects and destroyed more value in the software industry than anyone else!

Just think what happens when you separate requirements from design:

1. You don't validate assumptions to more than a very superficial level (the "I don't care about the technology, just give me the business needs"-level)
   
2. You never evaluate the system's constraints and how those may make the system possible or not (hardware, technology, etc.)
3. You never think of how testable the system will be (as in "is it possible to ever prove that this requirement is implemented?")
   
4. You don't write down the tests that will prove that the system works (until it is too late and you have to hire a gazillion testers to test the whole system using a script that was developed by people that never really participated in developing/defining the requirements)
   
5. You never visualize how the system will be used by real users
6. You probably - I'm being generous here - did not involve the testers/coders/designers in the requirements definition
7. You spend way too much time disconnected from the reality of your business. Remember that our business is software development (as in coding/testing and getting feedback from real users) not requirements management (even though some tool-vendors will make you believe otherwise).
   

Now, having been in the software industry for more than I care to admit I know why we had requirements documents back in the old days. The short story is: we did not know better, the long story is way too long to tell in one blog post and involves some famous academic people and a junior process guy called Winston Royce.

During those requirement writing sessions we tried our best to stay away from the design discussions because we knew that if we did go into design (which we wanted and felt we should do) the requirements would never be ready. So we decided to do the wrong thing because we knew there was no way we would ever get the requirements written and the Requirements Gate passed if we did not do it.

Now that I look back I find it amazing that we ever got the requirements right! Think about it, if you don't validate the assumptions you make but to a superficial level you are saying to yourself "I know that there's no way I can prove that I'm writing the right requirements for my system to succeed". How's that for a smart way to do software development? But that's how we got taught in University (still are today to a great extent), and that's what our process trainers brainwash us to believe in.

This separation of Requirements and Design is, in my view, the single biggest reason for software projects to fail, and it is still happening today!

Define the most important requirements first, then worry about the rest

This is why it is so important to delay writing the low-priority (as in "the ones you are not sure you need") requirements until the last responsible moment (aka "delay commitment"). You should take an appropriate set of requirements that fit in one iteration (using the highest-risk, highest-value heuristic) and develop those. Get a slice of the system working, don't get caught in the useless (and never ending) discussions that do not relate to the requirements being implemented (note that these requirements can be non-functionals also, e.g. performance or security).

Once you have a set of requirements implemented you have validated (or disproved) the most important assumptions about your system because those are the ones that relate to the highest-risk items of your requirements list.

Since you have avoided to think about all the requirements you have not either lost any time discussing requirements that make no sense to implement and therefore should have little (or no) importance when making the major decisions for the system, like architecture, technology, etc.

Another important thing to consider is that when you think about design, you are in effect creating feedback to the requirements cycle. If you separate requirements from design you are also removing all possibility of having early (and needed) feedback to your requirements writing!

If there are some times when I think I did stupid things in the past this is one. Boy, we were dumb back then! (and may are still today...)

Blogged with the Flock Browser