Why do we keep on giving up any control over our project? It would be so easy to keep it...

I get very often shocked by the comments that I hear from supposedly very smart people. Today was no exception. I heard the following comment:

There is content we don't want to timebox, therefore there's no need to link it to any timeline...

-- Quote from someone that has direct responsibility over scope in a project (my emphasis)

This quote betrays a complete misunderstanding of the dynamics of software development, and a complete (albeit unintentional) ignorance of the market forces we need to deal with.

Here's my point: by scope-boxing a particular Feature what we are doing is effectively giving up control of it's size. Once the team is given the Feature they will work on it until it is "perfect", that means we don't have a clue when it will be done and therefore the schedule for that feature is completely unpredictable! Yes, sure we will stop development on it at some point, but will it happen before it is too late?

The advantage of timeboxing the content for a Feature (Feature must fit in a Sprint) is that we have a clear deadline at which point we evaluate if the feature is ready to go to the market! Without this constraint the team is left "alone" to decide when the feature is ready. But the team is the wrong actor to decide when a feature is ready! The product owner should be doing that work based on market intelligence, user knowledge, etc.

By scope boxing (as opposed to timeboxing) our features we are effectively giving up control of our projects!

Why is it so hard to understand this point? Where is my reasoning missing clarity?

Can you help?

Photo credit: danielygo @ flickr

Agile scales, Waterfall doesn't! The myth is busted!

I have, for many years, been working in software development and for about 6 years with Agile Software Development adoption. First as a project manager and later driving the adoption of Agile in the organizations where I've worked.

One clear characteristic has become clear to me over time. You may not hear about it much today, but some years ago the biggest fault people would point to Agile methods was that they did not scale! (despite some work by Alistair Cockburn to prove them wrong).

Today, it is quite clear that Agile does scale, and scales to very large organization like the one where I'm working right now. This organization is adopting Agile at a scale that is probably unprecedented, a colossal scale.

So, the cat is out of the bag. Agile does scale! But during my years in this industry I've started to note another trend: Waterfall-like processes do not scale. Many of the problems that we are now solving with Agile are problems that were originally caused by the ideas developed for larger projects using the waterfall model of software development (I include the V-model in this group).

Take an example: Requirements Engineering. In the waterfall-like models the requirements were collected upfront with considerable efforts to try and be as exhaustive as possible (everybody remembers Boehm's cost model for late-found requirements). This approach basically created a huge inventory of requirements (overproduction anyone?) that would then require large groups of people to manage (in large organization, remember this post is about scaling).

These large groups of requirements would not deliver any value to the end-customers, and in fact many of them would never be implemented because others would be found later. This inevitable late discovery of requirements in turn created even more requirements that needed more people to manage them.

The end result was that large software organizations ended up with large requirements-related departments (analysts, user-specialists, user-research, product management, product concepts, etc.) and relatively small engineering departments. It is no wonder then that companies that followed that model would end up slow and unadapted to the fast changing world of software development (remember IBM?).

At the end, the conclusion can only be that Waterfall does not scale! But Agile does!

I'll be exploring this and other topics regarding Agile at scale in my talk at Agile Eastern Europe that will take place on September 18-19 in Kiev.

If you are there or near by don't miss the conference that promises to have extremely interesting talks!



Photo credit: "It's this big" by bobu @ flickr

The value Black Swan (or the killer improvement in software development)

Thinking back to how we value our work, we must recognize that, in software, quantity is not value!

The number of things we do in a sprint does not vary too much, we can consider it a Gaussian (assuming correct and consistent measurement) -- or in Black Swan (TBS) parlance: velocity is a variable from Mediocristan. However, the value of each item does not follow a Gaussian -- in Black Swan parlance value per item is a variable from Extremistan. It is conceivable to think that we can have one item that accounts for 90% of the value we deliver, in fact that is most often the case, but even knowing this we may underestimate (or lack understanding of) the real extreme value of an item.

What does that mean in practice? How does this affect our planning or item selection? Can we in anyway predict the value? (not, according to TBS).... The payoff of a "large value" item would however be much bigger! (altavista vs Google)

In my view, given the current knowledge I have of the software development process/world, I'd say that this assertion means that the most value in a software development process can be obtained by concentrating on selecting those extreme value items that can make the software a success. Current science and practice on this, however, seems to lack any repeatability or reliability when it comes to reliably selecting the "killer" features... Do you have any links to papers/articles about value focus in the requirements selection process? link them up in the comments section.

PS: if you don't know what a Black Swan is, you better read this, or the book.

Separation of design and requirements may have destroyed more software projects than waterfall...

The people that convinced us to never mix design with requirements may have wrecked more software projects and destroyed more value in the software industry than anyone else!

Just think what happens when you separate requirements from design:

1. You don't validate assumptions to more than a very superficial level (the "I don't care about the technology, just give me the business needs"-level)
   
2. You never evaluate the system's constraints and how those may make the system possible or not (hardware, technology, etc.)
3. You never think of how testable the system will be (as in "is it possible to ever prove that this requirement is implemented?")
   
4. You don't write down the tests that will prove that the system works (until it is too late and you have to hire a gazillion testers to test the whole system using a script that was developed by people that never really participated in developing/defining the requirements)
   
5. You never visualize how the system will be used by real users
6. You probably - I'm being generous here - did not involve the testers/coders/designers in the requirements definition
7. You spend way too much time disconnected from the reality of your business. Remember that our business is software development (as in coding/testing and getting feedback from real users) not requirements management (even though some tool-vendors will make you believe otherwise).
   

Now, having been in the software industry for more than I care to admit I know why we had requirements documents back in the old days. The short story is: we did not know better, the long story is way too long to tell in one blog post and involves some famous academic people and a junior process guy called Winston Royce.

During those requirement writing sessions we tried our best to stay away from the design discussions because we knew that if we did go into design (which we wanted and felt we should do) the requirements would never be ready. So we decided to do the wrong thing because we knew there was no way we would ever get the requirements written and the Requirements Gate passed if we did not do it.

Now that I look back I find it amazing that we ever got the requirements right! Think about it, if you don't validate the assumptions you make but to a superficial level you are saying to yourself "I know that there's no way I can prove that I'm writing the right requirements for my system to succeed". How's that for a smart way to do software development? But that's how we got taught in University (still are today to a great extent), and that's what our process trainers brainwash us to believe in.

This separation of Requirements and Design is, in my view, the single biggest reason for software projects to fail, and it is still happening today!

Define the most important requirements first, then worry about the rest

This is why it is so important to delay writing the low-priority (as in "the ones you are not sure you need") requirements until the last responsible moment (aka "delay commitment"). You should take an appropriate set of requirements that fit in one iteration (using the highest-risk, highest-value heuristic) and develop those. Get a slice of the system working, don't get caught in the useless (and never ending) discussions that do not relate to the requirements being implemented (note that these requirements can be non-functionals also, e.g. performance or security).

Once you have a set of requirements implemented you have validated (or disproved) the most important assumptions about your system because those are the ones that relate to the highest-risk items of your requirements list.

Since you have avoided to think about all the requirements you have not either lost any time discussing requirements that make no sense to implement and therefore should have little (or no) importance when making the major decisions for the system, like architecture, technology, etc.

Another important thing to consider is that when you think about design, you are in effect creating feedback to the requirements cycle. If you separate requirements from design you are also removing all possibility of having early (and needed) feedback to your requirements writing!

If there are some times when I think I did stupid things in the past this is one. Boy, we were dumb back then! (and may are still today...)

Blogged with the Flock Browser