Is estimation a scam? Can a non-expert estimate as well as an expert? Help me investigate these questions!

How do you prove that estimation is a scam? Well, it's not easy, especially if luminaries in the Agile community defend that you can honestly estimate a project where you don't have a team available and only a fuzzy understanding of Requirements or Technology involved, or both!

The fact is that many projects are indeed estimated before a team is even set up, but that is ok because we are armed with "expert" estimates! So I decided to test that particular aspect of the estimation "game": can experts estimate better than non-experts a software project?

To be able to answer this question I have to design an experiment. It should avoid bias, and focus only on the estimation process and it's outcome.

It should focus on the usefulness of an estimate for a particular project go/no go decision: accuracy of estimation. The parameters known should be a description of the problem (the problem has to be relatively common, or estimation would not be possible anyway) and the technology choices made for that particular project. And the resulting actual length of the project must be independent of the estimates.

The experiment

So here's my first try at defining the experiment:

• Collect information on a project that has already ended (i.e. the length can no longer be affected)
• Ask 10 people with expertise in that area (problem and technology) to estimate the length of the project given a team size
• Ask 10 people without expertise in that area (problem and/or technology) but with expertise in software development in general to estimate the length of the project given a team size

Here's my expectation (or Hypothesis 1): I believe that the accuracy of estimation will not correlate with the expertise in the technology and problem domains. If I am correct, this will invalidate the concept that expert estimates are somehow "better" (more accurate in the context of this experiment). The obvious conclusion should then be that the accuracy of estimates in this type of context (very common when bidding for a project) is at least significantly affected by other variables rather than expertise. If I am not correct, and the experts can indeed deliver reliably better estimates than the non-experts this should mean that expertise is an advantage when bidding in this type of projects.

Hypothesis 2 focuses on the usefulness of this type of estimates, i.e. their accuracy and therefore relevance for go/no go decisions. In this case I don't have a clear bad/good line between different deviations, I'm just exploring the actual deviation and hopefully will run other experiments later on to explore the impact/meaningfulness of these deviations.

Now I need your help

I need you to help me design/improve this experiment, so that we can continue to further the investigation into Estimates and its alternative #NoEstimates. Help me out by answering these questions:

• Can you identify in the experiment description bias for or against expert estimation?
• Do you have other Hypothesis that we could test with a similar experiment design?

Let me know what you think about this experiment. Once the experiment has been reviewed by enough people and we have improved its design enough I'll start preparing the data collection. For now help me fine-tune this experiment.
Photo credit: CC BY NC ND by Alvin K @ flickr

Learning from the “Real World”, inventory in Product Development

In 2000 I was frustrated with the problems I saw in the Software industry, so I decided to study Lean (or Toyota Production System) to see if I could learn something to apply in my projects. As I studied Lean/TPS I discovered a concept that clicked. The concept was Inventory. Inventory is bad (sometimes necessary, but bad). Put simply: Inventory represents money you already spent but has not yet been converted into a sale.

Later, when working on RandomManager.com (my super-cool T-shirt business for intelligent people like you!), I applied what I had learned about Inventory. All t-shirts we print are bought after customers purchase them (positive cash-flow) and shipped immediately after printing (1-2 days typically). So my Inventory at RandomManager.com is small (1-2 days) and only bought after I got paid (inventory is incurred *after* the sale is made). This allows me to focus on what matters: the order I just got, instead of stressing about being able to pay the server bill. In financial terms: it allows me to have a positive cash flow (money in the bank at all times).

This approach alone makes RandomManager.com even possible. If I had to buy inventory in advance I would have a lot money invested in it but no revenues: my cash flow would be negative which means I’d be financing *future* sales without knowing if they would happen. In fact, I would be making a bet on future events - I don’t like that as you probably know from my #NoEstimates posts.

So, I think I can understand why Inventory is *bad* for my t-shirt Startup RandomManager.com, but what is Inventory for Product Development (including software)? And how should we manage it? And what are the consequences of inventory in Product Development? And how can we avoid inventory in Product Development? Read on...

What is inventory in Product Development?

Today I’m going to focus on describing what I think is Inventory in Product Development (including software).

This just hit me: Software inventory is *not* unreleased code. Have to find time to write about this tomorrow... #NoEstimates #Lean

— Vasco Duarte (@duarte_vasco) July 18, 2013

As I published my insight on twitter, Nik Silver linked me to an interesting post where he describes 3 possible definitions of Inventory, let’s explore those different views:

Inventory as unreleased code

Nik pointed to a post that argues for unreleased code as Inventory. Although I understand the argument (and agree with it), I don’t think code is the inventory we have in Product Development. For one, code is not present in all Product Development environments so it does not fit the context I’m trying to describe, but most importantly code is a product of something that happened before: a requirement was created. Just like in my T-Shirt Startup the inventory is not the printed t-shirt, but rather *all* of the money invested into products that are not yet sold. At best we could argue that Requirements + Unreleased Code are Inventory, but that would not cover all Product Development environments.

Inventory as yet-to-be-delivered requirements

The second hypothesis that Nik considers in his post is that Undelivered Requirements are Inventory. Again, I do agree that they can be considered inventory, but where do requirements come from? Is that the complete investment-before-purchase? No. There’s something else that causes Requirements to be written...

Inventory in Product Development is...

My view is that Inventory in Product Development is - just like in RandomManager.com - everything that represents investment before a sale is made or a customer is served. In Product Development there is something that happens way, way before Requirements or even Code are written that represents (and leads to) investment that must be made before any customers are served.

My theory is that invalidated Business Models factors (including hypothesis on customer segments, problems to be solved, distribution chains, marketing needs, etc.) are the real inventory in Product Development (including software).

I’ll try to make this clear with an example. In RandomManager.com if we wanted to provide Same Day Shipping (a service concept) for our t-shirts we would need to have at least 1 t-shirt of each size times the number of colors times sizes available plus a t-shirt printer ready to run (also an investment). Why would we do Same Day Shipping? Perhaps we hypothesized that it would either drive more sales, customer satisfaction, higher prices or all of the above. This is a Business Model hypothesis, made before one requirement is written or a line of code is created. This is the real inventory: something that represents the beginning of an investment without a sale being made.

In Product Development Inventory is the start of the chain of events that will lead to investment that has no relation to actual (money-in-the-bank) sales. In short, Inventory is any investment that can make your business fail. Code, requirements untested code, unreleased code are way down the chain of events that was started on day Zero when you first thought about the Business Model for your business. (Like an Estimate, for example - sorry, could not resist ;)

Validating your business model

This post is large as it is, so I will leave this to a later post (hint: there’s many approaches that help you reach this goal), but for now: what do you think? Do you identify with the description above? Do you see this happening at your work? How would you solve the Inventory problem?

Photo Credit: images_of_money @ flickr

The #NoEstimates How To

You’ve probably seen a long, recursive discussion on twitter about #NoEstimates.

Some of you, like William Gill may be even interested in a more concrete description that you can apply in your own projects.

So here it is. This is my first attempt at a #NoEstimates How To, first presented at Turku Agile Day 2013, the coziest conference in the Agile circuit.

Before you continue, a disclaimer: #NoEstimates is not a one-size fits all approach. I have my own approach (see below) based on my own experience, but there are other views. Follow Woody and Neil for different approaches to the #NoEstimates ideas.

Here it is: the what and why of #NoEstimates...

Rule 1 - Focus on value, manage the risk

The biggest concern in my approach to #NoEstimates is Value. By value I mean “concrete functionality that we can test in the field”. My approach to #NoEstimates is to create conversations around what matters for our customers (like Product Vision) and prioritize that work around one single parameter: Value. If you don’t know what constitutes value in your context you should do small experiments (safe-to-fail experiments) that are designed to put you on the right track to define what is valuable in your context.

After you sorted out what is most valuable the next step is to manage risk. In Software, risk management is best done by breaking down your work into small enough chunks that you can implement and test (functionally and with users) quickly. These chunks I call “risk-neutral” chunks of work, i.e. if you fail to deliver those in time, your project will not suffer a major set-back. As an example of “risk-neutral” I use the rule of thumb: “one person can complete this workin one day”. Why? Because if you are wrong about that assessment, you will know about it already tomorrow! No need to wait for a long time before you know you are late/developing the wrong thing.

Rule 2 - Discover the product, measure the throughput

In order to define and select the right work-items (for example, the right User Stories) we need to create an environment where discovery is encouraged. In the Agile community several people have developed and written about concrete methods to “discover” the product (BDD or Impact Mapping being but 2 examples). The important aspect in the process of Discovery is that you define the product from your Vision to the Goals and develop the backlog as you go. The alternative (defended by Story Point enthusiasts) requires you to write down the whole Backlog before you can estimate the project. I don’t believe that works and advise people not to do that. Use Story Mapping, or BDD from a Vision and develop your product as you go. Set the target for your MVP and start measuring throughput.

The second aspect of this rule is that you don’t focus on estimating when the development will be finished, instead you let the rate of development (Throughput) inform your view on when the MVP will be ready. Then you skillfully manage the scope (after all you do have a Vision and a set of Goals for the product, right?) to meet the relevant market release deadline. In many products the market dates are set by the market, not you. #NoEstimates accepts that. For example: consumer products in China need to be launched before Chinese New Year’s.

Throughput becomes your GPS-navigation system that helps you manage the content and the Product Vision becomes your map that helps you deliver a coherent, beautiful product.

Rule 3 - Use the data you have, measure continuously

Finally, once you have set-up the system of development you can then continuously measure where you are (how many features left to be done? How many - on average - do we do per week?) and adjust (reassess MVP and backlog and continuously update it to reflect accurately your Vision given the time you have until release).

Share your experiences, read more

I hope this helps you to get on the way to experiment with #NoEstimates. Please share your thoughts, experiences and questions below.

You can read more about #NoEstimates in the following blog posts:

• Story Points considered harmful, why the future of estimation is really in our past
• A better way to predict project release date
• Focus on what matters, a #NoEstimates advocacy post

I am not the only one writing about #NoEstimates, you can find Neil's posts on this single page. Woody also has quite many posts here.

Happy reading!