This blog has moved. Go to for the latest posts.

Tuesday, September 24, 2013

Using #NoEstimates when deciding which projects to take on

One of the questions (Part I, Part II) that emerged from the #NoEstimatesQuestions discussion on twitter was: “How do I choose between options or opportunities (without having estimates regarding the cost)?”

The thinking goes: I can spend some time doing A or B, but which should I do first? To answer this question we can look at it from two angles:
  • 1) How much does A or B cost and how long to they take?
  • 2) How much will I learn from doing A or doing B? Which knowledge is more important for me now?
The first point of view assumes that both A and B have similar value (or Return) and that the investment part (the I in ROI) is the most important factor. I happen to think that in software this is not the case. In my experience the R (Return in Return On Investment - ROI) is often more important than the Investment in defining the overall value of a certain decision. In my experience, the second approach will help you make a better decision when selecting between different options. Asking yourself "what is the value I expect to get from option A or option B?" is often a much more insightful discussion regarding the Business Model for your product or service. Here is how I would answer the question: “How do I choose between options of opportunities (without having estimates regarding cost)?”
  • Step 1- Define what you expect from doing A and B. Example: A) If I implement “login in with facebook” I will expect to have more users to access the site due to the simpler registration. B) If I implement “invite a friend” I expect to have more visits to the site and ultimately more people registering than if I choose A.
  • Step 2- Now, break the work down into risk-neutral chunks (i.e. chunks that don’t kill your project if you fail to reach the expected goal). This leads me to goal decomposition: given that I can’t test any of the options by tomorrow, is there something that I can test by tomorrow which would help me choose futher? Can I implement the “invite a friend” feature by tomorrow? Yes: great! lets do that. The reason is simple: by tomorrow I can start to test option B, if I had gone with Option A I would have to wait longer to be able to test my expectation.
  • Step 3- Now it is time to get to work, deliver the smallest/easiest possible implementation of “invite a friend” by tomorrow, and review the impact of that feature. Once this functionality is live you can can either continue with B (if more changes are needed), or ask yourself what are the next options you need to decide on.

How to tackle Large Problems(™)

The example above will, rightfully, called a “simple” problem. So, the next question is: what if I have to decide between 2 Large Problems(tm), which can take several years of development and months of deployment? For example:
I work in the Ministry of Health and I need to decide between developing a large system to manage the health-care records of 10’s of millions of people, or maintain an existing system with potentially huge costs.
In this environment the Return part of the ROI equation is not easily accessible. In fact, one could argue that in this type of projects the I (or Investment/Cost) is the most important factor in making a decision. Most governments do that and spend considerable amounts of time estimating the costs of both options (with terrible consequences). #NoEstimates asks for a completely different approach to these problems (where cost is the key factor in deciding). In fact I would work to completely eliminate the “develop new system from scratch” option from the table. The reason is simple: we are terribly bad at estimating large software projects, so we should avoid them completely. Instead, we must use the flexibility of software to our advantage: start small and slowly grow the team as well as the final technical solution. How I would do it:
  • Step 1- Determine what is the most operationally critical part of the system (e.g. by number of people assigned to work on that part of the system, or by contingency plan size -- don’t spend time estimating criticality, there is plenty information already available). Don’t start by changing this part of the system - it is far too critical (as you just determined). Select any other part of the system that you can change without any long-winded approval processes (random selection is ok, but you probably have a hunch).
  • Step 2- With a small team (say 5-7 people) start a 1 week project to change this part of the system (a new version of the same software, including tests to ensure that the new implementation works as expected). Can’t do it in a week? Select a smaller part of that system component until you can. Don’t bother estimating, just ask the team until most agree it can be done. In parallel start listing, every day, other system components you will need to change next. Use the ideas in Step 1 to prioritize that list. By the end of week #1 you will have the first “changeable system component backlog” (don’t worry about the comprehensiveness of this list, you will update it every constantly).
  • Step 3- By the end of week #1 you should have a small part of the system component re-implemented to the new standards (maybe a new technology? maybe a new privacy law?). Now ask, how many such components do you have to change to complete the project. 10? good, then it will take 10 weeks or so (don’t worry about being right, you will update your projections as you go)
  • Step 4- Take the next part of the system and start another 1 week project. Later you may want to start more than one part of the project by adding one more team. But don’t add all teams at the same time
In this approach we try to achieve the following:
  • Evolving understanding of the system in terms that can be used for data-driven progress projection (a- changeable system components backlog, b- how many system components can we change per week?)
  • An evolutionary change to the system in a way that the system is always functional and the project - even if stopped prematurely - constantly delivers value to the owners of the system (in this case the taxpayer).


I’m sure that there are still quite many questions about the steps and approach I described here. Leave those questions below so that we can continue the dialog.

Labels: , , , , ,

at 13:02
RSS link

Bookmark and Share


  • Regarding how to start large change, from scratch or no there are two main alternatives. You described the first one here, start from system by iterating based on amount of work that can be fullfilled within a week. If I understand correctly the main reason for this is to get rid of the cost/estimate -> decision problem.

    I tend to believe that there is also another way. In this heatlhcare example I would spend a considerable amount of time with the users(customers) and try to nail down functions they need and squeeze them radically to the smallest possible deliverable. Most likely we will end with headache about hw or general architectural challenges. Those will be faced anyway, so better earlier than later. Then I would start to pilot with the nurses and doctors to get easiest and simplest UX and build a modular architecture on top of virtualized solution.

    Either way we should get feedback soon and provide real results for the decision makers.

    By Blogger hena, at September 24, 2013 9:22 PM  

  • @hena
    The reason to start with small changes is *not* to get rid of the cost/decision problem. On the contrary, it is to focus on delivering working software from the start (an Agile principle).

    What you mention about UX may make sense intuitively, but it does not make sense from a software development point of view.

    Here's why: having working software is a technical challenge. One that very often derails large projects. Having UX that works is a communication challenge that can only be solved by having real deliverables that can be tested and iterated.

    So I would take a different approach from what you describe, instead of spending considerable time with users before implementing anything I would bring users into the project (another Agile principle) and establish a cadence of UX validation (every 2 weeks? every 3 weeks?).

    UX becomes, in this case, a key process for each of the 1-week projects instead of an upfront activity.

    In the end, even if the project gets stopped, we will still have valuable software delivered to the paying customers (the tax payers in this case).

    By Blogger Unknown, at September 25, 2013 10:11 AM  

  • If I read correctly between the lines, you are describing a context where a company is doing internal development and is using projects to manage them. Projects can be started without a budget or estimated total cost, and I assume the company has an existing business with some products or services. To contrast the context is not, for example, a new start-up company, a software subcontractor, or a heavily regulated life-and-limb-critical domain. Furthermore, the scope of the situation we are investigating is software development and choosing between software projects (as your examples use features). Please correct me if I'm mistaken.

    In such a context your no estimates model for choosing between two projects is entirely sensible.

    If I may, I would like to expand the situation a little bit.

    What if we are a start-up, have no established business, we are not doing projects and we are investigating our business as a whole. We are sitting on top of some very interesting new technology with countless possible applications. We have one application for our nice technology which is our first product. There is some time and money to expand our business, so we are looking for new applications with possibly new products and markets. Let's say I have twenty cool idea for how the world could use our nice technology. Since we are only a handful of people we really cannot afford to start implementing chunks of work on twenty different ideas.

    I have described a situation that I've been to in my previous life a couple of times. The practical process of deciding which ideas to pursue usually involved a collaborative thought process of figuring out the best guestimate for effort spent (time/money/etc.) and expected gains (revenue/good PR/customer reference/learning/etc.). I like to call this roughly figuring out an ROI, and yes that thought process has more holes than data in it.

    Note a couple of things here. First, the process I describe does not use considerable amounts of time/money in estimating. Second, it does treat estimations as nothing but "today's best guess with our best brains, tomorrow we know more". There is no division of business and product people to developers, we are all happily on the same boat trying to do the best we can. As far as I can see, the dysfunctions the no estimates tries to address really are not present. But we do have to make a probabilistic decision with insufficient data, and we are always short on time, money, resources and information.

    I do not see that your no estimates model is really applicable in this situation. How do you see it? And finally back to Marko's original question: given many business options or opportunities, how do you choose which to pursue without estimating cost?

    Please note that my goal here is not to torpedo no estimates (a lot of which I agree with), merely to explore the contexts in which it makes sense. I think the complexity guys call this bounded applicability, but I digress.

    By Blogger Ari, at September 25, 2013 4:15 PM  

  • +1 hena! :)

    Iterating UX with working software can get expensive fast. Two weeks of a development team's time is several man-months of work. You can do a lot of validation and iteration with real users with much much cheaper methods than writing software. These include functional prototypes, service walkthroughs, wizard of oz, role-play and such.

    Vaidated learning as economically as possible is the key here. The second you come up with a design idea it starts piling up speculation inventory and risk increases. The faster you can validate the design the less risk and less money wasted. If possible, it is better to validate something in two hours with a prototype than in two weeks of coding.

    I dare say software is sometimes a bit of a golden hammer for agile developers. :)

    BTW I just talked about this last week, sorry for pitching my own stuff but you might find my slides elevant:

    By Blogger Ari, at September 25, 2013 4:27 PM  

  • @Ari
    I don't understand the difference you refer to between choosing one project/feature and what you describe.

    Can you list what are the key differences you see between both examples I use and what you describe? What is not applicable in your case?

    By Blogger Unknown, at September 25, 2013 4:29 PM  

  • @Vasco
    The situation is not "how to choose between project A and project B". The situation is "how to choose between projects A, B, C, D, E, ..., or Z while we can maybe work on two at a time".

    By Blogger Ari, at September 25, 2013 4:31 PM  

  • @Ari
    I don't think there is any significant difference between A,B choices and A,B,C,D,E,Z choices.
    Sure it takes more time to think about the impact you expect from each choice on your business model, but it fundamentally is the same process when it comes to cost.

    In any case, when you have multiple choices if none is clearly better (from a value/return) perspective, then what is the point of spending time evaluating them all? Just pick one, invest a little, learn and go.

    By Blogger Unknown, at September 25, 2013 4:46 PM  

  • @Ari

    Regarding iteration of UX:
    Sure, you should not develop a complete product before you test it. But this is a totally different thing from what I described in the post and in my answer to Hena.

    In the post:
    - invest a week, learn, iterate

    In the answer to Hena:
    - Get the users into the project and have them as part of the design/development team.

    None of these answers calls for "large, unrecoverable investment into a UX before you test it"...

    By Blogger Unknown, at September 25, 2013 4:48 PM  

  • @Vasco

    About choosing A/B or A..Z

    The difference is significant. Let me try to explain better. In the situation in your example with projects A and B you implement chunks of work from A and B with one team and one week iterations, and then decide. In my situation you implement twenty chunks of work from A through Z, with one team that means twenty weeks. Consider this from a WIP / Lean point of view, our startup is now busy experimenting with twenty ideas/projects, and not making much progress on any of them. You are trashing. Working on twenty different one-week experiments/iterations in a one-team startup is not feasible.

    About estimating cost

    I am suggesting that you want to spend time evaluating value/cost of your options just as long as that estimation brings value. No more, no less. In my start-up example that time may be hours or days. If I understand correctly, you are suggesting to spend exactly zero time thinking your options through.

    Evaluating value alone makes no sense at all. You may have an idea that you estimate is worth 100MEUR, but takes several calendar years to build. You have three months money in the bank. Oops, you neglected to estimate cost and just bankrupted your company. Maybe it would have been a good idea to look at your options for thirty minutes over coffee and see that you costs are at some level feasible?

    I don't think that anyone investing their own money/livelihood would just blindly pick one idea and start investing in it without considering alternatives & evaluating their potential outcomes.

    A Lean Startup way of looking at it (I think you are familiar with this from Happy Melly):

    A business model canvas represents a business model that should be in balance and make sense, meaning it should be profitable. That is why the canvas has both revenue streams _and_ cost structure. Sure it is all assumptions in the beginning, that is why you form hypothesis and experiments and start developing your model. But you sure as hell would not bother starting even a single experiment without a plausible cost/revenue structure even in theory.

    By Blogger Ari, at September 25, 2013 5:35 PM  

  • @Vasco regarding UX

    In the post:
    - invest a week, learn, iterate

    In the answer to Hena:
    - Get the users into the project and have them as part of the design/development team.

    My suggestion:
    - invest an hour or a day, co-design and build prototype with users, evaluate, fix, repeat
    - when prototype works, spend one week writing software, evaluate & iterate

    By Blogger Ari, at September 25, 2013 5:39 PM  

  • Ari, you say: "our startup is now busy experimenting with twenty ideas/projects, and not making much progress on any of them. You are trashing."
    Why? I'm not sure that is true. If a team doesn't make progress, that's learning. Stop! Especially if they are trashing :-)
    I'd say it's *value* in validating *all* ideas. As an entrepreneur I'd surely want to validate all my ideas. And if all ideas are great, I'd probably won't have any problem raising more money from e.g. investors. And then I can build more teams etc. And if an idea is not worth raising money for, then that's probably a sign you shouldn't do it or prioritize that idea.
    If you *must* choose, well I'd say you're not in a very great situation to start with. Then maybe you should start on focusing how to solve that. If that's not solvable, well then I think you're suggestion in your comment is a great start :-)

    By Blogger Henrik Ebbeskog, at July 18, 2014 6:39 PM  

Post a Comment

<< Home

(c) All rights reserved