This blog has moved. Go to SoftwareDevelopmentToday.com for the latest posts.

Wednesday, December 18, 2013

The most valuable question for your software project


Every time a software project is started, a dance starts: the dance of project approval. Decision makers and project delivery team take different positions at a table, some ask questions, others do their best to respond, given that these questions are typically about the future. Some of the questions are so much into the future that it would be foolish to pretend we know the answer. Yet we do.

This was the main lesson I learned in my life as a project manager: you have to learn how to dance.

How to dance...

In the dance-like meetings there are 3 types of people:

  • A) Those that pretend to know the future
  • B) Those that know they can't know the future
  • C) And those that don't want to know the future
The question below is only valid for the people in category B). The other two categories are beyond help from me or anyone else.

What is the likelihood this project will fail?

When a project starts, the best question anyone can ask about the project is this:
The most important question: What is the likelihood this project will fail? Why is this question important?

  • First, if you don't have an answer to this question you are either in the first or third category of those listed above, and there's nothing I (or anyone else) can do to help you.
  • Second, by discussing the possibility of failure we are actively looking for possible causes of problems for the project (rule 1 of risk management).
  • Third, having an actual conversation about the failure probability of a project will give you an indication of the "value at risk" for the project.

For example: let's say the project budget is about 100K dollar, and to make it a worthwhile investment we would need to make it between 120K and 90K in total cost (i.e. if it is likely it will cost more than 120K you should not do it). If the answer to the most important question is: 50/50, then you know that the project is not worth doing. It has very little chance of delivering the necessary value to justify it's existence, in fact: it is likely it will be above 100K and up to as much as 200K in cost.
However, if the answer is that we are 80% confident we will deliver the project on timecost, then the project is worth pursuing (although with some strict scope control in place) because the expected worst case scenario would be a total cost of 120K dollars (the 20% margin implicit in the confidence above). In practice an 80% confidence means you expect the project to cost up to 120K (100K + 20%) in the *worst* case.

In a later post I'll explain how we can answer this most important question using the #NoEstimates approach. For now, let me know what you think of the question and the concept of "value at risk" above.

Definition of Value-at-risk [PDF]: "value-at-risk summarizes the expected maximum loss (or worst loss) over a target horizon within a given confidence interval"

Note: A special thanks to @galleman for the idea of the 3 categories of people from which I derived the 3 categories I present in this post.

Photo credit: Steven Depolo @ flick

Labels: , , , , , , , ,

Posted by at 07:30
RSS link

Bookmark and Share

2 Comments:

  • I don't quite understand your use of probabilities here. It's important to note that "probability that project will fail" is NOT AT ALL the same as "probability of going over budget".

    When you write "probability of a project failing", I interpret this as a simplification to a binary event. We can then use classic expected value decision theory (which may or may not apply well to software projects in the real world). In this framework, if a project costs 100K, and I'm 50% sure I will deliver, then it's worth doing if it results in a revenue of about 200K or more.

    Of course that is a gross over-simplification, in particular it assumes that things work in the long run; if you don't have enough cash, though, the short run may kill you first.

    Your post focuses on cost only, not taking revenue into account. You go on to talk of "probability of going over budget", but here too the math doesn't seem right to me.

    You can't get the 120K and 200K worst-case numbers just from your assumed probabilities of .8 and .5; not unless you bring in more assumptions, e.g. a normal distribution for costs.

    I strongly doubt that the normality assumption holds for software project costs. The probability distributions for software project costs seem to be long-tailed, so even at p=80% of meeting the budget, your worst case could be well over 120K.

    There are additional problems with trying to estimate probabilities.

    One problem is that different participants in the project might well have wildly different answers to your Most Important Question.

    Another problem is that they might all be poorly calibrated; most people are. People are so poorly calibrated that I doubt an 80% confidence actually means anything like 80% chance.

    You can improve your calibration by repeated practice at making probability assessments and tracking (for example) a Brier score over time. (This is the theme of my talk "The Art of Being Wrong" that I delivered at a half-dozen conferences this year.)

    However, improvement requires frequent and rapid feedback, and the typical duration of a software project is way too long for such feedback to be useful if you're only going to have it at the whole project level.

    I'm pretty sure it is An Important Question, and one that we would do well to ask before every project. I rather doubt that it is The Most Important Question (if there is even such a question).

    By Blogger Unknown, at December 18, 2013 1:01 PM  

  • Vasco, The probability of failure is not actually calculable (if that's a word). What you want to calculate is the probability of project success. The inverse is not an arithmetic subtraction, since the probebility is the cummualtive probability distirbution. There uus a large body of rsearch and practice in the "probability of program success" (PoPS) domain https://acc.dau.mil/pops

    When using the 3 classes of uncertainty. But the ones you're using are not the meaning or intent of the ones I've stated and use on our programs

    (1) Didn't know - didn't do their homework
    (2) Couldn't know - couldn't afford or the information wasn't available at the time the decision was needed
    (3) Don't want to know - this is a political issue.

    Does not mean the uncertainty information is not knowable.

    Your (B) is technically not correct. The future is "knowable" in most cases for our software development domain. In other domains - politics, international affairs and other "chaotic processes" the future may not be knowable. But were I work it is always "knowable," we may not be able to avoid to "know." but that doesn't remove the knowability. The statement you've used is not consistent with Bayesian statistics. Confusing the two removes your decision options. You may nit "choose" to seek the future outcomes. That's different than they are not available for your understanding.

    So to follow Laurent, it's the probability of "success" that we're after. When that probability is unacceptable, specific actions are needed - it
    s the decision making processes your after using probability and statistics. This is also the basis of Bayesian statistics.

    When you use the 50/50 "chance" you need to state the Probability Distribution Function and if the 50/50 is the Most Likely or the Mean. This is a common misunderstanding. We want PDF before stating any value along the x-axis of the PDF.

    As well you need both the cost and time confidence. Being on cost and late or being late and on cost may be unacceptable, depending on the project.

    By Blogger Glen Alleman, at December 18, 2013 8:57 PM  

Post a Comment

<< Home

 
(c) All rights reserved