But aren't Story Points much better at predicting the release date and scope delivered than the method you propose?
Despite all of the other reasons not to use Story Points, I decided to tackle this question specifically. And the results are in! Story points are less accurate when predicting the release date and scope delivered, than just counting the number of stories (or items) delivered per iteration! This seems counter-intuitive because we have less "detail" when we merely count the number of stories delivered. Many asked me:
But if you don't know the size of the work how can you predict when it is going to be done?
In God we trust, all others must bring dataBefore speculating, let's look at the data! The case I want to present is: a long project (24 iterations) for which we collected both Story Points and number of items completed per iteration. I had one question with two sub questions in mind:
Which metric (SP's or # of items) was a more accurate predictor for the output of the whole project?
a) When we calculated based on the averages for the first 3 iterations
b) When we calculated based on the averages for the first 5(!) iterations
Why this question is important is that, if we can predict with high accuracy the output of a project based on the first 3-5 sprints, we have a good case to stop doing up-front estimation altogether! After all, investing 3-10 weeks in actual development delivers much more information about the product then spending 2-4 weeks in Requirements/Architecture/Design discussions (not to mention that it bores people out of their minds!)
So what were the results? First of all a disclaimer: this is data from one single project; we do need more data to make a better case for not estimating at all! See below for more on how to contribute data to this project! The results are in, and counting the number of items is a better predictor than Story Points based estimations!
In this project, the answer to the question: "which metric is more accurate when compared to the actual output of the project?" is: Counting the number of stories/items delivered at the end of each iteration is a better predictor for the output of a project than estimating based on Story Points delivered!
Final note, how to contribute data to this studyThe case I presented above is based on one single project. We currently have data for more than 20 projects and 14 different teams; but we need more data to investigate the claims I make here and in the previous post.
I call upon the community to share the data they have. I have made my contribution by sharing the data I have collected over the last years in a world-accessible spread-sheet that you can see and download here.
Please share the data for your projects in a google-doc or similar world-accessible spreadsheet and leave a comment below with the link to the data. For us to learn more about how to better predict project outcomes we need to be able to look at a large data set. Only then we will be able to either verify or destroy the claim that Story Points are useful for our projects. Thank you all in advance for your contribution! Photo credit: NASA's Marshall Space Flight Center @ flickr