How to accurately forecast a software team's performance

Touched off by a my recent article What is my software team's capacity, I thought I'd briefly share how I do probabilistic forecasting for software teams.

Throughput and Variance

Ok, the two biggest numbers to know are the team's throughput, which is the rate at which they complete work, and the variation within that.

Have a list of the throughput over the last few sprints or whatever increment you operate in.

I'd tell you to go ahead and eliminate the variation, but that's a little outside of the scope of this article.

Breakdown Rate

The next thing you need to measure is how the team tends to break work down. Every team does this a little differently, but each team tends to do this very consistently. You just need to observe or go back through the tracking tool of choice.

What you want to see is something like:

• 8 pt stories turn into 5 items 80% of the time

Completion Point

Know what is funny? How everyone is sure there is too much work, but nobody knows where the finish line actually is.

Find it.

For most teams, there is an infinite backlog in some sort of condition. Find out where the last essential item is, and then count how far down it is. You're going to model the team executing on all the items between now and that completion point.

If you ever wanted to motivate someone to cut scope, ask them to count how many items there are until it's done.

Sure, if the priorities change, so will your results. That's life. The good news is, only one data point will change.

Model the Execution

Now for the part where I expect my readership will collapse.

You're going to build and run a monte-carlo simulation of your team with the data you collected.

I'm not going to get into exactly how to build one of these. You can ask one of your friendly neighborhood PMP folks to do this. It was in their exam.

Oh, they don't know how to do it either?

Here's an example from How to Measure Anything.

Basic Model

Your very basic model simply plugs in the throughput, and the full item count. Now, this is going to be pretty good, but we can do better.

Variations and Anomalies

Remember that variation you captured earlier? That variation represents surprises. You can probably find lots of them as well as other events that derail the team.

You can begin to add to the model these "quirks."

Vacation isn't a quirk. A re-org every 6 months is.

You basically will represent what these quirks are as a probability and an impact to throughput. It can be tempting to go crazy adding in all kinds of things here, but until you do this a lot, you won't have any sense of if your model is better or worse than before.

Results

The simulation will run for a thousand times, each one showing how long it will take for the team to complete that number of items. You can then create a histogram of that and see what percentage of the time they finish by a certain date.

Now you can say with quite a bit of authority of when they'll be done with what chance they have.

Challenges

This style of forecasting works really well in a lot of cases, but for many teams and groups there are immediate challenges to deal with.

Dependencies

Yep, the not-so-silent killer. You'll need to model this behavior. What you do is look at how the work is arranged by dependency. If you're lucky you'll be able to say something like 1-in-3 items has a dependency.

Now you need to look at what happens with those dependencies. Some will go smoothly, some will not. You need to get counts for how often this happens, and the impact. If you're lazy (Don't be lazy), each dependency has a chance to delay work by 1.5 times the estimate.

Don't be lazy.

Bugs, Defects, and Outages

Want to see the cost of poor quality? Model how much time they lose to this. Again, you just observe how often they create bugs and how often they close them. When you add this to the model you'll be able to see just how impactful improving quality really is.

Multiple Teams!

This isn't as hard as you think. These are new additions to the model. Each simulation will simulate how each team contributes to the success.

Priority and Scope Changes

I kind of covered this before, but this is normal. In fact, you'll likely advocate for doing exactly this once you get your initial numbers in.

The truth is that many teams don't radically change how effective they are, so their throughput and variance will usually stay the same. You just need to update the backlog depth numbers as you go.