Enter your search text in the box above

Select a result to preview

Why DORA Metrics Are Bad

#article #metrics
Cover image for Why DORA Metrics Are Bad

Ever since the book Accelerate came out, organizations flocked to set up their DORA metrics, but this left countless leaders and organizations wondering, "What now," and "Are DORA metrics even useful?"

The bottom line is DORA is still useful, but only if you know what you're doing. They're just numbers, not magic.

What Are DORA Metrics?

DORA metrics come from the Google DevOps Report and research by Dr. Nicole Forsgren. There are four metrics that are used and the research used those metrics to benchmark organizations.

The four metrics are:

While officially no description of how to measure these was provided, they offer organizations and leaders hope that tracking these will lead them to becoming a high-performing group.

Why Do Teams Game DORA Metrics?

Gaming metrics is nothing new and DORA metrics aren't unique in that regard. This behavior will happen when folks believe that improving a metric will help them because their managers or bosses expect it.

For DORA, this is most likely to mean that the goal posts for what it means to deploy, ship, fix an issue, and what counts as an issue will become increasingly favorable to the teams who wish to look good.

Need an example? Deployment frequency shifts away from code in production to code completed by the team. Mean time to recovery shifts from a verified fix to a plan for a fix. Change failure rate shifts from any unexpected incident to only certain types of incidents.

Admittedly, every group can define the boundaries of these metrics, but each of these compromises allows teams to look better and better while not addressing any underlying issues that lead to actual improvement or high-performance.

Are DORA Metrics Still Relevant?

The reality is that DORA is still useful. Though it all depends on how you use it. One of the best aspects of DORA metrics is that they are balanced and trying to blindly make one look good will likely show up in another looking worse. This balancing force helps keep everything in check. Many metrics leaders use lack any balance, which is one of the biggest mistakes they make.

Also, the research that led to DORA's popularity is sound. The part people struggle with is that the work it takes to go from where you are to higher performing isn't a simple checklist. The work of improvement is much harder than folks realize, and all DORA can do is show you where you stand but not where to go next.

Consider a team leveraging a lot of AI coding assistance. Are they becoming high-performing now that they produce 10x the output? Well, with DORA we could weigh the fact that they are producing lots of code, but is the "ask-to-get" of lead time any better? Is the code even making it to production? Is it slop that is causing failure that is harder to fix? DORA can help leaders see these challenges.

DORA Vs SPACE Vs Flow Metrics

Should you use DORA or the more recent set of SPACE metrics, or flow metrics? This question is a bit like worrying about which of several great restaurants to eat at. These are sets of metrics. They do not show you how to improve, lead, or what actions to take. They will, however, provide a little transparency to key aspects of how things work.

The challenge with all sets of metrics is that you have to know what they mean, what they don't, and what would prompt any action. Here's an example using SPACE: Your Satisfaction is showing an upward trend, but your Activity dropped while your Collaboration is flat. What would you do with this?

The challenge that all metrics frameworks present is that just having the metric is never enough, and without developing that extra meaning they're just numbers moving.

Having said that, I'll provide a concrete recommendation. Begin with the area you are most comfortable in. Flow metrics (Lead time, cycle time, throughput, etc.) work well on a small process or set of processes. DORA works well across numerous teams or an organization, and SPACE adds layers that many leaders consider to be intangible. So, if you're comfortable working at a team level or specific processes within team work, begin with flow metrics. If you're more comfortable one level higher, where you need to see how many teams or an org are doing, begin with DORA. I will recommend you keep SPACE off the table for one reason. Most folks aren't good with this stuff and SPACE has you begin to leverage lots of new proxy metrics for hard-to-understand things like Satisfaction. These can be very difficult to wrangle without gaming or to take any action on. DORA is much simpler, and a better starting point.

Framework Works best at Start here if
Flow metrics (lead time, cycle time, throughput) A single team or a specific process You're comfortable at the team level and want fast feedback on process changes
DORA Many teams or a whole organization You need to see how the org is doing operationally, one level up from the work
SPACE Org-wide, adds proxies for intangibles like satisfaction You're already fluent with metrics. Most leaders should hold off since the proxies are easy to game and hard to act on

What Should You Use Instead?

It should come as no surprise that I sit in the camp of saying that you use the metrics that assist you in making decisions and taking action. DORA can do that, but most people lack the skillset needed.

So if you're in the position of wanting metrics but experiencing doubt and uncertainty you can try out Signal Mapping instead. Signal Mapping is a process that will help you map key decisions, actions, or questions to observable data so that you become a data-informed leader. It's a practice that you can do privately to develop your data-driven leadership and it grows your competency with metrics so that things like DORA become obvious and intuitive.

If you are already using DORA and want to improve upon what it's showing, you can do a few things:

With any of these I highly recommend balancing your metrics so that you don't invite problems.

Frequently Asked Questions

What Are the Four DORA Metrics?

DORA metrics come from the Google DevOps Report and research by Dr. Nicole Forsgren. There are four metrics that are used and the research used those metrics to benchmark organizations.

The four metrics are:

While officially no description of how to measure these was provided, they offer organizations and leaders hope that tracking these will lead them to becoming a high-performing group.

Do DORA Metrics Measure Developer Productivity?

They do not measure developer productivity, but they can provide insight into common aspects of software organization performance. There is a significant difference between a "productive developer" and an organization that safely and quickly ships changes the business needs. DORA focuses on that bigger picture.

What Are the Alternatives to DORA Metrics?

Metrics frameworks like DORA or SPACE are easy to find and show up every few years. The research behind them is sound, but again they require a deep understanding of leveraging metrics that many leaders simply don't have.

My recommendation is to grow into frameworks like that by beginning with Signal Mapping and then learning to categorize your metrics into the categories of: Investigation, Operational, Signals, and Targets. These will help you not only know why you have the metric, but also how you should use it. This works well with frameworks like DORA or SPACE and also allows for you to measure what is important today for your group and teams.

Should Small Teams Use DORA Metrics?

I think DORA is a great set of metrics for many organizations to use, but they can be overkill. Small teams and small companies often have the benefit of being very close to their customer and overall business results. I think that closeness beats DORA. Now, as companies and groups get larger, they often lose that closeness and need another way to know if they're doing good work. DORA becomes a proxy to say how you are performing operationally. So, the value of DORA goes up the further away you get from business outcomes.

If this kind of straight talk is useful, I write a short letter every Friday. Stories, tips, techniques, and the occasional bits of beekeeping. Join it below.

Read More