Why are metrics so hard 🩻

Happy Friday!

I want your help. I've decided to write a booklet on metrics and measurements. I want to make sure it answers your questions. Please reply with any questions, frustrations, and hopes you have for a booklet that helps everyone develop and use metrics better.

Why metrics? Well, it's one of those thorns in most leaders' and organizations' sides. Plenty of data, dashboards, and reports, but a shortage of meaning and insight. Everyone knows they should be using data to make decisions, but it's just noise.

An integral part of my practice is getting paid based on measurable results, not time. I literally put myself on the line with my ability to wield data successfully. Not only that, in every single engagement I've developed metrics to target radical improvements. It's a skill set I've put a lot of effort into.

You don't read this newsletter for that stuff, though; you want something you can use. So while I've been writing a lot on the subject, I realized there's one angle related to metrics I don't write enough about.

Setting targets and goals. I want to share some advice about this.

By the way, here's what I wrote this week:

Everyone has been burned by seeing a target number or metric move in the right direction while things paradoxically get worse. This happens because it's folks' job to make that number look good. Full stop. The secret is that nobody has a conversation and sets boundaries around the question, "At what cost?"

Consider the classic desire that velocity should improve. Set a goal or target for it, and it will magically go up. Oddly, though, not much more work will actually get done, and often less will get done. Also, the work will have lower quality than before. Behind the scenes, folks have taken the shortest route possible to improve velocity: Skew estimates and cut corners.

Velocity isn't unique. Almost all targets and goals invite this behavior, compromising something to make the number look good. It's usually quality, though, just saying.

Does this mean you can't set targets and goals? Of course not, but you have to be aware that you're working against this behavior. Let me give a specific technique that helps.

A long time ago, I read a book about how Toyota's manufacturing manages for improvement, and one technique it introduced is called Target Conditions. I believe the book is Toyota Kata by Mike Rother.

The way it works is that you have two columns representing the current condition and the target condition. For each of these columns, you put down the metrics that represent how things are operating, not just the one you care about. So, in a development organization, your current condition might have a list of things like:

Throughput
Cycle time
Defect Rate
Deployment Frequency
Change failure rate
Weekly outages
Critical static analysis issues
Test coverage

Curating metrics to represent the operational health of your team's and org's processes is out of scope for this newsletter, but you're looking for a suite of metrics that show how things actually work.

Now, with that list, you take your baseline and record it. This activity alone will likely expose problems with instrumentation and understanding around these metrics. That's something to work through. For example, your team may not want cycle time to include QA, but depending on the process you're looking at, it has to be included. Does a minor bug count toward the change failure rate?

Next, move on to the target conditions, where you'll duplicate the list, but next to them you'll indicate how these new metrics should look after you've finished improving. My advice around this is to use relative improvement. For example, you want cycle time to drop, so you say your target is for cycle time to drop by 10%. You can calculate what that turns into for convenience, but starting with a relative number helps frame the scope of the target.

Also, it is generally a good idea to not go crazy trying to alter every metric. Improvement is challenging work, so start off small and grow as you and your teams get better at this.

What makes Target Conditions interesting is that it does two things many orgs struggle with: It sets balanced metrics and isn't prescriptive.

Take, for example, how many groups use OKRs, and while they aren't supposed to be task lists, they inevitably turn into one. Did checking that box make things better? Target conditions leave no space for what to do or how to do it, just how you'll know things are better. This allows for infinite possibilities to experiment.

As for the balanced part, admittedly this depends on the metrics you track, but by focusing on process metrics you're likely to create a more balanced set than you would otherwise. This means that if you wanted to improve throughput and the other metrics stay the same, those hidden sacrifices will be laid bare as the other metrics begin to degrade.

So there you go, a useful tool that changed how I used metrics over 10 years ago, and that I somehow forgot to write about. If you give this a shot, or want to work through this together, give me a call.

Sincerely,

Ryan