Select a result to preview

Measures, Metrics, and Signals

#guide

One of the biggest requests I hear from clients is that they want more data and metrics. I get it, but the underlying reason is pretty telling, and in my opinion, illuminating. The reason for the request is they need to know what is going on, and what decisions to make.

This is an overall guide to how I look at this topic and some specific techniques.

Principles

Less is More

It is addictive to want more and more data, but it almost never helps and, in fact, overwhelms most people. There are, in my experience, just a few concrete cases for needing data and knowing that helps eliminate this feeling that you need more and more.

  1. Make a specific decision or intervention
  2. Monitor health and trends
  3. Investigation

I'll get into these further on, but these three categories nicely limit what you actually need. If you can describe the core decisions or interventions you need to make, you can develop signals. If you know what matters in terms of health you know what metrics need to exist. I should note that "health" can be operational or goals, but either way you don't need much. As for investigation, this is the broad category when something anomalous comes up. You may temporarily need to act like an archeologist to understand something but that should be rare and a prompt to look at how this event factors into gaps in the other categories.

Balance

This is one of the more challenging topics as it requires some careful thought and planning. I've written about that in an article called How I Set Goals. The main idea is that you need to balance your measures across two dimensions. Leading and lagging as well as tradeoffs. If you use one singular unbalanced metric you'll experience tunnel vision where the small thing you see looks good even if everything else is falling apart. Balancing your metrics helps avoid this.

Target Your Efforts

It's tempting to try to make each important measure or goal move forward at once but I'm going to advise against it. Instead, recognize that each metric you have and each goal associated to it are acting as constraints against one another. They often behave in unintuitive ways.

I recommend making your efforts focus on improving one metric while holding the others steady. This will help you more deeply understand the relationships between the work without letting things spiral into chaos.

As an example, improving the rate of delivery without diminishing quality is almost never explicit and almost always the opposite of what happens.

Get Started

What Goals Are Important

This is a great place to start as many leaders are living with goals from higher up in the organization. While goal setting and framing is a bit of an art, you can cheat a little by answering a few questions like:

Lets take typical business outcome like revenue. We can apply this in several ways using these three questions. Here are two examples:

Each statement answers those four questions and leads to a concrete goal with clear metrics.

Even if you feel like you cannot possibly impact these, they are your guiding light and must be ever-present with you and your groups. They are what everything else is about and help make a lot of decisions easier when you realize you're working on things that nobody can justify.

Leading Indicators

Ok, after goals which are lagging measure we need leading indicators for them. This is an art as well but these are often what are easier for us to change.

Good looking leading indicators are pointless if your goals aren't moving.

Leading indicators measure what we are doing right now and tend to focus on operational excellence.

There is an implicit hypothesis you need to make explicit when developing leading indicators. The hypothesis goes like this:

"We believe that a positive leading indicator will lead to a positive lagging indicator."

Don't skip that bit. It is what protects you from getting lost in worrying about leading indicators that have no connection to what is actually important. This is the biggest trap I see for most leaders dealing with data. Don't fall for it.

You can be the slowest teams in the company and be the most successful if you have the most significant impacts.

Popular examples here are:

Balancing

There is some balance between leading indicators and the goals, but now we're going to add more balance. Aside from the trap with leading indicators, it is very easy to simply focus on what is obvious and assume the rest will be fine.

It won't be.

For every measure you've identified you will need to ask yourself, "What might we lose or sacrifice if we pursue this?" That will tell you what you need to add balance with.

The single biggest fraud in software is the idea that speed costs quality. Balance measures of speed with measures of quality.

The DORA metrics are a great example of balance that are worth looking at and developing your own sense of how they balance so you can do this yourself.

In DORA there are four metrics: Lead time, Deployment frequency, Change Failure Rate, and mean time to recovery. Lead time and change failure rate balance themselves and deployment frequency and MTTR balance themselves.

If you do not add these balancing measures, you'll experience degradations in your operations, teams, and other key areas that will eat up your ability to be an effective leader. You must balance your metrics.

Done

If you've made it this far you have a few things written down. You have your business goals written, leading indicators under your control that you hope will get you there, and balancing measures to make sure things don't go sideways. That might look like this:

# Goals
- Improve revenue by 20% by Q3
- Reduce customer complaints by 10% by Q4
  
# Leading Metrics
- Cycle time
- Commit frequency
- High/Critical security issues in static analysis
- Defect rate
- Hotfix rate
- How long the build is red

I put in a collection of metrics in there so you can get a sense of some roughly balanced leading indicators. The idea is that with only this you can now always report on how you are performing as well as if you're making the desired impact without too many surprises. The scope and work you do exists to move towards your goals and the leading indicators show you how well you're operating to that end.

Signals

Signals are a powerful tool for leaders that exist in this bucket of measures but in a unique way. What we produced above is what we need to monitor health and trends, but we don't know yet what decisions to make or where to intervene. There's a loose sense with what we developed but as I pointed out above, there is only a loose connection between the leading metrics and overall goals so a change in them may not warrant your intervention.

Signals are indicators you develop to help you with that. They're not always as formal as the metrics above, and can be personal to you or operational with your groups.

The most well-known example of a signal is a "Red" or broken build. That signal acts as a prompt for someone to go investigate and fix something broken.

Develop Key Decisions

The hardest part in this is to list out a set of decisions you need to make with this project and your teams. You don't have to be perfect or complete, but you do need to develop this list. Knowing what types of decisions you need to make or what interventions are required are the key ingredient in developing signals.

Here are some examples:

You can tell this list can grow infinitely and the art with this to focus on the most important few, and grow it over time only if you realize you have a blind spot. You know you have a blind spot if you got surprised by one of these things or it went poorly last time.

How Will I Know

Time to turn these into a signal. Each signal is paired to an intervention or decision from above. So you'll want to keep that pairing together so this isn't confusing in a few months.

For each decision you've started with, ask, "How will I know?" or "How might I detect the need for this?"

There is a tendency for a lot of folks to get uneasy with this question because they believe they need some perfect combination of data. You don't. All you need is enough to make you learn more.

Don't fall into the trap of believing that without perfect data that you cannot move forward. Here are some examples that I use quite a lot happens in 1:1s.

They wonder if they'd be a better fit in a different role as a signal that they may be dissatisfied for their job an I need to talk with them.

If 40% of folks in a 1:1 bring up the same basic topic, I need to investigate it now.

Is this scientific, proven, or certain? No it isn't. It is just enough of a signal for me to act and that is the entire point.

Developing signals for the work around you helps you develop not only your internal compass as to what to watch for and ask about, but also the level of instrumentation in your operations to learn about.

Most leaders I've met are shocked by some sort of event that their team dealt with that they were unaware of entirely. Those leaders had no signals and chose to hope that people would tell them when something important happened. Don't be that leader.

Done

I didn't give a lot of examples here because signals are something that really comes with practice and reflection. Start with just a few focusing on things that have caught you off guard in the past or things that worry you. Be honest about if they're helpful or not and iterate.

Do take the time to write these down and keep them visible near you. These things are easy to forget in the chaos of every day, but if you keep them close you'll notice how sharp your observations become about what is going on, and how much useful information was slipping by before.

Visualize the Data

I want to mention a few bits about how to actually track data and make it useful since this is another area folks tend to get wobbly.

Relative over Absolutes

There is a tendency to make metrics into absolutes like reach X revenue or get cycle time to Y days. These are targets. Generally speaking, you can do this but it is something I am going to tell you to avoid for one reason: You don't know if thats good or not.

Part of using metrics effectively is monitoring their change over time, and a target hides this. Your efforts to make improvements are overshadowed by the fact that you haven't hit your target when you should be focusing on how you're seeing a positive trend.

To that end, use relative metrics or measures not absolutes. One of the easiest ways is to describe a percentage change.

This adds some complexity to your tracking but it makes a difference. You want to know two things with your metrics: Where it is, and if that is movement in the right or wrong direction.

This helps you assess the efforts relative to the impact on what you care about.

This is also how your investment portfolio works, it shows you what you have as a snapshot and shows you its trend over different time periods.

So let us say you were wanting to reduce cycle time by 10%. You'd want to see that as your goal and then the actual cycle time and then an indicator if that is better or worse than the previous one or rolling average.

10 Data Points

Folks get all bent out of shape over the idea that you need lots of data for it to be significant. This is way less true than people realize. First, having just one point of data is better than none at all. So if your concern about the data not being significant enough is preventing you from having any, shame on you.

Second, if you look into the subject of sampling you'll learn that you need surprisingly little amounts of data to have what most people would consider an acceptable margin of error.

Third, these metrics and signals aren't meant to be proof they're meant to act as indicators. A thermometer can't tell you you're healthy but it can indicate you're sick. So don't get confused about the purpose of these metrics and signals with the idea of proof and correctness.

The tl;dr of this is, you can feel good about just 10 points of data, and if you're starting off with less than that, that's ok too.

Some Measures I Use Regularly

I thought I'd share metrics I tend to start with since they are often very applicable in most organizations.

Flow Metrics

Cycle Time

How long a part of a process takes. Generally this is applied to development teams from when they "Start" work and it is "Done." You have to be very specific about what it means to start work and when done is.

People get upset about things like blocked work inflating the numbers, but that work is sitting not getting done so it counts. The goal is to remove the blocker.

Throughput

The rate at which work completes. Many tools measure this for you. This is more important than cycle time but very few companies understand it. Little's Law helps understand the relationship about this.

Wait Time

How long items sit waiting to move forward is one of the biggest sources of waste there is. This is not a metric that is useful to many people but very illuminating when you're trying to figure out how to improve process efficiency.

Quality

Defect Rate

One of my go-to measures for quality is simply work done compared to bugs/defects created. It yields a percentage that I've observed between 60% and 120%. This is how often problems are created when doing work.

Note that creating problems and fixing them are two different things. Though the message still matters that we're working with a leaky bucket.

Defect Efficiency

The rate of creating bugs compared to closing them. Plenty of teams track bugs dutifully but they don't always get fixed. Conceptually this is fine, but in practice most groups justify not fixing bugs because they feel it takes too long and isn't worth it.

This indicates the group and company's tolerance of shipping problems to its customers.

Change Failure Rate

When I calculate defect rate I do it in absolute terms. If you want to get more specific about something like something going wrong in production this is the more focused version. You measure the completed work that went into production compared to the issues uncovered in production.

This nuance isn't useful for many teams, but it again points out how leaky the processes are and tolerant we are to ship problems to customers. Be careful with QA on this one as many times they are very sensitive to the fact they are never given time or resources to test anything and this feels like an attack on them.

MTTR

Mean Time To Recovery is another cousin to defect efficiency that measures how long it takes from when an issue is surfaced in production until it is fixed. This is a rolling average. Much like cycle time's issue where blocked things sit there running the clock, small bugs that are surfaced also run the clock here.

Again, it surfaces how tolerant we are of our customers living with issues and our own operational capacity to safely resolve them.

Product

Pirate Metrics

Acquisition, Activation, Retention, Revenue, and Referral, or AARRR. Acquisition is a user arriving at our service, activation is their entry into our service, retention is their return to the service, revenue is revenue, and referral is organic growth.

Each of these will have specific meaning for your product and how you instrument this will be unique depending on the goals and product maturity.

Churn Rate

Where customers leave. Now this can also mean in a specific workflow within the app but in engineering groups I like to create a churn rate tied to customer loss related to quality. Customer support or customer success teams will know this.

Lead Time

This is the classic, "Ask to get." I keep an eye on this but almost never bring it up. The reason it isn't relevant is that most companies are comfortable with their planning processes and committees and budget cycles so pointing out that it takes 1.5 years to get something isn't a conversation most are willing to have in a productive way.

Still, this is useful to keep because there will be pressure to, "deliver faster" and the bottleneck isn't likely where anyone thinks it is.

Leadership

1:1 Sentiment

This is fuzzy but I have signals I use that are developed by tracking sentiments across 1:1s for individuals and across groups. This will be something simple like X/Y or X% of folks expressing a frustration or sentiment prompts me to investigate.

KPI/North Star

These things almost always exist somewhere but often get lost in the frenzy to track and report on "progress" and "activity." I like to put these back front and center in every report and communication so it can always be a part of every conversation.

When this doesn't exist, I create one and test if it's resonant with my senior leaders and use that as an excuse to develop a more appropriate one or find the real one that has been left out.

Miscellaneous

Process Failure Rate

Everything has a process attached and it is often shocking to folks how rarely that process is successful.

This is a ratio of items that make it through compared to items that got kicked back at some point along the way.

It is important to note that many processes explicitly do this as they have numerous quality gates built-in for this purpose. So this rate isn't always indicative of bad things happening, but rather how much waste there is.

Interruptions

Another shocking metric for many groups is how often work that was unplanned gets accomplished compared to planned work. This is also called side-loading by some folks. This happens when someone taps someone on the shoulder with a personal request. Also this could be an emergency issue.

Either way, tracking the rate at which this happens is often surprising to folks. While fixing outages isn't a bad thing, having them is, and this begins to show the impact of how poor upstream quality eats into downstream productivity.

It also begins to point out how hard it is for a team to reliably or predictably accomplish work when so much of it is a surprise. Be warned that almost everyone wants to put an end to this even if they are the person side-loading work into the team.

Deployment Frequency

Part of DORA. Deployment is a complicated thing at most companies and being able to deploy quickly and safely requires operational excellence. It also allows for better risk recovery.