Enter your search text in the box above

Select a result to preview

How to Use the Metrics and Signals Worksheet

#article #metrics

metric-and-signal-worksheet.png

Developing signals and metrics can be overwhelming, but this guide will help you make use of the Metrics and Signals Worksheet that encapsulates the techniques I use to help clients with their impossible asks.

This page will assume you're familiar with the background information on developing engineering metrics.

What Is the Metrics and Signals Worksheet?

The Metrics and Signals Worksheet is a one-page worksheet that helps you develop signals, balanced metrics, and operational metrics.

It is something I developed to help leaders put pen to paper and develop useful signals and metrics to lead their teams and organizations. It is built on my own career of developing useful metrics that lead to insight and action.

If you don't have the worksheet, put your email below and I'll send along a copy. You'll also get my newsletter. Unsubscribe anytime.

The worksheet helps you develop signals, balanced metrics, and operational metrics that form the core of leveraging data to make better decisions. It isn't meant to replace things like DORA or any other metric framework. It helps you bring those metrics, as well as the ones unique to your situation, to life.

How Do I Fill Out the Signals Section?

Signal Mapping is all about connecting something you observe to an action or decision. Signals take the form of a statement "When I see___, I will___."

Start by listing out decisions you have to make or actions you need to take. Don't focus on the ones that you only make a few times a year. Start with frequent ones to build practice.

Now that you have that, go through each one and ask yourself, "What would I need to see that would make me act?" You're not looking for concrete proof here, you're looking for just enough to act. These are indicators, nothing more.

Now, you keep these signals handy and when they go off you'll likely experience doubt. That feeling is useful because it will spark a feedback loop of wondering if the signal was correct or not. While the feeling is useful to prompt reflection, the feeling doesn't mean the signal is wrong.

How Do I Fill Out the Balanced Metrics?

Balanced metrics pair what you care about with leading, lagging, and sacrifice measures so no metric can lie to you or be "gamed."

Start by thinking of what you care about that you want metrics for. It could be quality, productivity, satisfaction, or anything else.

Now, you're going to come up with metrics that help you see into that while providing balance. For leading indicators you want to think of what you can measure today that gives you insight. This won't be a perfect measurement, but it is typically fast and easy to come to.

Next, you're going to think of how you'd measure afterward as your lagging indicator. This deserves a little explanation. Let's say we care about productivity and the thing we measure today is velocity. The question about what we'd measure afterward is saying, "How do I know we were productive?" This answer will likely not be velocity but rather how much work we actually did, so maybe issue completion.

Now we move on to the other balancing force that helps counteract gaming behavior. When you care about one thing, many people will sacrifice something else to get that number to look good. You will need to anticipate that here. You are going to guess what will be sacrificed, then come up with a metric that will help you catch if it happens. My advice would be that you start with quality. Quality is almost always the thing that folks sacrifice.

This is a kind of complicated topic, so I recommend you read a longer guide. Also, DORA metrics create this balancing force naturally and it can be useful to think through that as an example.

How Do I Set Operational Metrics?

Operational metrics answer the question, "How are we doing?" There may be metrics for performance, system stability, etc. The main thing you need to know with operational metrics is what the threshold is for any given indicator.

Without a threshold you have numbers that dance without meaning.

So the exercise here is to choose some operational metrics. You likely already have some. Write down the metric and the current value for that metric. Now you'll want to set a threshold. This threshold is a value that will prompt you and your team to take some sort of action.

Here's an example.

I have an indicator for stability that is server disk space. It is currently at 45% full. I've set a threshold of 85%. This means that while I'm lower than that number, I'm fine. If it hits 85% I treat this as a threat to my operational stability and I act.

Some of these thresholds will be easy to develop. Some won't. For ones you're unsure about consider using standard deviations as a threshold.

Once you get the hang of this, you can begin to weave this into your strategy using horizon planning.

What Do I Do After Two Weeks?

The signals you develop are the best thing to practice with to develop your skills. Pick signals that happen regularly so you get the feedback on what makes a good signal. This feedback loop will naturally develop your ability to balance metrics and develop operational thresholds.

Having said that, the balanced metrics and operational metrics are living things. Where possible, keep the data in front of you. Set a cadence of pausing to reflect on these and ask if they're helping you, and if not, what is missing?

When your numbers jump, find out why. You'll learn about how things work and that will, in turn, help you refine your metrics. Learning to do this takes practice and you'll make mistakes. If you want to skip some of that and hit the ground running, I offer a workshop to help you and your peer leaders do just that.

What Does a Completed Worksheet Look Like?

Want to see an example of a sheet that is filled out? You can take a look right here. It is worth mentioning that this example isn't from any one place, but does represent real metrics and signals I've used or seen used successfully.

signal-worksheet-example.jpg

Frequently Asked Questions

What Is a Signal in Engineering Metrics?

A signal is data with an action attached to it. "When I see interest in other jobs come up twice in a month of 1:1s, I focus on morale" is a signal. Signals serve as indicators that it's time to act.

Do I Need DORA or SPACE Metrics to Use This?

You don't need these frameworks to use the worksheet and develop functional metrics for you and your teams. DORA, SPACE, and other metric frameworks work alongside this content. Where the other frameworks tell you what to measure, this helps you understand how and why to measure.

Can I Fill This Out With My Team?

Yes, but with a caveat. Signals are something you can do with a group, but also work well to develop signals for your own leadership needs. Balanced metrics and operational metrics, on the other hand, benefit strongly from collaboration. Not only will it help sharpen what you come up with, it creates shared meaning that you'll benefit from going forward.

How Many Signals Should I Track?

When you're starting off, fewer signals that fire more regularly is the recommendation. I wouldn't recommend going any higher than 5-10. It's not likely you'll wind up with lots over time either. These signals wind up feeding into other metrics and become internalized over time as they work.

If this kind of straight talk is useful, I write a short letter every Friday. Stories, tips, techniques, and the occasional bits of beekeeping. Join it below.

Read More