Enter your search text in the box above

Select a result to preview

How to Measure Developer Productivity

#article #metrics
Cover image for How to Measure Developer Productivity

Measuring developer productivity is a thorny subject often faced by leaders who feel a need to know how their developers are performing and if there are any bad apples. The main recommendation though is that you should focus your measurements on the overall system's performance and team productivity, and not the individual.

Can Developer Productivity Be Measured?

Yes, you can measure anything. Having said that, let's be honest about why this hasn't ever really worked out for you and likely never will. A lot of folks turn to things like individual velocity, pull-requests, lines of code, or the number of tasks they complete. The problem everyone has is that these, at best, have a weak correlation to the deeper questions you want answers to. What's more, often measuring individual productivity leads to sabotage or worse performance.

In fact, there is academic research, like Robert Austin's Measuring and Managing Performance in Organizations, that suggests that measuring organizational and individual performance works against itself just about every time. Having said that, there is also research showing that leading with measures and metrics improves outcomes and performance.

One of the key things to understand is, what questions you want to answer beneath, "Are my developers productive?"

These are more specific questions you have that build up to the larger productivity question, and the more specific your questions get, the easier it becomes to use data to answer.

Why Measuring Individuals Backfires

There is an often shared story about when Britain colonized India, and they set up a bounty on poisonous snakes. The thinking was that this would incentivize folks to kill the dangerous snakes and make folks safer. Sure enough, folks started killing snakes. In fact, the supply of dead snakes increased.

It turns out folks started breeding the snakes so they could make some money.

This story, while amusing, is a great illustration of what happens when measurements that affect folks go wrong. As soon as a measure is known, folks will work to make that number appear good. Even if it messes something else up, they'll do it.

If you make velocity a measure, it will go up even if less work gets done. If you measure test coverage, it will improve even if bugs go up too.

The challenge here is that there is a huge difference between using data to understand, and using data to manage. Managing individuals with data is extremely hard and unless you know how to balance your metrics and develop appropriate ones, there will be unintended consequences.

I also want to point out that teams are far more than the sum of their parts. This is counterintuitive. Why this is the case is due to numerous factors and few of them have to do with the summation of individual contributions and productivity. You've seen teams of highly skilled people struggle just as you've seen teams of inexperienced juniors do impossible things. How would you know if someone that contributes little individually is the source of the team's overall radical performance?

How Does Google Measure Developer Productivity?

Google has produced lots of interesting contributions to the tech industry over the years. They became infamous for their interviewing methods that folks began to emulate before Google admitted that they made no difference whatsoever.

What Google does may not be the right fit for you.

Having said all that, there are lots of things that have ties back to Google. OKRs were invented at Intel but became popular after Google shared its success. DORA was bought by Google and Dr. Forsgren who popularized those metrics went on to then create the SPACE framework.

Each of these frameworks provides a method to apply metrics to the question about productivity and performance. What is most important to point out is that none of these are really about individual performance or productivity. Even when OKRs are applied to an individual their use isn't for assessing productivity at all.

Google's best work on measuring produced frameworks that measure systems and teams, not individuals.

Also, many of these are in use by companies and yet, they still don't have a useful answer or actionable insight. Does that mean these frameworks don't work? No. It means that the framework isn't enough.

What to Measure Instead

So here's the rub, you need data to help you understand what is going on and most attempts you've seen haven't worked out. What do you do?

I'll recommend you use Signal Mapping to start. Unlike most measurement frameworks that install specific measures, this is a way to develop measures that are tied directly to action. Now, as for what you should start measuring I'll recommend you start small. Also, don't start with measuring people, start with a question that you personally need answers to.

Most folks don't have a good background with using data to inform their work, and if that's you, you'll need to be patient while you build up that skill set. Trying to develop a dashboard yourself will exhaust you, and then overwhelm you with numerical noise that doesn't help you make any decisions.

How Do You Measure Productivity When AI Writes the Code?

AI tools are now helping developers produce ten times the amount of code. It really is a marvel. For folks who struggled to assess productivity before, this adds a unique wrinkle.

Or does it?

Let me put it this way, are your developers more or less productive based on the amount of cloud services they use? Are they more or less productive based on how much they use their IDE? Would measuring IDE usage or cloud service consumption help you know who is productive and who isn't? Nope. Neither will measuring AI usage.

Now, token cost is a different thing. This is something that you should develop thresholds for as an operational metric.

It's not the tool that is important, it's how individuals, teams, and organizations use it. A poor practitioner can produce volumes of defect-ridden unwanted code, and a skilled person will produce volumes of perfectly acceptable code.

If you believe your team should be able to produce ten times the amount of valuable code, are you? Are your problems increasing tenfold? Is producing tenfold actually overproduction? These are likely more important questions to develop metrics for than anything else.

The problems are the same, regardless of the tools in play. Make sure your system and teams are performing well. AI will only amplify what is happening.

Frequently Asked Questions

What Metrics Show Developer Productivity?

Team productivity is better than individual, and that is where you should focus that effort. An individual that isn't productive might be the glue the team needs to have explosive growth and productivity. Almost any attempt to measure individual productivity will not only miss that, but punish it.

Leaders need to understand how their teams really work and understand their individuals through 1:1s. They can make use of signals to detect when things are shifting out of balance.

Now, common metrics that I use in the space of team and organizational productivity are:

Should You Measure Individual Developers?

No, you really shouldn't. Having said that, you do have to know what is going on with your team and the folks on it. The best way to keep up with what is going on at an individual level is through your 1:1s. You'll be able to detect morale changes, resentments, strengths, and weaknesses if you leverage your 1:1s correctly. This coupled with team and organization level measures will help you with the insight you need.

Is Velocity a Measure of Productivity?

Velocity is a rolling average of story points completed. Am I more productive if I double or triple all my estimates to force my velocity to go up even if I completed less work? Nope.

If there is perceived incentive around higher velocity, you'll see velocity climb. Most of the time through inflated estimates.

Velocity is easy to get a hold of, but has nothing to do with productivity any more than a thermometer telling you that you're healthy.

How Does AI Change Measuring Developer Productivity?

AI amplifies what is already happening, so the same need to see if the team and organization as a whole is performing is similarly amplified. AI will amplify the types of problems you already experience as well as the things you're already good at.

Focus your efforts on measuring your group's ability to safely deliver on outcomes, and you'll likely see far more clearly if AI is helping or not.

The new thing to consider is that token cost becomes something to operationalize and weave into your cost of ownership or ROI for your organization.

If this kind of straight talk is useful, I write a short letter every Friday. Stories, tips, techniques, and the occasional bits of beekeeping. Join it below.

Read More