Do Metrics Under Deliver?


DILBERT © 1996 Scott Adams. Used By permission of UNIVERSAL UCLICK. All rights reserved.

Contributed by Tom Huntington

On the second Friday of each month Neil Baron assembles a handful of product executives to share best practices. Last month the ranks swelled to three-dozen executives as we discussed product metrics.

How does the seemingly simple task of measuring progress get so distorted? The following themes emerged.

Create responsible ground rules

We’re biased toward self-congratulation. Neil kicked off the discussion with the Dilbert cartoon above. Should metrics reporting be moved out of the Product organization to an impartial group like Finance? Such impartiality comes at a cost. The goal is to improve behavior, not just generate data. If the Product organization does not feel ownership in the metrics, it will not use the analysis. Responsible ground rules can ensure effectiveness, regardless of who tallies the metrics.

  • Buy-in: The Product organization should have an active voice in designing metrics so that it takes ownership of results.
  • Context: With proper context, metrics become less subjective. Two useful contexts are historical trends and prior expectations of current results.
  • Transparency: Share results and allow for comment. Sunlight is the best disinfectant.

Share commitment

Should we assign a single owner for each metric? Proponents argued that a single neck to choke is necessary to produce results. Opponents pointed out that each metric is part of a larger chain that impacts others. Narrow ownership invites misalignment. In the pursuit of overall sales, salespeople will overlook new product introduction. In defense of the status quo, a product owner will fight creative destruction. Producing results is table stakes, but teams that share commitments can be more strategic.

Identify the resulting action

Several executives distinguished between tactical and strategic metrics. For example, defect rates may be tactical while new customer penetration is strategic. However, this classification is not stable. If product defects are causing customer attrition then defects are highly strategic. The importance of each metric is relative to the resulting action. If defects rise above 2%, then a resulting action could be to shift a resource from Development to Quality Assurance. This resulting action brings relevance into focus. With this context, the team can then figure out the right metrics, meaningful thresholds and who to involve.

The right metric gives you leverageMetrics_BPMA

The group ended with a brief discussion of specific metrics themselves. What are the best metrics to track? My personal favorite is referral rate. When referrals are high, you have a virtuous cycle. However this answer is no better than off-the-shelf metrics like revenue and cash flow. The right metric is specific to each company. It should describe the unique economic engine of your firm. In his book “Good to Great,” Jim Collins defines the magic number as a denominator because it drives desired results like revenue and cash. The right metric gives you leverage.

Referral rates are high at the Product Executives Forum. Enough people were seated along the walls that it’s not exactly a roundtable any more. Nonetheless in Neil’s capable hands, the sessions remain rich and intimate in their pursuit of better product management.

Tom_HTom Huntington is co-founder and CEO of Windshield, collaboration for purpose-driven teams.




[Ed. note: Additional thanks to Tom for all images in this post.]

One thought on “Do Metrics Under Deliver?

  1. Great summary of the discussion, Tom. One other thing that came up is ensuring goals are aligned both with company strategy and across groups. You want incentives to encourage the right behavior (including cooperation).

    I’ve seen groups where people a product was ready for retirement but the team responsible for it was goaled on its revenue. This made their paycheck dependent on keeping the product alive at any cost.

    I’ve also witnessed QA being measured on reducing bugs while engineering is measured on output. This sets up conflicts that the teams can’t resolve on their own.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s