Measuring Engineering Performance

Just got a great query from an Engineering Director. Love it.

I’d love to pick your brain about engineering team performance measurement. How do you do it? Do you have any experience with what works/what doesn’t work?

I’ve got lots of thoughts on measuring the performance of engineering teams. No doubt you do too, so please Tweet @VelocityCounts and share your perspective. Okay, onwards…

Long story short: if you measure it, and the team is aware you are measuring it, they will game it. So, if the team is going to game the number (as you are measuring it) then you’ll want to make sure it is a metric that, when gamed, leads to the behaviour you want to see. For instance:

  1. Estimates: some teams may think it is about getting more story points completed in a sprint, so they inflate their estimates over time to make that metric look better. Measuring story points leads to poor behaviour as teams inflate estimates at the expense of consistency and predictability. Estimates are for the team. Consistency from sprint to sprint is the key. And you can not compare output between teams due to a different baseline for the estimates.
  2. Code Reviews: a great way to lower the cost is to catch bugs before they hit production, right? Code reviews are one way to do that. Some teams like to measure the time spent awaiting code review, not a bad idea. Just avoid measuring the time spent in code review as teams will optimise for lower time, which may mean lower the quality code reviews and lead to the team missing things.
  3. Incidents: one thing that incidents measures is the direct impact on our customers (outcomes). I see no downside to measuring incidents and I see a positive impact when people try to game this – less incidents, makes the team look better, better for the customer. Also, you may like to ask yourself: can you leverage this alongside code reviews to see who are the people that give ShipIts to deploys that lead to incidents.

At present I’m really interested in measuring outcomes for customers and maximising for that, rather than engineering team performance specifically. For instance, a small change does not mean much output in terms of code, story points, etc, although it may have a massive outcome in terms of customer experience, conversion, etc.

That make sense? What metrics are you thinking of measuring? Let me know @VelocityCounts.

Leave a Reply