DORA Is Not Enough For Engineering Teams

Debo Ray is the cofounder and CEO of DevZero.

Just a few years ago, many engineering organizations were focused on growth at any cost, intensely expanding their teams and tools. Now, organizations are tightening their belts and trying to do more with the same amount of resources. The pressure is on for teams to measure their output and make improvements across the board, from onboarding to overall productivity.

Five key metrics developed by Google's DevOps Research and Assessment (DORA) team have become the industry standard for assessing a development team’s performance: deployment frequency, lead time for changes, change failure rate, time to restore service and reliability. This framework may offer high-level takeaways about a team's performance, but I believe it falls short of providing truly valuable insights. I hear the same sentiment echoed repeatedly by other engineering leaders: DORA metrics aren't enough to plan, manage and lead an organization effectively. We need to find a more holistic approach to measuring the progress and success of an engineering organization.

Evaluating Processes, Not People

The main limitation of the DORA framework is that it evaluates developers through a narrow window of inputs and outputs. But a development team isn't a factory. Individual developers are knowledge workers, and you can't accurately measure their contributions by how many lines of code they write or how many pull requests they complete. Every team contains a wide range of experience and expertise, and every task requires a different amount of time and effort. There's a big distinction between a senior developer and a junior developer, or a simple bug fix and a complex cloud migration.

Instead of measuring developers, let's measure the development process. We need to start closely examining operating metrics and figure out what's really holding developers back from streamlining and enjoying their work. If we can optimize the process to remove bottlenecks, we can improve productivity and the developer experience.

My company has created a new framework, Developer Experience Index (DXI) to combine operating metrics with productivity metrics. As part of that, we use Open Developer Analytics (ODA), to measure the process and provide more granular insights. We use ODA to track neglected metrics, such as how long it takes for engineers to run commands, and identify inefficiencies. We found that each developer on our team was spending about eight hours a month just looking at their screen, waiting for something to happen. If a process takes 30 seconds and a developer has to execute it a dozen times a day, the minutes add up quickly. Multiply that time across an entire team, and you accumulate significant developer frustration and lost productivity.

Once we had this data, we investigated which processes and commands failed frequently and how we could troubleshoot the root problems. By focusing on smoothing out wrinkles in the process, we were able to reduce each developer's average wait time from eight hours to one hour per month.

We also applied a similar data-driven approach to the onboarding process for new developers. In the past, onboarding time took our team around 80 hours per developer. With remote environments, we cut that time to around four hours per developer, freeing up over 70 hours to devote to mission-critical priorities.

Best Practices For Measuring What Matters

It’s more important than ever to make the most of your existing resources. I recommend starting with the following four strategies that have been invaluable in my organization.

1. Gather Data

Before you can make any meaningful changes, you need to understand your current development workflow. Add instrumentation to the development workflow, collect comprehensive data on how developers spend their time and look for opportunities to remove obstacles or reduce friction. Measure how long specific tasks and processes take, how often commands fail or lag and how these factors affect your team’s experience.

2. Utilize The Cloud

The data we gathered showed us that many of the processes slowing down development processes that we were running locally could actually be executed faster remotely. The performance of individual computers is dependent on factors such as operating system version or processing chip speed, while a remote environment offers more consistent, robust resources. Consider making it standard practice for your team to execute resource-intensive commands in the cloud to minimize variability.

If you operate in a shared environment, you can also right-size your infrastructure to your team's actual needs. My team and I have found that 99% of the time, we don't need to work on machines with powerful CPUs or GPUs. By working in the cloud, developers can scale their processing or storage needs up or down for the task at hand while reducing our overall infrastructure costs.

3. Optimize Caching

Development teams tend to work on the same repositories day after day, and frequent caching is key for reducing database load and improving performance and scalability. Develop processes to identify and recognize only incremental changes and directly impacted services. With this approach, my team was able to reduce build time from around 30 minutes to five minutes.

4. Enhance Visibility

Measuring and improving development processes shouldn't be a one-time activity. Develop an ongoing strategy to gather data and observe what's happening in your environment. Teams are under constant pressure to deliver products on tighter timelines, and bottlenecks and other frustrations can negatively impact developer experience. The costs of an unhappy team are high, including increased developer churn and delayed time to market.

It's time for our industry to update the metrics we use to evaluate development teams. Real productivity isn't only measured in dollars and cents; it's also shaped by how happy, engaged and satisfied developers are at work. Implement metrics that give you better visibility into the development workflow and not just the developer workflow.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?