Rob Zuber on quality, metrics, and what it means to move in the right direction at CircleCI

Podcast

Rob Zuber on quality, metrics, and what it means to move in the right direction at CircleCI

Spotify

Apple

YouTube

Ganesh Datta
Host
CTO & Co-founder of Cortex
Rob Zuber
CTO

March 26, 2026

In This Episode

In this episode of Braintrust, Cortex co-founder and CTO Ganesh Datta sits down with Rob Zuber, CTO at CircleCI, who has spent over a decade at the center of how engineering teams build and ship software. Rob shares his thinking on two challenges that are becoming harder to ignore as AI accelerates output: the quiet erosion of software quality, and the pressure to move fast without a clear sense of direction.

They discuss what made the best QA engineers so effective and why that mindset largely disappeared, how LLMs could help bring it back, and why engineering leaders need to think about metrics very differently depending on whether their teams are scaling a mature system or exploring uncharted territory.

You’ll learn

As testing shifted to automation, engineering lost the adversarial instinct that made great quality engineers effective. They would smell something off in a system and dig in—behavior that no scripted test can replicate.
LLMs have absorbed an enormous amount of public code, incident reports, and known failure patterns. With the right framing, such as asking what will break rather than what to build, they could surface edge cases no one thought to script.
LLMs are rewarded for producing things, so they produce things. Give them a different goal and they pursue that instead.
The real question is whether teams are testing the right assumptions. AI tools have dramatically lowered the cost of experimentation, which should mean more ideas tested more quickly, not just more of the same delivered faster.
For exploratory teams, the metrics that matter are adoption, user feedback, and whether assumptions are being validated. PR throughput is the wrong thing to optimize for in that context.

Quotes

"An LLM has a massive bag of tools because it reads every line of code that's ever been public. With a little bit of guidance in the right direction, you should be able to say: I built this thing. What's going to go wrong?"

Rob Zuber

CTO

"Knowing that a number has moved is not interesting. It's certainly not the end of a conversation. It's the very beginning of a conversation."

Rob Zuber

CTO

"If you're obsessed with metrics that are like delivery metrics, you're gonna be deeply disappointed."

Rob Zuber

CTO

Timestamps

(2:57)
The lost art of QA and what made the best quality engineers so effective.
(06:17)
How LLMs could rebuild an adversarial, curiosity-driven approach to testing.
(11:37)
Why framing the question correctly is what unlocks LLM potential for quality work.
(18:11)
Spec-first development and UAT as a model for LLM-assisted quality testing.
(21:34)
Moving fast matters, but direction and assumption-testing matter more.
(26:17)
Running in dual mode: rapid prototyping for discovery vs. scaled, resilient systems.
(32:37)
Why DORA and delivery metrics are only the start of a conversation, not the end.
(34:52)
Why teams adopting AI tools need freedom from delivery metric pressure to actually learn.