Product Impact Podcast | Secrets to unlocking the value of AI

Christian Crumlish watched Elon Musk's DOGE team take a buzzsaw to the federal government's own product organization — the teams he'd spent years helping build — and he has a specific, ungentle theory about what they got wrong and the surprising thing they may have gotten right.
But the deeper thread of this conversation is what happens to the rest of us as agents absorb more of the work and the cost of shipping falls toward zero. Christian walks us through the real math of replacing people with agents — and the "reverse centaur" trap waiting on the other side of it — why some AI salaries have stopped tracking value entirely, and his own mid-career decision to trade a bigger paycheck for work he could still defend to himself.
His argument is oddly hopeful. Your work still has meaning; it just might not live inside the industry or the company you built your career around. This is a conversation about reinventing your career in the age of agentic AI, from someone who has done it more than once.
In this episode you'll learn:
➜ Christian Crumlish on what Elon Musk's DOGE team got wrong tearing through civic tech — and what it may have improved.
➜ The reverse unicorn: Cory Doctorow's term for when your AI agent keeps the fun work and hands you the boring parts.
➜ AI engineers and PMs pulling $1M+ comp — Christian argues that points to something more than value-based outcomes.
➜ Why the meaning in your work may not live inside the industry or business you built your career around anymore.
➜ Building a "Product OS" — skillifying the tacit product knowledge locked in one person's head.

..................
Featured quotes:
"DOGE was never about efficiency. It was about rearranging the relationship between the government and Elon Musk's business interests." — Christian Crumlish
"The money doesn't come for no reason. It comes with expectations and strings attached." — Christian Crumlish

About Christian Crumlish:
Christian Crumlish has led product and UX teams for more than two decades — Director of Messaging Products at AOL, curator of Yahoo's design pattern library, Director of Product at CloudOn, and Director of Product at 18F and California's Office of Data and Innovation. He's the author of The Internet for Busy People, The Power of Many, Designing Social Interfaces, and Product Management for UX People, and he's currently building his own AI product management assistant, Piper Morgan.
Find Christian on LinkedIn: https://www.linkedin.com/in/mediajunkie/
His newsletter, What Now?: https://designinproduct.com
Christian's profile on Rosenfeld Media: https://rosenverse.rosenfeldmedia.com/people/christian-crumlish
..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/
Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com
Join our Slack community to get honest and strategic advice for how to navigate the AI era: https://join.slack.com/t/aiproductimpact/shared_invite/zt-49ccrauua-CUJGWYPRKI89TUu6SRrPJg

Brought to you by:
➜ PH1 (https://ph1.ca) — We solve the first and last mile of AI: use cases, KPIs, evals, and scaling impact.
➜ AI Value Acceleration (https://aivalueacceleration.com)

Two years ago, Josh Clark and Veronika Kindred previewed a book about a coming era of sentient digital experiences on this show. That book, Sentient Design, is out now — and the future it predicted arrived faster than most companies' data, governance, or restraint could keep up with.
We talk with them about why most "agentic" features are still just a chat sidebar bolted onto an old product, what a gym-booking AI agent's very bad week reveals about designing for agents instead of people, and why Google running the biggest sentient design experiment alive — its own AI Overviews — is already rewriting how businesses get found.

In this episode we cover:
➜ Don't wait for perfect data — legacy data, not ambition, is blocking three live Fortune 50 sentient design projects.
➜ Personalization is quietly becoming individualization — reading the moment, not building a permanent profile.
➜ Defensive design: teaching AI systems humility and users healthy skepticism, so neither is taken at face value.
➜ Agents need their own front door — if you don't design one, somebody finds the back door instead.
➜ AI is a reckoning, not a feature: the real opportunity is removing interface entirely, not adding more of it.
➜ Google's AI Overviews are the biggest live sentient design experiment, and businesses never agreed to it.
..................
"Putting a chat sidebar on an existing thing, it's like bolting a spoiler onto a crummy Honda." — Josh Clark
"With LLMs, it's a really siloing technology." — Veronika Kindred
"We're not gonna have a single interface that swallows them all — although clearly OpenAI would like to see that happen." — Josh Clark

About Josh Clark & Veronika Kindred:
Josh Clark is principal of Big Medium, a design studio working with organizations on next-generation interfaces, and coined the term "Sentient Design" in 2024. Veronika Kindred is a designer and researcher at Big Medium who leads Sentient Design workshops for startups and Fortune 100 companies. Together they're the co-authors of Sentient Design: Crafting Intelligent Interfaces with AI (Rosenfeld Media, 2026) — and returning guests, having first previewed the book on this show in October 2024.
Find Josh on LinkedIn: https://www.linkedin.com/in/joshclark/
Find Veronika on LinkedIn: https://www.linkedin.com/in/vkind/
Get the book: https://rosenfeldmedia.com/books/sentient-design/

Josh & Veronika set up a coupon code for Product Impact listeners: 20% off Sentient Design at Rosenfeld Media.
➜ Use coupon code: SENTIENT-PI at https://rosenfeldmedia.com/books/sentient-design/
➜ Expires September 12

..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com
This episode was brought to you by:
➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles.
➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Metalab has spent the last 20 years designing products for Windsurf, Midjourney, Suno, The Atlantic, and Uber. AI has made feature parity between products fast and cheap — anyone can ship the same baseline in a fraction of the time it used to take. Design Director Myles Palmer's answer to that isn't "taste." He thinks that word is a hollow way to dodge a much harder question.
Myles discusses what actually separates Metalab's AI-native design process from how most teams are building right now, why a vibe-coded prototype can create a dangerous false sense of confidence, and what design looks like once a product moves past the chat box.

In this episode we cover:
➜ Why Metalab's 20-year discipline, not any single tool, is what actually differentiates its AI-native design process.
➜ Myles Palmer on why "taste" is a hollow answer, and what actually differentiates once AI makes feature parity cheap.
➜ Brittany's own vibe-coded travel app cost thousands in API calls and still lost to Google — the false confidence trap.
➜ What founders actually pay for later when they skip design early, and why it defends against a "sea of average."
➜ What AI design looks like past the chat box — ambient transit, try-on shopping, and rethinking university sites.
➜ Digital twins, the 23andMe cautionary tale, and why Apple's health-data model is where consent design is headed.
.
.................
"I hate the idea of taste. The idea that taste is this differentiator drives me crazy because it's like, well, everybody's taste is different anyway." — Myles Palmer
"Investing early in design defends against that sea of average." — Myles Palmer
.................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built ⁠https://productimpactpod.com⁠ to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

.................
Myles Palmer is a Design Director at Metalab, where he leads and shapes meaningful products and experiences for both startups and global brands. Before joining Metalab, he founded and led Companion, a design studio built on strategic creativity and sustainable team culture. Myles also founded Pair Up, a platform connecting thousands of creatives for mentorship and peer learning. His work blends strategic thinking with human-centered design and thoughtful leadership that elevates product experiences.

LinkedIn: https://www.linkedin.com/in/myles-palmer-b1b70519/

Metalab Website: www.metalab.com
.
.................
Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — Elevate your AI-Powered Product & Customer Experience. PH1 is the leading AI UX strategy consultant, leveraging behavioural science and model evaluations to gain you a competitive advantage. We've worked on innovative AI product discovery & corporate AI CX transformations.

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Tech workers are so pessimistic about their own careers that most wouldn't recommend the field to a friend starting out. Eighty-two percent say AI made them more productive — but the bar resets every quarter, so the exhaustion never ends. Burnout hit 55.7%, up from 44.7% a year ago.

Noam Segal and Lenny Rachitsky's 2026 survey put a number on that pessimism: the industry's overall willingness to recommend a tech career is negative 39. Even directors land at 33% negative; founders, the most optimistic group surveyed, still score negative 5. How AI has changed your sense of yourself as a professional — Amplified, Redefined, Destabilized, Diminished, Unchanged — predicts that outlook better than role or seniority.

Glean's Work AI Index names the hidden cost behind the productivity number: "botsitting," the 6.4 hours a week the average worker spends supervising and correcting AI instead of it saving them time. One respondent put it bluntly: "My brain is rotting. My work feels worse." Another: "I just follow Claude. I don't understand what I merge."

Join  the new AI mentorship Slack community here: tinyurl.com/productimpactslack.

In this episode we cover:
➜ 82% of tech workers say AI makes them more productive; 55.7% say they're more burned out than a year ago.
➜ A 5-way AI identity split — Amplified to Unchanged — predicts career outlook better than role or seniority.
➜ The paradox of getting good at AI: the more confident you get, the more replaceable your output feels.
➜ Optimism climbs sharply from manager to director to VP to founder — revealing who AI adoption actually serves.
➜ Three postures — AI-Pilled, AI-Learning, AI-Governing — decide where research and adjacent roles go next.
➜ Why AI that's "impressive in a demo" keeps failing multi-turn users — a pattern this show keeps finding.

..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/
Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:
➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles.
➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

...........
Sources referenced in this episode:

Lenny's 2026 Tech Survey: AI Burnout Is Surging, Layoff Fear Is High — https://productimpactpod.com/news/lennys-2026-tech-survey-ai-burnout-layoff-fear
How tech workers are feeling in 2026 (original survey) — https://www.lennysnewsletter.com/p/how-tech-workers-are-feeling-in-2026
Writer's 2026 enterprise AI adoption survey — https://writer.com/blog/enterprise-ai-adoption-2026/
New Report Says You're Wasting More Time Botsitting Than Getting Value from AI — https://productimpactpod.com/news/botsitting-work-ai-index-2026/
The Future of UX Researchers — https://productimpactpod.com/news/future-of-ux-researchers
s02e16 — Moritz Sudhof on AI's invisible failures — https://open.spotify.com/episode/13d4ua1scY81zmQNQEkOXy
s02e17 — Microsoft Copilot's UXR team on UX Evals and loss patterns — https://open.spotify.com/episode/3V08ELNqgGyMpypYeSgfXP

In our last episode, Stanford NLP researcher Dr. Moritz Sudhof showed that 79% of AI product failures are invisible — they don't fire alerts, don't surface in telemetry, and don't get flagged by users, and they quietly erode trust and accelerate churn. This episode is the operational follow-up: the team that built a method for catching exactly that class of failure.

Microsoft's Copilot UX research team spent a year running evaluations on real conversations — every archetype, every industry, every use case — and found that more than half of their quality failures weren't in any eval they were running. At the world's most widely deployed enterprise AI product, with sophisticated engineering and testing infrastructure, standard evals were still missing the majority of what users actually experienced as failure.
That finding isn't limited to Copilot's scale. It's a structural gap in how the industry evaluates AI quality — and if you're running automated evals and calling that sufficient, the gap in your own product is almost certainly larger than you know.
In this episode we cover:
Token usage and adoption tell you if your AI is being used — not whether it's actually working for anyone.
Users bring real prompts, test one model fully, then compare — that's what produces honest signal at scale.
More than half of Copilot's loss patterns — user-driven gaps in model behavior — weren't in any existing eval.
LLM judges get you to baseline quality. Users catch what automated testing structurally cannot.
The flywheel: UXR evals → loss pattern taxonomy → log inspection → prompt changes → retention gains.
This team started with 10 users and one comparative question. Signal strong enough to scale to an entire org.

"More than half of the loss patterns that we've detected were not things that we were measuring in our evals." — Wendy Wang

About the team: This work was developed by Christopher Monnier, Wendy Wang, and Chuck Kwong, UX researchers on the Microsoft Copilot team. Together, they built and continue to refine an interactive evaluation method that brings real user tasks, side-by-side product comparisons, quantitative results, and qualitative feedback into one process. The team uses this work to identify where Copilot succeeds and where people run into issues, understand the reasons behind user preferences, and turn the findings into clear opportunities for product and prompt teams to improve the experience.
Christopher Monnier on LinkedIn: https://www.linkedin.com/in/christophermonnier/
Chuck Kwong on LinkedIn: https://www.linkedin.com/in/charleskwong/
Wendy Wang on LinkedIn: https://www.linkedin.com/in/wendy-wang-mertensmeyer-8386b018/
Microsoft Copilot: https://www.microsoft.com/en-us/microsoft-copilot
..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:
➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in pinpointing how to best leverage AI and improve the impact of your AI product
➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.