Product Impact Podcast | Secrets to unlocking the value of AI

Metalab has spent the last 20 years designing products for Windsurf, Midjourney, Suno, The Atlantic, and Uber. AI has made feature parity between products fast and cheap — anyone can ship the same baseline in a fraction of the time it used to take. Design Director Myles Palmer's answer to that isn't "taste." He thinks that word is a hollow way to dodge a much harder question.
Myles discusses what actually separates Metalab's AI-native design process from how most teams are building right now, why a vibe-coded prototype can create a dangerous false sense of confidence, and what design looks like once a product moves past the chat box.

In this episode we cover:
➜ Why Metalab's 20-year discipline, not any single tool, is what actually differentiates its AI-native design process.
➜ Myles Palmer on why "taste" is a hollow answer, and what actually differentiates once AI makes feature parity cheap.
➜ Brittany's own vibe-coded travel app cost thousands in API calls and still lost to Google — the false confidence trap.
➜ What founders actually pay for later when they skip design early, and why it defends against a "sea of average."
➜ What AI design looks like past the chat box — ambient transit, try-on shopping, and rethinking university sites.
➜ Digital twins, the 23andMe cautionary tale, and why Apple's health-data model is where consent design is headed.
.
.................
"I hate the idea of taste. The idea that taste is this differentiator drives me crazy because it's like, well, everybody's taste is different anyway." — Myles Palmer
"Investing early in design defends against that sea of average." — Myles Palmer
.................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built ⁠https://productimpactpod.com⁠ to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

.................
Myles Palmer is a Design Director at Metalab, where he leads and shapes meaningful products and experiences for both startups and global brands. Before joining Metalab, he founded and led Companion, a design studio built on strategic creativity and sustainable team culture. Myles also founded Pair Up, a platform connecting thousands of creatives for mentorship and peer learning. His work blends strategic thinking with human-centered design and thoughtful leadership that elevates product experiences.

LinkedIn: https://www.linkedin.com/in/myles-palmer-b1b70519/

Metalab Website: www.metalab.com
.
.................
Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:

➜ PH1 (https://ph1.ca) — Elevate your AI-Powered Product & Customer Experience. PH1 is the leading AI UX strategy consultant, leveraging behavioural science and model evaluations to gain you a competitive advantage. We've worked on innovative AI product discovery & corporate AI CX transformations.

➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Tech workers are so pessimistic about their own careers that most wouldn't recommend the field to a friend starting out. Eighty-two percent say AI made them more productive — but the bar resets every quarter, so the exhaustion never ends. Burnout hit 55.7%, up from 44.7% a year ago.

Noam Segal and Lenny Rachitsky's 2026 survey put a number on that pessimism: the industry's overall willingness to recommend a tech career is negative 39. Even directors land at 33% negative; founders, the most optimistic group surveyed, still score negative 5. How AI has changed your sense of yourself as a professional — Amplified, Redefined, Destabilized, Diminished, Unchanged — predicts that outlook better than role or seniority.

Glean's Work AI Index names the hidden cost behind the productivity number: "botsitting," the 6.4 hours a week the average worker spends supervising and correcting AI instead of it saving them time. One respondent put it bluntly: "My brain is rotting. My work feels worse." Another: "I just follow Claude. I don't understand what I merge."

Join  the new AI mentorship Slack community here: tinyurl.com/productimpactslack.

In this episode we cover:
➜ 82% of tech workers say AI makes them more productive; 55.7% say they're more burned out than a year ago.
➜ A 5-way AI identity split — Amplified to Unchanged — predicts career outlook better than role or seniority.
➜ The paradox of getting good at AI: the more confident you get, the more replaceable your output feels.
➜ Optimism climbs sharply from manager to director to VP to founder — revealing who AI adoption actually serves.
➜ Three postures — AI-Pilled, AI-Learning, AI-Governing — decide where research and adjacent roles go next.
➜ Why AI that's "impressive in a demo" keeps failing multi-turn users — a pattern this show keeps finding.

..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/
Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:
➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles.
➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

...........
Sources referenced in this episode:

Lenny's 2026 Tech Survey: AI Burnout Is Surging, Layoff Fear Is High — https://productimpactpod.com/news/lennys-2026-tech-survey-ai-burnout-layoff-fear
How tech workers are feeling in 2026 (original survey) — https://www.lennysnewsletter.com/p/how-tech-workers-are-feeling-in-2026
Writer's 2026 enterprise AI adoption survey — https://writer.com/blog/enterprise-ai-adoption-2026/
New Report Says You're Wasting More Time Botsitting Than Getting Value from AI — https://productimpactpod.com/news/botsitting-work-ai-index-2026/
The Future of UX Researchers — https://productimpactpod.com/news/future-of-ux-researchers
s02e16 — Moritz Sudhof on AI's invisible failures — https://open.spotify.com/episode/13d4ua1scY81zmQNQEkOXy
s02e17 — Microsoft Copilot's UXR team on UX Evals and loss patterns — https://open.spotify.com/episode/3V08ELNqgGyMpypYeSgfXP

In our last episode, Stanford NLP researcher Dr. Moritz Sudhof showed that 79% of AI product failures are invisible — they don't fire alerts, don't surface in telemetry, and don't get flagged by users, and they quietly erode trust and accelerate churn. This episode is the operational follow-up: the team that built a method for catching exactly that class of failure.

Microsoft's Copilot UX research team spent a year running evaluations on real conversations — every archetype, every industry, every use case — and found that more than half of their quality failures weren't in any eval they were running. At the world's most widely deployed enterprise AI product, with sophisticated engineering and testing infrastructure, standard evals were still missing the majority of what users actually experienced as failure.
That finding isn't limited to Copilot's scale. It's a structural gap in how the industry evaluates AI quality — and if you're running automated evals and calling that sufficient, the gap in your own product is almost certainly larger than you know.
In this episode we cover:
Token usage and adoption tell you if your AI is being used — not whether it's actually working for anyone.
Users bring real prompts, test one model fully, then compare — that's what produces honest signal at scale.
More than half of Copilot's loss patterns — user-driven gaps in model behavior — weren't in any existing eval.
LLM judges get you to baseline quality. Users catch what automated testing structurally cannot.
The flywheel: UXR evals → loss pattern taxonomy → log inspection → prompt changes → retention gains.
This team started with 10 users and one comparative question. Signal strong enough to scale to an entire org.

"More than half of the loss patterns that we've detected were not things that we were measuring in our evals." — Wendy Wang

About the team: This work was developed by Christopher Monnier, Wendy Wang, and Chuck Kwong, UX researchers on the Microsoft Copilot team. Together, they built and continue to refine an interactive evaluation method that brings real user tasks, side-by-side product comparisons, quantitative results, and qualitative feedback into one process. The team uses this work to identify where Copilot succeeds and where people run into issues, understand the reasons behind user preferences, and turn the findings into clear opportunities for product and prompt teams to improve the experience.
Christopher Monnier on LinkedIn: https://www.linkedin.com/in/christophermonnier/
Chuck Kwong on LinkedIn: https://www.linkedin.com/in/charleskwong/
Wendy Wang on LinkedIn: https://www.linkedin.com/in/wendy-wang-mertensmeyer-8386b018/
Microsoft Copilot: https://www.microsoft.com/en-us/microsoft-copilot
..................
If you found this episode useful, please like, share, and send it to anyone on your team who'd find it helpful.
We built https://productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.

Hosted by:
➜ Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
➜ Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:
➜ PH1 (https://ph1.ca) — a strategy & research consultancy specialized in pinpointing how to best leverage AI and improve the impact of your AI product
➜ AI Value Acceleration (https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

KPMG pulled a report this year after its team accepted hallucinated information without question. Latham & Watkins submitted a court filing built on fabricated legal citations an AI invented, delivered with total confidence and perfect formatting. McDonald's AI ordering chatbot became a public failure for the same underlying reason: the system looked like it was working. Dr. Moritz Sudhof, CEO of Bigspin AI, analyzed 100,000 real conversations between users and live AI systems with Stanford NLP Group's Chris Potts and found why: 79% of AI failures are invisible to standard monitoring because they are behavioral failures, not technical ones.

Sudhof built this research on hard-won experience. As VP of AI at BetterUp, he shipped an AI coach that beat ChatGPT on every expert coaching benchmark and still lost users — until his team changed nothing but how the AI introduced itself, and outcomes doubled. That gap between what an eval measures and what actually happens in the room with a user became his research question, and then his company.

In this episode:
The Confidence Trap: AI states something false, dressed in precise numbers and total confidence.
Silent Mismatch: when the AI can't finish a task, it quietly answers a different question instead.
The Drift and the Death Spiral: how a long conversation loses the plot and burns the user's patience.
Models are post-trained to answer, not clarify — a default that gets worse as models get more capable.
The Paradox of AI Fluency: the users who push back hardest also hit the most failures, and succeed most.
Evals catch what you already know to test for. Reading real transcripts catches what you don't.

"The behavioral layer is where most of the damage is actually happening." — Moritz Sudhof
"The more specific and helpful it often gets, the more fake it often is." — Moritz Sudhof, on the Confidence Trap

About Moritz: Shipped conversational AI to hundreds of organizations as VP of AI at BetterUp. Lived the problem of knowing something's wrong but not what to fix.
Guest resources:
LinkedIn: https://www.linkedin.com/in/sudhof/
X: https://x.com/mmooritz
Personal site: https://msudhof.com/
Bigspin website: https://bigspin.ai
Key research:
Invisible Failures in Human–AI Interactions: https://arxiv.org/abs/2603.15423
A paradox of AI fluency: https://arxiv.org/abs/2604.25905

We built productimpactpod.com to be your AI product insights and strategic playbook hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.

Hosted by:
Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/

Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by:
PH1 (https://ph1.ca) — a strategy & research consultancy specialized in delivering evidence about the highest value use cases and customer profiles. AI Value Acceleration
(https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.

Four data reports from 2026 tell a consistent story, and none of it matches the adoption narrative. Writer surveyed 2,400 global workers and C-suite leaders: 97% of executives deployed AI agents in the past twelve months, 29% reported significant ROI. Glean's Work AI Index found a name for what most knowledge workers are actually experiencing: botsitting — spending more time supervising and correcting AI than gaining anything back. Section's biannual proficiency survey: 67% of workers use AI weekly, 5.5% are proficient enough to generate consistent value, and 79% of managers haven't demonstrated their own AI use to their team in the past month. Token consumption per organization grew roughly 320 times in twelve months while the share of organizations reporting significant ROI stayed at 29%.
Brittany Hobbs and Arpy Dragffy work through what's causing the gap, why the teams trying hardest to close it keep hitting structural walls, and what it takes to move from measuring adoption to generating defensible value — for individual contributors, for teams, and for the organizations responsible for this investment.

What you'll learn:
OpenAI, Writer, Glean, Section: four reports, one consistent signal — the adoption story hides the value failure.
Glean 2026: botsitting is the dominant AI experience. More knowledge workers are losing time to AI than gaining it.
67% of workers use AI weekly. Only 5.5% are proficient. The problem isn't more training days. It's the model of change.
Four years of measuring seats over outcomes has left AI leaders unable to defend their budgets. The window is closing.
Salesforce agreed to acquire Fin for $3.6B. What they built before that exit is the lesson most orgs are ignoring.
Boris Cherny no longer prompts — he builds loops. What that means for every team not yet running autonomous evaluation.

Articles referenced in this episode:
97% of Executives Deployed AI Agents. Only 29% See ROI. — Brittany's breakdown of the Writer 2026 survey and the 68-point deployment-to-value gap
The 10% Problem: AI's Value Gap Is Wider Than Anyone Is Admitting — Why AI value is concentrating at the top and what it means for the rest of the organization
WTF is an AI-native org anyways? Let's compare Airbnb & Meta's opposing plans. — The competing models for AI-native organization design
OpenAI & Anthropic are charging us way more than we need — Arpy on token economics, model selection, and the cost side of AI value creation

We built productimpactpod.com to be your AI product strategy and AI product news hub. Check it out.
Thank you for listening to the Product Impact Podcast — if you have feedback, guest recommendations, or want to chat — contact us.
Hosted by:
Arpy Dragffy Guerrero — https://www.linkedin.com/in/adragffy/
Brittany Hobbs — https://www.linkedin.com/in/brittanyhobbs/
Go to Substack to get AI strategy frameworks, news, and jobs: https://productimpactpod.substack.com

This episode was brought to you by: PH1 (https://ph1.ca) — an AI strategy consultancy specialized in improving the measurable success of AI products. AI Value Acceleration
(https://aivalueacceleration.com) — The consultancy specialising in enterprise value creation. Make sure that your spending doesn't go to waste. Find out exactly where the value creation of adopting AI products stalls.