The Agile Embedded Podcast

Linux Profiling with Mohammed Billoo

We sit down with Mohammed Billoo, founder of Mab Labs and author of the Embedded Linux Essentials Handbook, to explore the world of embedded Linux profiling and optimization. Mohammed shares hard-won lessons from the field, including debugging a scientific instrument that mysteriously crashed after 60-minute runs and optimizing a sophisticated MANET platform that took a 20% throughput hit.

The conversation reveals a fundamental truth: in embedded Linux, the CPU is rarely the bottleneck. Mohammed walks us through his systematic approach to performance problems, starting with simple tools like HTOP before diving into specialized instrumentation. We discuss the critical difference between VM size and VM RSS for memory analysis, why dumping console output can kill boot times, and how to leverage kernel configurations for maximum diagnostic bang-for-buck. Mohammed emphasizes the importance of building instrumentation into systems from day one—not for premature optimization, but to give your future self the data needed when problems inevitably surface. The discussion also touches on how LLMs can accelerate the learning curve for complex tools like Valgrind and perf, while stressing that physical reality remains the ultimate arbiter of system performance.

Key Topics

[03:15] The surface area problem: why embedded Linux profiling requires a tool chest, not just a toolbox

[06:30] Case study: debugging a scientific instrument that crashed after 60-minute runs

[08:45] VM size vs. VM RSS: understanding the critical difference in memory analysis

[14:20] Why the CPU is rarely the bottleneck: coprocessors, DMA, and crypto engines

[18:50] Essential kernel configurations: function tracer, perf, and config kallsyms

[24:10] File system bottlenecks: moving from CSV files to SQLite for data integrity

[28:40] Boot time optimization: why console output is one of the biggest time sinks

[32:15] Premature optimization vs. smart instrumentation: building in diagnostic capability from day one

[38:25] Leveraging LLMs for visualization and analysis of perf data and Valgrind output

[43:50] The first five commands: starting with HTOP and working down to specialized tools

Notable Quotes

"When you first get started, you have generally this arrogance that like, oh, it works fine. I've tested it. It's good to go. But then as you get more experience, as you become a more senior-level engineer, that arrogance, you start to kind of strip away a lot of that arrogance. You get humbled pretty quickly." — Mohammed Billoo

"The CPU is very rarely the bottleneck because it's meant to, and the drivers are implemented in Linux in such a way that they're intelligent enough that they can hand off a lot of the things of the CPU to coprocessors so that the CPU is really idle." — Mohammed Billoo

"I don't convince myself of a claim that I'm making until I have data to back it up. So I don't say, oh, you know, this is working fine. Like, well, again, what does fine mean? Or, you know, what does well mean? And what is the data to prove that?" — Mohammed Billoo

Resources Mentioned

HTOP - Interactive process viewer for Linux - Mohammed's first tool for getting a high-level view of system performance

perf - Linux profiling tool with performance counters - requires kernel configuration to enable

LTTng - Linux Trace Toolkit Next Generation - provides visibility across both user space and kernel space

Valgrind - Memory debugging and profiling tool for detecting memory leaks

iperf - Network throughput measurement tool with server and client components

GStreamer - Multimedia framework with built-in tools for per-frame timestamp analysis

Tracealyzer - Visualization tool for LTTng and other performance data

SQLite - Embedded database recommended for data integrity over CSV files in embedded systems

Embedded Linux Essentials Handbook - Mohammed Billoo's book published by Packt

Mab Labs - Mohammed Billoo's embedded solutions consultancy with blog on embedded Linux topics

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.
Want to join the agile Embedded Slack? Click here
Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Requirements Engineering Fundamentals - Part 1

We kick off a multi-part series on requirements engineering by exploring what requirements actually are and why they matter - even for Agilists. Jeff shares his medical device expertise while Luca brings his automotive and aerospace background to discuss the different levels of requirements (from high-level user needs to testable system requirements), the importance of traceability, and why proper tooling beats Word and Excel every time.

We dig into practical aspects like the EARS format for writing requirements, the crucial distinction between requirements and design choices, and why glossaries aren't as boring as they sound. Along the way, we tackle the tension between regulatory compliance and actual engineering value, emphasizing that documentation should be an artifact of diligent work - not the work itself. Whether you're in safety-critical industries or just want to build better products, understanding requirements engineering helps manage complexity and prevent costly mistakes.

Key Topics

[02:30] What is requirements engineering and why it matters beyond safety-critical industries

[06:45] Don't Agilists hate requirements? Debunking the myth and discussing iteration vs. waterfall

[11:20] The hierarchy of requirements: user needs, system requirements, and subsystem requirements

[18:00] Requirements vs. design choices: where to draw the line and why it matters for testing

[24:15] Writing good requirements: EARS format, must vs. shall vs. may, and the value of glossaries

[32:40] Traceability: linking requirements across levels and to test cases

[40:30] Why Word and Excel don't cut it: the case for proper requirements management tools

[48:20] Risk analysis and mitigation in safety-critical development

[52:00] Documentation as artifact of diligent work, not the work itself

Notable Quotes

"The whole agile movement was a reaction to the one time through the requirements specification build test loop that took several years. By the time you got to the end, the requirements no longer applied." — Jeff

"Do not use an LLM to manage requirements. Do use the LLM to write tools that help you manage requirements." — Luca

"I view any medical device that I work on as if it's going to be used on my child. What do I need to do to convince myself that it is safe and effective? Once I have done that, if there are remaining boxes to check to get it through FDA, I will check those boxes." — Jeff

Resources Mentioned

EARS (Easy Approach to Requirements Syntax) - A grammar format for writing clear, verifiable requirements that constrains how requirements are written to reduce ambiguity

FDA Guidance on Agile Development - Regulatory guidance describing how to do Agile development in medical device context

ISO 26262 - Automotive safety standard mentioned as having similar traceability requirements to medical devices

DO-178B - Aerospace software safety standard with similar requirements engineering principles

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.
Want to join the agile Embedded Slack? Click here
Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We talk with Tobias Kästner, a physicist-turned-software-architect and technical consultant at Inovex, about his journey from painfully slow hardware-software integration cycles to achieving three-week hardware sprints. Tobias shares hard-won lessons from medical device development, where fuzzy requirements and constant feedback from life scientists forced his team to rethink traditional approaches.

The conversation centers on practical techniques: breaking monolithic PCB designs into modular "feature boards" connected via shields (think Arduino-style), using Git for hardware version control with SHA-1s printed on silkscreens, and leveraging tools like Zephyr RTOS to enable plug-and-play firmware that matches the modularity of the hardware. Tobias explains how relaxing constraints like board size and using automation to merge schematics allowed his team to iterate rapidly while maintaining a clear path to final form-factor designs. We discuss how this approach scaled to projects with 120+ people across multiple teams, and why the interplay between system architecture, organizational structure, and information flow matters more than most realize.

Key Topics

[02:30] The painful reality of traditional hardware development: six-month wait for hardware, nine months of debugging

[08:00] Breaking apart monolithic PCB designs into modular feature boards with shield connectors

[12:45] Relaxing constraints: larger board areas, autorouting, and prioritizing testability over final form factor

[18:20] Version control for hardware: putting schematics in Git and printing SHA-1s on silkscreens

[22:00] Using automation to merge feature board schematics into final form-factor designs

[26:15] Firmware architecture: NuttX, Zephyr, KConfig, and device trees for modular, plug-and-play software

[35:40] Scaling agile hardware-software co-development to 120+ person projects across multiple teams

[39:00] The interplay of system architecture, organizational architecture, and information architecture

Notable Quotes

"When the board arrived, not a single line of code had been written for it because no one had been able to touch it. It took us nine additional months to debug all the things out of it." — Tobias Kästner

"I've never seen any board working the first time. I've never seen any prototype without thin wires patching things out, but that's maybe a different story." — Tobias Kästner

"We cannot think these architectures as independent of one another. If we have limitations in two of these architectures, we will see these limitations in the third architecture as well." — Tobias Kästner

Resources Mentioned

Inovex - Tobias's company offering engineering consulting services, trainings, and expertise in embedded systems, IoT, and full-stack development

Zephyr RTOS - Open-source real-time operating system with KConfig, device tree support, and extensive driver library that Tobias recommends for modular firmware development

NuttX RTOS - Apache Foundation RTOS with clean device driver model and KConfig support that Tobias used in earlier projects

KiCad - Open-source PCB design software with emerging Python API support for schematic automation

Services and Contact

Through Inovex, Tobias provides trainings for both Zephyr and Yocto Linux, as well as consultancy and engineering support for embedded projects -- from 1-2 day workshops evaluating architectural state and cost/benefit analysis, to first prototypes, to full-fledged software development. With partners such as alpha-board (Berlin) and Blunk electronic (Erfurt), they also offer agile hardware services and help teams get started with the methods discussed in this episode.

Email: [email protected]

Tobias Kästner on LinkedIn

tobiaskaestner on the Zephyr Discord Channel

Links

Companies:

Inovex -- Embedded Systems

Blunk electronic

alpha-board

Navimatix

Talks and publications:

Modular and Agile HW Development (2018 talk)

Leveraging Zephyr's HW Abstraction for Agile Systems Engineering (2023 talk)

Whitepaper: Agile in der Hardware -- by Gregor Gross, Christoph Schmiedinger, and Tobias Kästner

Leveraging Zephyr for Functional Architecture Decomposition (2025 talk)

Books recommended by Tobias:

Small Groups as Complex Systems -- Holly Arrow et al., SAGE Publications

The Dao of Complexity -- Jean Boulton, DeGruyter

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.
Want to join the agile Embedded Slack? Click here
Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

We explore how test-driven development (TDD) remains essential—perhaps more than ever—when working with AI coding tools. Luca shares his evolved workflow using Claude Code, breaking down how he structures tests in three phases: test ideas, test outlines, and test implementations. We discuss why TDD provides the necessary control and confidence when AI generates code, how it prevents technical debt accumulation, and why tests serve as precise specifications for AI rather than afterthoughts.

The conversation covers practical challenges like AI's tendency toward "success theater" (overly generous assertions), the importance of maintaining tight control over code quality, and why the bottleneck in AI-assisted development isn't code generation—it's expressing clear intent. We also touch on code spikes, large-scale refactorings, and why treating AI development as pair programming keeps you in the driver's seat. If you're wondering whether TDD still matters when AI writes your code, this episode makes a compelling case that it matters more than ever.

Key Topics

[02:30] Why TDD still matters with AI: confidence and control over generated code

[06:45] Tests as specifications: describing desired behavior to AI rather than writing prompts

[09:20] The three-phase test workflow: test ideas, test outlines, and implementations

[15:30] Pair programming with AI: staying at the conceptual level while AI handles implementation

[20:15] Code spikes and exploration: using AI to answer questions before writing production tests

[24:40] AI failure modes: over-mocking and "success theater" with weak assertions

[28:50] Large-scale refactorings: how AI excels at updating hundreds of tests simultaneously

[32:10] The real bottleneck: expressing intent and specifications, not code generation speed

Notable Quotes

"As far as I am concerned, test-driven development is just about writing prompts for the AI that it can then use to build what you want it to build." — Luca

"If you expect that a five-line prompt resulting in 10,000 lines of code will not result in 9,995 lines of uncertainty, you're just deluding yourself." — Luca

"You can be five times faster than you were before and still maintain a very high production level quality code, but you probably can't be a hundred times faster." — Jeff

Resources Mentioned

Claude Code - Terminal-based AI coding assistant that Luca uses for TDD workflows, keeping conceptual work separate from code-level work

Embedded AI Podcast - Luca's separate podcast focusing on AI in embedded systems, co-hosted with Ryan Torvik

Luca's AI Training Courses - Hands-on trainings for using AI in embedded systems development (and much more!)

links to all of Luca's work - Training, consulting, podcasts, conference talks and everything else

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.
Want to join the agile Embedded Slack? Click here
Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this second part of our series on engineering organizations, Jeff and Luca explore how companies that build products should focus their efforts differently depending on their stage and scope. We start with startups and early-stage companies desperately searching for product-market fit, where the brutal truth is: quality doesn't matter yet. Your MVP should embarrass you—if it doesn't, you waited too long. We discuss the critical mental shift from throwaway prototypes to proper engineering once validation arrives, and why technical founders often fail by solving the wrong problem brilliantly.

Moving up the ladder, we examine narrow-focus companies that have found their niche—like the German firm that does nothing but maintain a 100-year-old anchor chain machine, or specialists in medium-power electrical switches. These companies win through efficiency and deep expertise, but face existential risk if the market shifts. Finally, we tackle wide-focus companies introducing multiple product lines, where the challenge becomes running internal startups while managing established products, each requiring radically different approaches. The key insight: your focus must match your product's lifecycle stage, whether that's ruthless speed, cost optimization, or high-level process learning.

Key Topics

[02:30] Startups and early-stage companies: the existential search for product-market fit

[06:45] The MVP philosophy: if you're not embarrassed, you waited too long

[11:20] Quality vs. speed vs. scope: why quality doesn't matter in early stages

[15:40] The Potemkin village approach: building facades to validate demand

[19:15] Embedded products and MVPs: when physical products need creative shortcuts

[23:50] The critical switch: from prototypes to proper engineering after validation

[28:30] Narrow-focus companies: German hidden champions and deep specialization

[34:10] Wide-focus companies: running internal startups within established organizations

[40:25] Product teams and parallel focuses: managing different lifecycle stages simultaneously

[45:00] Large established companies: high-level process learning and avoiding organizational weight

Notable Quotes

"If you read the Lean Startup, they will explicitly say: if you weren't embarrassed by your MVP, you waited too long. It really has to be painfully flimsy because you cannot afford to do it well." — Luca

"Quality doesn't even factor because you're very explicitly building mock-ups from chewing gum and paper mache. They are fully intended to be thrown away." — Luca

"Getting that product-market fit is existential. You will die if you do not get it and get it relatively quickly." — Jeff

Resources Mentioned

The Lean Startup - Eric Ries' book discussing MVP philosophy and the importance of being embarrassed by your first product

The Mom Test - Rob Fitzpatrick's book about getting real customer feedback and validation through financial commitment

The Art of Innovation - Tom Kelley's book on IDEO's design process, including the clothespin switch story

Luca's Website - Trainings on embedded agile, AI in embedded systems, and more

Jeff's Website - Consulting services for medical device software development

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.
Want to join the agile Embedded Slack? Click here
Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/