DataTalks.Club

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.

CHAPTERS
00:00 Podcast intro
00:08 Tejas's path to AI engineering
12:24 Why evals go beyond a manual check
17:37 Sourcing evaluation cases and calibrating a judge
28:33 Refreshing datasets and pinning severe cases
33:10 Judge agreement versus agent performance
50:01 Retraining judges and keeping data fresh
55:59 Human preferences, sample sizes and audience questions
59:46 Evaluating final answers versus the agent's trajectory
1:02:46 Faster judging and further evaluation resources

Resources
AI evals: https://tej.as/blog/ai-evals-llm-judge
Code, demos and slides: https://github.com/TejasQ/basically-agent-evals
Tejas: https://tej.as

Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw
Join DataTalks.Club: https://datatalks.club/slack.html

In this talk, Dewang Sultania, ML Engineer at Netflix and co-author of Designing AI Systems, shares his technical journey from building early inference platforms at Adobe to deploying multimodal AI models at scale. We explore the complete architecture of enterprise AI platforms, focusing on evaluation datasets, tool calling orchestration, and the practical realities of shipping production agents.

You’ll learn about:
- The essential components needed to build an enterprise AI platform from scratch
- How traditional software engineers can transition into AI platform engineering
- The critical technical differences between session management and LLM memory
- Strategies for building, versioning, and trusting custom evaluation datasets
- Optimizing LLM tool calling and selecting the right performance benchmarks
- When to use multi-agent architectures versus single-agent systems
- Techniques for overcoming the lost in the middle problem in RAG pipelines

TIMECODES:
00:00 Core Components of Enterprise AI Platforms
11:46 Career Path to AI Platform Engineering
18:33 Session Management Versus LLM Memory Management
26:47 Best Practices for Custom LLM Evaluation Datasets
32:28 LLM Tool Calling Optimization and Benchmark Selection
39:25 Context Window Limits and AI Agent Tool Scaling
45:37 Single-Agent Versus Multi-Agent System Architecture
52:16 LLM-as-a-Judge Alignment and Human Evaluation Strategies
58:11 Solutions for the Lost in the Middle Problem in RAG Systems
01:03:27 Designing AI Systems Book Announcement

This discussion is perfect for software engineers, machine learning practitioners, and technical leaders building real-world AI applications. It offers practical insights for anyone transitioning into AI platform engineering or tasked with scaling LLM features for enterprise production.

Connect with Dewang:
- Linkedin - https://www.linkedin.com/in/dewang-sultania/
- Github - https://github.com/iron-stark

Connect with DataTalks.Club:
- Join the community - https://datatalks.club/slack.html
- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
- Check other upcoming events - https://lu.ma/dtc-events
- GitHub: https://github.com/DataTalksClub
- LinkedIn - https://www.linkedin.com/company/datatalks-club/
- Twitter - https://twitter.com/DataTalksClub
- Website - https://datatalks.club/

In this talk, Suhas Suresha, Senior Machine Learning Engineer at Adobe and co-author of Designing AI Systems, shares his expertise in building large-scale generative AI infrastructure from deploying real-time ECG analysis models at his previous startup, QALY, to architecting enterprise LLM platforms. We explore the massive engineering effort required to take an AI application from a weekend prototype to a scalable production system, and how to build a unified AI platform that eliminates "AI sprawl."

Links:
- Book - https://www.manning.com/books/designing-ai-systems
- Code - https://github.com/designing-ai-systems/designing_ai_systems_repo

You’ll learn about:
- The nine core components every production-ready AI platform needs to succeed.
- How to solve "AI sprawl" and prevent fragmented engineering efforts across teams.
- Architecting scalable Data Services and Model Services for robust LLM applications.
- Implementing comprehensive observability and request tracing to identify system bottlenecks.
- Managing long-term session memory, tool registries, and complex agent orchestration.
- Navigating the "build versus buy" dilemma for enterprise AI infrastructure.

TIMECODES:
0:00 AI Prototypes to Production and the Book Origin
5:49 AI Sprawl and the Need for a Centralized AI Platform
12:42 Definition of an AI Platform and the Build vs Buy Dilemma
23:17 AI Platform Engineering Skills and Data Service Architecture
31:26 System Observability and Request Flow Bottlenecks
37:32 LLM Evaluations, Prompt Experimentation, and Cost Tracking
43:57 Session Memory, Tool Registries, and Agent Orchestration
50:56 Invocation Modes and Durable Execution for AI Agents
56:14 The Nine Core AI Platform Components and Memory Management

This episode is designed for machine learning engineers, backend developers, and platform architects looking to transition into AI platform engineering. It is highly recommended for technical leaders and teams who need to scale LLM prototypes into reliable, cost-effective production systems.

Connect with DataTalks.Club:
- Join the community - https://datatalks.club/slack.html
- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
- Check other upcoming events - https://lu.ma/dtc-events
- GitHub: https://github.com/DataTalksClub
- LinkedIn - https://www.linkedin.com/company/datatalks-club/
- Twitter - https://twitter.com/DataTalksClub
- Website - https://datatalks.club/

Connect with Suhas:
LinkedIn - https://www.linkedin.com/in/suhas-suresha/
Twitter - https://x.com/inceptor92
Github - https://github.com/suhas1992
Website - https://suhassuresha.com/

In this talk, Sahil Walia, Senior Technical Architect at Snowflake, shares his extensive data engineering expertise from writing simple SQL queries early in his career to architecting enterprise-scale Apache Iceberg lakehouses today. We explore the evolution of the modern data platform, the mechanics of open lakehouses, and how decoupling storage from compute is transforming data architecture.

Links:
- https://medium.com/@sahil-walia/
- Iceberg Spec- https://iceberg.apache.org/spec/
- https://ossie.apache.org
- https://github.com/Snowflake-Labs/schemachange
- https://opensource.guide/how-to-contribute/

You’ll learn about:
- What defines a modern data platform and the importance of interoperability.
- Key differences between traditional storage, Hive metastores, and Apache Iceberg.
- How Iceberg metadata simplifies GDPR compliance, data deletions, and file compactions.
- When a lakehouse architecture actually makes sense (and when to avoid it).
- Standardizing business logic across compute engines using Apache OSI.
- Actionable tips for making impactful open-source data engineering contributions.

TIMECODES:
00:00 Defining the Modern Data Platform
11:34 What Exactly is a Lakehouse?
17:20 Comparing Hive Metastore and Apache Iceberg
22:34 Handling GDPR and Data Deletions in Iceberg
27:33 Apache Iceberg vs. Delta Lake
34:10 Do You Need a Lakehouse for Small Data?
40:11 Understanding Apache OSI (Open Semantic Interchange)
49:00 Managing Schema Changes in Data Warehouses
54:20 How to Contribute to Open Source Projects
01:00:40 The Role of AI in Open Source Contributions

This talk is designed for data engineers, data architects, and analytics professionals looking to modernize their data infrastructure. It provides essential architectural insights for technical teams evaluating Apache Iceberg, dealing with large-scale data transformation, or aiming to scale their open-source contributions.

Connect with Sahil:
- Linkedin - https://www.linkedin.com/in/sahilwaliasyracuse
- Github - https://github.com/sahil-walia
- Website - https://medium.com/@sahil-walia

Connect with DataTalks.Club:
- Join the community - https://datatalks.club/slack.html
- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
- Check other upcoming events - https://lu.ma/dtc-events
- GitHub: https://github.com/DataTalksClub
- LinkedIn - https://www.linkedin.com/company/datatalks-club/
- Twitter - https://twitter.com/DataTalksClub
- Website - https://datatalks.club/

In this talk, Philip Christos, Associate at M2VC, shares his expertise in frontier AI investing from his early days as a data analyst to evaluating cutting-edge deep tech startups. We explore how AI agents are reshaping modern work, the realities of venture capital in the LLM era, and why the future of software engineering relies more on high-level systems thinking than writing syntax.You’ll learn about:- How venture capitalists formulate AI investment strategies and evaluate startup pitch decks.- The rapid adoption of AI in software engineering compared to traditional sectors like accounting.- The crucial difference between data progress and algorithmic progress in frontier models.- Why LLMs still struggle with high-level software architecture, business logic, and design "taste."- How AI is accelerating iterations and breakthroughs in classical engineering and scientific research.- What specific deep tech and AI infrastructure ideas are actually attracting VC funding today.TIMECODES:00:00 Venture Capital Career Journey05:14 VC Analyst vs Data Analyst Roles10:13 AI Investment Thesis Strategy16:00 Emotion Analytics and Voice AI Startups22:09 LLM Impact on Venture Capital28:19 Software Engineering AI Adoption34:18 Data vs Algorithmic AI Progress40:15 Classical Engineering AI Acceleration46:12 Software Architecture AI Limitations51:11 Future Software Engineering Careers57:55 Systems Thinking in DevOps1:03:00 Pitching Deep Tech StartupsThis talk is perfect for software engineers, AI founders, and technical product managers who want to understand the evolving tech job market and venture landscape. It provides essential insights for anyone looking to build VC-backed AI startups or future-proof their career as AI coding tools become ubiquitous.Connect with Philip- Linkedin - https://www.linkedin.com/in/philipchristos/- Twitter - https://x.com/phil_christos- Website - https://t.me/stacked_channelConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/