Machine Learning Street Talk (MLST)

• Tau Language: The Software Synthesis Future (sponsored)

  This sponsored episode features mathematician Ohad Asor discussing logical approaches to AI, focusing on the limitations of machine learning and introducing the Tau language for software development and blockchain tech. Asor argues that machine learning cannot guarantee correctness. Tau allows logical specification of software requirements, automatically creating provably correct implementations with potential to revolutionize distributed systems. The discussion highlights program synthesis, software updates, and applications in finance and governance.SPONSOR MESSAGES:***Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/***TRANSCRIPT + RESEARCH:https://www.dropbox.com/scl/fi/t849j6v1juk3gc15g4rsy/TAU.pdf?rlkey=hh11h2mhog3ncdbeapbzpzctc&dl=0Tau:https://tau.net/Tau Language:https://tau.ai/tau-language/Research:https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdfTOC:1. Machine Learning Foundations and Limitations [00:00:00] 1.1 Fundamental Limitations of Machine Learning and PAC Learning Theory [00:04:50] 1.2 Transductive Learning and the Three Curses of Machine Learning [00:08:57] 1.3 Language, Reality, and AI System Design [00:12:58] 1.4 Program Synthesis and Formal Verification Approaches2. Logical Programming Architecture [00:31:55] 2.1 Safe AI Development Requirements [00:32:05] 2.2 Self-Referential Language Architecture [00:32:50] 2.3 Boolean Algebra and Logical Foundations [00:37:52] 2.4 SAT Solvers and Complexity Challenges [00:44:30] 2.5 Program Synthesis and Specification [00:47:39] 2.6 Overcoming Tarski's Undefinability with Boolean Algebra [00:56:05] 2.7 Tau Language Implementation and User Control3. Blockchain-Based Software Governance [01:09:10] 3.1 User Control and Software Governance Mechanisms [01:18:27] 3.2 Tau's Blockchain Architecture and Meta-Programming Capabilities [01:21:43] 3.3 Development Status and Token Implementation [01:24:52] 3.4 Consensus Building and Opinion Mapping System [01:35:29] 3.5 Automation and Financial ApplicationsCORE REFS (more in pinned comment):[00:03:45] PAC (Probably Approximately Correct) Learning framework, Leslie Valianthttps://en.wikipedia.org/wiki/Probably_approximately_correct_learning[00:06:10] Boolean Satisfiability Problem (SAT), Varioushttps://en.wikipedia.org/wiki/Boolean_satisfiability_problem[00:13:55] Knowledge as Justified True Belief (JTB), Matthias Steuphttps://plato.stanford.edu/entries/epistemology/[00:17:50] Wittgenstein's concept of the limits of language, Ludwig Wittgensteinhttps://plato.stanford.edu/entries/wittgenstein/[00:21:25] Boolean algebras, Ohad Osorhttps://tau.net/tau-language-research/[00:26:10] The Halting Problemhttps://plato.stanford.edu/entries/turing-machine/#HaltProb[00:30:25] Alfred Tarski (1901-1983), Mario Gómez-Torrentehttps://plato.stanford.edu/entries/tarski/[00:41:50] DPLLhttps://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf[00:49:50] Tarski's undefinability theorem (1936), Alfred Tarskihttps://plato.stanford.edu/entries/tarski-truth/[00:51:45] Boolean Algebra mathematical foundations, J. Donald Monkhttps://plato.stanford.edu/entries/boolalg-math/[01:02:35] Belief Revision Theory and AGM Postulates, Sven Ove Hanssonhttps://plato.stanford.edu/entries/logic-belief-revision/[01:05:35] Quantifier elimination in atomless boolean algebra, H. Jerome Keislerhttps://people.math.wisc.edu/~hkeisler/random.pdf[01:08:35] Quantifier elimination in Tau language specification, Ohad Asorhttps://tau.ai/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf[01:11:50] Tau Net blockchain platformhttps://tau.net/[01:19:20] Tau blockchain's innovative approach treating blockchain code itself as a contracthttps://tau.net/Whitepaper.pdf

  --------  

  1:41:19
• John Palazza - Vice President of Global Sales @ CentML ( sponsored)

  John Palazza from CentML joins us in this sponsored interview to discuss the critical importance of infrastructure optimization in the age of Large Language Models and Generative AI. We explore how enterprises can transition from the innovation phase to production and scale, highlighting the significance of efficient GPU utilization and cost management. The conversation covers the open-source versus proprietary model debate, the rise of AI agents, and the need for platform independence to avoid vendor lock-in, as well as emerging trends in AI infrastructure and the pivotal role of strategic partnerships.SPONSOR MESSAGES:***CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!https://centml.ai/pricing/Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/***TRANSCRIPT:https://www.dropbox.com/scl/fi/dnjsygrgdgq5ng5fdlfjg/JOHNPALAZZA.pdf?rlkey=hl9wyydi9mj077rbg5acdmo3a&dl=0John Palazza:Vice President of Global Sales @ CentMLhttps://www.linkedin.com/in/john-p-b34655/TOC:1. Enterprise AI Organization and Strategy [00:00:00] 1.1 Organizational Structure and ML Ownership [00:02:59] 1.2 Infrastructure Efficiency and GPU Utilization [00:07:59] 1.3 Platform Centralization vs Team Autonomy [00:11:32] 1.4 Enterprise AI Adoption Strategy and Leadership2. MLOps Infrastructure and Resource Management [00:15:08] 2.1 Technology Evolution and Enterprise Integration [00:19:10] 2.2 Enterprise MLOps Platform Development [00:22:15] 2.3 AI Interface Evolution and Agent-Based Solutions [00:25:47] 2.4 CentML's Infrastructure Solutions [00:30:00] 2.5 Workload Abstraction and Resource Allocation3. LLM Infrastructure Optimization and Independence [00:33:10] 3.1 GPU Optimization and Cost Efficiency [00:36:47] 3.2 AI Efficiency and Innovation Challenges [00:41:40] 3.3 Cloud Provider Strategy and Infrastructure Control [00:46:52] 3.4 Platform Independence and Vendor Lock-in [00:50:53] 3.5 Technical Innovation and Growth StrategyREFS:[00:01:25] Apple Acquires GraphLab, Apple Inc.https://techcrunch.com/2016/08/05/apple-acquires-turi-a-machine-learning-company/[00:03:50] Bain Tech Report 2024, Gartnerhttps://www.bain.com/insights/topics/technology-report/[00:04:50] PaaS vs IaaS Efficiency, Gartnerhttps://www.gartner.com/en/newsroom/press-releases/2024-11-19-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-total-723-billion-dollars-in-2025[00:14:55] Fashion Quote, Oscar Wildehttps://www.amazon.com/Complete-Works-Oscar-Wilde-Collins/dp/0007144369[00:15:30] PointCast Network, PointCast Inc.https://en.wikipedia.org/wiki/Push_technology[00:18:05] AI Bain Report, Bain & Companyhttps://www.bain.com/insights/how-generative-ai-changes-the-game-in-tech-services-tech-report-2024/[00:20:40] Uber Michelangelo, Uber Engineering Teamhttps://www.uber.com/en-SE/blog/michelangelo-machine-learning-platform/[00:20:50] Algorithmia Acquisition, DataRobothttps://www.datarobot.com/newsroom/press/datarobot-is-acquiring-algorithmia-enhancing-leading-mlops-architecture-for-the-enterprise/[00:22:55] Fine Tuning vs RAG, Heydar Soudani, Evangelos Kanoulas & Faegheh Hasibi.https://arxiv.org/html/2403.01432v2[00:24:40] LLM Agent Survey, Lei Wang et al.https://arxiv.org/abs/2308.11432[00:26:30] CentML CServe, CentMLhttps://docs.centml.ai/apps/llm[00:29:15] CentML Snowflake, Snowflakehttps://www.snowflake.com/en/engineering-blog/optimize-llms-with-llama-snowflake-ai-stack/[00:30:15] NVIDIA H100 GPU, NVIDIAhttps://www.nvidia.com/en-us/data-center/h100/[00:33:25] CentML\'s 60% savings, CentMLhttps://centml.ai/platform/

  --------  

  54:50
• Transformers Need Glasses! - Federico Barbero

  Federico Barbero (DeepMind/Oxford) is the lead author of "Transformers Need Glasses!". Have you ever wondered why LLMs struggle with seemingly simple tasks like counting or copying long strings of text? We break down the theoretical reasons behind these failures, revealing architectural bottlenecks and the challenges of maintaining information fidelity across extended contexts.Federico explains how these issues are rooted in the transformer's design, drawing parallels to over-squashing in graph neural networks and detailing how the softmax function limits sharp decision-making.But it's not all bad news! Discover practical "glasses" that can help transformers see more clearly, from simple input modifications to architectural tweaks.SPONSOR MESSAGES:***CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!https://centml.ai/pricing/Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/***https://federicobarbero.com/TRANSCRIPT + RESEARCH:https://www.dropbox.com/s/h7ys83ztwktqjje/Federico.pdf?dl=0TOC:1. Transformer Limitations: Token Detection & Representation[00:00:00] 1.1 Transformers fail at single token detection[00:02:45] 1.2 Representation collapse in transformers[00:03:21] 1.3 Experiment: LLMs fail at copying last tokens[00:18:00] 1.4 Attention sharpness limitations in transformers2. Transformer Limitations: Information Flow & Quantization[00:18:50] 2.1 Unidirectional information mixing[00:18:50] 2.2 Unidirectional information flow towards sequence beginning in transformers[00:21:50] 2.3 Diagonal attention heads as expensive no-ops in LAMA/Gemma[00:27:14] 2.4 Sequence entropy affects transformer model distinguishability[00:30:36] 2.5 Quantization limitations lead to information loss & representational collapse[00:38:34] 2.6 LLMs use subitizing as opposed to counting algorithms3. Transformers and the Nature of Reasoning[00:40:30] 3.1 Turing completeness conditions in transformers[00:43:23] 3.2 Transformers struggle with sequential tasks[00:45:50] 3.3 Windowed attention as solution to information compression[00:51:04] 3.4 Chess engines: mechanical computation vs creative reasoning[01:00:35] 3.5 Epistemic foraging introducedREFS:[00:01:05] Transformers Need Glasses!, Barbero et al.https://proceedings.neurips.cc/paper_files/paper/2024/file/b1d35561c4a4a0e0b6012b2af531e149-Paper-Conference.pdf[00:05:30] Softmax is Not Enough, Veličković et al.https://arxiv.org/abs/2410.01104[00:11:30] Adv Alg Lecture 15, Chawlahttps://pages.cs.wisc.edu/~shuchi/courses/787-F09/scribe-notes/lec15.pdf[00:15:05] Graph Attention Networks, Veličkovićhttps://arxiv.org/abs/1710.10903[00:19:15] Extract Training Data, Carlini et al.https://arxiv.org/pdf/2311.17035[00:31:30] 1-bit LLMs, Ma et al.https://arxiv.org/abs/2402.17764[00:38:35] LLMs Solve Math, Nikankin et al.https://arxiv.org/html/2410.21272v1[00:38:45] Subitizing, Railohttps://link.springer.com/10.1007/978-1-4419-1428-6_578[00:43:25] NN & Chomsky Hierarchy, Delétang et al.https://arxiv.org/abs/2207.02098[00:51:05] Measure of Intelligence, Chollethttps://arxiv.org/abs/1911.01547[00:52:10] AlphaZero, Silver et al.https://pubmed.ncbi.nlm.nih.gov/30523106/[00:55:10] Golden Gate Claude, Anthropichttps://www.anthropic.com/news/golden-gate-claude[00:56:40] Chess Positions, Chase & Simonhttps://www.sciencedirect.com/science/article/abs/pii/0010028573900042[01:00:35] Epistemic Foraging, Fristonhttps://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2016.00056/full

  --------  

  1:00:54
• Sakana AI - Chris Lu, Robert Tjarko Lange, Cong Lu

  We speak with Sakana AI, who are building nature-inspired methods that could fundamentally transform how we develop AI systems.The guests include Chris Lu, a researcher who recently completed his DPhil at Oxford University under Prof. Jakob Foerster's supervision, where he focused on meta-learning and multi-agent systems. Chris is the first author of the DiscoPOP paper, which demonstrates how language models can discover and design better training algorithms. Also joining is Robert Tjarko Lange, a founding member of Sakana AI who specializes in evolutionary algorithms and large language models. Robert leads research at the intersection of evolutionary computation and foundation models, and is completing his PhD at TU Berlin on evolutionary meta-learning. The discussion also features Cong Lu, currently a Research Scientist at Google DeepMind's Open-Endedness team, who previously helped develop The AI Scientist and Intelligent Go-Explore.SPONSOR MESSAGES:***CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!https://centml.ai/pricing/Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/**** DiscoPOP - A framework where language models discover their own optimization algorithms* EvoLLM - Using language models as evolution strategies for optimizationThe AI Scientist - A fully automated system that conducts scientific research end-to-end* Neural Attention Memory Models (NAMMs) - Evolved memory systems that make transformers both faster and more accurateTRANSCRIPT + REFS:https://www.dropbox.com/scl/fi/gflcyvnujp8cl7zlv3v9d/Sakana.pdf?rlkey=woaoo82943170jd4yyi2he71c&dl=0Robert Tjarko Langehttps://roberttlange.com/Chris Luhttps://chrislu.page/Cong Luhttps://www.conglu.co.uk/Sakanahttps://sakana.ai/blog/TOC:1. LLMs for Algorithm Generation and Optimization [00:00:00] 1.1 LLMs generating algorithms for training other LLMs [00:04:00] 1.2 Evolutionary black-box optim using neural network loss parameterization [00:11:50] 1.3 DiscoPOP: Non-convex loss function for noisy data [00:20:45] 1.4 External entropy Injection for preventing Model collapse [00:26:25] 1.5 LLMs for black-box optimization using abstract numerical sequences2. Model Learning and Generalization [00:31:05] 2.1 Fine-tuning on teacher algorithm trajectories [00:31:30] 2.2 Transformers learning gradient descent [00:33:00] 2.3 LLM tokenization biases towards specific numbers [00:34:50] 2.4 LLMs as evolution strategies for black box optimization [00:38:05] 2.5 DiscoPOP: LLMs discovering novel optimization algorithms3. AI Agents and System Architectures [00:51:30] 3.1 ARC challenge: Induction vs. transformer approaches [00:54:35] 3.2 LangChain / modular agent components [00:57:50] 3.3 Debate improves LLM truthfulness [01:00:55] 3.4 Time limits controlling AI agent systems [01:03:00] 3.5 Gemini: Million-token context enables flatter hierarchies [01:04:05] 3.6 Agents follow own interest gradients [01:09:50] 3.7 Go-Explore algorithm: archive-based exploration [01:11:05] 3.8 Foundation models for interesting state discovery [01:13:00] 3.9 LLMs leverage prior game knowledge4. AI for Scientific Discovery and Human Alignment [01:17:45] 4.1 Encoding Alignment & Aesthetics via Reward Functions [01:20:00] 4.2 AI Scientist: Automated Open-Ended Scientific Discovery [01:24:15] 4.3 DiscoPOP: LLM for Preference Optimization Algorithms [01:28:30] 4.4 Balancing AI Knowledge with Human Understanding [01:33:55] 4.5 AI-Driven Conferences and Paper Review

  --------  

  1:37:54
• Clement Bonnet - Can Latent Program Networks Solve Abstract Reasoning?

  Clement Bonnet discusses his novel approach to the ARC (Abstraction and Reasoning Corpus) challenge. Unlike approaches that rely on fine-tuning LLMs or generating samples at inference time, Clement's method encodes input-output pairs into a latent space, optimizes this representation with a search algorithm, and decodes outputs for new inputs. This end-to-end architecture uses a VAE loss, including reconstruction and prior losses. SPONSOR MESSAGES:***CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!https://centml.ai/pricing/Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/***TRANSCRIPT + RESEARCH OVERVIEW:https://www.dropbox.com/scl/fi/j7m0gaz1126y594gswtma/CLEMMLST.pdf?rlkey=y5qvwq2er5nchbcibm07rcfpq&dl=0Clem and Matthew-https://www.linkedin.com/in/clement-bonnet16/https://github.com/clement-bonnethttps://mvmacfarlane.github.io/TOC1. LPN Fundamentals [00:00:00] 1.1 Introduction to ARC Benchmark and LPN Overview [00:05:05] 1.2 Neural Networks' Challenges with ARC and Program Synthesis [00:06:55] 1.3 Induction vs Transduction in Machine Learning2. LPN Architecture and Latent Space [00:11:50] 2.1 LPN Architecture and Latent Space Implementation [00:16:25] 2.2 LPN Latent Space Encoding and VAE Architecture [00:20:25] 2.3 Gradient-Based Search Training Strategy [00:23:39] 2.4 LPN Model Architecture and Implementation Details3. Implementation and Scaling [00:27:34] 3.1 Training Data Generation and re-ARC Framework [00:31:28] 3.2 Limitations of Latent Space and Multi-Thread Search [00:34:43] 3.3 Program Composition and Computational Graph Architecture4. Advanced Concepts and Future Directions [00:45:09] 4.1 AI Creativity and Program Synthesis Approaches [00:49:47] 4.2 Scaling and Interpretability in Latent Space ModelsREFS[00:00:05] ARC benchmark, Chollethttps://arxiv.org/abs/2412.04604[00:02:10] Latent Program Spaces, Bonnet, Macfarlanehttps://arxiv.org/abs/2411.08706[00:07:45] Kevin Ellis work on program generationhttps://www.cs.cornell.edu/~ellisk/[00:08:45] Induction vs transduction in abstract reasoning, Li et al.https://arxiv.org/abs/2411.02272[00:17:40] VAEs, Kingma, Wellinghttps://arxiv.org/abs/1312.6114[00:27:50] re-ARC, Hodelhttps://github.com/michaelhodel/re-arc[00:29:40] Grid size in ARC tasks, Chollethttps://github.com/fchollet/ARC-AGI[00:33:00] Critique of deep learning, Marcushttps://arxiv.org/vc/arxiv/papers/2002/2002.06177v1.pdf

  --------  

  51:26

Meer Technologie podcasts

Trending Technologie -podcasts

Over Machine Learning Street Talk (MLST)