Latent Space: The AI Engineer Podcast

We’ve been on a bit of a mini World Models series over the last quarter: from introducing the topic with Yi Tay, to exploring Marble with World Labs’ Fei-Fei Li and Justin Johnson, to previewing World Models learned from massive gaming datasets with General Intuition’s Pim de Witte (who has now written down their approach to World Models with Not Boring), to discussing the Cosmos World Model with with Andrew White of Edison Scientific on our new Science pod, to writing up our own theses on Adversarial World Models. Meanwhile Nvidia, Waymo and Tesla have published their own approaches, Google has released Genie 3, and Yann LeCun has raised $1B for AMI and published LeWorldModel.
Today’s guests have a radically different approach to World Modeling to every player we just mentioned — while Genie 3 is impressive, its many flaws demonstrate the issues with their approach - terrain clipping, noninteractivity (single player, no physics/no objects other than the player move), and maximum of 60 second immersion.
Moonlake AI (inspired by the Dreamworks logo) is the diametric opposite - immediately multiplayer, incredibly interactive, indefinite lifetime, capable of MANY different kinds of world models by simulating environments, predicting outcomes, and planning over long horizons. This is enabled by bootstrapping from game engines and training custom agents:
In Towards Efficient World Models, Chris Manning and Ian Goodfellow join Fan-Yun in explaining why their approach to efficiency with structure and casuality instead of just blind scaling is sorely needed:
SOTA models still show physical or spatial understanding glitches, such as solid objects floating in mid-air or moving “inside” other solid objects.
If the goal is to plan for the next action, how often is a high-resolution pixel view necessary for modeling the world? Our bet is that there is a disproportionately large share of economically valuable tasks where such detail is not required. After all, humans with a wide variety of sensory limitations have little difficulty doing almost everything in the world. Furthermore, for a large number of purposes, describing a scene or a situation in a few words of language (“the car’s tires squealed as it cornered sharply”) is sufficient for understanding and planning.
Experiments also show that humans only partially process visual input in a top-down, task-directed way, often making use of abstracted object-level modeling. In almost all cases, partial representations combined with semantic understanding are sufficient.
…If the goal is to facilitate the understanding of causality in multimodal environments, then the world model—whether it is used in the virtual world or the physical world—must prioritize properties such as spatial and physical state consistency maintained over long time periods, and an ability to evolve the world that accurately reflects the consequences of actions. That’s what Moonlake is building.
Game engines are the right starting point abstraction to efficiently extract causal relationships, and building the interfaces and community (including their new $30,000 Creator Cup) to kickstart the flywheel of actions-to-observations.
We were fortunate enough to attend their sessions at GDC 2026 (the Mecca of Game Devs), and were impressed by the huge variety and flexibility of the worlds people were building with Moonlake’s tools already! Live videos on the pod.

Full Video Pod on YouTube!
Timestamps
00:00 Benchmarking Gets Hard00:47 Meet Moonlake Founders01:26 Why Build World Models03:12 Structure Not Just Scale05:37 Defining Action Conditioned Worlds07:32 Abstraction Versus Bitter Lesson14:39 Language Versus JEPA Debate20:27 Reasoning Traces And Rendering Layer37:00 Gameplay Over Graphics38:02 Fiction Rules And World Tweaks39:15 Code Engines Beat Learned Priors41:10 Diffusion Scaling Limits43:23 Symbolic Versus Diffusion Boundary46:14 Platform Vision Beyond Games50:24 Spatial Audio And Multimodal Latents54:23 NLP Roots Hiring And Moon Lake Name

Transcript
[00:00:00] Cold Open
[00:00:00] Chris Manning: Think this whole space is extremely difficult as things are emerging now. And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks.
[00:00:20] But these days so much of what people are wanting to do is nothing like that, right? You’re wanting to get some recommendations about which backpack would be best for you for your trip in Europe next month. It’s not so easy to come up with a benchmark, and it’s the same problem with these world models.
[00:00:41] Meet the Founders
[00:00:41] swyx: Okay. We’re back in the studio with Moon Lake’s, two leads. I, I guess there’s other founders as well, but, sun and Chris Manning. Welcome to the studio.
[00:00:54] Fan-yun Sun: Thanks. Thanks, Chris. Thanks for having us.
[00:00:56] swyx: You’ve got, you guys have, come burst onto the scene with a really refreshing [00:01:00] new take of mold models.
[00:01:01] I would just want to, I guess ask how you, the two of you came together. Chris, you’re a legend in NLP and just AI in, in, in general. You’re, you’re his grad student, I guess
[00:01:10] Fan-yun Sun: Actually my co-founder.
[00:01:11] swyx: Oh, yeah.
[00:01:12] Fan-yun Sun: I should give a lot of credit to my co-founder, Sharon. Yeah. She was, she was actually working with Professor Fe Androgyn and then she ended up working with, Ron and Chris Manning here.
[00:01:22] And then, so I got connected through to Chris initially, actually through my co-founder,
[00:01:26] What is Moon Lake?
[00:01:26] swyx: what is Moon Lake? What, what is, actually, I’m also very curious about the name, but like why going into world models?
[00:01:33] Fan-yun Sun: So I was working a lot. With actually Nvidia research during my PhD years on essentially generating interactive worlds to train reinforcement learning agents or embody EA agents.
[00:01:44] And then there’s two observations. One in academia and one in industry. An industry like folks at Nvidia are actually paying a lot of dollars to purchase these types of interactive worlds, whether it’s for the sake of evaluation or training the robots, or policies or models. And [00:02:00] then, in academia, same thing is happening.
[00:02:02] And more specifically, when I was actually working with Nvidia on the synthetic data foundation model training project, we were actually generating a lot of these synthetic data and showing that, hey, you can actually, these synthetic data are actually as useful as real world data when it comes to multimodal pre-training.
[00:02:16] But then, like I said, there’s a lot of dollars being paid out to like external vendors or, or like. Other folks to manually curate these types of data. It was very clear to us that, okay, on our way to, let’s call it embody general intelligence models need to learn the consequences behind their actions, which means that they need interactive data and the demand for those types of data are growing exponentially.
[00:02:38] But everybody’s sort of thinking about it from a pure, say, video generation perspective or something else. But we feel like the true actually opportunity is actually building reasoning models that can do these things, like how humans do these things today. So that’s a little bit on the genesis of Moon Lake, and I think the reason I got into world models was partly.
[00:02:59] A philosophical [00:03:00] take of the on the world where I like, believe the simulation theory and stuff like that. But on the other, on the other hand, it’s really just like, oh, like there’s an opportunity there that I feel like nobody’s doing it the way I think should be done.
[00:03:10] Structure, Not Scale: The Vision
[00:03:10] Chris Manning: I can say a little bit about that.
[00:03:12] Yeah. So of the overall goal is the pursuit of artificial intelligence and most of my career has been doing that in the language space and that’s been just extremely productive. As we all know, the story of the last few years, I don’t have to tell about how much we’ve achieved with large language models, but, uh.
[00:03:31] Although they have been extremely effective for ramping language and general intelligence, it’s clearly not the whole world. There’s this multimodal world of vision, sound, taste that you’d like to be dealing with more than just, language. And then the question is how to do it. And despite, a huge investment in the computer vision space, right, as the research field computer [00:04:00] vision has been for decades, far, far larger than the language space, actually.
[00:04:05] I think it’s fair. Say that, vision, understanding sort of stalled out, right? You got to object recognition and then progress just wasn’t being made right? If you look at any of these, vision language models, it’s the language that’s doing 90% of the work and the vision barely works. And so there’s really an interesting research question as to why that is and at heart, the ideas behind Moon Lake are an attempt to answer that, believing that there can be a really rich connection between a more symbolic layer of abstracted understanding of visual domains, which aren’t in the mainstream vision models, which are still trying to operate on the surface level of pixels.
[00:04:50] swyx: I think one of your blog posts, you put it as structure, not scale. Is that, a general thesis?
[00:04:57] Chris Manning: Yeah. Well, scale is good too.
[00:04:58] swyx: Yeah. Scale is good. Too
[00:04:59] lot,
[00:04:59] Chris Manning: [00:05:00] lots of data is good as well and scale, but nevertheless, you want the structure Yeah. To be able to much more efficiently learn.
[00:05:07] swyx: Yeah. The other thing I really liked also is you put out an example of what your kind of reasoning traces look like.
[00:05:12] Right. Which you would distill is the word that comes to mind. I don’t even think that’s a good, good description, but it would involve, for example, geometry, physics, affordances, symbolic logic, perceptual mappings, and what, what have you. But like that, that is the kind of example that involves, let’s call it spatial reasoning, role model reasoning as as compared to normal LM reasoning.
[00:05:35] Yeah.
[00:05:36] Defining World Models vs Video Generation
[00:05:36] Vibhu: But also like taking it a step back. So how do you guys define world models? A lot of people see okay, you can do diffusion, you can do video generation. But, you guys put out quite a few blog posts. You put out a essay recently, we can even pull it up about efficient world models. You have a pretty like structural definition here, but for the general audience that don’t super follow the space, right.
[00:05:55] What’s, what’s the difference in what we see from like a video generation model to [00:06:00] a world gen A simulator? How do you kind of paint that last
[00:06:02] Chris Manning: year? Yeah, so I think this is actually a little bit subtle because, people look at these amazing generative AI video models, SAWA VO three, one of these things, and they think Genie, they think, oh, this is amazing.
[00:06:17] This is we’ve solved understanding the world because you can produce these generative AI videos, but. The reality is that although the visuals do look fantastic, those visuals actually are accompanied by an understanding of the 3D world, understanding how objects can move, what the consequences of different actions are, and that’s what’s really needed for spatial intelligence.
[00:06:49] So I mean, a term we sometimes use is that you need action condition, world models. That you only actually have a world model if you can predict, [00:07:00] given some action is taken, what is going to change in the world because of it. And in particular, that becomes hard over longer time scales. So if you’re simply, trying to.
[00:07:12] Predict the next video frame. That’s not so difficult. But what you actually want to do is understand the consequences, likely consequences of actions minutes into the future. And to do that, you actually much more of an abstracted semantic model of the world.
[00:07:32] The Bitter Lesson & Data Abstraction
[00:07:32] swyx: Yeah, the question comes where you want to have more structure than is available in just predicting the next token.
[00:07:41] And typically, well, let’s, let’s call it the experience of the last five years has been that is just washed away by scale, right? So what is the right middle ground here that, you don’t ignore the bitter lesson, but also you. Can be more efficient than what we’re doing today.
[00:07:57] Chris Manning: One possibility [00:08:00] is, look, if we just collect masses and masses and masses and masses of video data, this problem will be solved.
[00:08:11] Under certain assumptions that could be true, but there are sort of multiple avenues in which it could not be true. The first is what’s really essential is understanding the, the consequences of actions producing an action conditioned world model. And if you are simply, collecting observational video data, which is the easy stuff to collect, when you’re sort of mining online videos, you don’t actually.
[00:08:41] Know the actions that are being taken to see how the video is changing. And so if you are never collecting directly actions and you are having to try and infer them from what happened in the observed video, that’s not impossible. But it’s very [00:09:00] hard and it’s not really established that you can get that to work at any scale yet.
[00:09:05] And so there’s a lot of premium on collecting action condition video data, which is part of why there’s been a lot of interest in using simulation so that you can be collecting data where you do know the actions, which isn’t quite limited supply, but there’s also in the limit of as much data as you could possibly have.
[00:09:28] Maybe the problem is eventually solvable, but. Even though we collect huge amounts of text data is always at a great level of abstraction, right? Language is a human designed, abstracted representation where there’s meaning in each token and it’s representing and abstraction of the world, right?
[00:09:51] As soon as you are describing someone as a professor, and as soon as you are saying that they’re condescending, right? These are very [00:10:00] abstracted descriptions of the world. It’s not at what you’re observing as pixel level, and to get to that kind of degree of abstraction, starting from pixels is orders and magnitude of extra data and processing.
[00:10:14] And so, although, we absolutely want to exploit, get as much data as possible, use the bitter lesson. Nevertheless, if there are ways in which you can work with five orders of magnitude less data than people working purely from pixels, you’re gonna be able to make a lot more progress, a lot more quickly.
[00:10:34] And that’s the bet here. And so you could just say that’s only wanting to be able to, do it more efficiently, do it more quickly, do it more cheaply. But I think it’s actually more than that, I think. One should be making the analogy to how human beings work at one level. You know? Yes, we have these high [00:11:00] resolution eyes and we can look and see a scene like a video, but all of the evidence from neuroscience and psychology is that most of what comes into people’s eyes is never processed.
[00:11:13] Right. That you are doing fairly fine ated processing of exactly what you’re focusing on. But as soon as it’s away from that of yeah, there’s another guy over there that you’ve sort of only processing top down this very abstracted semantic description of the world around you. And so, that’s what human beings are doing.
[00:11:33] They’re working with semantic abstractions and so. I think it is just the right representation. ‘cause we also have other goals we want to be able to do, real time worlds. So that means there’s a limit to how much processing you can do and we want to do long-term planning and consistency. And again, that favors abstraction.
[00:11:55] I mean, I guess there was actually a recent. Blog posts that [00:12:00] came out from our Friends of physical intelligence and, they were sort of heading in the same direction they were saying Oh, to the pay
[00:12:06] swyx: pay model.
[00:12:07] Chris Manning: Yeah. Yeah. To maintain a long term memory of what’s happening in the world. So we can, do longer term we actually storing text of what is, been happening in the world.
[00:12:19] Right. It is not such a successful strategy of trying to keep it all at a pixel level.
[00:12:24] Vibhu: And yeah, I mean, you can see it in video models like that Temporal consistency. We’re at a scale of train on, all the video data we have. We have it for maybe 30 seconds, a few minutes. That’s not the same as a game state played for half an hour.
[00:12:37] Right. I thought you guys break it down pretty well. You have a, you have a blog post about. Building multimodal worlds with an agent. I dunno if you guys wanna talk about this. This is one of the things I read, I
[00:12:48] swyx: thought, yeah, it’s the thing I talked about with the reasoning chain. Yeah.
[00:12:51] Vibhu: So there’s like different phases to this.
[00:12:53] It seems like it’s more of an agent, a scaffold, very different approach than just, type in a prompt and you, you don’t have the same consistency. [00:13:00] It also, like, for people that are listening, I, I would highly recommend reading it. It breaks down the problem in a different light, right?
[00:13:06] So like, what do you need to consider when you’re talking about video, like world game models, right? How would, what do you need to consider? What are the factors? What are the elements? What’s the state? So I don’t know if you guys have stuff to talk about for this one.
[00:13:19] Fan-yun Sun: Yeah. Actually, I wanted to add on a little bit Yeah.
[00:13:22] On our previous point, which is just like, change topics so quickly. I, I do feel like sometimes people confuse like, oh, like we’re taking an an, an method with abstraction. That means they don’t believe in bitter lesson. Like that’s just false, right? Like we are believed is a bitter lesson. But then I feel like the question that we always discuss is like, what is the right abstraction level today?
[00:13:42] The analogy I like to make is like, let’s just say we can encode and decode. Represent all of images, videos, audio and bytes. Then the most bitter lesson approached is to train a next byte prediction model as opposed to the next token prediction model where it’s just like, okay, it’s natively multimodal, can just, but it’s like, yeah, like [00:14:00] to, to Chris’s point, it’s like the scale and computing you need to achieve that.
[00:14:03] So that’s why we always come back to like, okay, what is the most efficient way to do it? And reasoning models to the point of this blog post is a showcase of like, Hey, we’re actually just like reasoning about the world and reasoning about. The aspects of the world that CAGR that matter for me to learn what I want to learn from this role model.
[00:14:21] swyx: Yeah, it’s like you’re improving the en encoder of whatever you’re, trying to model. And like a better representation would just represent the important things in less space. Yeah. Which would just be more efficient.
[00:14:33] Fan-yun Sun: Yeah.
[00:14:34] swyx: So yeah, I, I, I fully agree that it is not, antagonistic to, bitter lesson.
[00:14:38] I do wanna wanna mention one more thing. Is there any philosophical differences with the JPA stuff that, Yun is working on? I gotta go there. You, you, you, you’re, you’re imagining like some latent abstraction. I’m like, okay, fine. Let’s, let’s talk about it, right? Like it’s an elephant in the room.
[00:14:52] Chris Manning: Yeah.
[00:14:53] JEPA & Philosophical Differences with LeCun
[00:14:53] Chris Manning: There are philosophical differences. Jan Lacoon is a dear friend of mine, but. [00:15:00] He has never appreciated the power of language in particular, or symbolic representations in general. Yarn is a very visual thinker. He always wants to claim that he thinks visually and there are no words, symbols, or math in his head.
[00:15:21] Maybe that’s true of yarn. It’s certainly not the way I think. Um. But at any rate, the world according to yarn is the basic stuff of the, the world and of intelligence is visual and language is just. This low bit rate communication mechanism between humans and it doesn’t have much other utility and it’s far inferior to the high bit rate video, that comes into your eyes.
[00:15:53] And I think he’s fundamentally missing a number of important things [00:16:00] there. Think of this evolutionary argument looking at animals, right? That the closest analogies, the things with chimps, right? So chimpanzees, have fairly similar brains to human beings. They have great vision systems, they have great memory systems.
[00:16:18] They’ve got, better memory than we do of short term memories. They can plan, they can build primitive tools that, humans. Massively ahead in what we understand about the world, what we can plan, what we can build. And essentially what took off for us was that humans managed to develop language and that gave a symbolic knowledge, representation, and reasoning level, which just, okay if this sort of vaulting of what could be done with the intelligence in brains.
[00:16:59] So the [00:17:00] philosopher Dan de refers to language as a cognitive tool and argues that, humans unique among the creatures in the world have managed to build their own cognitive tools and language is the famous first example. But other things like, mathematics and programming languages are also cognitive tools.
[00:17:21] They give you an ability to. Think in abstractions, in extended causal reasoning chains. And that allows you to do much more. And we use that for spatial representation and intelligence and planning and gameplay as well. So we believe, and this is, underlying the specific technologies that Moon Lake is making, that symbolic representations are powerful.
[00:17:50] And you want to use that in your understanding of the visual world when you want a causal understanding, when you want to maintain long-term [00:18:00] consistency and prediction. And as I understand it, that’s just not in ya Koon’s worldview. So I think that’s the fundamental philosophical difference. Then there’s the specific model.
[00:18:11] He’s been advancing jpa, that’s a reasonable. Research bed is a direction as to, to head for building out a model of the visual world. To my mind, it’s sort of one reasonable research bed. It’s not really established. It’s the best one that everyone should be following,
[00:18:32] swyx: at least developed at scale, at Meta.
[00:18:34] But it’s not just vision, right? Like, I mean, JPA is a, just joint admitting prediction can be applied to anything really. And people have done it. The argument is that there is a latent representation or that is probably more. Suited to the task, then why not let machines do it for us instead of predefining it at all?
[00:18:50] And isn’t something like a JPA shaped thing the right answer? And if not, why not?
[00:18:55] Chris Manning: So I think there’s a part of jpa that’s right, which is [00:19:00] you do want to have a joint. Embedding that gives you a consistent model of the world. And Jan’s argument is you can never get that from auto aggressive language models ‘cause they’re sort of left to right churning out one token at a time.
[00:19:22] I guess this is where we’re the research arguments of the field, I’m not actually convinced that’s right. ‘cause although the token production is this auto aggressive, process that’s heading, left to right, I guess don’t have to be left to right. But anyway, in sequence of tokens we could have right to left Arabic.
[00:19:40] But although that’s true, all of the weights of the model that are internal to the transformer, they are a joint model of the model’s understanding of the world. And so I think you can think of the weights of the model as a form of. Joint representation, [00:20:00] and therefore it is plausible to think that could be the basis of a world model, which avoids, ya’s objections.
[00:20:10] swyx: I think I follow, and obviously that would touch on what Moon Lake eventually ends up doing as well. Right. Like, which it’s hard to tell because you put out the end results, but we don’t know the inputs that go into it. So it’s, it’s, that’s something that we have to figure out over time.
[00:20:25] Vibhu: Yeah. I mean, I guess this kind of breaks down some of the outputs. Do you wanna walk us through it?
[00:20:31] Reasoning Traces & Interactive Worlds
[00:20:31] Fan-yun Sun: Yeah. So this, this really just walks us through the reasoning traces of like, okay. So that just say, if we wanna build a world in this context, it’s really just a game demo that, that shows the, the variety of interactions that this world model can build.
[00:20:45] And yeah, it’s really just a reasoning traces of like, okay it prompted to create a bowling game. Like how did it achieve what you saw? That level of causality, interaction and consistency, right? So yeah, this is almost just like a, an example of [00:21:00] like a reasoning traces. Very
[00:21:01] swyx: detailed.
[00:21:01] Fan-yun Sun: Yeah.
[00:21:01] Vibhu: Very, very detailed.
[00:21:02] You gotta you don’t even realize it, right? Like when a video is generated, what happens when a ball strikes a pin, right? So first, like you, there’s audio in that, like audio triggers happens, score increments, the world changes. Like pins have to start dropping. There’s a timer that goes on. It’s just like very similar to how now we’re used to reasoning for language models.
[00:21:20] There’s a whole state of what happens. So geometry, physics, all this stuff. And then yeah, there’s kind of that single prompt. So asset, ation all this stuff. It’s like a, it’s a nice view to see what’s going on.
[00:21:32] swyx: I think Sun is also too polite to point out that, both like Google’s genie, demos as well as world Labs is marble, do not have interactive worlds.
[00:21:41] Fan-yun Sun: That’s the benefit of having a reasoning model, right? Like, because you can, you can say, oh, like maybe in this particular context, I want to learn how to bowl. And then you can say, okay, then what is it important when it comes to learning how to bowl? Okay, maybe it’s like I need to understand the, the basic of like, physics and I want to throw it over [00:22:00] them.
[00:22:00] I wanna know that when I, when it resets it’s a new game. So I know that yeah, basically, you know to pick up the ball, you know that ball’s gonna cause the pins to fall down. You know that what’s important to this particular bowling game is to score and you know that the score corresponds to the number of pins that fell down.
[00:22:19] So it’s just like, if it’s a model that sort of knows what it. Looks like, knows what a bowling game looks like, but doesn’t actually allows you to practice over and over again and to understand that, oh, like what it takes to actually get a high score. Then it sort of doesn’t actually allow you to learn what you set out to learn within the world model.
[00:22:38] And I think this is really just one example of showing like the advantages of the approach that we’re taking over most the, let’s call it the zeitgeist, is today, when people talk about clinical role models,
[00:22:51] Chris Manning: right? So it sort of seems like the question to ask when there’s a world model is.
[00:22:58] Can I not [00:23:00] only just wander around the world and look at the beautiful graphics, can I interact with the objects in the world and see the right consequences of actions?
[00:23:11] Vibhu: And you also understand what the consequences would be if you do something right. So it’s not just like, okay, there’s one thing if I pick it up, something will happen.
[00:23:19] But, there’s 50 options and I know I can expect, I can infer what would happen if I do any of them. Right. So very different when you can actually see it play around with it.
[00:23:28] swyx: There,
[00:23:28] Beyond Unity: Cognitive Tools for World Building
[00:23:31] swyx: there’s two cheeky elements of that. I mean, the, the, the I guess, less ambitious one is, let’s really establish for listeners, why is this fundamentally different than writing Unity code, right?
[00:23:40] Like just creating a model to translate a prompt into Unity code
[00:23:44] Fan-yun Sun: so there is an underlying physics engine. Yeah. In that sense, there’s some overlapping things to Unity, but the way we think about it is like physics engine. Tools or code are cognitive tools like borrowing Chris’s term, right? Like tools [00:24:00] that the model can employ as means to an end.
[00:24:04] So today maybe you say, okay, in this particular context we care about physics, we care about the long-term causality consequences. Then yes, we deploy it, employ physics engine, and then maybe tomorrow we say, okay, we’re we’re training that. Just say drones where we only care about really fluid dynamics and the visual aspect of the world.
[00:24:25] Then, then yeah, maybe we don’t actually, the model actually doesn’t have to use a physics engine. Or maybe it employs other types of representation or physics engine to achieve the task. So yes, writing code for Unity is sort of similar to a tool that our A model can employ, but our goal is for a model to take a representation conditioned reasoning.
[00:24:46] Approach or process.
[00:24:47] swyx: Yeah,
[00:24:47] Fan-yun Sun: internally.
[00:24:48] swyx: Yeah. Using these things as just like general two calls. Right. Which I think is very interesting. The other more ambitious one is, some kind of recursive element where it becomes multiplayer, right? Like here, there’s a single player element, you’re not [00:25:00] modeling any other people involved.
[00:25:01] And that is a whole other thing.
[00:25:04] Fan-yun Sun: But in fact, we can really do multiplayers. Oh yeah, okay. I haven’t seen any double situations. So just actually just like prompt our, our model to say, Hey, like configure to multiplayer. Then it’ll do like this. You’ll be able to configure multiplayer
[00:25:16] swyx: great
[00:25:17] Fan-yun Sun: persistency database for you.
[00:25:18] Easy. Yeah.
[00:25:19] Vibhu: So what, what are like some of the current limitations in where we’re at? So there’s one approach of like, okay, scale up video predictors. Obviously there’s data issues. With approaches like this, is it data constraints? What are like the next steps? Is it real time? Like, so there’s one side of, write an agent to write Unity code, but okay, I want to be streaming a game real time.
[00:25:38] I want to have characters being also like agent, but where, where do we kinda see this scaling up? Right?
[00:25:44] Fan-yun Sun: Yeah, there’s definitely a data constraint. Like the more data, the, the better. This reasoning model can almost basically act as humans to like operate a variety of tools and softwares to build whatever’s necessary.
[00:25:57] And then there’s a sort [00:26:00] of fidelity constraint, which we’re actually solving with another model, which we can talk about later. But it’s like, it’s not as easy to get to photorealism with the approach that we’re taking. But we think there are better solutions to that, which is we can dive into later.
[00:26:14] Later.
[00:26:15] Vibhu: The one one thing you note here is it’s a diffusion model, right? So there’s, there’s a few approaches, diffusion caution, splatting, yeah, so Ry diffusion model, you guys wanna
[00:26:25] Fan-yun Sun: Yeah.
[00:26:25] Vibhu: Introduce,
[00:26:26] Fan-yun Sun: yeah, totally.
[00:26:26] Rie: Neural Rendering & Skins for Worlds
[00:26:26] Fan-yun Sun: So within our world modeling framework, we think there are two models that we train, right?
[00:26:31] Like, there’s the multimodal reasoning model that we just talked about that essentially handles. Mainly the, the causality, the persistency and logic determinism of the world. And then RY is our bet on saying, okay, like while all those model, can take care of all these things that we just talked about, it’s limitations compared to existing, say, video models, is that it doesn’t have as high of a pixel [00:27:00] ality right off the gate, right?
[00:27:02] And EE is to say, Hey, we can actually take whatever persistent representation that we generate with our multimodal reasoning model and learn to restyle it into photo photorealistic styles or arbitrary styles you want. So this model is almost to say, Hey, I’m going to respect the persistency and interactivity of the world that you created, but my only job is to make sure that its pixel distribution is close to what we want.
[00:27:29] Vibhu: Yeah.
[00:27:30] swyx: Great example right there. You kept the KL divergence.
[00:27:33] Fan-yun Sun: Oh. Where,
[00:27:34] swyx: no, no. I mean this, this is a, a classic like, how you don’t stray too far from the source material as you, you kept the kl, which is Oh yeah. Kind of cool. Yeah.
[00:27:43] Fan-yun Sun: Yeah.
[00:27:44] swyx: I mean, and the
[00:27:44] Chris Manning: difference is, and I mean sun was pointing at this, where sort of saying it’s in one way a more difficult path, but a better path that, typically the diffusion models are producing the whole scene and it looks lovely, [00:28:00] but there isn’t spatial understanding behind it, which is allowing for the real time graphics gameplay, the spatial intelligence, understanding the consequences of worlds where this is, taking a path where it is assuming an abstracted semantic model of the world’s state.
[00:28:20] And then the diffusion model is then being used on top of that to produce the high quality graphics.
[00:28:27] swyx: Is there an intended practical, or business use for this, or is it like a, like a demonstration of capabilities?
[00:28:34] Fan-yun Sun: We actually believe that this is gonna be the next paradigm of rendering. So it’s gonna replace how ra raizer, it’s gonna replace DLSS today because it not only has these pixel prior that’s learned from the world such that you can literally play any game in photo realistic styles, which is a lot of people’s desire when they do GTA, right?
[00:28:51] Like,
[00:28:51] Vibhu: all the mods, all the people adding perfect lighting and all this.
[00:28:54] swyx: So
[00:28:54] Fan-yun Sun: skins
[00:28:55] swyx: for worlds, let’s call it
[00:28:56] Fan-yun Sun: skins, let’s call it skin for worlds. I,
[00:28:58] Vibhu: it’s also like, you can call it skin, you can call it [00:29:00] customization. You can play it how you want, right?
[00:29:01] Fan-yun Sun: Yeah, exactly. And I think another thing that we really pointed out specific specifically in this blog is the programmability of it, right?
[00:29:09] So what this means is that this render historically render is always a derivative of the game state, right? You’re saying, oh, here’s the game state, I’m rendering out a frame. But here I’m saying actually this render can be part of the gameplay loop. I can say something along the lines of, if upon getting 10.
[00:29:26] Apples, I’m gonna, my weapon of choice, my bullet’s gonna turn into apples. And that’s, that’s possible because we can say, we can basically dynamically have certain game state trigger the, the preconditions to the render such that the rendering is now part of the game loop too. One thing is to just say, okay, it’s, it’s, it’s the appearance.
[00:29:47] But the second thing is also to say there’s these novel interactions that are possible because this render now has actually priors of the world.
[00:29:57] swyx: It is up to the artist to figure out what to do with it.
[00:29:59] Fan-yun Sun: It [00:30:00] is up to the creators. Yes.
[00:30:01] swyx: Yeah.
[00:30:01] Fan-yun Sun: And I also think that’s actually another big argument that we’re making and the reason that we’re picking, taking the bet we’re baking is that a lot of the times, whether it’s for embody AI gaming, like you want a layer where human can inject their intentions.
[00:30:15] So, for example, let’s just say in the context of gaming, it’s obviously like my creative intent, but maybe in the context of embodied ai, it’s like, oh, like I take this foundational policy and I want to actually fine tune it to deploy in my house. So you want to almost say, inject, have a layer where human can say, oh, here’s the distribution of things I want to create to achieve my goal.
[00:30:35] And I think 3D graphics as it as it is today, is basic, the layer for people to say, Hey, what do I care about in this world? And it allows, basically human intent to be expressed in these worlds much more explicitly and distributionally as opposed to just saying, Hey, I’m gonna generate like, arbitrary.
[00:30:54] And it’s like just prompts,
[00:30:55] swyx: it’s one of those things where like, I think you, you’re going to build up a series of models, right? [00:31:00] This is just one of, this is probably like the highest utility or heaviest, frequency one, I don’t dunno what to call this. Where like you Yeah. You can immediately drop this in on any game and you don’t need anything else that.
[00:31:10] That you guys do. But, I, I could see, I could see that I think the, the human intent is something that people are not even used to because we’re so used to static worlds or, worlds that just don’t react, or, I don’t know. It’s, it, you’re kind of blowing my mind right now with like, I’m, I wonder if you’ve talked to people at GDC Hmm.
[00:31:27] And what are they gonna do with it?
[00:31:30] Fan-yun Sun: Yeah. Now the stance that we take on this front is like, we’re not gonna be more creative than our users to ship
[00:31:35] swyx: it out.
[00:31:35] Fan-yun Sun: Yeah. But we wanna make sure that we’re building things in a way that really allows them to express their intent.
[00:31:41] swyx: The thing that you said about, here’s the distribution that I want.
[00:31:45] I think text may be too low of a bandwidth to. To really demonstrate, because I, I, there, I’m, I’m probably just gonna want to drop in a bunch of, reference assets and then you can figure it out from
[00:31:58] Vibhu: there. But you probably wanna do a, a mixture of [00:32:00] both, right? Like you throw in a few images. I wanted this style.
[00:32:02] Yeah. I want it to look like this. So it, it’s, it’s a mixture, right?
[00:32:05] Chris Manning: I, I think it’s a mixture. I mean, yeah, I mean there’s clearly a visual component of this, and it’s not that, everything can be text. ‘cause of course you want to give a visual look, but there’s also a massive amount of giving the overall picture of the look of the world and the behavior of things that you can express in a few words of text.
[00:32:32] And it be very time consuming and difficult to do via visual means. So I think, yeah, you want a combination of both.
[00:32:40] Evaluating World Models
[00:32:40] Vibhu: So one question I kind of have is, how do we go about evaluating world models? So like, there’s many axes, right? One is like, okay. I have preferences. How well do we adhere to prompts? One is the simulation.
[00:32:50] One is like do things, is there core logic that’s broken? So coming from we know how to evaluate diffusion, there’s fidelity, there’s [00:33:00] stuff like that. But what are some of the challenges that most people probably aren’t thinking about?
[00:33:04] Fan-yun Sun: Yeah, I think this is like a great question and probably one of the hardest questions in role models because like, I think it always comes back to what are you building this role model for?
[00:33:13] And depending on your end goal and purpose, the evaluation should defer. So in the context of games, then the most direct way of measuring is how much behind are people actually spending in this world that you create? And if your goal is to say, for example, in the context that we just talked about, like, hey, deploying, deploying action in body, a agent, then your, your end.
[00:33:33] Metric is then, okay, after training in these worlds that you generate how robust it is to when you actually deploy to the target environment. But then, it’s, it’s hard to measure these end metrics. So today people have like these proxy metrics that I call that basically try to measure what we really care about, which is the end metrics, but then frankly it’s different for every use case.
[00:33:57] Yeah,
[00:33:57] Vibhu: which seems like quite a challenge, right? Like in [00:34:00] in language models or video models. Image models, your benchmarks are proxies, right? People aren’t actually asking instruction, following tool use questions. They’re proxies of how well it will do downstream. But for this, so like, should teams, should companies have their own individual benchmarks outside of games?
[00:34:16] If you think of stuff like, okay, video production, movies, stuff like that, that also want to use world models. Should, should they sort of internalize like. Their own proxy. Is this something you guys do? Where, where does that connect
[00:34:28] Chris Manning: go? Yeah, I think this whole space is extremely difficult as things are emerging now.
[00:34:35] And I mean, it’s not only for world models, I think it’s for everything including text-based models, right? ‘cause in the early days it seemed very easy to have good benchmarks ‘cause we could do things like question answering benchmarks and could you answer the question based on these documents and the various other kinds of, do pieces of logical reasoning or math.
[00:34:58] But again, these are sort of. [00:35:00] And there were sort of visual equivalents of things like object recognition, right? For these small component tasks. These days so much of what people are wanting to do also with language models is nothing like that, right? You’re wanting to, have an interaction with the language model and get some recommendations about which backpack would be best for you for your trip in Europe next month.
[00:35:25] And it’s not the same kind of thing, right? And it’s not so easy to come up with a benchmark as to does this large language model give you an effective interaction for guiding you in a good way for shopping, right? So, and it’s the same problem with these world models. So if we take the game design case, well success is that a game designer can.
[00:35:57] Produce what they are [00:36:00] imagining in a reasonable amount of time. And that’s really the kind of macro task. That’s a very hard thing to turn into a benchmark and I think a lot of this is actually going to turn into people walking, walking with their feet. Right? I mean, I guess that’s what’s happening, at the large language model level, right?
[00:36:23] When people are choosing to use, GPT five or Gemini or clawed, individuals are trying out these different models and deciding, oh, I like the kind of answers that GT five gives me, or no, I feel like I get more accurate detail from Claude, right?
[00:36:43] Vibhu: It’s a lot of
[00:36:43] Chris Manning: vitech, a lot of people just using it.
[00:36:45] It’s vibe checking. I realize that, but it’s actually whether. People feel it’s giving them utility in what they want. Right.
[00:36:52] Vibhu: And the the interesting thing there is like a lot of people prefer the visual, right? This looks pretty, which is not the objective of what this is [00:37:00] for, right? It’s if a, if a game designer is working on something, they care about the game engine, right?
[00:37:04] The state, it’s, it can look whatever. You can fix that up later. Or you can have a really good game state and you can quickly edit it to 20. 20 different versions, like Keep State,
[00:37:14] Chris Manning: right?
[00:37:14] Vibhu: So
[00:37:14] Chris Manning: that’s a really important distinction, for and for speaking to Moon Lake strength, right? So, yeah, great visuals are lovely to look at for a few seconds, but gains are really all about the concept, the game play.
[00:37:33] And a lot of the time that doesn’t actually even require great visuals. I mean, there are just lots of very successful games which have relatively primitive visuals, and there are other games where people have spent millions producing photo realistic, visuals, and the game sucks, right? So, keeping those two axes apart is really important in thinking about what’s important in a [00:38:00] world model for different uses.
[00:38:02] swyx: This conversation is reminding me of some game review and fiction discussions I’ve, had in my sort of non-AI related life. Some, for some people might know Brandon Sanderson, who’s a very famous, fiction author, had, is is a big game reviewer. And he, he’s a big fan of video games where you change one thing about a normal what you might assume about, about the world.
[00:38:22] For example, Baba is you, I don’t know if you might have come across that, where like the rules change as you play the game. And also like where, you can do things like reverse time selectively or like change gravity selectively. And I think this is also reminds, reminds me of other kinds of world models that are created by authors.
[00:38:38] Where Ted Chang is, is my typical example where he’ll take the world that, you know today, but change one thing about it and, but then create a consistent world based on that. Which is long-winded answer of me to, of. For me to say is it’s it easy to create alternative roles that don’t exist, but you change one thing and then let’s, let’s run a whole bunch of people through it to see if it works.
[00:38:58] Chris Manning: My first dance will [00:39:00] be, that seems a lot easier and more conceivable to do using Techn technology like Moon Lakes than with some of the other world models out there, where the sun can actually make it happen. I’ll let him give a second answer.
[00:39:15] swyx: If I guess for you, you’re constrained by the game engine tool, right?
[00:39:18] Like at the end of the day, that’s the, that’s the thought, partner that you have. If I ask for something where like, if it never is allowed to reverse time or if gravity only ever works one way, then well that’s it. But sometimes gravity might change,
[00:39:33] Fan-yun Sun: but it’s a lot easier to change with code as opposed to a model that is learned primarily on data of.
[00:39:42] Real world and virtual worlds that are, I guess, like for example, junior, like there’s actually trained on a lot of real world data and a lot of virtual gaming data, and it’s hard to say maybe it’s easier to say, okay, I wanna change the visuals in like the time period of, of the world. Like, you can’t change gravity, for [00:40:00] example.
[00:40:00] Vibhu: I feel like you can to light bounds, right? Everything comes down to like, code is a better way to execute it, but the models aren’t that diverse and creative, right? You can say, okay, make gravity slower. It can do that, but it’s limited to your representation of how you text it out, right? Like they’re, they’re only gonna do a few iterations, whereas programmatically, if there’s a game engine under the hood, you can kind of go wild, right?
[00:40:22] So one of the, I dunno, one of the limitations of most models is that they’re very overtrained to one style. Right. And extracting diversity is pretty difficult. At least that’s something we’ve seen.
[00:40:35] Fan-yun Sun: I mean, are there examples you have in mind where you Existing models? Yeah. Like it would be easier to do that’s not using code.
[00:40:43] Certain types of creative intent or like transition state transitions,
[00:40:47] swyx: Clipping, other models, other wo models are very good at clipping through things. Clipping my, my, my legs clipping through a rock because it’s, it’s just, it’s just bad. [00:41:00] Like, you would have to struggle very hard with your stuff to actually make that happen.
[00:41:04] Which I think is maybe a topic that you actually prepared on, Gian Splatting versus, the other stuff.
[00:41:09] Vibhu: Yeah. Yeah. It’s just for those not super familiar, right? There’s a, there’s gian splatting, there is diffusion. Like what works, what scales up. I feel like in February when Soro one came out the blog post was literally titled like,
[00:41:21] swyx: you bring it up.
[00:41:22] You never know.
[00:41:23] Vibhu: World, world, video generation models are world simulators. It’s super bitter lesson pilled. Yeah, emer, a lot of it is emergence, right? So, not to go through their blog post, basically their whole thing was as you scale up all this consistency, all this stuff just kind of solves, it’s a very simple premise, right?
[00:41:41] They just scaled up, diffusion, and from there, this is, this is Feb 2024, how much can we, it’s already been two years, which is basically five years. How much more in AI time do we need to just scale up or, or do we hit a data cap? But I think we already talked about this a lot, right? Like this is back to the beginning discussion of what’s [00:42:00] appropriate for the time.
[00:42:01] And that seems like your approach, right?
[00:42:03] Fan-yun Sun: Yeah. The point I’m trying to make is that they’re very many, many different types of world simulators and like having a world simulator that can produce pixel coherency is very, very useful for games and, marketing and all these things, but it’s not as useful as people think when it comes to causal reasoning.
[00:42:25] When it comes to embodied ai. Yeah, like it this title is true. We’re not saying that it’s, it’s like, not a great world simulator, but actually in the blog that we, we, we, we wrote, the bet is more so that there are gonna be disproportionately large share of value of real world tasks or, and virtual tasks where high resolution pixel fidelity is not needed.
[00:42:47] Yes. Video models have their values.
[00:42:50] swyx: Yeah. This is at the absolute limit of my physics understanding, but one example that comes to mind is basically having to solve like ba the equivalent of a three [00:43:00] body problem in a deterministic Well, where the video models, which is approximated good enough. Yeah.
[00:43:08] Right. Like there’s, there’s some point at which your approach kind of runs into like the you now have to simulate the world. Please, thank you very much. And like you’re trying to do that, but only to the extent that the game engine lets you and like game engines cannot do some things.
[00:43:23] Fan-yun Sun: Yeah, no, I mean, I think the interesting or more technical question here actually is where do you draw the boundary between.
[00:43:32] What’s handled with, let’s say, diffusion prior and what, when? What’s handled with symbolic priors?
[00:43:38] swyx: Yes.
[00:43:38] Fan-yun Sun: Okay.
[00:43:38] swyx: Okay.
[00:43:39] Fan-yun Sun: Right. Let’s go there. Because this, this boundary can actually be fluid. Like I think like maybe what you’re trying to get at is like, okay, people are saying pixel prior, everything. But what we’re saying is, okay, there’s a boundary that we draw where this is where we think provides the most economical value for the domains and things that we care about today.
[00:43:59] [00:44:00] And I actually do think, and it’s something that we do internally all the time, which is like, okay, given new equations that we learn or new elements of the world and that we, we learn, or maybe some other knowledge that we acquire in the process of developing the models. Should we still be maintaining this line exactly as it is today?
[00:44:22] Or should we move it a little bit left or a little bit right? Right. Like sometimes that we realize that, oh, like maybe customers or, or folks like want certain things that are better handled with preop pryor as opposed to, symbolic prior than,
[00:44:34] swyx: yeah. Your, your skin thing is a, is a example moving it, right.
[00:44:37] Yeah.
[00:44:37] Or left. Yeah,
[00:44:37] Fan-yun Sun: exactly.
[00:44:38] swyx: I dunno what the, the left right is.
[00:44:39] Fan-yun Sun: Yeah, yeah, yeah. No the, the model.
[00:44:42] swyx: Yes.
[00:44:42] Fan-yun Sun: Actually we have a few iterations of them. They’re actually at slightly different
[00:44:45] swyx: I know boundaries. You should, you should do that. That’s a cool dimension to show.
[00:44:49] Fan-yun Sun: Yeah.
[00:44:50] swyx: Is quantum mechanics the diffusion prior of our world?
[00:44:55] Right. It’s like that’s the boundary of classical mechanics versus quantum. Right? Like, that’s it. At one [00:45:00] point God plays dice and the other point doesn’t.
[00:45:02] Fan-yun Sun: I dunno if Chris, you wanna say it, but I think, I think generally I feel like physics is better with symbol P priors.
[00:45:08] Chris Manning: Even quantum physics.
[00:45:09] Fan-yun Sun: Even quantum physics.
[00:45:11] swyx: Yeah. This is starts against to, MLST territory is, is what I call it, where, he, he likes to get philosophical. We, we we’re quite friendly.
[00:45:18] Vibhu: I mean, we need to get, we need to get singularity. I heard some of that.
[00:45:23] swyx: No, no, I think that is actually really helpful and man, I just want you to productize this like, as a product guy, I’m just like, oh, also
[00:45:32] Vibhu: a gamer, I
[00:45:33] swyx: wanna, it’s like a researcher, like, it’s cool.
[00:45:35] Like this is a, the theoretical, like you have a very good, I don’t know, like the way of thinking about these things, but I just wanna see you like, express it. I do think like your fundamentally things when, when you leave open new tools, like, okay, use, use human intent to incorporate it into how you render.
[00:45:52] Artists are gonna have to take like two to three years to figure out what to do with this. And you just don’t know.
[00:45:57] Chris Manning: Right. But I think, this is, [00:46:00] gives a much more approachable and controllable world for the society, which is the beauty, the beauty of, NLP, that that will enable it to be adopted and used.
[00:46:10] And we are very hopeful about that. Yeah,
[00:46:13] Fan-yun Sun: yeah. Yeah. I mean, we are, we are very focused actually on commercialization in the sense that like we do, we do really believe in the data flywheel app approach. Yeah. Where, we put this in the hands of the creators and the users and then they will teach us when, what capability our model should improve.
[00:46:27] And that’s why we are, we are actually, like products and beta
[00:46:31] swyx: Yeah. Focusing on gaming. What, what’s like the adjacent thing to gaming
[00:46:34] Fan-yun Sun: embody adjacent, basically. So maybe we can, we can I’ll maybe start with where we see the platform in three years. Yeah. Which is like, okay. The users would tell us what they want to achieve.
[00:46:45] The end goal could be, Hey, I just, I wanna make something to teach my kids the value of humility. Or it could be, Hey, I wanna fine tune my, drones to be really good at rescue situations. I could be vacuum robots. I want to like train [00:47:00] my manipulation or like vacuum robot to be very robust to my office, right?
[00:47:04] But it’s like, whatever it is, scenario robust to
[00:47:06] swyx: my office
[00:47:07] Fan-yun Sun: or like navigate very robustly in my office. But then it’s like, whatever end goal that you want, our role model will say, okay, given what you want to achieve, let me generate a distribution of environments such that I can train and evaluate whatever it is you want.
[00:47:24] Yeah. Right. Maybe for the purpose of games, it’s just the end simulation and that’s the end product for certain policies. It’s like I can train it within these environments and then help you see where your policy is failing or not. Yeah. And then, so I think,
[00:47:37] swyx: so in that case, much more of a training tool.
[00:47:40] Than in other training
[00:47:41] Vibhu: evaluation? Both. Right?
[00:47:43] swyx: Sure. Same. Same thing.
[00:47:43] Fan-yun Sun: Yeah, same thing. I think it’s just this role model that allows people to train any policy that can act in any multimodal environments.
[00:47:51] swyx: Would it be harder to reward hack? Is there an angle here where it is harder to reward hack? Like it’s just, I’ll just put it generally because I think that’s a, that’s obviously a key [00:48:00] problem that a lot of people face when in training agents in these environments, and I don’t know, can you solve it?
[00:48:07] Chris Manning: I think not necessarily. To the extent that there’s a mis specified reward that. It seems like it could be hacked in a more symbolic world or in a more pixel based world. I dunno if Sun’s got any thoughts, but I don’t think that’s really being solved.
[00:48:26] swyx: The other thing that comes to mind is just you could just build a better sawa as a video generator model, right?
[00:48:31] Because then you, you would move the diffusion, side a bit more further to the right. I think if I got the directionality correct. And that’s it.
[00:48:40] Vibhu: It’s better on domains, right? Like on consistency over now, or for sure it exists versus something doesn’t, right.
[00:48:46] Chris Manning: So
[00:48:46] swyx: yeah. Yeah. Is
[00:48:49] Vibhu: is a question more like, like
[00:48:51] swyx: I’m just riffing on like, how do you, what can you build, you know?
[00:48:54] Oh, with the stuff that you have. I do think that the minor, the academic does go immediately to training [00:49:00] and in eval evaluation, but like art tends to take unusual directions. Like you might end up,
[00:49:06] Chris Manning: okay. Yeah. But the question is, can you use this piece of software to develop compelling gameplay and. I don’t think you can take SOAR and produce compelling gameplay, right?
[00:49:19] If you want to have a world that you can wander around in a bit, you are good. But what are your abilities to have gameplay mechanics implemented the way you’d like them to be and to have things stay, with the long-term history of your gameplay that influences future actions. I think there’s just nothing there for that.
[00:49:39] swyx: Yeah, I do tend to agree. I, I’m just trying to sort of test the boundaries. I would also make the observation that as AAA games industry has developed the line between what is a movie and what is a game has blurred. And you, you, you do end up basically producing a two hour movie as part of your game.
[00:49:57] Fan-yun Sun: No, honestly, there, there’s so many actually [00:50:00] applications in adjacent markets that our world model can go into. Yeah. But yeah, it, it’s sort of fun to riff, riff on. Although on the execution side, we we, we need to stay focused with like, okay, what are the capabilities we want to unlock over time?
[00:50:11] And there’s a roadmap for that. But yeah, if we’re just riffing on sort of like the possibilities, I feel like, whether it’s endless Yeah, it’s like classic
[00:50:18] swyx: and the embedding for a possibility and endless in my mind, it’s very close. Yeah. I do wanna, focus on one, like weird choice. I, I don’t know if it’s weird.
[00:50:28] Maybe I’m, I got something here. Audio, right? You could have just said no audio And audio in my mind has a lot of recursion, whereas in video you can just do recasting and that’s much computationally much simpler. Audio just seems way harder. I don’t know if you wanna just comment on just the special 3D audio.
[00:50:46] Problem. Did you really have to do it? I guess you do to be immersive, but like a lot of people do treat it as like, well, you just stick a, a tt S model on top of
[00:50:57] Vibhu: Well, there’s a lot more to game audio than [00:51:00] just speech. Right. It’s not just
[00:51:01] swyx: tts. Yeah. Tts. S Fxt, GM Spatial in my mind Echoes
[00:51:06] Chris Manning: Yeah.
[00:51:06] swyx: And reflections.
[00:51:07] And I, I don’t even know what’s, what else? I don’t know what, what other problems in this space.
[00:51:13] Fan-yun Sun: Yeah, I think this point like the, it’s sort of a more, more pointing to the benefits of using an game engine as a tool that’s available to the model, right? Because like part of the spatial audio is from the code that is underlying the simulation.
[00:51:32] And while we do give our model access to other types of audio models as. Tools.
[00:51:39] swyx: None of them would be spatial, I think.
[00:51:41] Fan-yun Sun: But that’s exactly sort of more 0.2. We’re giving our model an abstraction or a suite of tools such that it’s able to achieve that. And you can argue that sort of spatial is like a, like a emergence out of the, the tools that we and abstraction that we provide to the agents.
[00:51:59] And I think that’s the beauty of [00:52:00] this, this, this approach is like there’s a lot of things kind of like how human’s built technology and they’re like Lego blocks that build on top of each other. And it’s the same thing here. There’s gonna be things that sort of just sort of emerges from being able to put these things together in like combinatorially interesting ways,
[00:52:14] Chris Manning: right?
[00:52:15] So this integrated audio model exploits the understanding and semantics of the Moon Lake world, right? And whereas in general for the Gen AI video models. There’s no actual integration across to audio at all, right? That someone might stick some music or stick a soundscape or whatever else on top of their video.
[00:52:44] So it’s not a silent video, but they’re in no way connected into a consistent world model. And there’s nothing that’s okay. An action is happening in the video. Therefore there should be a sound that’s [00:53:00] coming from this part of the visual field.
[00:53:03] swyx: Yeah.
[00:53:03] Vibhu: Is that different than Sora too? Does it not have audio?
[00:53:06] Not to say it’s not like
[00:53:08] swyx: amazing
[00:53:08] Vibhu: isn’t a spatial
[00:53:09] swyx: audio.
[00:53:09] Vibhu: It doesn’t,
[00:53:10] swyx: no. I’ve played around it with it enough. It just sounds like someone put an 11 laps voice on top of it and just tried to do the lip sync.
[00:53:18] Vibhu: Oh, yeah. I’ve seen, okay. Generate a dog at the beach and reactions to big wave and move
[00:53:23] swyx: around.
[00:53:23] It’s definitely like, so have the dog, have the dog move away from camera and see if the, the song goes down. It doesn’t. ‘Cause they don’t have facial audio.
[00:53:32] Fan-yun Sun: We do want to basically like we, our moral model, like the one we’re training is basically towards the goal of having a combined latent representation across all these different modalities.
[00:53:42] Right? Such that it can like reason across these different modalities. So for example, if I close my eyes and like you play a video, you play a sound of like a car skidding away from me. I almost can like, visually extrapolate that trajectory in my mind. And I think that type of capability, we want our model to be able to reason, right?
[00:53:59] And that’s the reason that [00:54:00] we’re sort of taking this multimodal reasoning approach. It’s like we want this combine late in space that can
[00:54:05] swyx: Yeah. Oh, you said late in space. We like that. Here we have to play the, the bell Every time that someone says late in space, no, you gotta train daredevil one. Where you, you, you, it’s only audio, but you have to work out.
[00:54:15] Where everything is.
[00:54:19] Cool. I I think that that was, that was about it for our Moon Lake coverage. I do think that we have like a couple of, Chris Madden questions on, on IR and, just any, any other sort of attention topics or n NLP topics.
[00:54:31] Vibhu: Okay.
[00:54:31] swyx: Go ahead.
[00:54:32] Chris Manning’s Journey: From NLP to World Models
[00:54:32] Vibhu: Well, no, I mean, yeah, it’s just fun. We talked a bit about how you guys met, but you basically, you, you were like the godfather of NLP per se, right?
[00:54:39] You spent the whole career from early embeddings, early early attention. You did 2015 attention for machine translation, everything. You, you had information retrieval, so RAG before rag, we just wanna shout that out and admire a lot of that. Right? So what prompted the switch over to world models?
[00:54:56] How, how’d all that come about?
[00:54:58] Chris Manning: To some answer it [00:55:00] is, the enthusiasms and creativity of students, but there’s a bit of a history there, right? So, yeah. So clearly most of my career has been doing stuff with language and how I got into research was thinking, ah, this is just so amazing how humans can produce speech and understand each other in real time.
[00:55:21] And somehow they managed to learn languages from their kids. How could this possibly happen? And so, yeah, starting off I was very focused on language, but as it sort of got into the 2000 and tens, I started, going, I’d been working on question answering, and then I started to get, interest in visual question answering.
[00:55:42] And that was an area where it was very noticeable. That the visual understanding was bad. Right. These were the days when like, it sort of seemed like there’s almost no visual [00:56:00] understanding. You were just getting answers that came from priors. So, if you asked how many people are sitting at the table, it’d always answer two regardless of how many, how many people you could see in the picture.
[00:56:11] And so it seemed like, oh, these models actually aren’t able to get semantic information outta IMA images. And so I was interested in that problem and tried to work more on that. And so then that required. Knowing more about what’s happening in vision and how you can represent visual information.
[00:56:34] And then things start, there started to be this revolution of, doing generative AI images. And then I had students that started looking at that before the era of Moon Lake. I was also working with Demi Gore, who founded pika. And so, and
[00:56:50] swyx: Ian obviously
[00:56:52] Chris Manning: with gans. Yeah. Though Ian was never my student, but yeah, Ian I was very aware for the, the whole decade there of Ian with Gans.
[00:56:59] [00:57:00] Yeah. And I mean, Ian was a Stanford undergrad, but yeah,
[00:57:03] Vibhu: richard des u.com, I believe he was your student.
[00:57:06] Chris Manning: Yeah. Yeah. And there were, there were links across at that stage as well. So there were several papers in that era of doing, I mean, so Andre Cap was a, PhD student at the same time as Richard.
[00:57:20] And so there was some joint language vision work in that era as well. It seems kind of ancient by modern standards, but yeah, we’re trying to go from sort of textural dependency graphs to visual scenes
[00:57:32] Vibhu: at a time. The glove embeddings really took over a lot of. T-F-I-D-F, like one hot encoding, all that.
[00:57:38] The early vision language models we saw were like lava style adapters, right? It’s, it’s technically still just embedding latent space. Let’s add image, let’s like mixed modality. So, and that, that’s one of the things you super put out there too, right?
[00:57:51] swyx: Yeah.
[00:57:51] Vibhu: Yeah.
[00:57:52] swyx: Yeah.
[00:57:52] Hiring, Closing & The Name “Moon Lake”
[00:57:55] swyx: Well, thank you for all of that. Thank you for all advancing the worlds on, world modeling.
[00:57:56] I honestly, do think that if people deeply understand everything we just [00:58:00] covered, they will see what’s coming. I think you guys have, made some, a really significant contribution here. What are you hiring for? What is the, what do people find? We, we agreed that the CTA was a hiring call.
[00:58:10] Yeah. Don’t we have a GI You don’t need, you don’t need engineers anymore, right?
[00:58:14] Fan-yun Sun: Yeah. On the model side we are actually striving towards basically a self-improving system. But what that means is that we need people to set up the self-improving system. So more, more specifically people who have the intersection of knowledge within co-generation and computer vision and graphics, right?
[00:58:30] Yeah. That’s, that’s sort of the core research background that we look for within OTM and, and the majority of the team today do have like both backgrounds.
[00:58:38] swyx: When you say computer vision and graphics, are they the same thing or is it computer vision one thing, graphics, another thing. And how intertwined are they?
[00:58:46] Chris Manning: They’re intertwined but different.
[00:58:49] swyx: Yeah.
[00:58:49] Chris Manning: And I think, this relates to some of the themes that we’ve been talking about, that the more explicit underlying [00:59:00] world models that are being constructed inside Moon Lake really draw on the computer graphics tradition. And so it’s then combining that with the visual understanding of vision.
[00:59:16] swyx: Got it. Yeah. All right. So you’ve written a game engine, you’re come talk to us, right?
[00:59:21] Fan-yun Sun: Oh yeah, definitely. Definitely. But I do think that the line is blurred, like increasingly blurred these days where it’s like if you have a general understanding of group vision and graphics,
[00:59:31] swyx: I think for your standards it is, for me it feels like vision is, is.
[00:59:35] I’ll leave that to the big labs graphics. I, I, I can get that, you would want to do that from more first principles, but vision, there’s so many vision models off the shelf that I can take, but probably not good enough for your
[00:59:45] Fan-yun Sun: I see, I see. If, if you’re sort of like making that distinction then maybe we, we care a little bit more about having graphics
[00:59:51] swyx: knowledge.
[00:59:51] Yeah, exactly.
[00:59:52] It could be like, sometimes a hiring call can be as simple as like, if you know the answer to blah, you should talk to me. Like the sort of core known hard [01:00:00] problem in, in your world.
[01:00:01] Fan-yun Sun: Ah, I see. Yeah. In that case, if you, yeah, definitely. If you’ve written a game engine before, if you’ve rld a variety of coding models on different objectives, like
[01:00:13] swyx: easy,
[01:00:13] Many of those, yeah.
[01:00:14] Fan-yun Sun: If you’ve done multimodal lean space alignment, I, I intentionally include
[01:00:20] swyx: space.
[01:00:20] Fan-yun Sun: Again,
[01:00:21] swyx: a poor editor has a thing every time. Yeah. Lean space alignment. Honestly. Is it that hard?
[01:00:26] I, I, there’s some scripts out there that I’ve saved for the day. I someday have to do it, but I don’t have to do it.
[01:00:31] But it’s
[01:00:32] Fan-yun Sun: done, I think. Yeah. There, there’s, there’s a versions of that that are done. But I, I think we are aligning audio, text, language and video. Yeah. Right. Like, and basically we have these role models that are able to act as agents to like act in these worlds and extract long horizon videos and encoding that back to the model to sort of self-improve.
[01:00:52] So it’s an insanely exciting, but also technically challenge problem. Yeah. So people who wanna do their lives best work, that only [01:01:00] makes a place.
[01:01:01] Vibhu: How big are you guys? Where are you guys based?
[01:01:02] Fan-yun Sun: We’re currently based in San Mateo, although we’re moving up to sf. We’re about 18 folks right now.
[01:01:08] swyx: My ending question was gonna be why, what, what is the name?
[01:01:10] What’s behind the name?
[01:01:11] Vibhu: Yeah.
[01:01:12] Fan-yun Sun: Oh,
[01:01:14] Vibhu: Very cool. Graphics and design, by the way.
[01:01:16] Fan-yun Sun: Actually at the, at the time when the, when the, when we started the company, we were thinking a lot about how do we make a company name that gives people the vibe of like, open ai, but for like, almost like industrial light and magic vibes.
[01:01:28] Wow. Because it’s like we care about creativity and using that as a funnel to solve a GI. So then we were, we, we brainstorm a lot around like Dreamworks, right? Like industrial light magic. And, so there’s a few, few basically, space of things that we feel like are very, very semantically close to the company’s identity.
[01:01:47] swyx: Yeah.
[01:01:48] Fan-yun Sun: And then it ended up being Moon Lake, partly because of the Dreamworks vibe, the Dreamworks, moon
[01:01:54] swyx: Lake.
[01:01:55] Fan-yun Sun: Exactly. Yep. So that was a little bit of that inspiration. And then the moon was sort of [01:02:00] like a, it basically was like about the. Reflection. The reflection part also implies the self-improvement loop.
[01:02:07] Wow. That we sort of like, that’s really bleed and that’s the path towards multimodal general intelligence. So that’s, that’s that. I’ll leave that as I love a good
[01:02:15] swyx: name. I love a good name. This is great. It’s a
[01:02:16] Vibhu: very
[01:02:17] swyx: good name. It’s very good. Lo I’m glad I asked the question. I will also say, one, my favorite story, books or biographies ever is, creativity Inc.
[01:02:24] With Ed Kamal’s, story about Pixar and how he, was rejected as a Disney animation artist. So then he went into computing and brute forced his way into back. No, I love that story. Yeah. Disney.
[01:02:37] Fan-yun Sun: Yeah. And Walt Disney is also like one of my favorite founders. He’s like, his, his story. Like at the time you’re like, okay, I’m gonna create this like.
[01:02:44] Immersive park. Like people can’t, don’t even have that technology to create it virtually, but they’re like, you know what, let’s just build it physically such that people can,
[01:02:50] swyx: so he is the first world modeler.
[01:02:52] Fan-yun Sun: No, I, I I tell people that like, theme parks are world models too.
[01:02:56] swyx: Mm. Yeah. Yeah. Yeah. I mean, it’s a small world or it’s [01:03:00] a, like the Epcot center with all the little, replicas of the countries.
[01:03:03] Yeah. Those are very interesting. Okay. Well thank you, we’ve covered, a huge amount. Thank you for your time and thank you for inspiring us.
[01:03:10] Fan-yun Sun: Thank you
[01:03:10] swyx: for having us. Thank you. It’s fun
[01:03:11] Fan-yun Sun: chatting. Yeah. It’s been a good time.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Mistral has been on an absolute tear - with frequent successful model launches it is easy to forget that they raised the largest European AI round in history last year. We were long overdue for a Mistral episode, and we were very fortunate to work with Sophia and Howard to catch up with Pavan (Voxtral lead) and Guillaume (Chief Scientist, Co-founder) on the occasion of this week’s Voxtral TTS launch:
Mistral can’t directly say it, but the benchmarks do imply, that this is basically an open-weights ElevenLabs-level TTS model (Technically, it is a 4B Ministral based multilingual low-latency TTS open weights model that has a 68.4% win rate vs ElevenLabs Flash v2.5). The contributions are not just in the open weights but also in open research: We also spend a decent amount of the pod talking about their architecture that combines auto-regressive generation of semantic speech tokens with flow-matching for acoustic tokens (typically only applied in the Image Generation space, as seen in the Flow Matching NeurIPS workshop from the principal authors that we reference in the pod).
You can catch up on the paper here and the full episode is live on youtube!

Timestamps
00:00 Welcome and Guests00:22 Announcing Voxtral TTS01:41 Architecture and Codec02:53 Understanding vs Generation05:39 Flow Matching for Audio07:27 Real Time Voice Agents13:40 Efficiency and Model Strategy14:53 Voice Agents Vision17:56 Enterprise Deployment and Privacy23:39 Fine Tuning and Personalization25:22 Enterprise Voice Personalization26:09 Long-Form Speech Models26:58 Real-Time Encoder Advances27:45 Scaling Context for TTS28:53 What Makes Small Models30:37 Merging Modalities Tradeoffs33:05 Open Source Mission35:51 Lean and Formal Proofs38:40 Reasoning Transfer and Agents40:25 Next Frontiers in Training42:20 Hiring and AI for Science44:19 Forward Deployed Engineering46:22 Customer Feedback Loop48:29 Wrap Up and Thanks

Transcript
swyx: Okay, welcome to Latent Space. We’re here in the studio with our gues co-host Vibh u. Welcome. Thanks. Excited for this one as well as Guillaume and Pavan from Mistral. Welcome. Excited to be here.
Guillaume: Thank you.
swyx: Pavan, you are leading audio research at Mistral and Guillaume, you're Chief Scientist,
Announcing Voxtral TTS
swyx
Host
(00:05) Okay. (00:05) Welcome to Lean Space. (00:06) We’re here in the studio with trustee co-hosts, Vibhu. (00:09) Welcome.
Vibhu
Host
(00:11) Very excited for this one.
swyx
Host
(00:12) As well as Guillaume and Pavan from Mistral. (00:15) Welcome. (00:16) Excited to be here. (00:17) Thank you for having us.
(00:18) Pavan, you are leading audio research at Mistral and Guillaume, you’re a chief scientist. (00:23) What are we announcing today where we’re coordinating this release with you guys?
Guillaume
Guest
(00:26) Yeah, so we are releasing Voxtral TTS. So it’s our first audio model that generates speech. It’s not our first audio model. We had a couple of releases before.
(00:35) We had one in the summer that was Voxtral, our first audio model, but it was like a transcription model, ASR. Like a few months later, we released some update on top of this, supporting more languages. Also a lot of table stack features for our customers, context biasing, precision, timestamping and transcription. We also have some real-time model that can transcribe not just at the end of the level.
(00:56) You don’t need to fill your entire audio file, but that can also come in real-time. And here, this is a natural extension in the audio, so basically speech generation. So yeah, so we support nine languages, and this is a pretty small model, 3D model, so very fast, and also state of the art. Performed at the same level as the base model, but it’s much more efficient in terms of cost, and also much, in terms of cost, it’s also much cheaper, only a fraction of the cost of our competitors.
(01:22) And we are also releasing the work that this model is running.
swyx What’s the decision factor?
Guillaume It’s a good question.
swyx
There will be more. Yeah, Pavan, any sort of research notes to add on?
Architecture and Codec
Pavan: But it’s a novel architecture that we develop inhouse.
We traded on several internal architectures and ended up with a auto aggressive flow matching architecture. And also have a new in-house neural audio codec. Which, converts this audio into all point by herds latent [00:02:00] tokens, semantic and acoustic tokens. And yeah, that’s that’s their new part about this model and we’re pretty excited that it’s, it came out with such good quality and Jim was mentioning. Yeah, it’s a three B model. It’s based off of the TAL model that we actually released just a few months back and insert trunk and mainly meant for like the TTS stuff, but they need text capabilities are also there. Yeah.
swyx: So there’s a lot to cover.
I always I love any, anything to do with novel encodings and all those things because I think that’s obviously I creates a lot of efficiency, but also maybe bugs that sometimes happen. You were previously a Gemini and you worked on post training for language models, and maybe a lot of people will have less experience with audio models just in general compared to pure language.
What did you find that you have to revisit from scratch as you joined this trial and started doing this? At least
Understanding vs Generation
Pavan: when it comes to, for, I think the, there are two buckets, I guess the audio understanding and audio [00:03:00] generation. The audio understanding, like the walkthrough models that Kim was mentioning that we released earlier.
The walkthrough chat that we released I think July last year, and the follow up transcription only, models family that we released in January, that would be one bucket, and the generation is another bucket. I think. You can also treat them as a unified set of models, but currently the approaches are a little different between these two.
To your question on how audio is fed to the model? In the understanding model, it’s very similar to actually Pixar models that we also released,
swyx: yes.
Pavan: That’s
swyx: amazing.
Pavan: It was pretty, I, that was the first project I worked on after joined Misra. It was pretty, pretty nice. And Wtu was very similar in spirit.
I guess So we feed audio through an audio encoder similar to images through a vision encoder, and it produces continuous embeddings and which are fed as tokens to the main transformer decoded transformer model. Yeah. On the model output is just text. So on the output side, there is nothing that needs to be done in these kinds of mode.
I [00:04:00] guess the interesting part of what the generation stuff is, the output now has to produce audio and. The approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing attrition and a lot of models which are based off of this kind of approach.
And we took a slightly. A different, design decisions around this. But at the end of the day, the neural audio product converts audio into a 12.5 herdz set of latents. And each latent is, has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side.
There’s several ways to use this at each frame, but we just sum the embedding. So it’s like having key different vocabularies. Combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part on the output side, the, it’s not the, I don’t know if it’s the most popular, but one.
Popular technique is to have a depth transformer [00:05:00] because you have K tokens at each time step, like with a text, you just have one token at each time step. So you just do predict the token from the vocabulary with, yeah, with just, you get probability
swyx: This’s a very straightforward text. Very
Pavan: straightforward.
swyx: Yeah.
Pavan: But if you have K tokens, then the name thing would be to predict all of them in paddle. That doesn’t work. At least that doesn’t work that well because audio has more entropy. And the, one of the techniques people use is this depth transformer where you you almost have a small transformer, or it can be L-S-T-M-R in as well, but people use transformers and you predict the K tokens in auto aggressive fashion in that.
So you have two auto reive things going on.
Flow Matching for Audio
Pavan: So the thing we did differently is in, instead of having this auto aggressive K step prediction, we have a flow matching model. Instead of modeling this as a discrete token set we trained the codec to be both discrete and continuous to have this flexibility.
So we did try the discrete stuff too, and which it works well, but the continuous stuff works just better. So yeah, we took this flow matching, so the, it’s a flow [00:06:00] matching head, which takes the latent from the main transformer and like kind in fusion, it’s denoising, but in this flow matching itself, velocity estimate.
So you go from this noise t all the way to there. Audio latent, which corresponds to the 80 millisecond audio and then, which is sent through the work order to get back the 80 millisecond audio frame.
swyx: Yeah. Is this the first application of flow matching in audio? Because usually I come across this in the image.
Pavan: Yeah. Actually, in some sense there are models flow matching models in audio, but I think this specific combination I could be wrong. There could be somewhat. No. I haven’t seen. I haven’t seen much work in this, so I think it’s novel and a lot of it’s just a way bigger community, so they, I think they pioneer a lot of these diffusion flow matching work, and it’s interesting to adopt some of the ideas there into audio and,
swyx: yeah.
Pavan: Yeah, I’m, personally that’s the think part which is trying out about. One of more meta point is unlike text, even in vision, I think this is true, but in [00:07:00] audio step literature that there is no.
Winner model, yet there is no, okay, this is the way you do things. It’s it’s still by, I think people are still iterating and figuring out like what’s the best overall recipe. I guess the idea. Pretty sure there are models which are also completely end-to-end, like NATO audio. NATO audio, but it’s still not come to a convergence point where this, the right way to think that.
That also makes. A space pretty exciting to explore.
Real Time Voice Agents
Vibhu: What are some of the ways to look at it?
Vibhu: There are ways where you can do diffusion for audio generation, but if you want like real time generation, that’s a big thing with the approach I’m assuming that you took. Yeah. And also like how do you go about evaluating different axes of what you care about, yeah,
Pavan: good point. I think we so you can do just flow matching diffusion for the whole audio. We didn’t even go down that path because one of the main applications is voice agents and we want real time streaming, and that’s the use case. That’s not the only use case, but that’s one of the primary use cases we want to get to.
So we [00:08:00] picked the auto aggressive approach for that. And within the auto aggressive space, again, you can do chunk by chunk or you can do so we picked the. I think at least personally prefer the operations, which are the simplest, and so we try to see, can we just add audio as just another head to our regular transformer decode model because that kind of makes it easier for eventual end-to-end modeling of audio text native modeling.
Yeah. And it works pretty well. So I guess we went with that and we tried a little bit, but the flow matching head itself, like we had a discreet. Diffusion kind of approach, which also works well, but the flow matching work better.
swyx: I was just curious about how you also think about this overall direction of research.
Do you basically, when you work with the audio team, do you set some high level parameters and then let them explore whatever, or how does it work between you guys?
Guillaume: No I think the way it works is that we are the, we are prioritizing together, I think, what are the most important features because there are many things we can do [00:09:00] in audio.
Yeah, I think we try to. These are like how we should do things, for instance. Ultimately what we want to do is to build this through duplex model, but we are not going to start this start there directly, I think is. Some of the project people are doing, but
swyx: just to confirm, full effects means it can speak while I’m speaking or,
Guillaume: yeah.
Okay. Audio. Yeah. Yeah. So intimately we’re going to get there, but for us it was, we decided to take it like a step by step. So we start with whatever is the most important. I think support customers, which is the transcription is the most popular use case. Then the speech generation, Soviet time, just a bit before that.
And then actually to be like more, but try combining everything all together. But but yeah, we thought it was also important to like separate things and optimize each capability one by one before we
swyx: measure of that together. And the super omni model. But
Guillaume: very interesting because as Par said, it’s when you work on some other domains of this airline and everything, there are many areas where I think it’s not as interesting.
For instance. Many places, it’s essentially just around data or like creating new environments on a lot of kind [00:10:00] of easy things. But things were, I think the research is maybe not as interesting. Were in audio. There are so many ways to actually build this model. So many ways to go around it. That’s the sense I think is really interesting.
And what we also tried for speed generation is that we tried multiple approaches. What was interesting that even though they were extremely different, they under the big know the particles but the for matching turned out to be quite more natural. So we are happy with this.
swyx: Is there intuition why it maybe like flow matching is just models speech better in some natural fundamental, latent dimension?
Pavan: No, I think the main thing is e even at a particular time step, there is a distribution of things.
swyx: Yes.
Pavan: To be predicted like the way you inflate. So you already know the word that you’re speaking and Yeah. The intake space, let’s say the word maps register a single token for simplicity.
In most cases it does. So there is not a lot of so you just pick the word, but with within audio, even the same word could, even with your own voice, could be inflicted in so many different ways. And I think [00:11:00] any approach which like models this distribution and. And flow matching is one, one of the take.
It’s not the only one at all, but it’s a one which works pretty reasonably well. I think that’s better. So you have to pick across several different, the intuition I have is it’s, there are some, several different clusters each corresponding to some specific way you would inflict, pronounce that thing.
And you can’t predict the mean of it because that corresponds to some blurred out speech or something like that. But you have to pick one. And then like sharp
swyx: conditional inference.
Pavan: Yeah, exactly.
swyx: Is that all covered under disfluencies, which is I think the normal term of art. Pauses intonations. By the way, I have to thank Sophia for setting all this up, including like some of these really good notes because
Pavan: Yeah.
swyx: I’m less familiar with the audios for me.
Pavan: No. I think dis dismisses are definitely one such Eno defenses is more like
swyx: which is arms are.
Pavan: Yeah, arms. And also repeat like you like,
swyx: yeah.
Pavan: You do this full of words, your thinking, so you repeat the word.
swyx: Okay. Whereas intonation is like a diff, it’s up up [00:12:00] speak and all this.
Okay.
Pavan: Yeah. So I think there is a lot of like entropy. And modeling it as a distribution. And a, any technique which helps with it and the depth transformer is a conditional way of modeling this. And Transformers actually really good at it, even though that’s a mini transformers. So I think that worked pretty well too for us too.
It’s just that the main concentration is when you have a depth transformer. If you have K tokens, you need to do K auto steps, right? Even though it’s a small thing, it’s K steps, which is very vacant, say heavy, but flow matching. We were able to cut it down significantly. So we are able to do the inference in quad steps or 16 steps and it works pretty well.
And there are more normal techniques to bring it down even further to like, in extreme case, one step like we’re not doing it yet, but it at least the framework, LEDs itself to more efficient and Yes.
swyx: And the image guys have done.
Pavan: Yeah.
swyx: Incredible work guys. Yeah.
Pavan: It now you just. Send a prompt and you get an image.
swyx: Yeah. Surprisingly not enough. I think image model labs use those techniques in production. I think it’s, I feel like it’s a lot of research demos, but [00:13:00] nothing I can use on my phone today.
Guillaume: The thing, there’s a thing that would be interesting here is that since, indeed I’ve been so much sure that has been done in the vision community compared to radio dys, stomach, I think there are so many long infra Yeah.
And there are so many things we can do to actually improve this further. So it’s our first version, but we have so many ways to exist, much better and much more efficient, cost efficient, so
swyx: yeah.
Guillaume: So really it’s not a new field at all, of course, but there are still so many things that can be done.
Perfect. It’s
swyx: nice. I should also mention for those who are newer to flow matching, I think the creator, this guy’s name is Alex, he’s done I think in Europe’s maybe two Europes as ago. There was, there’s a very good workshop. There’s one hour on like this matching is I would recommend people look that up.
That’s the other thing, right?
Efficiency and Model Strategy
swyx: The efficiency wise, like I, I imagine like the reason is open weights the reason you pick 3.6 B backbone it you are 3.4 B you are, try to fit to some kinda hardware constraints. You kinda fits some kinda basic constraints. What are they?
Guillaume: Not necessarily, I think something we care about in our model that they’re efficient.
So we have a [00:14:00] lot of separate model, for instance. So we have this that is very small, very efficient. We also have a small OCR model that is available. Good, highly efficient as well. And I think on a project maybe there, I think companies are going to take is to have a coverage general model that will do a bit of everything.
But that is also going to be expensive. On here. What want say is if you care about this specific use case, if you can actually use this model, it just does that. It’s extremely good at it. Survey, very efficient. That’s why we can actually add. We do, but also OCR that are like really good at that.
And that would be much more cost effective factors and the general model that will contain a lot of capabilities you don’t really need. So yeah. So we’re doing like general model, but also like more customized model. This,
Open Weights and Benchmarks
Vibhu: how does it compare to other TTS models? It’s, we are going follow open wave.
We’re just dropping it. I think it’s pretty good.
Pavan: Yeah, I think it’s pretty good. Like it, it’s definitely one of the best. For sure. It’s probably I would say it’s the best open source model, but
Vibhu: decipher themselves.
swyx: Yeah.
Voice Agents Vision
Vibhu: Why now? How does it fit into broader ral vision? How do you see voice agents?
How do you see voice? I think every year I’ve heard, okay, you’re a [00:15:00] voice. You’re a voice. There’s a lot of architectural stuff. There’s a lot of end time that see it, your solving, but where do you see voice setting?
Guillaume: We had so many customers asking for voice. That’s also why we wanted to build it.
What’s interesting in this domain is that. In a sense, if you take something simple like transcription it doesn’t seem like something that should be very hard to do for a model. It’s essentially, it’s pattern recognition. It’s classification on this. Models are very good at classifying, right?
Or nonetheless, when you talk to them it’s not there yet, right? It’s not, you don’t talk to them the same way you talk to a person. On something, maybe people don’t realize it. It’s in English it’s still much better than in any user language, even compared to French instance. If you talk to this million in French, when you see people talking to this they’ll talk very slow.
They’ll articulate as much as they can. So it’s not natural, right? We’re not yet to this. And I think, yeah, maybe the next generation will not know this, but yeah, I think people that. But our edge will actually always keep this bias speaking very slowly when they talk to this model. Even if maybe, probably in a couple of years, maybe next year it’ll not be necessary anymore.
But yeah. But what’s interesting is to see that yeah, even for like languages [00:16:00] like yeah, French and Spanish Germans that are not no, no resource on religion. You have a lot of audios there on still it’s not as good. And I think a consequence. Because then for this, I suppose just is not as much energy, as much effort that has been put done in some other mod that for some vision or like coding.
But but yeah, there’s still a lot of progress to be done. I think it’s just a question of doing the work and it’s clear path I think to get there.
Pavan: It’s a little fascinating because I worked on Google Assistant I think while back at this point, but it’s, I think it’s, it like when you take a step back, it’s fascinating.
It’s not that long ago. It was like four years ago or five years ago, and it’s now it’s completely audio in, audio out and the function calling and the whole thing happens completely end to end. And in a very natural,
swyx: yeah,
Pavan: natural way and still ways to go. Kim was telling, even despite all the previous, it’s not like you’re speaking to a person.
When you talk to any of these agents, bots, or voice mode kind of situation, it’s still like a gap. I think that’s the great part and I feel like with even the existing [00:17:00] stack, we should be able to get to this very natural speech conversational abilities soon enough I guess.
And we’ll also hope. I get that
Guillaume: on this kind of the next step, right? Because when you talk to these agents, like usually people are just writing to them and sometimes they’ll this very clear, for instance, you are, you want to write code, but you are, you have a very clear idea of how you want the model to implement what you in mind.
But so here you are able to spend a lot of time writing. So it’s not really efficient on audio is really like a natural interface that is just not there yet, but I think it’s just gonna be the place.
Vibhu: How’s it like building, serving, inferencing, like we see a lot about, it’s very easy to take LMS off the shelf, serve them.
Fine tuning, deploying. I know you guys have a whole you have Ford, you have a whole stack of customizing, deploying. Is there a lag in getting that. Like distribution channel. Are you helping? There is. So like prompting, lms, you can have them be concise, verbose, all that.
They’re built on LM backbones, these models. How do you see all that?
Enterprise Deployment and Privacy
Guillaume: Yeah, I think this is a lot of what we’re doing with our own customers. Very [00:18:00] often they come to us, so it’s for different reasons. I think one reason is sometimes they have this lot of privacy concerns.
They have this data that it’s very sensitive. They don’t want data to leave. The companies, they wanted to stay. Inside the company. So we have them deploy model in-house. So either on a, either on premise or on private cloud. So they’re not worried that it’s given to a third party on the there some leakage.
Sometimes they have this kind of many companies have this different, sensitivity of data they have like sometimes channel chat can send it to the cloud has to stay there. So then it creates some kind of heterogeneous workflows where it’s annoying. You cannot send some data to the cloud.
This one you can, so here, when we actually deploy the model for them, they don’t have this consideration. They are like not worried that, this is going to leak. Everything is much easier. So we help them basically do this on the, so it’s one of the very proposition. But but the other is very often, when customers use this off the shelf close model, but very sad is that they are not leveraging, these data that have been collecting for four years or something for decades.
So much data. Sometimes it’s trillions of tokens of [00:19:00] data in a very specific domain. Their domain, which is data that you’ll not find in the public, on the public internet. So data on which, like close model, we actually not have access to one, which that’s going to be really good. So if they’re using like closed source models are basically not benefiting from all these insights.
All these data they have collected three years, they can always give it into the context that in France, but is never as good as if you actually train the modern analysis. So yes, that’s basically what we help them to do. We actually provide them some purchase, basically what we announced at GTC this week.
So we provide them with this, it’s basically like a platform with a lot of tools to actually help them process data. Trained on that. Yeah, it’s actually the same thing that we’re using in the science team. So it’s actually very better tested infrastructure, like a lot of efficient training cut base.
For a quality pre-training like a fine tuning, even doing S-F-T-I-L. So we help them do this using the same tools as what our science team is building is using. So since it’s tools that we’ve been using for two years now, it’s really better tested. It’s really sophisticated.
So it’s the same thing. We are giving to them, giving the company the same thing [00:20:00] that what are same still using internally actually build their own ai and it makes a really big difference. I think sometimes customers. And many in general don’t realize how much better the model becomes when you fine tune it on your own data.
And you can have a, your model is here. You start from there. You have a cross source model, which is sort here, but if you actually fine tune it can actually really go much further than this. And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything.
You don’t have to feed like 10 K tokens of contact at every query. So it’s it’s much easier. It’s a bit, I think using a closed source model is really sad because it basically puts. You are not leveraging all this data and you are going to be using the same model as all your old competitors when you’re actually using, everything you have been collected for years, which is really valuable.
So yeah. So we help basically customers do this. We have a lot of solution I mean deployed for engineers that go in the company that basically look at the problem customers are facing to look at what they’re struggling to do what we should do to solve it. So we help them solve them together.
So it’s I think our approach is a bit different, but here. [00:21:00] Some of their companies and competitors, it’s, we don’t just release an endpoint on sale, do some stuff on top of that, or we don’t just give a checkpoint. We really look very closely with customers. We look at the issues they have, we had them solve them.
We really make some tailored solution for the client are facing. Some example are also going to be, sometime we have some customers. They really wanted to have a really good model, really performance on some, like Asian languages on the, if you take some of the shelf models, they can speak it, they can write in this language, but it’s not amazing.
This language would be like maybe zero 1% of the mixture. So it has been included during training, but very little. So what we did here is upgrade. We trained a new model for them, but so this language was 50% of the mix, so it’s much, much stronger. It knows of the dialects, it knows the, so it’s yeah.
So it’s some example of things we can do and it’s really arbitrary, custom. I think you had some of their customers, for instance, they wanted some. They wanted some 3D model that can do audio with a very good function cable. So something you wanted to put in the car in particular, they wanted this to be offline because in a car you don’t necessarily have access to internet.
So [00:22:00] yeah. So here we can actually build the solutions. There is no like model out of the box on this. In the internet you have this very, you have this very general model generalist, like he’s strong model. But for things like this, they always want at specific solutions and on some other reasons.
Sometimes they come to us is because, like they, they experiment with some closed source model. They get some prototype. They’re happy with what they build. They, it works well. They’re happy with the performance, and then they want to go to production and then they analyze. But it’s extremely expensive.
You cannot push this. It’s so then they come back to us on this. They can help us build the same thing as this, but using something much cheaper on here. And here we can sometime be something 10 x cheaper by just functioning a model and it’ll be better OnPrem on their old server and also much cheaper as well.
So yeah,
swyx: that’s the drop pitch right there. Take all the
money.
Vibhu: And outside of that you do, we do put open wave models so people can do this themselves. I feel like not enough people go outta their way.
swyx: They’re not going to, they’re gonna ask them to do it as the expert. I
Guillaume: think initially we didn’t know, [00:23:00] we wanted completely short at the beginning of the company because, I think our study was not exactly the same as what it is today, but what we underestimated initially is the complexity of deploying this model and connecting them to everything to be sure it has access to the company knowledge on the, and it was, yeah, on, we were seeing customers struggling with this, but it was even, that was three years ago and no, things are much more complicated because now you don’t just have, text on SFT on a simple instruction following.
You have reasoning like your agents, you have like tools. You have a multimodal audio, so it’s much more complicated than before. And even back then it was hard for customers. So they really need, have some support and this is why actually providing like always some four D position as well. The process
Fine Tuning and Personalization
swyx: I’m curious is there also voice fine tuning that people do?
Pavan: So in this forge we also have a say unified framework. And the hope is like the er speech to text that we released earlier this year. And even the ER chart that we released last year. And I think a big people, I think there’s a big, rich ecosystem [00:24:00] of people fine tuning whisper, and people want the same thing with w so it’s much stronger than Whisper.
And yeah, the the platform offers that kind of fine tuning yeah, which could be any kind of fine tuning. Like for instance, even sometimes people want to support new languages to this, which are tail languages, which we hope to cover. Certain natively, but if there is a language where you data and you want to frank you, I think this is a good use case.
Or the other use cases, you, it’s the same language, like even English but it’s in a very domain specific way.
swyx: Yeah. Terminology, jargon, medical stuff.
Pavan: Exactly. And also there’s specific acoustic conditions like there’s a lot of noise or the, and. The model will do decently in most conditions, but you can always make it better.
And that those are some of the use cases where you can improve it e even further. And that’s one good use case for this and for text to speech. We’re just releasing it so we’ll have support for that soon too. I think it’s similar use case.
Voice Personalization
Pavan: It’s little different the kind of things that you want to extend a [00:25:00] text to speech model to, which could be like voice personalization, voice adaptation for enterprises.
Many enterprises need very specific kind of tone, very specific kind of like personality for this kind of voice. And all of those are like good use cases for fine tuning.
swyx: This one I was gonna ask you, we never talked about cloning voice clothing here. How important is it, right?
Like I can clone a famous person’s voice. Okay. But
Pavan: the main use case would be like for enterprise personalization, like enterprises need like a lot of customization. You don’t want the same. Voice for all the enterprises. Each enterprise want a customized, specialized something which is representative both their brand and also their, I guess safety considerations and the use case I think the kind of thing that you would deploy as a empathetic assistant in the context of a healthcare domain would be very different from the kind of thing that would be in a customer support bot and would be different from like more conversational aspects.
I think those are the. [00:26:00] Customizations you would expect from enterprise. And that’s the main use case, at least from our side.
Vibhu: My, my basic example is you don’t want to call to customer services and have the same exact voice. It’s just, it’s gonna be weird.
Long-Form Speech Models
Long-Form Speech Models
Vibhu: But also on the technical side of this, so there’s like a few things in TRO that I thought were pretty interesting.
He’s a big fan of this paper. Oh, he said very good paper. He said this is the best SR paper he’s ever read. Yeah. I’ve hyped up this voice paper enough. We covered it. Somewhere, but a big thing. So Whisper is known for 32nd generation a 32nd processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done.
Even little niches of how the padding is. So it’s very much needed. You need to have that padding in there, the synthetic data generation around this. I’m wondering if you can share the same about the new speech to text, right? Text to speech. So how do you. How do you generate long form, coherent?
How do you generate, how do you do that? And then any gems? Is there gonna be a paper?
Pavan: Yeah. Yeah. They would be a technical report. Okay. Yeah. I think I could have a lot of details.
Real-Time Encoder Advances
Pavan: But me I think the [00:27:00] summary of it, actually, some of the considerations in this paper were, because we started with the wipa encoder as the starting point, and now we have in-house encoders, like the bigger time model, for instance, which we released in January.
Also release a technical report for that real time model as well, which is this dual stream architecture. It’s an interesting architecture. You should check it out. And there we have a causal encoder and I don’t think there’s any strong, multilingual causal encoder out in the community. So we thought it’s a good contribution.
So that’s one nice encoder there. Other people want to adapt. That’s a good end code. And we train it from scratch. I think her. Post stack is now mature enough that we are able to train super strong ENC codes. And some of these considerations, like spatting and stuff, is a function of the Whisper ENC code.
And now that we train encoders, inhouse the design concentrations are different.
Scaling Context for TTS
Pavan: And for the question on text to speech, I think that’s also leans onto the original auto aggressive decoder backbone. I think, it says very, almost identical considerations. I think the long context in it’s not even long con, [00:28:00] so the model processes audio at 12.5 herds, so one second maps to like 12.5 tokens.
So I think one minute is like 7.8 tokens. You can get like up to 10 minutes in eight K context window and get half an hour and 30 K context window. So that’s and 30 2K context is something that’s we are very comfortable training on. We can extend it even much longer. 1 48 K. Okay. You can naturally see how it can extend to even our long generations.
Yeah. We need the. Like data recipe and the whole algorithm to work coherently enough through such long context. But the techniques are some way very similar to the text, long context modeling. And the key differences, it’s just doing flow matching order regressively instead of a text open prediction.
swyx: Okay. I think that was most, most of the sort of voice questions that we had. But
What Makes a Model Small
Vibhu: I have a big question on Mr. Al, Mr. Small. So what is small? How do we define [00:29:00] small? What is this? What is this? I remember the days of Misal seven B on my laptop. The snuff fitting on my laptop. I could run it on the big laptop, but
Guillaume: it’s just additional.
Question of terminology, like here what we did, baseball is north active parameters, but it’s true. Really not give it another name, but yeah, we could have called it medium, but only, I,
I suppose it’s a model that we released mixture of experts. It’s a model that combines different model before which we were doing the same, is that we had one model, general model for Israel. Doing instruction following, were like a separate model that was Devrel trial. So qu coding specify specific to code with another model for Reason Maal.
So this were separate artifacts built by different team at trial on what we’re doing is basically merging all of this. It was, you had pixel trial was the first vision model. We was like a separate model on the way we do things internally is that we have one team focus on one capability, build one model.
On the means mature, mature enough, we decide to merge this into the [00:30:00] matrix. But here it was the first time we basically match all of this into one. But there are some other things we did at first time to merge time, for instance, like more capabilities or function coding I think would be, are, it’s going to be much, much better in this trial, small platform.
But but yeah, so it’s our latest model on the working is,
Vibhu: and yeah, key things is it’s very sparse. Six, be active pretty efficient to serve. 2 56 K context. Yeah,
Merging Capabilities vs Specialists
swyx: I think what’s interesting is just this general theory of developing individual capabilities in different teams and then merging them.
Where is this going gonna end up?
Vibhu: Like we’ve seen the five things put together in this. Yeah. What are the next five teams?
swyx: I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model. This was what they were selling. All modalities and all modalities out.
But I feel like you might do it.
Guillaume: I think there’s some mod where it’s not competitive use, for instance for audio. For audio here, if you want to do transcription, I think it makes no sense to use a model. If you just want to trans tech it, it’ll be very inefficient. If you want to do audio, you probably just want to be the [00:31:00] one VR 3D model performance essentially
swyx: the same.
It’s going to be incredibly cheaper. So here, that’s why we want
Guillaume: to have a separate but just does this. Yeah, I think the question is just, yeah. If you are to, to your model. By speech and you asking like a very complex questions on how you do this on the, just to cascade things. Do you want to put a d in a model that has like a one key around it?
It’s like a, not a competitive discussion, I think unaware if you doing into the direction, but that’s possible. Of course. But yeah. But I think for us, the next capabilities we want to try to integrate into these models when we are going to be yes, like marketing or no reasoning better, I think more capabilities that people don’t talk too much about, but at high bottom, I think for our customers in our, on different industries, for instance, things are around like a legal computer.
I design all these things that is this males out of the box are to put at that. Because people, if you don’t prioritize this, there is not like too benchmark on that. But
swyx: this done how to
Guillaume: make this good and this just start to do the work. Extracting some that processing it [00:32:00] expression. So yeah.
But we are offering the imagine to this.
swyx: I think for voice. Yeah. The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right? Which people don’t understand spatial audio because like most TTS is just oh, I’m speaking to a microphone in perfect studio quality.
But when you have video, like the voice moves around.
Pavan: That’s true. The constitution was a little different in the sense that there it’s like a a standalone artifact where you get the whole thing and you consume it. But in a conversational setting, it’s a, you need the extreme low latency.
swyx: Yeah,
Pavan: streaming would be one of the primary concentrations.
swyx: You can build a giant company just doing that, right? So you don’t need to do the voice, but I was just know on the theme of merging modalities, that is something I, I am like, wow. Like I didn’t, everyone up till, let’s say mid last year was just doing these like pipelines of okay, we’ll stitch a TTS model with a voice thing and a lip sync [00:33:00] thing and what have you.
Nope. Just giant model. Yeah.
Open Source Mission
Vibhu: I have a two part question. So one is, it’s still open. It seems like open source is still very core to what you guys do and I just have to plug your paper. Jan 2024. This is the one trial of experts like. Very fundamental research on how to do good.
Moes paper comes out very good paper for anyone. That’s just side tangent. No.
swyx: This thing caused, we bring back, eight by 22 was like the nuclear bomb for open source. I think it takes Shouldn be more seven B more. Yeah. Yeah. But this is a bigger opposite than me.
Yeah. Yeah I don’t remember this. I remember, I don’t think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think. Yeah. The model was did as well.
Vibhu: It’s just a little update probably.
swyx: Yeah. No, but you have a point to make.
Vibhu: No, you gotta check that. But then, I just want to hear more broadly on open source for you guys, and when you had asked earlier [00:34:00] about what’s next, what are the other, side tapes working on you. You put out Lean straw. This,
swyx: it’s not necessarily surprise. I was like, I don’t, this doesn’t fit my mental model or Misra.
Guillaume: Yeah. First for open source in general, I think it’s really something which looks to the January of the company. I think we started it per once, is we so we have open sourcing with, since the beginning and even before this. So before this, so me and Tim were at Meta, we released LA and I think what was really nice.
To see that before this, for most researchers like universities, it was impossible to work on elements. There was no alien outside. And if you look at many of the techniques that were developed after, for instance, was open source all this post-training approaches like even DPOD, like preference optimization, all of this were done by people that had access to this portal.
And it’ll have been impossible to do without this. So it’s really making sense, move faster. So we really want to contribute to this ecosystem. I think like the deep and also like very lot of impact. All these papers that are I think in the open source community are really helping the science community as a whole to move faster.
So [00:35:00] we want contribute to this ecosystem. That’s why we’re releasing very detailed technical reports. So ma trial and our first reason model, and ation, lot of results, things that work, things that did not work as well. Think helpful on the, yeah, so for the audio model also to share a lot of details, share of them for real time model.
And the, yeah, so we really want to continue this, basically belong to this community of people who share science. I think we really don’t want to be, leading in a world where the smartest model, the best models are only behind, close doors. Only accessible to a shoe companies that we, as a power to decide we can use them on it.
I think it’s a scary future. We don’t want to live in, we really want this model to be accessible to anyone that want. Intelligence to be used unaccessible by anyone who can use it. So yeah, so that’s why we are pushing this mission and source model. Yeah. So not, so yeah, no strategy. So it’s open source, not the first model, so not the best on the Yeah.
Lean and Formal Proofs
Guillaume: LIN trial I think is also one step into this direction. So it’s yeah, a bit different than what we are usually releasing. But we have a small team internally [00:36:00] working on them. Formal proofing, formal math. So I think a subject we care about in general and we were working on reasoning. I think we started too early before doing reasoning without LMD is very hard, especially when you work with formal systems because the amount of data you have is negligible.
It’s addressable community of people writing like formal proofs. But the reason why we like it is because I think there is if you look at what people are doing with reasoning, is there, the problems that you can use. Are usually going to be problems where you can verify the output. So for instance, all this ai ME problem where the solution is a number between 100, like a thousand.
So you can verify, compare this with a reference or it’s an expression. You can actually compare the output expression generic with the reference. But there are many, most of them have problem and most of the reason problem. There is no like way to easily verify the solution. If the question is show that F is continuous, cannot compare in the reference, right?
If it’s a probe that this is true or probes is properties, there is no way to. You cannot act, simply verify the correctness of your proof. So it’s hard to apply the, there is no referable reward here. So [00:37:00] what you could provide is of course, like a judge and judge that will look at your proof. But it’s very hard and it’s very, you could do certain, some reward hacking happening there.
So it’s difficult. You could provide like a reference proof, but then there are also many ways to prove the same thing. So if the model says give negative reward because it’s a different poop, maybe it was still digit proof, just different. So it’s not going to work well. What’s nice with lean and with formal probing is that you don’t have to worry about this whatsoever.
We just,
swyx: they’re all function is largely compiles in lean is functionally the same. Exactly.
Guillaume: It’s like a problem if it compiles it’s correct. It’s very easy. And you can apply this and then you can,
swyx: it’s just way too small. So no human will actually go and do it.
Guillaume: Yeah, that’s exactly.
It’s the only people can do it. It’s like a very small committee of people doing a PhD on that. So it’s super small. And it’s sad because it’s actually very useful on not just mat, but also in software verification. So for instance, software verification today. So tiny market. Very few industries work on this and we need that.
It’s usually going to be like companies like building airplanes, air robotics,
swyx: like
Guillaume: things [00:38:00] where they absolutely want to be sure. Life depend on this, but it’s very rare that people formally verify the correctness of their software. But I think one of the reasons for this is simply that it’s just hard to do.
swyx: Are you think of TLA plus? It’s the language that some people do for software verification? No. That people use in a ference, but but yeah, it’s the reason I think why people don’t use it more and why this industry is not as big as could be is because it’s very hard. But now with cutting edges that are there, it’s going to be very different.
Guillaume: We’re going to see much more of this. So I think yes, industry there is going to be much larger in the future that we, these models. So yeah. Here also anticipating this a little bit, we wanted to work on that because it’s proving like a math theory and like a, essentially the same tools.
swyx: Yeah.
Reasoning Transfer and Agents
swyx: One of my theories is that because the proofs takes so long, it’s actually just a proxy for long horizon reasoning and coherence and planning. Maybe a lot of people will say okay, it’s for people who like math. It’s for being okay. It’s like a niche math language. Who cares? But actually, and you use this as part of your data mixture for [00:39:00] post-training and reasoning, actually, it might spike everywhere else.
Yeah. And I think that’s un under explored or no one’s like really put out a definitive paper on how this generalizes.
Guillaume: Yeah, absolutely. And
Pavan: I think even
Guillaume: that’s what we’re seeing already. For instance, you should do some reasoning on math as then the American should do reason even.
Yeah. In the early stage. So we, the, there is some transfer, some sort of emergence that happens. And I think some, it’s also interesting, it’s not just I think the topic in general, but it’s, there is a lot of connection with this on including agents because. Sometimes the model can see like a three that it has to prove it’s very complex, but then it can take the initiative to say, I’m going to prove this three lr.
I’m going to suggest three Rs, and I’m going to in parallel prove each R. So three of them in parallel with sub agents, but I’m also going to prove them in theory and the three tool so you can do this also. Pretty interesting. You can, even if you fail to put one of the LeMar, you can actually, maybe you succeed to put the normal lema too, so you get some possible reward here.
So it’s a bit less Spartan issue, just get to zero one for the entire thing. [00:40:00] So it’s pretty interesting. I think we can actually,
Vibhu: yeah, it’s also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting yeah, similar score wise, you are, thirty, seventy, a hundred fifty, three hundred bucks.
Smaller.
swyx: I think cost is a bit unfair, right? ‘cause this one is at like inference cost. It’s always there on top with their margins on top of it. But, we don’t know anything else, so we gotta figure it out.
Vibhu: Okay.
Next Frontiers in Training
Vibhu: I did wanna actually push on that more. Not on cost, but you mentioned about, okay, it’s a great way to have verifiable long context reasoning.
What are other frontiers that, I’m sure you guys are working on internally, there’s a lot of push of people pushing back on pre-training. Scaling, RL pushing, compute towards having more than half of your training budget. All on rl. Where are you guys seeing the frontier of research in that?
Guillaume: You mean the
Vibhu: just in foundation model training in the next, one thing that you guys do actually is you do fundamental research from the ground up, right? So you probably have a really good look at where you can [00:41:00] forecast this out.
Guillaume: Yeah. I think for us we’re still working a lot on the pre-training side.
I think we are very far from situational, the pre-training. I think ML four preprinting will be like big step compared to everything we have done before. So we are pretty excited about this. And I think on the other side, I think now we have more and more to think about this algorithm that will actually support this very long trajectories.
I think when it was, for instance, GRPO for it doesn’t really work this any bit of policy. Which was okay initially because you are solving math problem that can be solved in like a few thousand tokens. So the model can alize them pretty quickly. So when you do your update, the model is never too far off.
It’s never too far off. But now when you are moving towards this kind of problems where certain takes hours, like six hours to get a reward, then your model is co pick places. So you have bi new infrastructure that supports this, but also new A, so now everything we’re doing internally, we’re trying to. Build some infra that we actually anticipate is what we have in six months, one now, which is this extremely no scenarios on the, I think when we started Missal, part of me and [00:42:00] we wanted to, is very nice under element where people are there, they can do research, they like with a lot of resources.
So it was nice. I think things changed a lot when I think when J Pity came out. I think after that I think was. This one is same again. But but yeah, but it was nice. And I think we also want to work part of this descrip before
swyx: coming to the end.
Hiring and Team Footprint
swyx: We’re just, obviously, I think you guys are doing incredible work.
You’ve, they are a very impressive vision for open source and for voice. What are you hiring for? What’s the what are you looking for that you are trying to join the company?
Guillaume: Yeah, so we are hiring a lot of people in our sense team. We’re hiring, in all our offices. So we have a, our H two is in France in Paris.
We have a small team in London. We like a team in Pato as well. Co we open some offices in in SAU, in Poland. So one in Zurich. We also like some presence in New York as well on Sooner one in San Francisco. So we all bit either way also like hiring remotely. So we’re going the team trying to hire like very strong people.
I think we want to stay, so the team is not. Instead of fairly small team. [00:43:00] But I think we want to keep it that way. ‘Cause we we find it quite efficient. So like a small team they agile so yeah.
swyx: Okay.
AI for Science Partnerships
swyx: Let’s focus on science and the forward deployed. We actually are strong believers in science.
We started the our new science pod that focuses specifically on the air for science. What areas do you think are the most promis.
Guillaume: What we’re pretty excited about right now, and something we have already started doing or that we’d probably be able to share more about this in a couple of months, is that we are exploring AI for science.
And there are a lot of areas where we think that you could get some extremely promising buzz. If you were to apply AI in these domains. There are a lot of long inputs. You just have to find these domains where actually AI has not been yet applied, and it’s usually hard to do because the people working in those domains don’t necessarily know the capability of these models.
They don’t know. How I would just have to pair them with Yeah, exactly. Your researcher slashing, which is actually hard to do. But this matching, we’re doing it naturally with our customers. So we have some company we are very closely with. So for instance, ISM Andreesen are one of our partners, so we’re doing some research with them on their other, like tons of extremely interesting problems.
Columns in physics, in [00:44:00] science matter science that they’re essentially the only ones to work on. ‘cause they’re doing something No, no one else is doing on the, yeah. So there are many domains where AI can actually revolutionize things. Just you have to think about it on you familiar with what can do or to apply it.
So yeah, it’s something where more modeling with our partners, with our customers sort AI for s, but.
swyx: Yeah. Okay.
Forward Deployed Skills
swyx: And then for deployed what it makes a good four deployed engineer, what do they need? Where do people fail?
Guillaume: I think it’s usually you need people that are very familiar with the tech and not necessarily with a lot of research expertise, but that are actually pretty good at using this model that can actually like that know how to do functioning, that know how to like, start some error pipeline.
And it’s it’s not easy. It’s something that mucus. Majority of companies will not be able to do this on their own. So here I think we need people that are, that like to solve problems that are accept solving some complex, very concrete problem. It’s applied science basically.
And yeah, so I think it’s not too different. I think from the case you need in research because it’s essentially you are trying to find solutions to problems that in [00:45:00] customers have not yet. So sometimes it’s easy. Sometimes you’re here to do the work. You have to like create synthetic data.
Find some edge case. So it can be, yeah. Depends on the problem. But but yeah, you have to, I think it also a bit of patience on the be creative. I think very similar skill is Asian,
Pavan: the diversity of the work they do. It always surprises me. It’s it’s, it goes all the way from the kind of stuff they encounter in industries.
It’s just very interesting. I think.
swyx: Any fun like success anecdotes.
Guillaume: Yeah, it can be actually training this small model on edge that just we do one specific thing can be like training some very large model without some specific languages as well. Making models really good at some tube use, like for instance, computer ID design, these kind of things.
Is that pairing with vision as well? Yeah,
Pavan: and the fact detection for chips or like in, in factories identifying things like it, the. Diversity could be anything where you can deploy these foundation models. So yeah the work to make it work in that specific setting, basically whatever it takes to make it like add value in that, by the way, workflow.
Vibhu: Yeah. [00:46:00] And it goes across the stack, right? Like even just pulling up the website like.
swyx: It’s so broad on compute. It is so broad.
Vibhu: We didn’t even touch on if you have a coding CLI tool. One thing you guys were actually like, I think the first tool was agents, ral agents. You had the agent builder, you can serve it via API and all that.
And I’m guessing forward deploy people.
Guillaume: Yeah.
Vibhu: Help build that out and stuff.
Customer Feedback Loop
Guillaume: It is also why we are, so we’re doing many things, but I think that’s also part of the value proposition that sometime know customers. They’re always very. Extremely careful about their data and they don’t want to, they don’t like, trusting so many partners, trusting one partner for code, giving the data to another third party for like audios and another one.
So they don’t like this here. What they really like with our approach that we can help them on anything so they don’t have to send the data to so many clouds. So yeah,
swyx: I think that there can be many orders of magnitude more. F Ds then research scientists and they don’t need your full experience, but they’re still super variable to customers
Guillaume: in practice.
These two teams [00:47:00] are still quite intertwine, very often. Yeah. So first of all, they’re using the same tools, the same data pipeline and everything on the, it’s it’s very helpful for the science team to get the feedback and the solution team ‘cause they can. Look at these customers are trying to do this.
This is not working. It can really be show in the next version. Yeah. But this is basically a real world eval. Yeah, it’s real world eval and it’s not something, for instance, if you’re just working in the lab, it’s just ships model. But you don’t do this work of for customers. You have no idea for whether your model is good at this H case.
For instance, you even in year found this, right? So yeah, there is a very gap, big gap between the public benchmarks that are very like academic. On
Pavan: the rare cases are just very diverse and in the specific concept of a customer, you can fine tune and make it like first evaluate, create a solid eval, benchmark, and then measure in the context of their, the kind of audio.
Like for instance, one use case is literally just, there’s the word for kids and they have to just say it out. It’s a very specific thing. You’re just saying one word and then you have to you, you’ll grade the kid whether they did it right or not. It’s [00:48:00] like R for, but so there’re very diverse use cases and the idea is that they, the.
Applied scientist engineer will go and make it better. And then from the learnings we incorporate it into the base model itself. So it’s it’s just better out of the box.
Vibhu: Yeah. It’s a good full circle system. Like the foundation model evals are all just proxies of what you really, you’re never gonna have one that says it, it doesn’t make sense for there to be, a one word transcription like that.
It’s not something you wanna fit on. Perfect.
Wrap Up and Thanks
swyx: Everyone should go check out everything that Michelle has to offer and try the TTS model, which will link in the show notes. But thank you so much for coming tha thanks. Such a stretch.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Materials science is the unsung hero of the science world. Behind every physical product you interact was decades of research into getting the properties of materials just right. Your gym clothes contain synthetic fibers developed over decades. The glass screen, diodes, and chip substrate technology needed to read this blog post were only viable due to many teams of material scientists.
Our guest Prof. Heather Kulik was one of the first material scientists to realize that there was alpha in combining computational tools with data driven modeling — she did AI for science before it was cool. She has a hard-fought perspective for how to succeed in this field. Yes, she believes the wins are real. To get there you must work hard to deeply integrate domain expertise with AI techniques, and also maintain a discriminating mind. Ultimately what matters is you succeed in the lab, and nature doesn’t care about how hyped a model is. These lessons personally resonated with the Latent.Space Science team and our own experience.
This episode is a must watch for all aspiring AI for science practitioners. A few highlights:
Designing new polymers with AI: Heather’s group recently used AI to design new polymers that are significantly stronger. These materials were created and tested in the lab, and the scientists who built them were surprised by the designs. The AI had figured out certain building blocks could break in a novel way. The AI discovered a purely quantum mechanical effect, and after convincing their lab collaborators to actually synthesize it, the material turned out to be four times tougher!

The twenty-two-atom ligand challenge: When asked about the role and need of human scientists, Heather points out that AI has a strong understanding of academic chemistry, but is still lacking intuition. Every time an LLM is updated, Heather asks it to design a ligand that contains exactly twenty-two heavy atoms. She has yet to find one that can succeed at this seemingly simple task that any expert could do in a second! Is this the chemistry counterpart to counting ‘r’s in strawberry?

Side note: Heather joked that this comment would date itself immediately, so we decided to see if this was still true three months after recording. We found some interesting results! We asked both Claude and ChatGPT to design a 22 atom ligand for both a metal-organic framework (MOF) and a Kinase protein.
* For the Kinase, both models got it right: Claude pulled out RDKit in a python script and iterated on several designs, whereas ChatGPT just one-shotted it.
* For MOFs, both models got it wrong, generating ligands with 21, 23, or 24 atoms, yet stubbornly not getting 22 atoms.
Is there something different about how LLMs reason in the materials and bio domains?
Materials vs biology: The two biggest domains of AI in science have been biology and materials. We asked Heather if there could be an AlphaFold moment for materials. Her answer reframes how we should think about the field:
* First, the datasets in material science are woefully lacking in comparison to the bio world. The closest to ground truth in most cases are noisy DFT datasets. These are just approximations to the real world! The datasets that are accurate are all boring, as Heather quipped “We have really good datasets for really boring chemistry.” Furthermore, good experimental structures are hard to come by and require interpretation. So generating generating high-quality, novel datasets at scale would really drive the field forward.
* More philosophically, AlphaFold is making predictions in a fairly limited space: there are just twenty amino acids. Sure, even here AlphaFold doesn’t get everything right, but it seems plausible that one could learn the entire design space. For materials, each element is a new set of interactions and chemistry, with little to no transferability. This is a massive open problem in material science that we hope some of the smartest AI scientists will want to work on!
The difficulties of trusting the literature: Heather’s team has spent the last few years using NLP and later LLMs to extract data from literature. Even a few thousand data points from these papers can be valuable for guiding her group’s work. One surprising result: sometimes the reported values for a property (say temperature) do not match up with the graphs in the papers! So there’s lots of potential in using LLMs to mine data from the literature, just do it with care.
The role of academia in an ever-changing world: One theme that has been running through many of our conversations has been the changing role of the academic — and the scientist — in science. When startups are raising $100s of millions and hyperscalers and Big Pharma are all ramping up AI-for-science efforts, the academic researcher needs both resources and judgement about problems to chase more than ever.
Resources include data that is organized for machine learning, access to high throughput experimentation labs, and compute resources. These are all things that academics can build together. More importantly, Heather emphasizes curiosity about problems that haven’t hit the radar of the heavily capitalized AI companies. After so many years on the forefront of AI for Science, Heather’s judgement that Chemical Engineering and Material Science still need curious people asking questions with no clear path to money is a welcome beacon in the AI fog.

Full Video podcast
Is on Youtube!



This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Mar 23 update for Latent Spacenauts: this episode was recorded before the Dreamer team announced they were joining Meta Superintelligence Labs, and it turned out to be the last interview they did before the news became public. Consider this a snapshot from just before the transition!
In 2024, David Singleton left Stripe and joined forces with Hugo Barra for a buzzy stealth startup named /dev/agents. This month they emerged out as Dreamer, a consumer-first platform to discover, build, and use AI agents and agentic apps, centered on a personal “Sidekick” that helps users customize experiences via natural language.
Sidekick is nothing less than an “agent that builds agents”, with all the complexity that that entails:
You’ve seen many many website builder, app builder, and even agent builder startups by now, but our favorite detail is the sheer amount of work that has gone into the “full stack” nature of the platform, including shipping their own SDK, logging, database, prompt management, serverless functions, and so on. Most platforms restrict the tech stack you can use just to get off the ground — Dreamer does it “right” by letting you push whatever arbitrary code you want to their VMs.
Paying the Builders
Of course former leaders of Stripe and Android would not stop at just building the tools, but also building the ecosystem. Dreamer is deeply aware of the 4 sided network effect it has going on and is ready to fund all of it.

It’s time to Dream!

Full Video Episode
on youtube.

Transcript
[00:00:00] Meet Dreamer Purple
[00:00:00] swyx: Okay, we’re here in the studio with David Singleton. Welcome.
[00:00:08] David Singleton: Hey, Wix. It’s great to be here.
[00:00:09] swyx: It’s great to have you. Uh, we have very sympa that your company color is the same as Lean Spaces color.
[00:00:15] David Singleton: That’s right. Dreamer Purple.
[00:00:17] swyx: It used to be Devrel agents, which I thought was very cool. It’s like you call back to Devrel Payments.
[00:00:22] David Singleton: Yeah.
[00:00:22] swyx: And you were obviously CTO Stripe. And talk to me about just the origin or thinking process behind Dreamer. Yeah. And maybe, maybe start with like, what, what is Dreamer?
[00:00:31] David Singleton: Yeah.
[00:00:31] What Is Dreamer
[00:00:31] David Singleton: So Dreamer is a new product, uh, which everyone can come and play with today. Um, it’s a place where everyone, literally, everyone can discover, build, and enjoy and use AI agents and agenda apps.
[00:00:45] And we really did design it for consumers, for folks who are not necessarily. Uh, have any kind of technical background. It’s really aimed at everyone. I think often of my sister, she’s very smart. She’s not in the slightest bit technical. She has lots of problems in her life that [00:01:00] she would like to be able to have great software and intelligent software to solve.
[00:01:04] But you know, even with the rise of tools like Cloud Code and so forth, she’s got no way to get started. And Dreamer is a place where she can come in, grab some intelligent apps that other people in the community have built, start using them right away, and solve real problems in her life.
[00:01:19] Sidekick And Waitlist
[00:01:19] David Singleton: And at the core, we have a personal agent called the Sidekick.
[00:01:24] Um, you can give your sidekick a name, you can give it its own personality, and it really helps you across your entire day, your life. It helps you use all of the agents on the platform, and it also helps you build anything you want. And we’ve been working in this for a little while. We recently launched in beta.
[00:01:41] So anyone can go to dreamer.com, join the wait list. Um, and we have many, many, many people in the community now who are building really fun, really powerful, really useful. Agents and the agentic apps for themselves.
[00:01:54] swyx: I think we’re gonna go right into a demo. Yeah. I just wanna make an observation that, uh, you, you, [00:02:00] you put discover first before build.
[00:02:02] Mm-hmm. But actually, at least for the engineers in the audience. ‘cause we are primarily engineers and you’re primarily targeting consumers, right?
[00:02:08] David Singleton: Yeah.
[00:02:08] swyx: For engineers. Like, there’s a huge full stack of stuff, which we’re gonna dive into. Let’s write. It’s so impressive. I’m like, holy s**t, this, this is what I’ve always wanted.
[00:02:16] Cool. Uh, so, so I think that’s really good and I’ve, in some ways, I think given your background given, uh, Hugo’s, is it Hugo? Hugo.
[00:02:24] David Singleton: Hugo. Hugo Bar. Yeah.
[00:02:25] swyx: Hugo, it’s not surprising that you can basically kind of build an app store Yeah. For agents.
[00:02:30] David Singleton: Yeah. So Hugo was my co-founder. Yeah. Um, Hugo and I met with our other co-founder Nicholas Checkoff in the very early days of Android at Google, where we were building Google’s first mobile apps.
[00:02:41] Uh, we then contributed to very core pieces of Android itself. And you’re right, we were really excited about building two things. One, solving a bunch of problems. That this breakthrough technology here I’m talking about mobile needed to have solved in order to make it work for real people at scale. And then secondly, building this ecosystem, um, [00:03:00] of third party developers using the Play Store, um, and able to deliver way more value on the platform than we could have delivered on our own.
[00:03:08] And we think about Dreamer in exactly the same way. So I was working at Stripe, as you mentioned, and we had the opportunity to put some of the very first AI agent systems in the world into production. And from the moment we did the first of those, I was just struck with a strong sense of conviction that this is breakthrough technology that’s gonna change how all of us work with computers and phones and so forth, all of the, the technology in our lives, but.
[00:03:34] There’s a lot of problems to be solved, for real people to be able to make this approachable. Um, and it really is kind of a direct analog for what we were solving back in the early days of mobile apps at Google and, and Android. So it’s, it’s been fun to bring that to life.
[00:03:47] swyx: Yeah. Uh, let’s look at it.
[00:03:48] David Singleton: Yeah, let’s take a look.
[00:03:49] Dashboard And Daily Briefing
[00:03:49] David Singleton: So, uh, dreamer.com, this is our homepage. This is where you can come and, uh, watch some videos about what is here and sign up for the wait list. Once
[00:03:57] swyx: you, I, I just wanna say for those listening, ‘cause we have a lot, you [00:04:00] know, switch to YouTube, look at the animations. So much care.
[00:04:03] David Singleton: We, we really care about, uh, this product being fun.
[00:04:07] Uh, and, and interesting to use. Obviously a lot of people are using it to do real important stuff. You can do real work, uh, here, uh, but also you can build fun things too. Once you get off of our wait list, you’ll come into the product. The first thing that happens is you’ll have a conversation with your side cake, which is this little friendly, uh, character here.
[00:04:27] And psychic will seek to get to know you and understand you. What do you care about? And will help you discover and build your first AI agents or agentic apps. After that, you’re, you’re gonna have a dashboard. This is my dashboard. Everyone’s is different. Um, you can see I have a few things here. I have a feed.
[00:04:42] So a lot of our agents do things in the background when you’re not looking and the feed is how they let you know what they’ve been up to. I have, uh, some widgets, uh, from apps that I have built. Uh, this one is called Calendar Hero. Uh, this is something that I installed from the gallery. Uh, so built by someone in our community.
[00:04:59] It’s a [00:05:00] really powerful calendar app because for each of my meetings, if it’s with someone I don’t already know, well it’ll actually go off and research it, um, and give me both a history of my interactions with those people and also a bunch of, you know, public useful information to, to get started. One of the things I love about this particular app is that every day it generates a podcast, um, a daily briefing.
[00:05:24] And one of the things that we’ve done with the platform is we’ve made it possible for all the things that agents do to show up in places that you care about. So if you look over here, this is the screen in my phone, and if I go ahead and open my Apple Podcasts, you can see right here. Your Daily briefing podcast is ready.
[00:05:39] This was produced by an agent running in my Dreamer account, and it was very easy by scanning a QR code to connect it to my Apple podcast. That’s what I listened to in the car now every morning. Yeah. On my way to work.
[00:05:50] swyx: It, it
[00:05:50] David Singleton: preps me for, for my day.
[00:05:52] swyx: So one additional bit of context. I asked you immediately after seeing this was like, what, what about, I wanna talk back to my agent and you said you actually started with voice and then you went to [00:06:00] podcasts.
[00:06:00] ‘cause it’s nice to have it pre downloaded
[00:06:02] David Singleton: that, right? That’s right. Um, yeah, we, you, you can talk to your sidekick. So, you know, on mobile we have, uh, a dreamer app and you can talk to the sidekick right here. Um, but we’ve actually found that making things, uh, show up in the other apps that you already use in your life is incredibly powerful.
[00:06:19] So let’s take a look at what’s kind of under the hood here.
[00:06:21] Gallery Tools And Payouts
[00:06:21] David Singleton: So I already mentioned that we have a gallery, so this is where you’ll find a lot of agents from our community. Uh, there’s. Many at this point, hundreds. And they are solving all kinds of, uh, use cases. I’d say the the top use cases are on personal productivity, but also a lot of information management that can range from personal information like docs and so forth, managing your emails.
[00:06:42] It also ranges out to public information that you might be interested in, but you need something to help manage the, the kind of fire hose of stuff that’s coming at you. For instance, I have, um, an agent which looks at all the AI news, um, all the time. There’s a lot of it and it finds the stuff that I would actually be [00:07:00] interested in, um, and I find it incredibly useful.
[00:07:03] So these are agents that you can install that other people have built. Anything that you install on Dreamer, you can actually just say, I wanna start making some changes, and we’ll look at that in a second. But in natural language, with the sidekicks help, you can change any of these experiences to work just the way you want them.
[00:07:18] But the base layer of the system are tools. So you know, as well as anyone swyx, that any AI system is only as good as the quality of data that it can pull in and the quality of action it can take. So before we launched our beta, we worked very hard to make sure that we seeded our tools with a bunch of very high quality and powerful integrations.
[00:07:39] So, you know, for instance, this is real Google search, this is actual Gmail. Um, and you can do very useful things with those. But also this is a platform for everyone. And as we got started talking to people in our alpha community, a whole bunch of sports use cases popped out and we realized if you want to build something cool for sports with ai, you need really high quality live data.
[00:07:58] So look at these [00:08:00] Formula one M-L-B-N-F-L, uh, these are tools, uh, that we’ve built. We’ve done a, these are not data scraped off the web. This is a, a direct data feed integration. And because it’s live and ‘cause it’s high quality, you can build really powerful stuff. But tools is not something that we are just going to kind of control ourselves.
[00:08:19] The platform is open for tool Builders to contribute tools that anyone on Dreamer can use. So, um, this is actually the place in the platform where I think software engineers, um, well number one, would love for you to come and play with it. Uh, but software engineers are really gonna build, um, a lot of powerful stuff into the system.
[00:08:38] And we are actually sharing something for the first time on this podcast, which there is, uh, tool builders on Dreamer get paid. So if you publish a tool to the platform and a lot of agents use it, you’ll actually get paid, uh, in proportion to their usage. And we’d love for folks to come and give this a try.
[00:08:54] We’ve got good docs that help you get started and you can build things that, you know, scratch your own itch. For instance, someone built this [00:09:00] Ski Bum tool, which provides live snow conditions for a bunch of, uh, ski resorts. I’d love to show you how I’ve used that in a second. And also we have some tools, partners where the tools themselves are paper use.
[00:09:12] So for instance, parallel web systems is a premium tool. Uh, you can do really cool stuff with it. Um, it’s a a, an agentic web research tool. And that one, because it’s expensive to operate, is paid on a, on a per usage basis. But if you’re coming in to build agents on the platform, even the premium tools, you get a free trial.
[00:09:29] So you get a chance to actually try them out, make sure that the use case is good for you before you decide to, to to sign up. So that’s tools. So we have the gallery, we have tools, and then the sidekick helps us put all of this together to build agents. We do that in the agents studio. You can also do this on your phone, but if I open up Agent Studio here on Desktop psychic’s, just gonna start a conversation about what you want to build together.
[00:09:51] I’d love to show you one that I made recently.
[00:09:53] swyx: Let’s do
[00:09:53] David Singleton: it.
[00:09:53] Building A Conference App
[00:09:53] David Singleton: Um, let’s look at something that hopefully is kind of near and dear to your heart. So one of the things I love about Dreamer and this kind of moment in technology is that if you think about it. There are all these things in your life where, have you ever gone to a conference?
[00:10:09] I know you have. Right? And, uh, big conferences have apps. Um, and these apps are usually built by agencies and they’re, they’re usually actually quite expensive to build. I’ve been involved in running some of these myself. And how many conferences have you been to where the app was good? Zero. Honestly.
[00:10:23] swyx: Exactly. Zero,
[00:10:24] David Singleton: maybe one. I, I’ve, I’ve been to one conference. That was pretty good. Wait, wait session sessions. Um, but, but the point is, they’re rarely great pieces of software. Right. And they’re also expensive to build, but they’re, they’re interesting ‘cause they’re episodic, they last for this one thing. Um, and then they’re, they’re not relevant anymore.
[00:10:43] Um,
[00:10:43] swyx: and so it’s the worst feeling to invest in them because, you know, it’s like, it’s got a limited. Date?
[00:10:48] David Singleton: Absolutely. So I decided to build, uh, a conference app for your AI engineer conference. Amazing. Uh, on Dreamer. One of the things that Swix has done, uh, which I [00:11:00] thought was very forward-looking, is actually put a whole bunch of data about the conference on the webpage in an LLM readable way.
[00:11:06] There’s an LLMs txt file, there’s a feed of all of the sessions in js, ON. So I used the data from your conference last year and built this intelligent app, uh, just by talking to our sidekick, uh, in Dreamer. So just to give you a quick tour, this is my Dream Conference app. What I always wanna do for conferences is I wanna be able to search for speakers.
[00:11:28] I’m usually there because, uh, there, uh, is a speaker I care about. So, you know, SWIX, you’re the speaker I care about. I can actually see here who you’re on stage with. So here’s, here’s Greg Brockman. You’ve read even ai, uh, and this is his session. And look Greg and Swix for the speaker. So let’s add that to my schedule.
[00:11:45] Great. And then maybe there’s a couple others I might see here. Like on day two, I remember there were some keynotes. So, uh, building the open agenda web, that sounds fun. So I add that to my schedule.
[00:11:55] swyx: She’s now CEO of Xbox.
[00:11:56] David Singleton: Awesome.
[00:11:57] swyx: Which is interesting. So cool. So,
[00:11:59] David Singleton: so I’ve [00:12:00] gone through and picked out a couple of sessions that I cared about.
[00:12:03] That’s as far as I usually get with any conference app. But of course you’ve got the whole of the rest of the conference to figure out what to do. So here is where the native intelligence of, of these things you build on Dreamer can come in. So I’m gonna click guide me. So Dreamers sidekick actually parsed out the whole schedule and figured out what some of the themes are and I can choose what I’m interested in here.
[00:12:23] I’m definitely interested in agents. Uh, I’m definitely interested in code generation and also reasoning in rl. So now I’m gonna say build my schedule. So what this is doing is. It’s going across every time slot for the conference. And it’s choosing among the things I could go to, which one it thinks is best for me based on my interests.
[00:12:41] It also uses its own memory of me that’s part of Dreamer, uh, to understand what I might like best. And you know, there’s an LLM prompt running for each one of these time slots. So this is, it’s not super fast, but it’ll be done in about 30 or 40 seconds. And I’m gonna have a special custom schedule for the conference.
[00:12:57] This, like I said, is my [00:13:00] dream conference app is exactly what I’ve always wanted and I was able to build this yesterday morning. Um, I did it between some meetings. I think I spent a total of 25 minutes of wall clock time on it. I did it over the course of a couple of hours. And, uh, here is my schedule for the conference.
[00:13:15] I can see it in a calendar view. This is what I should do on Tuesday, this is what I should do on Wednesday. Oof, no conflicts, but, you know, I may not go to every single thing. And there you have it built in, you know, dreamer. So let’s take a look at what the building experience actually looks like. So this is the, the actual account that I made it on.
[00:13:32] Oh, of course I should say anything you build on Dreamer also works on your phone. So, uh, here is my AI engineer conference app right here on my phone. Got all the same functionality, and of course this is the best place to jump into my schedule.
[00:13:46] swyx: Yeah.
[00:13:46] David Singleton: Um,
[00:13:46] swyx: so you could generate a podcast about it just completely multimodal, absolute thing, right?
[00:13:51] To me, I mean, this is why I outsource, I mean, well, I, I posted the L-M-T-X-T, the JSON because you cannot run an engineer conference in 2025 [00:14:00] and not let engineers. Do whatever they want.
[00:14:02] David Singleton: Yeah.
[00:14:03] swyx: And since all conference apps suck, I’m just gonna put up a ba minimum viable app and just let people do whatever they want.
[00:14:09] David Singleton: Totally. And the cool thing about this on Bremer is I published this to the gallery and you can use it so you’ve got one that’s built to my taste of conference apps. I think it’s pretty cool. But you might want something different. Yeah. In which case you just start telling the sidekick how to change it.
[00:14:23] So let’s just very quickly look
[00:14:24] swyx: at our, what sports grid is also, you can fork it, right? That I can publish. That’s right. I can publish your one and go, this is the base starter. It’s, it’s got good defaults, but go customize, whatever.
[00:14:32] David Singleton: That’s right. That’s right.
[00:14:33] swyx: Yeah.
[00:14:33] Agent Studio Under The Hood
[00:14:33] David Singleton: So let’s take a look at how I actually built this.
[00:14:34] This is real. So I’m gonna say make changes. This experience we’re looking at now is our, uh, agent development studio. Um, like I said, you can do this on your phone as well. And in fact, this one I started out on desktop. Let’s look at my actual prompts. I said, let’s make an agent called AI Engineer Schedule Planner should be a custom schedule planner for the AI engineer conference.
[00:14:53] I’m not gonna read this all up. You get, you get the point and it told it where to get the data from. So that was the first prompt. And actually after I gave it that [00:15:00] prompt, I actually had a simple version of this app working, um, after the sidekick took one turn. So the Sidekick is a, like a professional software engineer, and we’ve worked very hard to make this work and build functional apps for folks that might not have any engineering experience whatsoever.
[00:15:14] So, you know, done here we have build logs that are technical, but you can hide those away. And sidekick, as it is building, will actually translate everything that is coming out of, uh, of the, the harness into English that you can actually read. And by the way, this English is in the personality of your sidekick, which is fun.
[00:15:32] Um. And the way that we build agents and agent apps, it’s a little different to what you might have seen in some other platforms for a couple of reasons. One, just the build process. The very first thing that Sidekick does, it understands all the agents you’ve got set up. It understands all the tools and it will come up with a plan for how to realize your goal, how to make sure it actually has the data and the capabilities to complete it.
[00:15:54] It will occasionally refuse. If it can’t do what you’re asking, it will tell you I can’t do that. It needs another tool. And that’s a good [00:16:00] jumping off point for any of the tool builders out there to build a new tool. So it’ll fi first figure out how, then it will build it, and then it will actually test it.
[00:16:07] So it will actually make sure that the thing that it has generated is realizing your goal. And you probably know as well as anybody that anytime you can get any. Modern state-of-the-art coding model into a loop where it can make changes and perceive its own output and then fix bugs. Magic happens. So these builds, the first build will often take 10 to 15 minutes on Dreamer, which is a little bit longer than you might’ve seen on some other platforms.
[00:16:31] But the first thing that it creates will work most of the time. And then of course, as you start making smaller changes, you can like ask it to tweak the UI in any way that you like. Those are much faster. And just to give you a sense, uh, for this one, here’s something I asked. Put a logo, I gave it a logo file in static files.
[00:16:48] Use that as the title. So for folks that actually really want to dig, uh, into a bit more detail, we’ve provided a powerful IDE here. So I can actually see here’s the code that was generated and some pieces of the [00:17:00] code are more accessible than others, like the prompts. So this is the prompt that’s used by a powerful LLM in order to do that schedule picking.
[00:17:08] And I can actually read it here directly. I can edit it without having to ask the sidekick if I want to do that.
[00:17:12] swyx: So this is very nice.
[00:17:13] David Singleton: This is for the more, the more, uh, sophisticated users.
[00:17:16] swyx: Yeah. This is other people’s entire startup is prop management.
[00:17:21] David Singleton: This is true. The other thing that is different about Dreamer is once you’ve built something here, it’s ready to go.
[00:17:28] We host it. So you don’t have to worry about getting a database from a database provider signing up, getting API keys. You don’t have to worry about your LLM provider tokens. All of that is hosted on the platform. And you can use it yourself. You can share it to the gallery for other people to, to riff on it.
[00:17:46] You can also share it with your friends and coworkers to use your instance of the agent or agentic app. And we’re seeing that happen a lot in our community. We’ve seen a whole bunch of folks who built little applications for their personal life [00:18:00] and shared them with their significant other. We’ve seen people who are building little productivity apps for their team at work and sharing it, uh, among them.
[00:18:07] And we actually do this a lot inside of the company. So at this point we, we pretty much run the company on Dreamer agents for all kinds of important things. Uh, maybe a good example of that is, um, our wait list. People are signing up every time someone signs up for our wait list. A dreamer agent will actually research, uh, that person.
[00:18:25] And we’re looking for folks who are builders, not super technical to build agents and come in, uh, and give us a lot of feedback and we’re prioritized bringing those people off of the wait list First,
[00:18:35] swyx: just a quick question on that one is there’s, it may not come up again. Do you find enrichment APIs to be useful like the ZoomInfo?
[00:18:42] Uh, clear bit
[00:18:43] David Singleton: enrichment is a very, uh, common use case. Um, on dreamer. Any application on Dreamer can kick off a sub-agent to do a particular task. Um, so this actually is a powerful agentic harness that runs inside of its own [00:19:00] vm. Uh, we call them sidekick tasks ‘cause they actually run in the context of the sidekick.
[00:19:04] I’ll talk more about Sidekick in a second and. Enrichment is a very common use case. And the cool thing about a sidekick task is that it has access to all the tools on the platform, but also public data as well. And so very frequently enrichment on our platform happens using public data that it can be found in the web.
[00:19:24] There are some tools for getting people data, uh, from, uh, from various bespoke systems. And so that works pretty well. But actually, you’d be surprised. I mean, we would love if someone out there would like to build a ZoomInfo tool, we don’t have one today. We’d love to see that on the platform, and I’m sure it’ll be very powerful.
[00:19:39] But we’re also seeing that this powerful agent harness can pull a lot of data in on that note of tools that make experiences better, we’re constantly adding more tools because people in the community are building them and publishing them. We review the tools carefully and then they go live for everybody.
[00:19:54] Yesterday we added granola. And that was pretty cool. So I was talking to actually, uh, Sarah on my team was [00:20:00] talking to, uh, someone building on the platform this morning and they actually, they have an agentic app that they built, which is a kind of magic to-do list. So they put stuff on their to-do list and for each thing it kicks off one of these, uh, sidekick tasks to figure out how to move the ball forward thing.
[00:20:14] Sometimes it’ll complete it
[00:20:15] swyx: entirely. Yeah.
[00:20:16] David Singleton: Often by calling another agent on the platform and sometimes it just kind of researches it and helps ‘em take the first step.
[00:20:21] swyx: Yeah. Do you know, this is Sam Altman’s number one, ask for an AI app. It’s the self-completing to-do list.
[00:20:26] David Singleton: Yeah. The self-completing to-do list is something that a lot of people have built on Dreamer and are getting a lot of use out of.
[00:20:32] Yeah. And, and finding it actually genuinely I shouldn’t, I should, I should try that. Mm-hmm. Please do. And you’ll even find some in the gallery that you can remix. So he was saying this morning that he’s, he built this self completing to-do list, uh, on Dreamer already. But he connected the granola tool yesterday and now something really magical happens, which is when he says in meetings that he’s gonna do a thing, it magically shows up on his to-do list and then it can magically get completed.
[00:20:56] And then, as I mentioned, all the agents, all the [00:21:00] apps on Dreamer can actually work together. So our coding agent, as it builds them, does something very special where it exposes the internals of each of the experiences to the system. And then Sidekick can manipulate those to get stuff done. So he has built another agent, which he uses for recruiting.
[00:21:18] It kind of keeps track of candidates and also it’s got a kinda mini CRM function, so he’s able to introduce candidates to each other. He told us this morning that something he’d committed to do in a meeting that was recorded on granola yesterday showed up in his magic to-do list and his magic to-do list.
[00:21:34] It was like introduce a person for recruiting, used his recruiting agent to get it done.
[00:21:39] swyx: Ah,
[00:21:39] David Singleton: um, and this is, this is the dream. This is why we started the company. It really is the case that you can build and use these very powerful, bespoke experiences that can automate your life by working together. And I’d love to talk a little bit about how they work together.
[00:21:55] Ecosystem Trust And Monetization
[00:21:55] David Singleton: So obviously it’s really cool to have [00:22:00] software that will work on your behalf, but it’s only useful if you can trust it, right? So privacy and security is very important to us making these things accessible and. While also being trustworthy is hard. So the model that we have, which is working very well, is that the sidekick is at the core of everything here.
[00:22:22] So it is both your companion, your helper, but it’s also the traffic cup in the system. So when, when one agent wants to work with another agent and dreamer, it doesn’t do it directly, it does it via the sidekick, well ask the sidekick to do the thing. And the sidekick understands both everything, all the expectations that have been set with me as a user about what agents can do, which tools I’ve given them permission to use.
[00:22:45] And it will make sure that whatever is is going on is actually aligned with my own interests. And you know, that’s part of the background that I bring to this problem domain. I’ve. Worked for years, uh, keeping very important information, safe and secure. And [00:23:00] so as we started to think about this problem, we realized that we actually had to build something that’s a bit like an operating system.
[00:23:06] You know, the sidekicks, like the kernel, the agents and apps are like users. Yeah. Different rings. Exactly. Because if you try to pick off just one piece of this, you can’t actually make it work for people at scale. Uh, because you could build little vibe coded apps, but they’re gonna grab all your data willy-nilly.
[00:23:23] They won’t be able to work together. You actually have to invest in the fundamental core in order to make it work well for people. And that’s what we’ve been doing and it’s, uh, it’s been a lot of fun. One other thing I wanted to mention is, um, I’ve obviously talked about two things, tools and agentic apps.
[00:23:42] We really designed Dreamer to be an ecosystem and a platform, and one of my favorite quotes about platforms, I think it’s from Bill Gates, is that you can only be a platform. If you create more value for the folks participating and using the platform than, than the platform itself creates. [00:24:00] And that’s our goal here.
[00:24:01] So we at every step have been thinking about how do we make sure that other people are deriving even more value from Dreamer than we are? So in that vein, I already mentioned tool builders get paid and people can build agents that solve their needs and share them with others, and we are already thinking about ways that they can actually monetize those as well.
[00:24:24] Against that backdrop, one of the things that we are launching today is our Builders in Residence program. So there are tons of people building really cool stuff and contributing it to the gallery already, but we’ve been really inspired by programs we’ve seen at other companies where artists might be in residence, people that are very creative.
[00:24:43] And might have ideas outside of what the, the folks at the company or in the ecosystem already have. And so we are looking for creative people who have fun ideas and, you know, want to really figure out how to apply their creativity at the cutting edge [00:25:00] of technology today to come and work with us. So, uh, if you go to dreamer.com/latent space, you’ll find, ooh, well, we love Latent space.
[00:25:09] Uh, you’ll find a link both to, uh, our tool Builder information and our builder in residence program. And for builders and residents, we’ll let you in off the wait list quickly, build an agent, and then for a small number of, of the most creative folks, we’re going to pay you to build agents. Uh, you can work directly with our team.
[00:25:29] You know, this is like building Legos. So, you know, we’ve got some of the basic blocks together already, but if you need a Ron steering wheel and we don’t have one already, like we’ll build it for you. Yeah. Um, we really want to be inspired by, by these, uh, these builders in residence.
[00:25:43] swyx: This Legos thing is pretty common as an analogy.
[00:25:46] And there’s a, there’s a thing I call the master builder. Uh, we, the actual Lego company has master builders that they employ Yeah. To inspire people and post on socials.
[00:25:56] David Singleton: That is exactly what inspired us as well. Honestly, we talked about the Lego Master [00:26:00] Builder program, so that’s our builder in residence program.
[00:26:02] swyx: Yeah.
[00:26:03] David Singleton: Um, and then, uh, finally back on, on tools. Like I said, anyone can come in and build tools today. If you follow the latent space link dreamer.com/latent space, again, we’ll get you off. Directly off the wait list. So you can build right away, you can monetize by publishing onto the platform. That’s for everyone, the very best tool that gets added to the platform by mid-April.
[00:26:23] Uh, we have a $10,000 prize that we want to give out really, because we just want to seed the creativity of everyone out there. So we’re excited to do that.
[00:26:31] swyx: Yeah. And you know, uh, this is completely a flywheel, right? Like the more tools, the more builders, the more the third thing agents, you know, it just feeds into each other.
[00:26:39] David Singleton: That’s right.
[00:26:39] swyx: Yeah. Just on the payments thing, because we probably won’t touch on that again, but I have to ask the former CTO Stripe on payments as presumably you’re using Stripe Connect.
[00:26:48] David Singleton: Yeah.
[00:26:48] swyx: Um. Any pain points that you’re, people are very interested in agent commerce and micropayment and all these things.
[00:26:55] Presumably stable coins get into a conversation at some point, but maybe not now.
[00:26:58] David Singleton: Yeah, we are [00:27:00] really, really excited about e agent commerce. The first step we are taking is help people in the world who have never been able to build these kind of experiences and software before to build stuff that meets their passions, share it with the world and get paid.
[00:27:14] So that’s all commerce that happens on our platform, and so we don’t need anything new to facilitate that. Stripe Connect has existed for quite a while and is the perfect solution for this kind of stuff, so, um, we we’re excited about that. First and foremost, however. A lot of the things that people are already doing on Dreamer, we just talked about a self-completing to-do list.
[00:27:34] A lot of the ways that you want to complete to-dos is by actually closing the loop in the real world, and that’s going to involve the exchange of value. So we have some folks that are building tools already that actually do have money move in order to, to complete that, that loop. So far, we just want to be open and agnostic to all the protocols out there.
[00:27:54] I honestly think this moment in time is a little bit like the early web. So I personally started coding as a kid [00:28:00] and I think I got access to the internet in about 19 95, 19 96. And back then, uh, the web existed, you know, HTTP was a protocol, but there were also other protocols I was using all the time, like Gopher and UUCP and uh, various others.
[00:28:15] So the point is like the web, HTTP and HTML. Was just one among many protocols. And of course it became the winner and it’s awesome. Yeah. Um, but the others were also kind of interesting and viable at the time as well. And I think the world of agentic commerce is like this right now. Also,
[00:28:30] swyx: acp.
[00:28:31] David Singleton: Acp, exactly.
[00:28:32] All the, all the cps, you know, on Dreamer. We hope that folks will build tools that kinda make use of all of these things, but I’m sure that at a certain point. One or two will emerge as the winners, and then we’ll be able to build like really deep support in,
[00:28:44] swyx: yeah. This is like maybe a complete tangent, but I do think about how a lot of these companies in AI companies in particular have to switch from c based to usage based because of course, but then, then they end up, end up having to sort of [00:29:00] obscure the margins a little bit and then they inventing end up inventing their equivalent of rob robots.
[00:29:04] David Singleton: Mm-hmm.
[00:29:04] swyx: Uh, where they’re like, well, okay, well every company should have their own currency. And it’s, it’s like very short lead to a token.
[00:29:11] David Singleton: Yeah.
[00:29:11] swyx: Or, and I’m like, okay, well where does this end? I can’t really play out the next step as to like, is this chaos? Is this,
[00:29:18] David Singleton: yeah.
[00:29:18] swyx: Okay.
[00:29:18] David Singleton: Well, I think it is kind of like the wild west.
[00:29:21] I don’t mean that in a completely, it’s all completely disorganized way, but there’s just so many things that could happen from here. The Overton window is very wide, right? Not far how this might land. And I’m just very excited to be building a platform that can take advantage of all of those opportunities and we’re just gonna be there.
[00:29:36] Uh, working for our users to make sure that things that emerge work,
[00:29:39] swyx: you’re gonna own the consumers, you’re gonna be up the OS for the app store for everything.
[00:29:43] David Singleton: So one of the ways to think about this is, um, dreamer actually uses all of the state-of-the-art models as a user. You don’t have to think about should I be using, you know, Opus four six, or should I be using the five four model from [00:30:00] OpenAI?
[00:30:00] We are continually doing evals and so forth to make sure that the best things are there for you. You can just build on the platform and know that as the world ships around, you’re gonna get the right stuff for you. Um, and I think that’s something that is needed to actually have folks take advantage of this technology at scale.
[00:30:19] I’d love to show you another example of something I built.
[00:30:21] swyx: Let’s do it.
[00:30:22] David Singleton: This is another example of software that just lasts for a certain moment in time. So recently I went on a ski trip with a bunch of friends,
[00:30:31] ski
[00:30:31] David Singleton: Bum. Uh, so it uses ski bum. Yes. I went on a ski trip to Big Sky. I’d never been there before.
[00:30:38] And I made this little intelligent app for us. And you can see it says it’s loading big sky conditions. So it’s actually calling the Ski Bum tool that I just showed you, which is, uh, published in our, uh, in our gallery. So what is this? This is a little app that was just for our weekend trip. It shows the current status of all the lifts of Big Sky.
[00:30:54] Using that tool from the ecosystem, it shows the forecast for the upcoming weekend. It shows our [00:31:00] accommodation. This is just like where my group was staying. This is just for us and also a bunch of dining information that one of our friends, uh, put together who, who’s an expert on Big Sky. So I was able to take this app, share the link with my friends.
[00:31:12] They weren’t on Dreamer yet, just send it to them on iMessage and they get a version they can use on their phone. And of course, here’s the real kicker. So I’ve been on ski trips before and other weekend adventures with my friends. Yeah, people pay for different things and at the end of the weekend it’s always a pain to figure out who needs to pay, who to settle up.
[00:31:29] So we use this during the weekend. We added all of our expenses in here. Uh, too close are it’s drill data. It’s only too closely. And then at the end of the trip, we press split. And we’re, we settled up and we’re done. So there’s another dreamer. This was all through dreamer. So the, the actual payment? No, no.
[00:31:47] We, it happened because, because we paid for stuff in the real world, it was like, okay, this person needs to pay that person 20 bucks. Right? Right. This person already paid in that. Right. So it just helped us all settle up. We didn’t move the money on Dreamer. You could do that. And in fact, if you’re a tool builder [00:32:00] thinking about this and getting excited, like come build a tool to do that stuff.
[00:32:02] We really think of our tool builders as design partners.
[00:32:05] swyx: Yeah. I got, I got the tool. Uh, what, like, I hate, I use Bank of America. I hate bank, I hate the app. Mm-hmm. I hate the web. All banking websites just horrible.
[00:32:13] David Singleton: Yeah.
[00:32:13] swyx: So just build me, like build a thing on top of Plaid.
[00:32:15] David Singleton: Yeah. Right. And then just So
[00:32:17] swyx: five code by banking app,
[00:32:18] David Singleton: there’s already a tool for that.
[00:32:20] Oh. So, um, attain Finance is a tool, a builder in our community built. Okay. Um, and it uses a secure system like Plaid. To access your, uh, financial data and you can build powerful personal finance agents on Dreamer today using this tool. And like I said, we review tools carefully. So when bringing Attain Finance onto the platform, we did actually quite a detailed security review with that company to make sure that if folks build stuff with it, it’s, it’s gonna work well.
[00:32:49] So yeah, check that out. I think, uh, I’m, I’m pretty certain it connects to Bank of America. So you’ll be able to build the, the app that you wanted already?
[00:32:55] swyx: Yeah. There’s a couple of points I wanted to sort of dive in on, maybe highlight to folks, [00:33:00] because I, obviously, I spent more time with Dreamers. So we’re making a point where you choose on behalf of your users because they’re meant to be consumers.
[00:33:07] So maybe less technical,
[00:33:08] David Singleton: right?
[00:33:08] swyx: But obviously people can, how users can override. If you read that’s, but it’s not just lms, it is also the, the transcription. It, it’s like all, like there’s, there’s a first party curated set of here’s the house opinion. That’s right. On what?
[00:33:21] David Singleton: That’s
[00:33:21] swyx: right. The thing is, that’s right.
[00:33:22] Is what’s the list? Is there like,
[00:33:24] David Singleton: yeah, so actually if you look in the tool gallery, the first party kind of curated set are all the ones that have these grayscale icons. So we have a built in tool for image understanding, for image generation, for RSS, exploration, text to speech and so forth.
[00:33:38] swyx: Recipes.
[00:33:39] David Singleton: Uh, we actually do have a built in recipes tool.
[00:33:41] It turns out that a lot of people in our alpha wanted to do stuff for cooking. Yeah. Um, and you know, you can scrape the web to get good recipes, but we were able to quite quickly find a good repository of recipes. It works great here. Yeah.
[00:33:55] Stable Tool Interfaces
[00:33:55] David Singleton: So the point behind these though is that we’ll keep the interfaces stable, so they’ll always work.
[00:34:00] But you know, the best translation model and, you know, there are people using this translation tool to translate Chinese podcasts into English. It’s, it’s pretty powerful. It can deal with very long text, but the best translation tool today might be different from the best translation tool sometime next year.
[00:34:15] And we’re just gonna make sure that that translation tool is always pretty close to state of the art. So you can build something and you know it’s gonna continue to work well. Of course, some of our tools are branded. You may actually have a preferred way of buying groceries, like maybe you prefer Instacart and that’s great.
[00:34:29] You can use the Instacart tool specifically.
[00:34:31] swyx: Yeah.
[00:34:32] Partnerships And Ecosystem
[00:34:32] swyx: Your partnerships, uh, I mean, I don’t know if you ever hit of partnerships, but this is gonna be a bonanza for anyone on to do deals.
[00:34:38] David Singleton: We have an amazing person who, uh, works on all of our partnerships. Um, and it’s part of what you have to do to build a platform like this that’s gonna work for people.
[00:34:46] Like, we’ve gone and done that. Schlep has a lot of work, one talks lots of different companies, um, in order to make sure that you’ve got good tools at the core.
[00:34:54] swyx: Yeah.
[00:34:54] David Singleton: And then of course, because we’re open to tool builders contributing to the platform, this is only gonna get better and better and [00:35:00] better.
[00:35:00] swyx: Yeah.
[00:35:01] Agent Lab Routing Layer
[00:35:01] swyx: One observation I have this, this is gonna master a thesis I’ve been pursuing, which is, uh, what I’ve been calling an agent lab
[00:35:05] David Singleton: mm-hmm.
[00:35:06] swyx: Where you sort of different than a model lab in, in, in the sense that you never train your own models, but you are the router evaluation layer, ex subject domain expert for choosing between, uh, models.
[00:35:18] David Singleton: Yeah.
[00:35:18] swyx: And you’re explicitly doing these things. And so like in my sort of construction, every agent lab does some version of this where like, here’s the image understanding endpoint and we will route for you and don’t worry about it. Yeah. Sally, I think it’s kind of cool.
[00:35:32] David Singleton: I, I think it makes total sense. Um, and again, to make this work for folks that don’t follow the AI news every day, it’s an actually, it’s a, it’s a really important thing to do.
[00:35:42] Yeah. And it, it’s been, it’s been a real pleasure. I mean, I’m a, I’m personally a total geek for this stuff. I love it. And being able to go and dive into all those details in order to make it work well for other people. It’s a true pleasure. I cannot imagine working at anything else right now. It’s just so much fun.
[00:35:56] swyx: The tricky part is multimodality when some of these things do [00:36:00] merge.
[00:36:00] David Singleton: Mm-hmm.
[00:36:01] swyx: And you are, you’re sort of, this is your imposing structure on things that fundamentally don’t want to be structured. And so sometimes that might work against you, but for 99% of these cases, this is fine.
[00:36:10] David Singleton: Yeah. I mean, I think it’s gonna be very interesting to see how the, the, the world matures because a lot of the power of dreamer is the ability to kick off these subagents, so these powerful agent harnesses, which can actually change how they work based on the data.
[00:36:25] I actually think that we will be able to. Kind of keep up with and stay at the forefront of the changing landscape of how tools and systems work together. And that’s, that’s new. You know, software didn’t used to work like this and now it does. Um, so even, even just figuring out how to design the right pri to make that possible has itself be a lot of fun.
[00:36:44] Builders Can Publish Tools
[00:36:44] swyx: This is, is a sort of maybe two part question that why can’t streamer make its own tools? And then why don’t you let you builders maybe stand up their own routing group? I call this a routing group, right? Like where it’s like collect Yeah. Things.
[00:36:58] David Singleton: So two things, to [00:37:00] some extent, dreamer does make its own tools in that agents appear to the system as tools.
[00:37:05] So they can be, they can be used to accomplish things. So you can build an agent that is essentially a tool. Yeah. Um, and it it,
[00:37:12] swyx: which is to me very useful for reuse.
[00:37:14] David Singleton: Right.
[00:37:14] swyx: Right. Exactly. ‘cause I, I like, this is the way I like it. Now my next five apps, I don’t want to do this whole series of back and forth again.
[00:37:20] David Singleton: Right.
[00:37:21] swyx: Yeah.
[00:37:21] David Singleton: Um. Then at the tool layer of the system, it’s open to anyone. So it’s actually quite powerful and flexible. So if you wanted to add a tool, which was, uh, imagine that you were training your own foundation model, Swyx. That might be fun. And imagine you wanted people to be able to play with, I don’t know, maybe you make like, you know, nano chat or whatever and you want to Yeah.
[00:37:42] Let people play with your own nano chat and see how I change themselves.
[00:37:44] swyx: Now.
[00:37:45] David Singleton: You could, you could publish a tool that is Nano Chat and it nano image generation behind a tool, and it could be your own writer if you wanted to. I see. And honestly, if that’s the kind of thing that gets you excited as a builder, please come and do it.
[00:37:57] Like we, we really are [00:38:00] believers in this idea that we aren’t going to figure out every single detail ourselves. We’re gonna make sure it’s a safe and fun place to build this stuff, but we’re really open to these ideas coming from other people. Um, and so I’d like nothing more than you come in and build a tool that does some of that cool stuff that you, that you have in mind.
[00:38:15] swyx: Yeah. Awesome.
[00:38:16] David Singleton: And just as a reminder, if you’d like to do that, the way to find the links is dreamer.com/latent space. Um, and for a limited time on that page, um, anyone who’s listening to this podcast will also get directly off of our wait list. Uh, it’s quite long right now. We are working hard to bring Zika.
[00:38:32] Wait, so skip the wait list.
[00:38:33] swyx: You know, I think, I think that’s fantastic. I, I think it’s, it is really sort of probuild way to do it. I wanted to jump back to the, the bar. Yeah. You know, you know, I get excited about this.
[00:38:41] David Singleton: Yes. Okay. Let’s set it back in there.
[00:38:43] swyx: Like, let’s, you know, this is the engineer podcast that’s get
[00:38:46] David Singleton: Yeah.
[00:38:46] swyx: As technical as you can.
[00:38:47] David Singleton: Yeah.
[00:38:47] swyx: On everything you’ve built, like have a show off.
[00:38:50] David Singleton: Yeah. Okay.
[00:38:51] Under The Hood Debugging
[00:38:51] David Singleton: So let’s go wild in the aisles in the Asian studio. So as you can see, over on the left here is a conversation with the sidekick where you ask it what to do and it will explain in English that anyone can understand what’s going on.
[00:39:03] But, um, if you want to pull back the covers and look under the hood, um, if you’re, uh, an engineer like me, then we have this, uh, this kind of debug drawer at the bottom. So you can see the full build logs here, but you can actually also dig in and see the files and prompts that have been generated. Uh, you can upload files from your computer in static files.
[00:39:24] Um,
[00:39:24] swyx: very important,
[00:39:25] David Singleton: uh, indeed. You can actually read the prompts that have been generated for you. We intentionally put an example in here just that you can see what the format looks like. And then, you know, we already looked at this one that was generated for this particular, um, app, but if you actually want to bring the code out of Dreamer and work on your own local machine, you can.
[00:39:45] So at the core of everything here is an SDK with a powerful command line interface and we built that first. It’s actually possible to build agents on Dreamer without talking to the sidekick. You can write code with your fingers on a keyboard if you want to. I know that’s very [00:40:00] antiquated, not, but actually this can be a lot of fun.
[00:40:02] So if you wanna pull it out onto your laptop, you can use our, our CLI and, uh, you can edit it in cursor or in cloud code. You know, you don’t have to use our sidekick. And the CLI actually has full access to the rest of the platform with you as the user. So, you know, obviously it is, uh, secure and privacy sensitive, and this is a way that, um, some of our most technical builders do build stuff on the platform.
[00:40:24] The really cool thing is the side cake. When it’s in coding mode, it uses exactly the same CLI. So the way it. Build stuff on Dreamer is using the same tools that you might as an engineer. Um, and that’s actually a very powerful abstraction because it turns out that the right way to give a lot of context to agents to use CLIs is to write great documentation.
[00:40:46] Make sure that all of the things that you could do are actually possible. And guess what? That makes it a delightful developer experience for real heroes as well.
[00:40:53] swyx: Yeah. So that’s pretty cool. We’ve been telling developers to do this and they ignore this until now they have to for content.
[00:40:58] David Singleton: I, I’ve been saying this for a [00:41:00] long time.
[00:41:00] Uh, we actually Stripe docs.
[00:41:02] swyx: I mean, come on. Absolutely. Come on.
[00:41:03] David Singleton: Absolutely. But actually, I was chatting with folks at Stripe last week and saying, Hey, you gotta make the Stripe CLI actually tell agents what they can do on Stripe because that way they’re gonna use more stuff on Stripe. I think this is a real trend for the entire industry.
[00:41:16] swyx: Yeah.
[00:41:16] David Singleton: So we, we’ve been doing that.
[00:41:17] swyx: To me, this, this download and, uh, GI push mm-hmm. Everything is complete confidence in that you’re not hacking it. Right. Because there’s other, let’s call them AI builder platforms that impose their stack on you and if you, if you, and so therefore they don’t allow you to do this because they cannot.
[00:41:34] Right. ‘cause they, they impose some degrees of freedom, uh, restrictions so that they can get it to work. Yours is a fully general like VM running the full code. Correct. Do whatever you want. Correct. Any language you want. Correct. Yeah.
[00:41:46] David Singleton: Correct. Well, in terms of language, if you use the SDK, you could build stuff in other languages.
[00:41:51] We’ve actually found that TypeScript is the best language for building these experiences. Yes. Because it’s strongly tight. So you find out at compile time if you’ve made mistakes [00:42:00] and there’s nothing better than getting in. A coding agent in a loop where it can see its mistakes and ask them. So TypeScript is the language that everything gets built in by default here.
[00:42:08] swyx: Did And did you see that TypeScript overtook Python? I did. I did. Yeah.
[00:42:12] David Singleton: And for what it’s worth, when we started the company, we started writing stuff in Python, and I love Python. Um, if I do, uh, a vendor code, I always write it in Python. It’s my favorite language as a developer with my fingers on the keyboard.
[00:42:23] Um, but TypeScript is an amazing language for AI because there’s tons of training data in the models, um, and it’s strongly tight. And actually at the company we built most of the stack in TypeScript, and we have this amazing property, which is, we have type safety all the way from the database to the front end.
[00:42:40] And there’s nothing better for working with coding agents than being able to have them check their correctness, compile time. So the same ideas behind building the company’s code base, we’ve put into the agent SDK here as well.
[00:42:51] swyx: Yeah. Do you know if you’d use one of those tools, like Prisma or whatever, or is it Tool Lab for you?
[00:42:55] David Singleton: We, we actually have crafted most of our own tools. Um. For [00:43:00] instance, we had LLM Driven Code Review, uh, before the thing that got published from philanthropic this week. You know, we, we’ve been doing this stuff, uh, on our own bat
[00:43:07] swyx: email, we’ll pay $25 per review.
[00:43:09] David Singleton: We, we pay a lot less than that. However, I hear that those reviews are excellent and possibly worth $25.
[00:43:14] swyx: Yeah. You know, it’s an option. Right. It’s good, good to have it.
[00:43:17] David Singleton: Just to give you a tour of some other stuff here. So, um, I can also see all the versions. Yeah. Um, this is not gi, this is not gi, this is built into dreamer. I can see all the versions that have been pushed before. Why is it
[00:43:27] swyx: not gi?
[00:43:28] David Singleton: It’s not gi because we can make it work more efficiently than Git.
[00:43:32] And we actually, we do some work behind the scenes to kind of understand what’s in each of these versions. Yeah. Um,
[00:43:37] swyx: so one of the things I’m pursuing, and I have a lot of thesis, right? Mm-hmm. One of the thesis is like, does GI go away? Does GitHub go away? And like, what, what is the active reinvent
[00:43:46] David Singleton: you for, for what it’s worth to some extent.
[00:43:48] And anything you build, there’s a lot of path dependency. If we started over, we might make this gi There’s, uh, you know, within the company we use, uh. For our, you know, platform source code. And we like it and it [00:44:00] works well with coding agents as well. The very first versions of this, we wanted to be able to make it possible for the sidekick to manipulate it easily.
[00:44:06] Um, and this, this was an expedient way to do it.
[00:44:08] swyx: Yeah.
[00:44:08] Workflows Logs And Databases
[00:44:08] David Singleton: Um, you can also see all the activity that has happened in the workflows that you build. A lot of agents, you’ll build on Dreamer, do things in the background, so they run on triggers. These are stimuli from the outside to kick them off, and this is a nice way to see all of the things that might have kicked off your agent.
[00:44:24] You know, you can have an agent that kicks off on a webhook, so you can plug it into external systems. You can have an agent that runs when you receive certain emails that match filters, including LLM filters. And so here you can see, oh, when did it run? What did it do? You know, if I open up one of these guide me prompts or guide me, uh, events.
[00:44:41] Oh my can see God. Well, I told you it was calling an LLM for every one of those time slots. Here’s all of the LLM calls, here’s the actual prompts.
[00:44:49] swyx: And you don’t mind exposing all of this, right?
[00:44:51] David Singleton: No. We want builders to see what’s going on under the hood. It’s haiku to,
[00:44:53] swyx: okay. Yeah. So,
[00:44:54] David Singleton: okay. Right now that one was haiku.
[00:44:56] Like I said, we work with all the models and sidekick will actually pick the best one [00:45:00] for the job. And you saw that was pretty high quality and pretty fast. So Haiku four five is the one that it picked for that job. Exactly. Uh, we also have logs, as I mentioned, there’s a database spun up on demand for every, uh, agent.
[00:45:12] You don’t have to go and figure out how to do your own hosting. This is a SQL Light. This is a SQL Light database. Yeah. Um, it’s a multi-user SQL light database. And then, uh, but, but each one is you, you get a database that is unique to this agent. But then if you share the agent with multiple people, we take care of like who are the owners in each row?
[00:45:31] And all of that stuff is just there outta the box. Um,
[00:45:34] swyx: and again, in-house?
[00:45:35] David Singleton: In-house.
[00:45:36] swyx: Oh my God.
[00:45:37] David Singleton: Yeah. Um, well we do work with a bunch of infrastructure providers, but the technology for how to manipulate this is in-house. Fun fact. We actually did a lot of our own infrastructure development early on at the company and realized we need to spend our energy in the stuff that we’re uniquely doing in the world.
[00:45:53] So we’re very delighted to partner with a bunch of great designer and some of this stuff. And then finally, um, I mentioned that agentic apps agents [00:46:00] expose all of their internals to the system so the psychic can manipulate them and use them just like a user can. So you can see how it’s decided to break this problem up into functions.
[00:46:09] Some of the functions, the ones with the little I here are exported. That means that there’s probably the visible from outside. Exactly. And others are internal. And if you want to, you can dig right in here and call individual functions and see what happens. But mostly. You don’t need to think about that at all.
[00:46:24] Yeah. Uh, you can keep that little drawer closed and you can talk to your sidekick and build really powerful and enchanting experiences.
[00:46:30] swyx: Yeah. I mean, to me, like showing this gives the engineer a complete mental model of what you’ve done and what you can do with it. Yeah. For example, the first thing I, I, I look for.
[00:46:39] A mental checklist of things, right? Like is off in the database, off looks like it’s not right. So that’s a separate layer. That’s probably me means it’s hard to do multi-user apps on the same app, right?
[00:46:50] David Singleton: So you actually, we’ve solved that. So, um, see, yes, the platform builds in off, so you as a user sign into the platform, if you’re using an [00:47:00] agent that was published by someone else, then your identity is, is kind of taken care of by the system.
[00:47:05] And when you query the database, you’re gonna get the stuff that is for you. Unless the builder specifically said, this is public data that everyone should see. So they, they actually get a chance to think about that. And again, sidekick can guide you through building, uh, agents and apps that work that way.
[00:47:19] So you’re right, that’s another thing that people have to think about when they’re trying to figure out how to build software experiences on Dreamer. You, it’s built in. You talk to the sidekick as if it were a human being about what you want and that’s what you get. So, you know, my, my Big Sky app that I just showed you that was designed for multiple people to use it.
[00:47:38] And of course the things that we were putting in as expenses were supposed to be visible to everybody, and I just told the sidekick that’s the way I wanted it. Uh, but by default, if I built an app like that, the data from each user would not been visible to the others.
[00:47:49] swyx: Yeah. Yeah. Uh, this is, I presume this is a mood question, but basically you’ve had to build your own coding agent, right?
[00:47:55] Which is sidekick slash whatever is in Inside Psychic. Obviously there’s a lot of [00:48:00] people with a lot of desire for cloud code and Code X and attachment to it. Mm-hmm. I know under the hood data basically reduced to a loop, but like, would you let people use cloud coding and Code X or is the harness too specialized?
[00:48:12] David Singleton: Yeah. If you, if you want to use, um, cloud code and Code X, then you go down here. Yeah. Hit get the S St K. And we even say this right here, edits your heart’s content Z cursor code.
[00:48:22] swyx: Like people want to use it inside of Ick, right? Yeah. They want to switch the engine.
[00:48:26] David Singleton: Yeah.
[00:48:26] swyx: That’s the coding engine.
[00:48:27] David Singleton: Yeah. We are not doing that right now.
[00:48:29] Um, you know, again, the goal really is abstract the complexity. Yeah. Um, because the real target for. Building agentic apps is folks who can’t do this already today. I can’t tell you how many users in our community I’ve spoken to who are like Dreamer has changed my life because I used to have all these ideas.
[00:48:50] If only I could find an engineer to help me implement them, I’d be able to get them done. They’re free, and now I can talk to my sidekick and, and get it built. I think that’s like really how we think [00:49:00] about the people that should get a ton of value and fun, um, out of the platform. And so they’re not asking to be able to plug in their their own, you know, coding agent.
[00:49:11] And for those folks, the opportunity is massive. If you’ve never been able to do stuff in code, now you can build stuff for you, for your friends, for your family, for your coworkers. And also there’s a huge opportunity for folks who do build stuff in code to actually contribute to this ecosystem. So that’s how we think about it.
[00:49:28] swyx: Yeah. Amazing.
[00:49:28] Personalization And Memory
[00:49:28] swyx: That’s most of what I wanted to cover Dreamer wise. I think personalization and memory yeah. Is probably like the single most important job of, uh, of the os. Maybe we could talk about that and then I’ll, I wanted to zoom out on company building stuff.
[00:49:40] David Singleton: Yeah, yeah. Sounds good.
[00:49:41] swyx: Yeah. So how do you handle memory?
[00:49:43] What, yeah, what have you found? What have you tried and failed?
[00:49:45] David Singleton: Yeah. Okay. So, uh, first of all, at the core of dreamer is the sidekick. The sidekick gets to know you and it builds up a memory about you over time, and that turns out to be very important. So Dreamer, that’s your moat. That’s Dreamer gets better the more you use it.[00:50:00]
[00:50:00] For instance, a lot of agents in the platform, when you start using them, the first thing that they’ll show you, here’s what I think is relevant to you for this particular use case. Uh, a very popular kind of agent on Dreamer is a weekend activity planner. So, um,
[00:50:14] swyx: like, just tell me what to do.
[00:50:15] David Singleton: Well, tell me what to do, especially if I’ve got kids, right?
[00:50:17] So I have two kids and a dog, and my wife and I often spend a lot of time trying to figure out what are we gonna do with the crew this weekend. And, you know, we have interests that are very consistent. It actually can take a ton of work during the week to figure this out. So there is an agent on Dreamer called Weekend Activity Planner, and it helps me find things to do with, with the family of the weekend.
[00:50:39] In fact, this morning I got a message from my weekend activity planner telling me about the St. Patrick’s Day parade on Saturday. Oh, at Civic Center. I’m Irish. My kids are technically Irish as well. I mean, they, they, they have multiple citizenships, but you know, they’re, they’re Irish. Um, what a better thing to do than take them to the St.
[00:50:56] Patrick’s Day parade. Why did that get recommended to me? Because in the [00:51:00] profile that we can, activity Planner knows about me. It knows that I’m Irish, right? So all of that comes from the memory that Psychic builds up about me over time. We have invested in this a bunch. We will continue to invest in this more.
[00:51:11] We’ve tried actually many different techniques. As, you know, the, the kind of, um, cutting edge of a agentic memory has changed over time. You know, very early on we were putting lots of facts into a vector database and, uh, and doing embeddings and pulling them back out, um, using, you know, reverse lookup of embeddings rag that actually worked, but turned out to be much more complexity than was actually required.
[00:51:33] So, you know, today we’ve replaced it with a different system. Uh, I think we use a system that’s pretty similar to what you’ll find in lots of other products, but it’s an area that we’re actively, uh, investing in. Like, there’s, there’s. More than one person at the company specifically working on memory. And so expect us to just continue to make it better.
[00:51:50] swyx: Did you try knowledge graphs?
[00:51:51] David Singleton: We’ve tried knowledge graphs. The system that we have now is not a knowledge graph. Yeah. Um, but we’ve probably implemented most of the papers you’ve seen out there on agent [00:52:00] memory and the current system is working pretty well.
[00:52:02] swyx: Yeah. Excellent. Zooming out just on the company stuff.
[00:52:06] Mm-hmm. Um, uh, this is your first time in the CEO seat. Correct. You were CTO before. Correct. What’s different?
[00:52:11] David Singleton: Yeah. The difference between being a CEO and A CTO really is just. Like making sure you’re looking across everything. So, um, I have run products before, so for instance, Android wear, you’re basically a CEO
[00:52:25] swyx: of
[00:52:25] David Singleton: that product.
[00:52:26] I, I, I was running that as a general manager.
[00:52:28] swyx: Yeah.
[00:52:29] David Singleton: However, when you do it for your own company and the buck truly stops with you, it definitely kind of raises the temperature a little bit. Um, but it’s been a lot of fun for me to think about a lot of go to market topics. Um, I spend a lot of my time these days meeting users, uh, talking to folks about what they could do on the platform, being very active on X and LinkedIn, uh, which by the way is a huge pleasure.
[00:52:51] It is so much fun to be able to engage with users and potential users directly and understand what they would like to do. Um, and that’s the biggest difference [00:53:00] between this role and being the CTO, um, of, uh, of a company. At the same time, I am someone who always likes to look for why are we doing this?
[00:53:10] Who are the people that. Benefit from it. So, you know, even as A-C-T-O-I was always paying a lot of attention to topics across the company. So I feel very grateful for all I learned in my previous roles that kind of got me ready to, to do this at this kind of scale.
[00:53:24] swyx: Yeah.
[00:53:24] Tiny Teams Hiring And Taste
[00:53:24] swyx: To me this is like the natural lead into when I went into your office.
[00:53:27] Yeah. It’s surprisingly small.
[00:53:28] David Singleton: Yes.
[00:53:29] swyx: So, and I have a, another thesis I’m pursuing for latent space, which is the emergence of tiny teams. Yeah. Where, uh, you know, the, the classic sort of image is that teams with more millions in revenue than employees, right? Yeah. So you, that’s revenue efficiency definition.
[00:53:43] But I do think as a CEO, you are going to run a smaller team than you used to.
[00:53:46] David Singleton: Yeah. So I believe very strongly in the power of small teams. So the more people you add to a team, the more communication overhead there is. And it doesn’t even grow linearly. If you think about it, the more people you add, everyone cares [00:54:00] about getting to know everybody else.
[00:54:01] And sharing what they’re doing with everybody else. And that’s great. I’m not saying they shouldn’t, right? The very, like, I wanna work in teams that are fun, where people are talking to each other and, and sharing ideas and so forth. But, you know, there’s just a kind of gravitational weight that comes from larger and larger teams.
[00:54:16] So just inherently large organizations are less nimble than small ones. And if you run a large organization, you have to keep thinking about how do I kinda like prune things so that it can act like a small team. So a dreamer, the, the core team that built everything I just showed you was, was honestly about six people.
[00:54:34] Uh, we’re larger than I we’re about 17 people at the company now because as, but
[00:54:38] swyx: still, uh, for everything you just showed,
[00:54:40] David Singleton: it’s, it’s still a small team, which is great. Very, very high talent density team. We’ve been very, very careful and kind of obsessed as we grew to make sure that everyone that’s joining the company is joining a team that they’re gonna get a lot of, uh, learning out of, but also they’re actually going to kind of.
[00:54:57] Help everyone else a lot as well. There’s something very [00:55:00] special about that too. You know, every single person at our company I would be delighted to do any project with at any time because, uh, they’re just all great. And, you know, the smaller you keep the team, the easier it is to make sure that, that that talent density is there as well.
[00:55:14] Of course, it’s a real luxury to be building a company. We started this company in late 24, but it’s a real luxury to be building a company today because we can build with agents. So we’re using coding agents.
[00:55:26] swyx: Yeah,
[00:55:26] David Singleton: we’re using Dreamer marketing agents. All of our operations. We’re looking at how we can, can actually accelerate what we’re doing, uh, using our own tools.
[00:55:36] swyx: Um, any, actually any agents that you don’t build that you wanna shout out? Just that, that you love?
[00:55:41] David Singleton: Yeah. Is it
[00:55:41] swyx: other people’s
[00:55:42] David Singleton: agents that we built for the
[00:55:43] swyx: company? No, no, no. Other people’s, uh, stuff like you shout out granola.
[00:55:46] David Singleton: Yeah. So I showed you Attain finance. Uh, attain Finance has an agent as well, which like helps you manage your money.
[00:55:53] I find this really amazing. So, um, I always have this like lingering feeling that I’ve got a whole bunch of [00:56:00] subscriptions that if I just had a bit of time to go across them, I could, you know, figure out how to consolidate them. And the person who built Attain Finance doesn’t work at our company. What they were part of the early Alpha group.
[00:56:10] So they gotta kind of look at how all this stuff works pretty early on. And they built this really amazing experience that actually helps you. Like, save a lot of money because it will kind of help you analyze your purchases. It’s almost like a kind of a financial fitness coach. He’s called Andrew, uh, who, who built it.
[00:56:26] He came and showed it to us and the first thing it did was it recommended that he should buy fewer burritos. And, uh, he was like, it’s true. Like that is actually how I could save the most money. So, uh, that’s a, that’s a pretty cool example.
[00:56:38] swyx: Uh, I noticed he was first. Because he’s order alphabetical order.
[00:56:43] So I’m, I’m wondering how come there are no like Avar? Uh,
[00:56:46] David Singleton: yeah. Well if you’re a builder right there and you’re wondering how do I seo o myself on the Dreamer platform, Swyx suggest you name your tool Avar. In all seriousness though, those are the tools I have connected. So they’re in alphabetical order.
[00:56:58] If you haven’t yet connected them, we actually [00:57:00] kind of put them in the right order for you. So if Sidekick understands you and puts in the right order, uh, but I’d say a arc is gonna come before, uh, anything else,
[00:57:06] swyx: right? Yeah, exactly. Um, and, and then I, I think how has hiring changed? Yeah. You’ve hired plenty of self engineers in your life.
[00:57:14] David Singleton: Mm-hmm.
[00:57:14] swyx: I assume something’s changed.
[00:57:15] David Singleton: Yeah, absolutely. So one of the main things that I look for now when hiring engineers is. How well do you work with coding agents? Our team actually is quite experienced a good number. Everyone at Dreamer, other than, well, I guess I write a lot of code too. Everyone’s an ic, an individual contributor.
[00:57:32] Many of the folks that work on the team have previously been managers. And it turns out being an engineering manager, as long as you stay very close to the code and are able to continue to craft it yourself, is actually a great skill profile for being able to make agents work for you and for your team in this, uh, in, in this age.
[00:57:50] And so that’s definitely something that we look for quite intently when hiring engineers. And, um, we still have folks write some code like with their fingers. It’s just important to know [00:58:00] that the kind of core of the craft is there. But the vast majority of what we spend time doing is building quite significant and elaborate stuff together in a fun, collaborative environment with coding agents.
[00:58:09] swyx: Right.
[00:58:09] David Singleton: Um,
[00:58:10] swyx: so what, what is the interview loop like? Sit there with Codex, do something.
[00:58:13] David Singleton: Yeah, I mean, our interview loop is one a coding. Screen to make sure that the, the base is there. And then we actually do a couple of short projects, uh, with an engineer on our team and whoever is thinking about joining, where we’ll actually put out a very fully formed product idea, we’ll riff on it together and make sure that we can test product sense a little bit and we’ll actually try to build the whole thing with x or cloud code or whatever, uh, whatever the person is most familiar with.
[00:58:39] Um, and watching how someone thinks about prompting the agents, what they do while the agent is working. ‘cause you know, you can actually, this is a kind of interesting, uh, dynamic in the industry. Anytime I’m working on code these days, I always have more than one agent going at the same time because while one agent is going and reviewing the output of the next one, and if you [00:59:00] get them in a nice round robin, you can be very, very productive.
[00:59:02] You can also chain agents together. You can have one agent producing code, another agent reviewing it. And actually just seeing how folks have adapted their workflow, um, is a big part of what we’re we’re looking for in our interview process.
[00:59:13] swyx: Amazing. I guess last question, but also open to you to bring up any topics that I haven’t touched on, have you wanted LLMs to do that they still cannot do today?
[00:59:23] David Singleton: That’s a great question. Um, and it’s amazing ‘cause the capabilities of LLM just, just advanced so quickly. You know, if you’d asked me a year ago, I would’ve said, well, you know, music generation, I, I like music. Um, and Suno is amazing by the way. And, but previous generations i’d, yeah, I can kind of tell that that’s AI generated today.
[00:59:42] I listened to the latest tracks made by Suno. I’m like, that’s, that’s pretty impressive. If we went back six months, I’d be asking for better image generation. The latest nano banana, uh, which by the way is a tool on the platform that you can use on Dreamer is producing spectacular infographics.
[00:59:58] Spectacular [01:00:00] painterly images when I ask for those as well. So, so that’s quite impressive. I still think I, so I think as we go forward into the future, there is still a lot of room for human creativity and so that’s also maybe where I’m going to have to say that LLMs are most lacking. So I think that when you think about building software, the thing that’s really important and that we all need to bring is taste.
[01:00:24] Mm-hmm. Right? You have to like actually truly understand people, their motivations. How do I build something that’s really delightful? So, you know, we had to do a lot of work on Dreamer to make it possible for the experiences that we build to not look like AI generic slop.
[01:00:43] swyx: Right? We go,
[01:00:44] David Singleton: um. And we’ve done that by putting a lot of our own taste into the templates and the prompts and the, the harness.
[01:00:52] Um, so I hope you have fun playing with it. I, I, I think Dreamer today generates experiences that don’t feel super generic, but that’s a ton of [01:01:00] work, right? The LMS do not do that by default. And in fact, I, if I see a, if you ask for a simple like to-do list app or something, uh, built by the models, I can tell which model built it just by kind of how it looks.
[01:01:10] So, um, taste, creativity, sense of individuality is still something that I think the LLMs are not producing out of the box. And I think that’s gonna be an interesting frontier. What do you think?
[01:01:21] swyx: Usually that’s, this is by, uh, from builder to researcher question. ‘cause uh, we do have researchers listening.
[01:01:27] Yeah. And I’m just like, well, that’s it. But like soft taste for me please is, is like a very broad topic. Uh, what do I think? I mean, I agree. I just think that it’s too big of a topic to break down. Mm-hmm. Particularly because there’s a lot of, I’ll know it when I see it type, uh, eval, which is unverifiable for, for researchers to do so.
[01:01:45] David Singleton: Yeah, I mean I, I do talk to researchers quite often and, uh, we talk about this topic and I think most people agree
[01:01:51] swyx: uhhuh
[01:01:52] David Singleton: that, you know, one of the great things about building models to generate code was just, you know, it’s so verifiable. So Yeah. Um, you know, it’s [01:02:00] very tractable and they agree that the next problem is how do you kind of step up that hierarchy of needs and get into these taste questions.
[01:02:08] And quantifying taste is hard, but I’m actually kind of excited that some people are gonna start doing this. And you know, that’s when I think that some of the really iconic companies in the world will start to become places where, you know, folks really try to like. Debug and understand the creative process.
[01:02:23] And I get pretty excited about that.
[01:02:25] swyx: Yeah. Uh, I, I think we are slowly uncovering what intelligence really means and, and the, the benchmarks that we adopt and then abandon because they’re solved is, is basically us evolving the machine intelligence in the way that we, the different way than we evolved, but we are slowly understanding what it means to be intelligent.
[01:02:44] Right. And, uh, and it’s, it’s interesting. I wonder how they suppress us in the future, but like, we’re not even there yet. We’re just like, get, get it to a place where we like what we get. Mm-hmm. From the machinist sometimes. You know, it used to be 30%, now it’s like 95%, but still there’s that 5%. [01:03:00] That’s right.
[01:03:00] Yeah. Any other topics we should have touched on?
[01:03:02] David Singleton: No, I think we’ve covered everything, but I wanna remind everyone,
[01:03:06] swyx: ct
[01:03:06] David Singleton: dreamer.com/latent space.
[01:03:09] swyx: Yes. No, it’s a, it’s a very good deal. I mean, like, come on. Like, yeah. So thank you for offering that.
[01:03:14] David Singleton: Cool. Well Swyx, thank you so much. This was fun.
[01:03:16] swyx: Yeah, thank you.
[01:03:17] Uh, we, we’ll get Alejandro to put like flashing neon signs on the, on the YouTube. Cool. Wonderful. Um, alright. Thanks. So my cool,
[01:03:23] David Singleton: awesome, thank you.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Claude Cowork came out of an accident.
Felix and the Anthropic team noticed something interesting with Claude Code: many users were using it primarily for all kinds of messy knowledge work instead of coding. Even technical builders would use it for lots of non-technical work.
Even more shocking, Claude cowork wrote itself. With a team of humans simply orchestrating multiple claude code instances, the tool was ready after a brief week and a half.
This isn’t Felix’s first rodeo with impactful and playful desktop apps. He’s helped ship the Slack desktop app and is a core maintainer of Electron the open-source software framework used for building cross-platform desktop applications, even putting Windows 95 into an Electron app that runs on macOS, Windows, and Linux.
In this episode, Felix joins us to unpack why execution has suddenly become cheap enough that teams can “just build all the candidates” and why the real frontier in AI products is no longer better chat, but trusted task execution.
He also shares why Anthropic is betting on local-first agent workflows, why skills may matter more than most people realize, and how the hardest questions ahead are about autonomy, safety, portability, and the changing shape of knowledge work itself.
We discuss
* Felix’s path: Slack desktop app, Electron, Windows 95 in JavaScript, and now building Claude Cowork at Anthropic
* What Claude Cowork actually is: a more user-friendly, VM-based version of Claude Code designed to bring agentic workflows to non-terminal-native users

* Why “user-friendly” does not mean “less powerful”: Cowork as a superset product, much like how VS Code initially looked simpler than Visual Studio but became more hackable and extensible
* Anthropic’s prototype-first culture: why Cowork was built in 10 days using many pre-existing internal pieces, and how internal prototypes shaped the final product
* Why execution is getting cheap: the shift from long memos, specs, and debate toward rapidly building multiple candidates and choosing based on reality instead of theory
* The local debate: why Felix thinks Silicon Valley is undervaluing the local computer, and why putting Claude “where you work” is often more powerful
* Why Claude gets its own computer: the VM as both a safety boundary and a capability unlock, letting Claude install tools, run scripts, and work more independently without constant approval
* Safety through sandboxing: why “approve every command” is not a real long-term UX, and how virtual machines create a middle ground between uselessly safe and dangerously autonomous
* How Cowork differs from Claude Code: coding evals vs. knowledge-work evals, different system-prompt tradeoffs, longer planning horizons, and heavier use of planning and clarification tools
* Why skills matter: simple markdown-based instructions as a lightweight abstraction layer for reusable workflows, personalized automation, and portable agent behavior
* Skills vs. MCPs: why Felix is increasingly interested in file-based, text-native interfaces that tell the model what to do, rather than forcing everything through rigid tool schemas
* The portability problem: why personal skills should move across agent products, and the unresolved tension between public reusable workflows and private user-specific context
* Real use cases already happening today: uploading videos, organizing files, handling taxes, managing calendars, debugging internal crashes, analyzing finances, and automating repetitive browser workflows
* Why AI products should work with your existing stack: Anthropic’s bias toward integrating with Chrome, Office, and existing workflows instead of rebuilding every app from scratch
* Computer use one year later: how much better it has gotten, why vision plus browser context is such a superpower, and why letting Claude see the thing it is working on changes everything
* Why many “AI verticals” may get compressed: specialized wrappers may matter in the short term, but better general models and stronger primitives could absorb a lot of narrow use cases
* The future of junior work: Felix’s concerns about entry-level roles, labor-market disruption, and whether AI can compress early-career learning into denser simulated experience
* Why Waterloo grads stand out: internships, shipping experience, and learning how real teams build products versus purely theoretical academic preparation
* The agentic future of the desktop: what it means for Claude to have its own computer, whether AI should act on your machine or a remote one, and how intimacy with personal data changes the product design space
* Why Electron still mattered: shipping Chromium as a controlled rendering stack, the limits of OS-native webviews, and why browser engines remain one of the great software abstractions
* Anthropic’s Labs mentality: wild internal experiments, half-broken future-looking prototypes, and the broader effort to move users from asking questions to delegating increasingly long and valuable tasks
* Why the endgame is not just more capability, but more independence: teaching users to trust AI with bigger scopes of work, for longer durations, with fewer interventions
Felix Rieseberg
* X: https://x.com/felixrieseberg
* LinkedIn: https://www.linkedin.com/in/felixrieseberg
* Website: https://felixrieseberg.com/
Anthropic
* Website: http://anthropic.com
Full Video Pod
Timestamps
00:00 — Cheap execution and building all the candidates00:44 — Intro in the new Kernel studio02:47 — What Claude Cowork is04:18 — Why user-friendly can be more powerful05:33 — How Anthropic built Cowork07:09 — Prototype-first product development08:00 — Why local computers still matter09:20 — Skills, primitives, and platform leverage12:13 — Cowork’s architecture: VM + Chrome + system prompt15:38 — Felix’s own bug-fixing Cowork workflows17:38 — Local-first agents20:16 — Evals, planning, and knowledge-work optimization23:14 — What Anthropic means by evals24:21 — Scaffolding, tools, and why skills matter27:44 — Demo: YouTube uploads and self-generated skills31:03 — Calendar automation and cleaning your desktop34:47 — Browser context and why DOM access matters37:47 — Skills portability and plugins44:36 — Which AI categories survive?46:19 — Junior jobs, simulated work, and labor disruption52:00 — Gradual takeoff vs big-bang takeoff53:42 — Finance, taxes, and enterprise verticals56:24 — Vision and the improvement in computer use57:31 — Why Claude writes its own scripts58:06 — Should Claude have its own computer?1:01:26 — Windows 95 in JavaScript1:03:19 — VM tradeoffs and sandbox design1:07:23 — Approval fatigue and safe delegation1:11:18 — The future of Cowork1:12:27 — What comes next for agentic knowledge work1:15:13 — Electron, Chromium, and desktop software lessons1:22:16 — Multiplayer agents and coworker-to-coworker workflows1:26:05 — Anthropic Labs and closing thoughts
Transcript
Alessio: Hey everyone. Welcome to the Latent Space Podcast, our first one in the new studio. This is Alessio, founder of Kernel Labs, and I’m joined by swyx, editor of Latent Space.
swyx: Yeah, so nice to be here. Thanks to, uh, TJ, Alessio, Allen helping to set everything up. It looks beautiful. We even have the logo outside.
Yeah, kind.
Felix: It’s like really nice, right? When you walk in here as a guest, you’re like, ah, this is a serious production. You’re like, feel it immediately.
swyx: Yeah. Felix, you’ve been, you’re, you’re currently a product manager of Cowork or,
Felix: uh, really Technic
swyx: Eng. Yeah. The, the identities are kind of vague member technical staff.
Felix: I know member staff is like, the official title will carry around forever.
swyx: Yeah. I basically kind of wanted, like we’ve been. Kinda obsessed. I, I’ve been using it a lot, even for managing latent space. Like, uh, cowork helps me upload videos and like title things and like edit and everything. It’s, it’s like really amazing.
Alessio: Cool. He said multiple times Cowork has said gi in the group track.
swyx: Yeah, yeah, yeah. So, so we have a second, uh, we have a second channel, uh, for latent space tv. Uh, and I, uh, and uh, we basically, this is our Discord meetup. Um, and I I, we have like Claude Coworks, it might be a GI, I don’t know if we, we have, uh, uploaded it yet, but one of the sessions was like a, like a Claude cowork thing.
Felix: I, you have to see, I would love to see it. Like, I’m so curious, like one of the most fun parts of my job is like constantly see the weird things people use Cowork for because it’s obviously like very hard for us to actually design for specific use cases we do. But like every single person who’s like most amazed is usually amazed about a thing that I didn’t even expect cowork would be good at.
Um, we have a new designer and it’s one of the first small tasks. I was like, Hey, we need like a new emoji for cowork for our internal stock. It’s like a pretty small thing. I like, can you please do it? And he drew an SVG and just gave it to coworker was like, can you animate this emoji? And now it has like this beautiful loopy animation.
Um, and I mean, I think obviously this goes down to like, it turns out you can do more things with code than you expected, but it, it’s like that kind of stuff that is really fun to me. So, long story short, I would love to see like, the kind of things you’re doing.
swyx: I’ll pull it up. I’ll pull it up.
Felix: Yeah. Yeah.
swyx: Uh, but before we get into it, I, I think always wanna start with like a top level. What is Claude Cowork for people who haven’t heard of it? Haven’t tried it out.
Felix: Okay. Uh, real quick, Claude Cowork is a user friendly version of Claude Code. So the way it basically works is we have Claude Code and for us, fairly impressive agent harness that over December we noticed more and more people are using either, even though they’re not technical, they, they’re not at home in the terminal or they are at home in the terminal, but they started using Claude Code for non-coding workloads, right?
Like managing expenses or like filling out receipts or organizing a knowledge base. Like there was a big obsidian moment that a lot of people liked and we wanted to capitalize on that, but also bring, bring this capability to people who are not terminal native and who might not know how to like brew and store something.
So cowork is Claude Code running in original machine with a little bit of padding, a little bit more guardrails, making it a little safer and a little bit more convenient for people who don’t wanna first open up the terminal when they go to work.
swyx: It’s interesting, uh, that is kind of. Pitch that way as a more user friendly thing because I always feel like it, it, to me, I I treat it as like why I’m familiar with Claude Code.
Like we, we did a Claude Code episode Yeah. A year ago. But this one is like even more power user tools ‘cause it, uh, it kind of integrates much better with like clotting Chrome and, uh, in all the, all the other tooling. But like, maybe, maybe that’s like a perception thing, right? Like
Felix: No, honestly, I don’t think you’re wrong.
This is like a, a thing I’ve been thinking a lot about for like the last two weeks. So,
swyx: but when they say user friendly, it’s like, oh, it’s the dumb down version. But no, actually this is the superset.
Felix: Yeah. Like, I think a similar thing happened, A similar thing happened to me about 10 years ago, like maybe 12 years ago when I was at Microsoft and we started working on, on Electron and like browser-based technologies and cross-platform stuff.
And one of the first use cases was Visual Studio Code, which used to be a website. And the initial narrative was, or Visual Studio Code is, is like a more user-friendly version of Visual Studio. But in a similar vein, I think there was some voices saying, oh, this is. For serious developers, like, we’re not gonna use this.
Right? For like anything. And I think in the end what happened is people have different stories about why Visual Studio Code became such a big thing. But my personal, my personal belief is that the Hackability and the extendability has like played a pretty big role, right? You can hook in Visual Studio Code that like almost any workload, it’s so easy to hack on, so easy to put extensions for it.
And I think cowork might be hitting a similar thing where it’s very easy to extend and it’s very easy to bring into your workflows. Uh, so the convenience I think is a bit of a, it’s obviously the thing we strive for as developers, but I think the way people find value in it then is by probably mapping it onto whatever they actually have to do in their job.
Alessio: So end of last year, you see the spike of like non-technical usage and clock code. What’s the design process to say we should make clock code work? Because I mean, you built it in only 10 days. Um, I’m sure there was some discussion before on whether it’s easier to use mean. You know, like making, making like a desktop GUI is obviously one way to do it, but like there’s a lot of nuance in the product.
Like maybe talk people through what was like the trigger of like, we should build a separate thing. We should not build like a different plot code thing. And then maybe some of the more interesting design decisions that maybe you didn’t take.
Felix: Yeah, I think philanthropic, we’ve been thinking about ways to move people who are comfortable with using Claude to answer questions and bring more of the power of like this thing to now like, execute tasks for you.
I can like solve problems for you can like build things for you. How do we bring that capability to people who are currently mostly comfortable with like a like question answer paradigm within the chat. And we’ve had a lot of prototypes around that. Just going back as far as like easily a year and a half.
Like we had a lot of people working on that. Um, and internally philanthropic is a very prototype demo, first culture. We have a lot of like internal prototypes that don’t reach the public. What Cowork actually became is like we sort of picked the right pieces out of the many prototypes that we had.
Right. And that’s, that’s maybe also like, I think an important qualifier whenever people mention this like 10 day number. I do think it’s important to me to mention that within Double Scratch there was like a lot of stuff already happening, right? Like, and I think it’s important for people to remember that when you build a website, you use React, you use like a bunch of other things.
And this is like a similar scenario with like a lot of pieces we already had. Um, and in terms of decision path, I think we live in like an interesting new world where execution is actually quite cheap.
swyx: Mm-hmm.
Felix: So maybe, maybe what you would do That’s so crazy. The year. I know it’s wild.
swyx: You should be, ideas are cheap.
Execution is the hard part. I
Felix: know. And like the, we, we used to live in this world maybe where you would take a product manager and the product manager would go to a number of potential customers and in this like very low bandwidth way, would try to. Try to like tease out what are the problems they’re having, what are they willing to buy?
Um, and then maybe what can you build to like drive out that need and then you go back and you like draft a spec and you think about it and then like you make a design and you execute it. We internally philanthropic app, not pretty much closer to the point where we’re like, don’t even write a memo, just like build, like let’s build all the candidates very quickly.
Let’s just build all of them and then pick the best ones. I think the, the decision that is most impactful both for the product as well for the users right now is like the way we put value on your local computer. I think that’s a big decision point a lot of people have thought about. Should this thing, whatever it is, should it ultimately run into computer or should it run in the cloud?
‘cause they’re big trade offs, right?
Alessio: I guess like if we solve auth, it would be easy to do in the cloud. But I think like the fact that I can just download any file from anywhere and then put it and cowork there, it’s like a big unlock. Um, I mean it’s interesting you mentioned reusing certain pieces. I think this is something I’ve been thinking about even with Claude Code, right?
The price of like writing code is going to zero, blah, blah, blah. But it actually seems like the value of having some sort of platform substrate is like increasing because as you build these new things, you can kind of plug them together.
Felix: Yeah.
Alessio: So I almost feel like when people are saying, oh, the value of a lot of software is gonna zero because you can recreate it, to me it’s almost like the opposite.
It’s like having an existing platform to build on top of. It’s like even more valuable because you can kind of bolt things on.
Felix: Yeah.
Alessio: You have obviously mcps, you have skills, you have like obviously the models, which is a big part. All these things kind of come together. Do you feel like that’s a valid way to think about it, where people should invest even more in kind of like primitives.
To rebuild on or are you like recreating a lot of it each time because like things change and it’s easier to rewrite than reuse?
Felix: You know, I think, I think you’re right. I think you’re right that the holistic platform is really useful. And this is maybe a whole like a somewhat contrarian view to a lot of people in ai.
I actually don’t think that the future is going to be hyper personalized software down to the point where everyone is running their own version. Like, I actually think it’s going to be quite hard for all of us to have our own internal chat tool and like, if I wanna talk to you, like
swyx: how
Felix: is that gonna work, right?
In the, in the context of cowork and how we build it, I think it’s a bit of a combination. Like what the, the execution that gets cheap is not necessarily rebuilding all the primitives. I think our priori, there’s also not a lot of value in it. So for instance, my team did not think about rebuilding clock code.
We’re like very much started with the. The core thesis of this should be Claude Code.
Mm-hmm.
Felix: And then we’ll like build things on top of it. The part of the execution that gets a little cheaper is like, how do you take all of these Lego pieces and put them together in a way that makes sense for users?
It’s like actually valuable. You have so many different approaches now in terms of what kind of, what kind of things do you actually elevate to a primitive, do you strongly believe that all your products should be built by just combining primitive that the public also has available? Do you keep some things internal?
Um, and I think that’s still evolving, but I think what’s probably gonna go away is like, I’m not sure if it’s gonna fully go away, but I’m gonna say, I think for me personally, I will probably no longer try to come up with a really good product without testing up with people. This is not a new concept, but wherever you used to have to make costly decisions around, do we pick technology A or technology B, or do we like, um, build it this way, build it the other way.
I really strongly believe now you just build all of them and try them out with a small focus group and then whatever, whatever is better is what you go with. Right. And that, that is probably quite different even from how we maybe worked a year ago. Right. Like, I think, I think this happened very recently.
Alessio: Yeah. I started building something in on Electron since you’re here. Coincidence. Uh, but then Electron and like SQL Light are like, there’s like some issues that like between development and like, uh, building anyway. And I was like, let’s just rebuild the whole thing in Swift and just recreated the whole thing in Swift.
And it’s like, I. It’s done.
swyx: You know, I didn’t take any effort. I, I, I don’t even know Swift.
Alessio: Yeah, exactly. I was like, I’m the, I’m not reviewing it anyway, whatever. You can write in whatever language you pick, but the important stuff that I did was not write the electron bindings. Yeah. It was like the logic of what happens in the app, you know, and then the model is like, yeah, I can just recreate the same thing as with
swyx: Yeah.
I, I think you still want, especially for people who are doing like high performance software or like very complex software, uh, you still want like, some view of the architecture. Uh, but you can use markdown for that,
Felix: right? Yeah.
swyx: Uh, you don’t actually have to read the code again. I, I’m still like on a sort of like a definitional thing.
Um, can we build a good mental model of Claude Cowork? Um, this is what I have, right? Like you you said it’s like fundamentally cloud co. We don’t wanna touch it. There’s the cloud app, there’s clouding Chrome. I think you guys do something different in planning, but, uh, I’ve been talking with Tariq who is on the cloud co team, and you guys are, he’s like, no, we just exposed planning.
Maybe we can clarify like, what are the major pieces. That people should be aware. It goes into cowork, like,
Felix: okay, I think you basically have them. So really, um, you can, you can take planning more or less out. I think there’s a few things that are really valuable in cowork. Um, the virtual machine is probably the most powerful thing.
So we currently run like a, we currently run like a lightweight VM and we put clocked out into the vm and we do that for, for, um, a number of reasons. Safety and security is a big one, but even if you, even if you ignore for a second safety and security and you’re just like, okay, Yolo, I want this thing to do whatever.
It is quite powerful to give Claus on computer that is like generally a good idea. And in terms of architecture and UX and everything else that we’ve been working on, philanthropic, it often is quite useful for you to like anthropomorphize, um, clot aggressively and just be like, this is a person. What will you do if you give a, if you had a person, right?
Yeah. And the analogy I’ve given my dad this morning who is still like quite insistent on using chat even for like coding things, is if you were a developer and your employer told you that you don’t need a computer, they’re just gonna like, send you emails with a code and you send emails with code back like that, maybe work for Patrick Miles in the back, but that it’s not very effective.
Um, so what we can do with the VM is because it’s a, it’s a Linux system, Claude Code has more or less free reign to install whatever needs to install. It can install Python, it can install no js. We do have strict network ingress and egress controls. So you can still, as, as a user in like plain human language, make it clear to, to the entire system what you’re okay with and what you’re not okay with.
But at no point do we have to ask a real person, like a, like a person who might be in marketing or a lawyer. I’d have to go to a lawyer and be like, are you okay with me installing Homebrew?
Alessio: Yeah, yeah.
Felix: Right. Because the implications of the question and the answer are complex and nuanced and like, not, not easy to reason about.
This gives us a lot of distraction that makes Cloud very powerful. Now then around it, we, we do probably have a number of things that also keeps growing almost every single week that you’re probably noticing that make cowork maybe better for certain tasks than just cloud. Cloud on its own. Yeah. But most of those actually live in the system prompt.
They’re about like, what can we infer about the work that you do? What can we, what can we intru in the system prompt to make that more effective? It’s of course the like very tight integration with Cloud and Chrome. You’re noticing that a lot of people, especially as the models get better, a lot of people throw up their hands when it comes to MCP connectors in this area.
I’m not gonna, I’m not gonna go through like 25 M CCP connectors, click off everywhere and then like half of them don’t let me do the things anyway. So Cloud and Chrome is quite powerful because we can just talk to the cloud and Chrome sub agent and that will just do things for you.
swyx: Yeah, so, so one example right in MCPI, honestly, I think that the state of MCP is kind of, kind of.
Really hard to integrate. Um, I need to, I needed to add, uh, Figma MCP to the coding agent that I use.
Felix: Yeah.
swyx: Uh, and, but I didn’t wanna read the docs, so I just had caught to it. And it’s, it’s great at reading docs and the same, same way I had to set up like a Google Cloud, um, account for some project I was working on and get some API keys somewhere.
And Google Cloud is famously super hard to navigate, so I just didn’t wanna deal with any of it. I just used Claude Cowork
Felix: within the first week of developing on Core. This happened very, very quickly. Um, I caught myself by starting to use cowork for coding tasks, which is not ostensibly what we built it for, right?
We don’t need to. But I found myself, um, I found myself like on our internal, internal tool that we have for, to collect crashes and just like debugging information and I found myself sort like picking out the ones that I think we can easily fix versus the ones that might be like kernel corruption or something else on the operating system.
And I found myself sort of picking these out and then just telling Clark, go fix this bug. I was like, what am I doing here? Go one level up, tell a cowork, I want you to go to all these crash tools. I want you to find all the bugs that you think are fixable and not like an operating system crash. And then I want you to tell another cloud to like fix all of that.
Um, and that’s, that’s, that’s sort of another cloud,
swyx: just so it can spin up another instance or,
Felix: uh, it, currently what I do is, um, and this is a bit of a hack, but I tell it to use clockwork remote to which website itself? Yeah, that’s interesting. So you basically take, if you, if you imagine like a dashboard with like 20 bucks, you, this is remote control or clock or remote, or, sorry, I just wanted to confirm what, the way I’m using it is.
I have cowork running and I’m telling cowork, here’s where I normally go every morning to find the latest bugs. Go read the entire bug list, separate out which ones are fixable, which ones are, are fixable, and then for the fixable ones, four is this almost loop. For each bug, write a markdown file with a prompt.
And then for each markdown v, that is a prompt. Start of a cloud set. So natively Claude Code has
swyx: this concept of subagents. Mm-hmm. And this is basically a subagent, but you’re not using the subagent functionality.
Felix: I’m not using the subagent functionality. And the reason I’m not is because I’m firing that off as a Claude Code remote
swyx: task.
Felix: Yes. That’s kind of nice. ‘cause then I can just fire it off. I can go to my next meeting and in Claude Code remote. Now the work is happening.
swyx: Mm-hmm. Yeah. You, you see like you’re already starting to use the cloud over your local machine. And I think this is one of those things where like. Shouldn’t just everything just be cloud first, right?
Felix: Ah, this is such a good group. I’m like solely bad about this. I have so many thoughts about that. Okay. So I generally believe that Silicon Valley overall is undervaluing the local computer. And my default argument for that is always how come we’re all using MacBooks and not like an iPad or a Chromebook?
Um, that there is like still value in, in having a local machine. And now when I think about Clot, it’s this entity that is supposed to be very useful to you, like it tremendously useful to you. I think that entity needs to have access to all the same tools you have access to. Otherwise it’s gonna be hamstrung in like all these complex ways.
And there’s, there’s sort of two approaches we could take. We could say, okay, we’re gonna like one by one chip away at everything that is at your computer and move it into the cloud. That’s, that’s one way to do it. Um, and I think other products have taken that path. I personally, this is a very personal opinion, but I personally, for the amount of tools that I use.
Just don’t have the patience to give another tool like permissions to every single thing and keep those permissions up to date. The second thing that I’m still grappling with, and I don’t have a good answer for anyone just yet, but the second thing I’m still grappling with is what does it look like for someone to slurp up your entire work and put that in the cloud?
Like if I, just as an example, like if you could click a button and it just clone your entire computer into the cloud, is that something that you would want? I’m not totally convinced yet that all everyone will. Mm-hmm. And that is sort of like upstream of all the technical issues we’re gonna have. ‘cause like in general, I think the world is not ready for this kind of stuff.
Like, I’ll give you one quick example that would probably be very easy for us. So as a desktop app, we in theory with your permission, can do a lot of things on your computer, including reading your Chrome cookies. If we really want to do right, we could take your Chrome cookies, you would have to decrypt them for us.
We could put those on the cloud if we really felt like it. Pretty easy solution. That would be super cool. We could just be like, oh, we can do all your tasks in the cloud now. Um, a lot of websites, thanks, include it. If, if they see the same authentication from like two different locations, we’ll just lock down your account and now you have to go to the branch and be like, okay, I, I’m here with my passport.
You actually know that. Wow. Yeah. As tired as well are of the term agent for the age agent future, I think there’s a lot of stuff that sort of slowly needs to catch up and until that’s the case, the way I, as someone’s working on clock and make Cloud most effective is to like put it where you are working.
swyx: Anything else? I thought with our mental model, so like, basically like, uh, part of me also just want, like the more I understand how it works, the more I can use it to its full potential. Right?
Felix: Yeah.
swyx: And so what I’m get hearing from you is you told me to delete the planning thing. You’re not doing anything special on, on the, that’s only exclusive to Qua cowork.
Felix: We have some tricks for this sort of like change week over week. We eval cowork maybe against different use cases than he would evil clock code, right? If you think about it this way. Okay, so like clock code is our eval clock cowork. Yeah. So clock code is like quite optimized for coding tasks and we mostly value it whether or not we’re getting better or worse depending on how good it is at like a typical suite job.
And Clark Cowork on the other hand, we evaluate more against typical knowledge work, the kind of stuff he would find in finance or in like maybe a, like in like a legal office. Um, my personal use case is always like managing my things, like managing my personal mortgage or something like that, right? Or like wealth planning for me and my family.
Those are the kinds of use cases we eval, clock cowork on. And what you might be picking up on is like the subtle changes we make to the system. Prompt what we put in the system, prompt how we steer, clot with the tools we give it. Um, like either it’d be better in one or the other direction and whether there’s a trade off, try us exist a lot.
CLO code will be better of a code and Claude Cowork will be better. For non-coding tasks, will those gaps still exist in the next three generations of models? It’s like a little unclear to me though.
swyx: Yeah,
Felix: because right now these like hyper optimizations we make, I’m not sure for how long they’re still be relevant.
swyx: I think what I was referring to was also, it, it just, uh, it qualitatively felt different when I probably, it’s just all prompting and I’m reading too much into it, but like the, the fact that it comes out with like a nine step plan, I can edit the plan and give feedback and, and, and see it execute the plan.
Yeah. It felt more long range than in Claude Code, but maybe that already existed in Claude Code and you just build a nicer UI for it.
Felix: It’s kind of both. Um, like if the Clark Code people who build the planning functionalities would city, they probably say yes, we have all of those things in Clark code and they do.
Um, I think people tend to give cowork. Tasks that are maybe of longer time horizon, I thought is
swyx: so long. Yeah.
Felix: That’s like one thing, right? It’s just like that the, the chunk of work tends to be maybe a little bigger. And then the second thing is that because the work, when it gets longer, it gets a little bit more ambiguous.
We do tell co-work to make heavy use of the planning tool or to make heavy use of the ask user question tool, right? We do want it to come up with like. Different scenarios of, okay, tease out what the user actually wants. Don’t go off to work for like four hours and then come back with the wrong thing.
And you’re probably picking up on that.
swyx: Yeah.
Felix: Um, I wish I could tell you I like built this magical thing and it’s like, there’s some secret sauce,
swyx: but No, no, no. I mean, it’s, it’s just clarity is good that, you know, engineers just want to know. Yeah. They can, they can plan around it. And then I think also for me, um, I am realizing I have to switch to my, my other machine because this is a new machine that doesn’t have my session.
But, uh, yeah, the, the, the planning is really important for, for me to like approve or like to see whether it’s like, it’s right. The ask is, the question is so beautifully presented. I mean, it also, it also available in like cursor and, and in Claude Code. But like, I, I think like it’s so nice to see that it, like it’s kind of for me like to understand that it gets me, it gets what I want to do.
Felix: Yeah.
swyx: Yeah.
Felix: It probably very hard
swyx: just on the topical evals. Mm-hmm. When you say eval, I think people are very vague about what it means. Is it just like vibe testing or do you have like automated programmatic evals of Claude Cowork?
Felix: When we say eval, uh, what we really mean is that we essentially take the entire transcript, including all the tools that clot has available ultimately to it, and we then measure what are the outputs, depending on what we tweak, right?
So we do run that a lot. We use that in training. Um, we use that in, in like, if you sort of separate out post training from like the scaffolding around it. Cowork sort of exists in the scaffolding space, but obviously we also train on it a little bit. Um, so when we say eval, we mean given the certain transcript, what do the outputs look like?
Including the file outputs as well as like the actual token outputs, like the ones that you see in the chat window.
Alessio: I’m curious, um, how much of the failure modes are the model intelligence versus like the usage of the end tool to put the intelligence in? Like the well planning is like a good example, right?
It’s like one thing is to come up with a plan. The other thing is like make a nice spreadsheet. Yeah. That kind of runs you through the plan. Like how have you seen that? Well,
Felix: the thing that I grapple with a lot is that whatever scaffolding you come up with, I think we still have a bit of sort of like model overhang where the model is dramatically more capable than right.
Users end up using it for. And I think part of that is that we’re just not getting the model all the tools to do all the things that’s theory capable of, right? There’s like one thing, um, however, whenever you do build the scaffolding, I’m sort of wondering at what point, at what point will that scaffolding go away and like how much you invest in figuring out what the right scaffolding is.
It’s kind of up to, it’s a little bit of a bet. And one thing that I as an NJ quite enjoy is that like working in philanthropic and working at a frontier lab, I maybe have a little bit more insight into what’s coming, coming down the chute in terms of like, what’s the next model, what is the model capable of?
What is good at, what is it bad at? And I’m, I’m increasingly wondering, is the right thing for us to like really invest too much in sort of these like scaffolding corrections where the model might otherwise not misbehave, but just not do the thing that you want?
Alessio: Yeah.
Felix: Or is it to just like give it as many capabilities as possible, try to make those safe so there’s the worst case scenarios, likeno status might be otherwise.
And then just simply wait a second for the next model drop. I’m personally, currently more leaning into the ladder. I think we’re gonna see a lot of like applications and companies that do very impressive things with ai that in the short term might seem very effective ‘cause they’re very specialized to individual use cases.
But I think once models get better generalization and get better at like those specific use cases without being super guided on those, I’m not sure how long that’s gonna stick around. And you can kind of, kind of already see this in like skills and NCP servers, right? Mm-hmm. We’ve, we’ve already seen sort of this like slow shift from MCP service to skills.
And like, maybe a good example is Barry who made skills. He was initially hacking on something that honestly looked a lot, looked, looked a lot like what Cowork does today. It was sort of thinking about what if cowork, but for like people who don’t wanna build code. Mm-hmm. And, um, he too did that as a prototype inside the desktop app.
One of the first use cases we thought of were, okay, what, what are like coding like use cases that could really benefit from graphical interfaces and like from being a little separated from the actual underlying code. And everyone comes with the same answers. Data analysis,
Alessio: right?
Felix: Yeah. Or saying how many users do we have today?
How many, like, it’s always data analysis. And I think the thing that ultimately led to skills is that we wanted to connect this little prototype to our data warehouse and. The team very quickly discovered that like instead of building a custom tool for the thing to talk our data warehouse, they just like meet and embarked on follow like mm-hmm.
Dear Claude, if you want to get data, here’s the end point. Here’s what the API looks like. You’ll figure it out.
swyx: Ah.
Felix: And then it be hand over control. Yeah, yeah. Also just like maybe go one step up in the layer of abstractions, right. Just, yeah. Instead of, instead of telling the thing, here’s ACL I, please call the CLI, or here’s an MCP.
Please call this ECT shape. Just like this is the end point. If you wanna know something, if you post here, maybe you can do post sql. It’s gonna be okay. And that ended up being so effective that they started trying the same pattern of like just giving the model a markdown file that describes whatever it needs to do.
That the whole thing eventually became skills and we’re like. We should package this up. This is a good idea.
swyx: Yeah. Um, we’ve had Barry Mahesh, uh, on, on our conference and uh, he’s uh, definitely got a good idea there.
Felix: Yeah.
swyx: I wanted to show you the, how I’ve been using Claude Cowork.
Felix: Uh, this is was my favorite part.
swyx: This is this. So this is like me, uh, this is how we run the Discord. Uh, we literally, uh, at first I didn’t trust Cloud Core. This was my very first usage.
Felix: Okay.
swyx: Right. So then I was like, okay, I will just try to manually download from Zoom all my recordings and upload it to YouTube. Yeah. Because this is a very laborious process.
I got a click, click, click YouTube, um, isn’t super user friendly. Uh, and it just did it. And then I was like, actually, you know, even the download from Zoom part, I should also. Put into Claude Cowork, and then I did it right. Here’s a bunch of, and it starts compacting here, and it, and it, it starts to even be able to do things like look through the individual frames of the video to name the video so I can upload it auto automatically.
Oh, that is, and this replaces my job as a YouTuber. We will forever appreciate your creative Yes. You know, and so that’s great. Uh, but then by the way, it compacts and makes, makes like a new thing, right? So I, I don’t, I don’t have the initial, initial thing, but then I asked it to make its own skills so that it, so that something that’s repetitive and one-off and human guided becomes more automated and I can use the skills independently and reuse them.
Uh, and it obviously you can write skills and that goes into context and skills at the bottom here, which is, which is so nice. Um, so I have all these skills that, that I now sort of do on a weekly basis. Uh, I know you’ve released scheduled Coworks, which I haven’t done yet, but
Felix: course I should try them. I, I think this is like so wonderful and fun for me to see because.
One thing that is very fun for me about skills in particular is that they’re so easy to make. Like anyone can make a skill, like a text message, could be a skill, and they can be so hyper personalized to you. And this is like sort of the subtraction layer, right? Like, um, I, I’m just guessing, but I assume, heck, you are very good at your job.
You’re probably given this thing some guidance about how to do it, right? I,
swyx: I just said, wrap everything up into, into a skill, right?
Felix: Yeah.
swyx: And then, uh, and then I was like, actually, sometimes I might need to break, uh, things apart because some parts fail or some parts might be needed in individually. So I told it to split one skill into three skills.
So it’s like a skill splitting thing, and then there’s like a parent skill that just orchestrates all of them if I want to use that. You know, like, um, I think that’s, that’s like really good. Uh, and, and, uh, there’s, there’s one more part, which is the, uh, Google Chrome thing that I told you about.
Felix: Yeah.
swyx: Where I’m like, okay, you know, what’s better than uploading, using Claude Coworks to YouTube?
Like actually. Looking at the docs to like programmatically upload to YouTube and then putting that in a skill. And I’ve never done that before. I don’t want to deal with Google Cloud. Yeah. So Claude Cowork does it for me.
Felix: That is really cool.
swyx: So, so I, I just, I don’t care. I just, like, I do a thing. I don’t, it doesn’t really matter.
Felix: That is really cool. And then you’ve, I assume paired the skill just with the script that it’s built.
swyx: Yeah, no, I just update, update the skills.
Felix: Oh, that is beautiful. Yeah. That’s wonderful.
swyx: It’s kind of like a skill, like, uh, uh, basically I think like the way that people ease into Claude Cowork is like take a knowledge work task that you would normally be clicking around for and then, uh, try to turn, turn that, and then you do the, okay, well what if you went further?
Okay. And then when, if you went further, when, if you, and it sort of expand the scope of cowork as you gain trust with it and, and also teach it how to replace you.
Felix: Yeah. It’s like a little bit like playing factorial, but for your own life. Uh, like you say, you start really small.
swyx: Yeah.
Felix: You start automating something really tiny and like.
Once it clicks, you keep adding onto this like automation empire. Just like make your life easier and easier. My favorite skill has been, um, every single morning Kohlberg starts looking at my calendar and make sure that there’s conflicts because people tend to schedule a lot of meetings, sometimes last minute, sometimes miss it soft and painful.
And a lot of products have existed like that A lot. I’ve written in the custom prompt there. I haven’t made it a skill, um, honestly should.
swyx: Yeah.
Felix: But I’ve given it like pretty clear instructions about okay, here are some people, if they book over other meetings, I’m probably gonna go to their meeting. Like if Dario schedules a meeting.
swyx: Right.
Felix: Not try to reschedule down. Right. Um, and I think there’s some other rules in there about like what kind of meetings I care more about what kind of meetings I care less about. What is okay to like, maybe pun like when I want to be, when I want to be working, when I don’t want to be working. And it’s those really small things that I can think kind of click with people.
Right. When we launch co-work, I think one of the US races that went most viral on Twitter. X was clean up your desktop, which is stuff, because silly, that’s such a smart thing, right? Like you don’t need to model to clean up your desktop. Not really. Um,
swyx: like this, like clean up my desktop.
Felix: Yeah, exactly. Yeah.
swyx: I need to, I need to choose my desktop, right? I guess give it access to my desktop.
Felix: Yeah.
swyx: Okay. Uh, okay. This is very scary. Oh, we’ll do it.
Alessio: I did, I did it with my downloads folder. It was like, you have so many term sheets and there’s like eight copies of your rental lease for your office. I was like, all right.
Like, don’t yell at me.
Felix: It’s like, it’s not such a small task. And then like, I, I would never go out there and normally otherwise and tell people I’ve pulled a product. It can organize your folder. Right. Um, because it feels small. But I think to your point like,
swyx: oh, here’s, here’s the, here’s the ask user questions.
Felix: Yeah.
swyx: Uh,
Felix: beautiful. Right. Elite obvious junk. You probably shouldn’t click that.
Alessio: No.
Felix: If he’s not done right.
swyx: As long as it’s reversible, I don’t
Alessio: make up blend to,
swyx: yeah. Uh, yeah. No, I, I have a, I have a typical, everything is super messy folder. So, yes. I think this, this is super helpful. So this is a pretty simple task.
Mm-hmm. But I’ve, okay, here it is. Right. Here’s the progress. I don’t see this in, that’s why I’m like, this gotta be something different than, uh, than Claude Code, because I’m like, we
Felix: do. Yeah. That’s, we do system prompt that. We’re like, all right. We want you to think about like, this task Yeah. Methodology.
Yeah.
swyx: And then I can, I can, I can do like little suggestions for, for, for these things. It’s beautiful. Look at this. I, I can, I can like say like, oh, don’t do that. Don’t do this. It’s amazing.
Felix: I’m so happy. You like it. Um, I mean, the other way around, like we’re part of the Clark core team, if you would like this in Clark COVID.
swyx: Yeah. Yeah. Yeah. Uh, so, so yeah, I mean, uh, this is really good. Obviously I, I’m like kind of raving about it. Uh, you know, I have other things like sign up for pg e so if you can do phone calls for me, that’d be great. Um, I, I do, people
Felix: have done that. Obviously you can’t do that natively, but people have done that with like, various other providers.
swyx: Yeah. Uh, and then this is like signing up for the Figma MCP. Um, I, I really am trying to do like everything, um, data analysis as well. I do think, um, oh, design to code, uh, very, very good. Right? So like, here’s a Figma file, take it. And then this is where like a lot of other tasks is like knowledge work, like replace my manual clicking, but this is no, I would normally use Claude Code or uh, Claude Code for this, but because I perceive that you have better Chrome integration
Felix: mm-hmm.
swyx: I, I think you can actually do a better job of this. And I, this, this is one shot at my, uh, conference website.
Felix: That’s pretty cool. Like at some point I would love to like, hear how you feel about code. In the desktop apps, which is like I never use, which is the, the same team. Same team.
swyx: So I use the call code in terminal, which I, I perceive to be the default way of cloud coding.
Felix: So one thing this has,
swyx: sorry, I’m just like, I’m not
Felix: here, I’m not here. All products. Can I talk about other stuff? Like I, I’m not sure if people out there wanna like hear me advertise my stuff for like an hour. Please do that. Um, this thing is like a builtin browser, which is a thing a lot of products have said.
Yeah, it’s a builtin browser. And I think giving cloud eyes into like what you’re actually working on makes it so much more effective. And that’s probably what you’ve seen in cohort because it can see Chrome, it can like debug the dom, it can like see things. Um, that does make it more powerful.
swyx: Yeah. So, so I think, uh, my mental model was kind broken.
‘cause I only use this cowork because I thought it had a, a browser thing in it. But I understand that the Claude Code app. The app version of Claude Code does have a built-in browser. I’ve seen, I’ve seen this preview thing.
Felix: Yeah.
swyx: I just, I’ve never used it.
Felix: But in the end, in the end, you sort of have it by hard.
Yeah. You basically get the same thing. Right? Like the, the, the additional skill that you’re describing is chart is better if we can see what it’s working on. Right. That’s, that’s sort of like the summary here and like whether it’s using your Chrome
swyx: Yeah.
Felix: Or it’s just like making up its own little like browser.
It doesn’t really make a big difference because either way it’s gonna see what it’s working on and that just makes it much better. And then you don’t have to run QA for your cloud.
swyx: Why doesn’t it pick up my existing Claude Code sessions? ‘cause I, I mean, obviously I’ve used Claude Code, but Excellent question.
Um, don’t have a good answer other than like, we’re honest. Just haven’t Yeah. This is what the Open AI team does. Okay. Uh, cool. I I I don’t have other, like, I, I just, I, I do wanna expand people’s minds and also maybe show people if they haven’t really done it, but like, I, I think it’s very interesting how I sometimes use this more than I use, I mean, I use dia, right?
Yeah. Um, I, and I use, uh, I’ve used like all the other agentic browsers and philanthropic didn’t have to build an agentic browser because you just had Claude Cowork and that’s enough.
Felix: Yeah. I also think like maybe integrating with number of excellent browsers out there, it’s like currently on my personal priority list, a little higher than like trying to rebuild a browser from scratch.
Yeah. You know, never say never, but I think going back to this idea of like, we wanna plug this into an entire existing workflow, I think our goal is actually to not replace any of the applications we have in your computer. But instead of like, work really well within a new workflow,
Alessio: make the new one. Yeah.
Are, it seems that nowadays, especially on the browser, most of the innovation is like user ergonomics. It’s not really like the underlying browser engine. So I feel like to call it, it doesn’t really matter if it’s like the, uh, or Chrome or Alice, whatever.
Felix: Yeah. We wanna, we wanna meet you wherever you are.
Which is like, like obviously I would say that, but it’s also just generally true because I don’t wanna shrink my potential user base artificially by saying, okay, like, I’m gonna start building for the people who are willing to switch browsers.
Alessio: Right.
Felix: That’s such a, like, you know, like many lawsuits have been filed over who gets to review the browser and like a lot of money has switched hands over the question of like, which browser is default and which search engine is default within the browser.
Um, I just wanna build for, yeah, I wanna build for swyx essentially. Like, I wanna, I wanna, I wanna build for people who have a number of annoying tasks that they feel like. Maybe clock could do it. Could do it for them.
Alessio: Yeah. What do you think about skills portability? I think there’s been one thing, I use another thing called zo, which is kinda like a cloud computer plus agent.
And I have a skill to add visitors to the office. Yeah. So whenever somebody has to come in after hours, they need to check in downstairs. Um, but I wanna like text the thing, so it doesn’t really work in, in cowork, but now that skill is in the zone harness and it’s not in my cowork thing. And then if I make a change, it’s gotta, I gotta sync them.
How do you see that going? Like I see memory as like. Cloud personal, kinda like, I don’t necessarily want my memories to be cross thing.
Felix: Yeah.
Alessio: But I do want my skills to be cross agent that I use. I think with MTPs, people do the same thing. It’s like, oh, Mt. P Gateway. Mt P registry. I don’t really know if that’s like a business.
So I’m curious like if you’ve had any thoughts in the area.
Felix: I think for me, this is sort of where I go back to the really basic primitives for our skills are file-based instead of like this complicated thing that exists inside a place somewhere that is like super proprietary. I’m really leaning into the idea of like, it’s all just files and vultures, and that makes it very portable on its own.
Right. We do have skills as part of this container format, which was just called plugins.
Alessio: Mm-hmm.
Felix: And plugins are available both for Claude Code and Claude Code work the same format, and you can install plugins. This works in cowork today. You can basically say, I’m gonna add a whole, like just a GitHub repo as a.
Skills marketplace or like a plugin marketplace. And that’s how we’re doing portability. I think we have a lot of room left to grow in. How do we make it easy for people to know that they can write skills? How do we make it easy for them to just like, share a skill with you? Because obviously all the words I just said, right?
Like I’m losing most of the knowledge worker base out there, right. And start by saying, oh, you can connect to GitHub repo. It’s not exactly how most people will end up working in like a general knowledge worker space. Um, but I think there’s something there. And another thing that’s there that I think has not really been properly explored is the, the, the combination of which part of the skill is very portable and then which part of the skill is like very personal to you.
Right. And I think that’s something we haven’t really solved as an industry. Hmm.
swyx: It’s like, which, how you wanna introduce more structure to the skill or have always have like. Public skill, private skill, you know, pair. Yeah, yeah. Kind of. I think there’s
Felix: like a, like the easiest way to do this, which is we do like use string interpolation or something.
Right, right. Yeah, yeah. Insert username here, insert like phone number, insert, like known folder, locations, that kind of stuff. Um, that’s probably clunky. That’s why we haven’t built it. Um, but I do think someone is going to come up with like an interesting way to keep everything we like about skills. The portability is just a file, it’s just marked down.
It’s just text, honestly. Right. Like a text file words. The complete lack of structure, which means you don’t need any kind of tutorial to write a skill. Just like explain it to Claude the way he would explain it to me and Claude will probably get it before I work. Mm-hmm. Right? You’re just like, for booking a flight, tell Claude how to book a flight the same way we tell him somewhere.
I just started working here today. But combine that with a very like, personal thing. Um, maybe we’ll stick with a booking a flight example. I don’t actually think. AI should be booking flights. I think the tools we have is yes.
swyx: Yeah. Finally, somebody says it. It’s the default demo that everyone’s making.
Felix: I’m
swyx: like, I even against like booking demos, it is not a good showcase.
Felix: Yeah. I’m like, I just wanna book my flight myself. But, um, I think there’s a lot of things that have a personal and a non-personal component and that’s maybe why people reach for flight booking because some things are very universal. Yeah. Super flight is usually better, right? Like few people try to book the most expensive flight.
And then some things are quite personal about like what times you prefer, which seat you prefer, which airports you prefer. Combining that and like a skill format that is actually portable, compatible, easy to understand for people. I think that would be very exciting. We just haven’t figured it out yet.
Alessio: Yeah, I think the text part every, I think everybody by now has some sort of like cloud file thing. Either Dropbox, Google Drive, whatever. So it feels like in a way it should basically like sim link. My skills into all my agent harnesses. Yeah. Just keep those ing like we have internally this like valuable tokens repo, which is like all the commands sub agents.
It’s good. Uh, and then I build like a TUI where you can start it and be like, you know, install this command and this three sub agents into this agent in this folder and just copy paste this. It doesn’t do anything. It literally cp the file into that. But I feel like there should be something similar where like whenever I go into a new thing, it’s like, hey, here’s like the link to exactly the cloud folder and just bring down these skills into this.
Yeah. Like today it doesn’t quite work like that. Like if I install a new agent, I cannot, I have to like copy paste all the skills and I don’t even know where they are.
Felix: Yeah.
Alessio: That’s like the big problem. It’s like where do I find them?
Felix: Yeah.
Alessio: Um, so I’m curious like in the future like that, that almost feels like my personal productivity thing will be my skills.
Felix: Yeah.
Alessio: Is not really the product that I use. Everybody has access to the same product. But today there’s, that just looks like copy pasting ME files, I
Felix: think so many things I, I really like thinking about agents and LLMs just as like another coworker. So many attempts have made to build documentation companies that are like, oh, we’re gonna solve oil documentation problems.
Um, I myself, like spend a little bit of time working in notion, right? I’m like deeply familiar with the concept of let’s get everyone on the same page. Mm-hmm. Right? And what you’re basically saying here is you want all your agents to be on the same page about your preferences, about the skills, about the way they ought to work and like how they ought to execute.
And I’m not sure what the right thing is going to be if it’s going to be some, some company that can say, all right, we’re as an independent body, we’re not trying to like, push into any particular product. It’s our job to be like the skill authority, and we provide, I don’t know, we’re gonna be the Dropbox of skills and we can just sim link us into all the products we want to use.
I’m not sure that’s gonna be viable business, but as, as an idea, it would be cool.
Alessio: Yeah. Yeah. I think so many things are just going away as businesses. It’s like, how am I supposed to do it? I’m not even asking somebody to make a product about it. Like yeah. I wanna personally know. And there’s things like you said, it’s like you almost wanna skill and then interpolate it between personal and work.
So if I’m booking a fly for work, it’s different than I’m booking a flight personally.
Felix: Yeah.
Alessio: In some ways, yeah. But like a lot of the scaffolding is the same, you know? Cool.
Felix: I mean, as an engineer I will tell you like, you know, technic a person to technic a person. I will just be like siblings.
Alessio: Well that’s what, that’s what I do.
We call that MD and agents that MD’s just the same how sim length. And so it is like, that works, but it feels like, yeah, I don’t know. Maybe
Felix: you can always go one, you can always tell cowork problem and then cowork will solve it for you. Just make the siblings. That’s like one way to do it.
Alessio: That’s true.
That’s true. All right. Everything is called cowork.
Felix: Uh, potentially spicy. Question for both of you.
swyx: Uh, which of these industries will go away?
Alessio: Okay, so what Felix was saying before is interesting. There’s busy like. The short term pressure of like, we need to turn these tokens into valuable things, which is I should build the last mile product that harness the model.
And then there’s the question of like, long term, which ones are gonna still be valuable? And I think you’re kind of seeing this today with like, uh, you know, the coding space in a way is kind of like everybody’s moving up and up in stack because you need more than just turning tokens into code. I think search, like enterprise search is kind of saying the same thing.
Like with G Clean and like all these different companies is like, at the end of the day, if Cowork is the one doing all the work, the search itself is like such a small part that like, I don’t know if I’m really gonna pay that much money just to do search. It’s almost like everything is like a cowork vertical.
So like how much can cowork first party support?
swyx: Mm-hmm.
Alessio: And how much can it not? I think for a lot of these things, the planning thing that you were showing do Which one? The planning. The planning.
swyx: Okay. Yeah. Yeah.
Alessio: That’s one thing where like most of the value that these agents provide is like they’re better at planning for specific tasks.
Yeah. And have better tools for it.
swyx: Yeah.
Alessio: But I think the models are now moving in that direction and they have the right harnesses and they’re on your computer. So for me it’s almost like if for the end customer trusts your startup to be the provider of that task result, then I think that works. This is, uh, something that, this is a short
swyx: spike that we’re, we’re working on.
Uh, yeah.
Felix: I think, look, I’ll, I’ll, I’ll tell you this, like I don’t think I’m the best person to like actually estimate which industry is going to be hit the hardest. But I do think that at philanthropic as a group of people, we’re deeply worried about the impact. That the tools are going to have on the labor market, especially for like junior employees that, because I think, I think it’s only honest to say that when we talk about automating a lot away, a lot of the work that we personally find annoying that we maybe think’s not the best use of our time.
In a lot of industries, that kind of work would’ve been given to a junior entry level employee. Yeah. Right. And I think it’s, it’s only, it’s only right to be really worried about that and like worry what that’s going to do in particular to people like enter the shop market.
Alessio: Mm-hmm. I have a solution for that.
Which you make them, you create simulative jobs for them.
Felix: Okay.
Alessio: So this is, this is like half joke, half true. So if you think about software engineering, when you’re like a junior engineer, you work like 1, 2, 3 years. And in those three years there’s like maybe like a handful of moments where like you really learn something.
And then a bunch of other days where like you’re not really progressing.
Felix: Yeah.
Alessio: I think now we can use AI and these models to actually like shortcut these careers and almost like simulate the early years of your work and like just make them like super dense and like these learnings, it’s like, hey, we’re working on this feature, which is like a distributed system and you need to learn this thing that might take three months at a company.
And so you take three months here, it’s like we’re just simulating the whole thing. It’s actually not a real thing. And in one week we kind of speed run through the whole thing and you kind of learn your lesson from there. And we kind of repeat that in like one year. You basically get like three years worth of like projects and experience.
Yeah. I think it’s harder for like things like sales or for things like, you know, marketing because you don’t really have a way to get the feedback loop. But I think a lot of it, it sounds kind of silly, it’s like you’re making the new effect job, but it’s almost like you go to college, right? People pay to learn how to do it, and this might feel similar where it’s like, hey, we have the.
Jane Street Simulator is like, you wanna come work at Jane Street? We’ll just put you in the simulator for like three months.
Felix: Wow.
Alessio: And you’ll come out of it. It’s like, you know, I’m ready.
Felix: So there, there is an aspect here. I’m not an expert enough to like actually know what, what is going to happen to marketing or legal or finance, right?
Like, I don’t work in those jobs and I, I don’t think I should talk about them, but I am an engineer and I think I have a pretty good idea of what engineering is like. And I think one thing we’re sort of seeing is that as a company and also as, as the public, we’re like deeply worried about entry level, but we’re also seeing more senior engineers accelerate it.
If like they’re more productive. They, they actually increase the value they provide. And the thing that I’m thinking about a lot is the fact that even before all of this happened, um, I’ve always had a lot of respect for the University of Waterloo and the, the new grads that have joined my teams as from coming from the University of Waterloo always felt like.
More ready than new grads will like literally spend their entire time at the university regardless of how good, but never actually had to work inside an environment where you have to ship things that eventually will be used by users. And I’m, I’m, I’m German. I like initially went to German University and I think the, the, the like information systems programs, there tend to be very theoretical, right?
Like I often give people the example of like trying to become a doctor, but you first have to do four years of biology and as a result when you get a new grad, you sort of have to teach them what it’s like to actually build products and to work in a company and like work with other people. And like some people will have different opinion and like, how do you do all of those things?
And the University of Ulu, it seems like they just. Spend half of their time. I dunno if it’s true, but I think it’s, it’s a year, right? They spend so much time,
swyx: part of your job, uh, a cu a curriculum to do spend a year in internships.
Felix: Yeah. They just like go from company to company. They show up on your team as like a junior engineer who spend like 20 companies.
Not really, but like, it seems like a lot of my new grads have also briefly worked at Apple, Google, Tesla. Yes. And uh, there’s a common meme where they like collect all these logos, like infinity stones, but, and they always put it on LinkedIn and it is very unclear that they’re an intern. Like Yeah, yeah, exactly.
But it does actually make them so much better compared to other new grads. And I wonder if that’s a useful model maybe for the future when we also have to like, crunch down the amount of time you have as a junior employee. ‘cause the value you have as a junior employee is going to like, be impacted.
swyx: My sort of pro young people take is that they’re, you’re more, uh, you have higher neuroplasticity, you can learn more, you have less preexisting biases.
And, uh, what I is assuming is true for you, what OpenAI often says is that. Actually it’s the, the younger, like fresh grad engineers that use Codex or their coding stuff, uh, more innovatively than the, uh, experienced engineers who have a set and preferred way of doing things.
Felix: Yeah. As I talk to people, I, I someone experience.
swyx: Yeah. So maybe you’re more AI native. Yeah. And therefore you’re, you, you get cut. But like, I think the problem is you don’t need that many of them.
Felix: I mean, philanthropic is on the record as saying we do believe that the impact on the market is going to be sizable and we do not think that people overall are ready.
Right. And we do actually think we should probably talk about it as a society much more. Yeah. I’m not sure that I’m like the individual that can add like anything useful there. But I think as societies with economists and, and governments that need to wrestle those questions in a way that is probably more meaningful than me wrestling with them, we’re probably not doing good enough.
swyx: Well, we, we’ll try to educate and then I think also just releasing frequently as, as, as you guys do, or probably maybe too frequently
Felix: Yeah.
swyx: Uh, is helping people to adjust over time. Right. Rather than one big bang thing. There’s like sort of this gradual takeoff that people are living through that we
Felix: Yeah.
swyx: Waking people up. Right.
Felix: Yeah. And I, but I think a lot of us like wondering at what point do we actually have full takeoff, right? Like at what point is there, we’re all sort of expecting this like big bang moment where things will accelerate so quickly that it becomes a self-reinforcing loop.
swyx: Mm-hmm.
Felix: And at that point, it’s sort of like off to the races and there will be no more like slowly catching up.
You notice just have cloud being so good at everything.
swyx: Yeah. It’s when cowork is training models, it’s when it’s looking at tensor board and Exactly. Weight and biases and training things.
Felix: I like we can all debate like how many years it’s away, right? Like some people make a better route, like maybe it’s 10 years away, maybe it’s a year away.
Um, I’m not entirely sure where, where I come on this time, but I’m not totally sure that ultimately it matters all that much, whether or not it happens in four or five years. If we have a decent one, certainly that’s going to happen. It’s probably something we should wrestle with.
swyx: I wanted to talk, so by the way, the, the scheduled task complete, uh, the, the, there’s the clean my desktop task complete and it did it organized by file type, which, okay.
But, you know, I was trying to get it to do more sort of thematic, like read the file, understand what it’s about, group by, uh, the, the topic rather than the file type. But
Felix: I mean, you can just follow up and have it do that. Oh yeah. Here, like it did, it is proposing That’s right.
swyx: Yeah. So it’s, it’s got some like topical things, but uh, yeah, I could probably do better.
Like, yeah, so like I probably need to give it a skill to read video files so that it understands here’s how I like to,
Felix: honestly though, like, um, I see that you’re using Opus 4.6, right? Like my recommendation for people is increasingly don’t worry about it anymore. Just like tell it what you want it to do.
swyx: Yeah.
Felix: And it’s probably gonna figure out a way to do it. It might not be the way that you like necessarily or the way that you’ve gone about it.
swyx: Videos, deeper,
Alessio: lower outsourcing, organizing all of this. So let’s fight. Yeah. Yeah.
Felix: I’m honestly like, so curious what cloud is gonna come up with.
swyx: I’ll kick that off.
I wanted to also just talk about the, the overall, uh, you know, you talk about data analysis, you talk about like, uh, your, your personal finances. You also said, uh, which by the way for us is very timely tax season, right? Like Yeah. Use cloud core for tax season. It is not responsible for any mistakes, but might as well, right?
Like it’s, it’s free knowledge work for you. Yeah. Uh, so I just like, I think cloud for finance is a big deal. Um, and this is definitely like in that mix. I wonder, is it like, do you, is it a separate team? Do you talk to them? How important is it? Right. Like, because you can also natively output Excel files now.
Felix: Yeah.
swyx: Just
Felix: talk about the
swyx: finance effort
Felix: grow. Yeah. We care about the verticals quite a bit. So we do have a dedicated verticals team. We have a dedicated enterprise team,
swyx: and those is business engineering, not sales.
Felix: It’s engineering. Yeah, yeah, yeah. It’s engineering. So we do have people who sort of come to work every single day and they, they ask themselves, how do we make co-work extremely effective for people in those specific industries?
How do we make it easier for them to understand, how do we make it easier for them to plug into this and like sort of get the same value out of it that software engineers get? I think it’s no real surprise that software engineers ended up being sort of at the forefront of the entire AI moment because so much of it is this like Rub Goldberg machine nest where like we’re already used to automating things, right?
Like it’s part of our job. Yeah. So we care about it quite a bit. I think it also like really matches what we see. Cloud being very good and as a model, I think it provides tremendous amount of value to those customers in particular because. We can do so much with the amount of data they have. Those are like data heavy industries.
Their industries for correctness matters quite a bit.
swyx: So for us of, I’ve used it to analyze my business, I just can’t show it. So
Felix: it’s two sense. I had a similar question about, about taxes. Like, I did tweet, I did tweet about the fact, I did tweet about, oh, COVID is doing my taxes. This is honestly incredible.
And, um, it’s like annoying. He is like, this is so cool, but I’m not gonna, Twitter is maybe not the audience that needs to like see my tax return.
swyx: Yeah. That way. Here, here it is. It’s it’s reading on the videos, so it’s like Yeah, it’s getting more, yeah.
Felix: How did it actually do it? I’m actually curious.
swyx: Oh, usually it just like, takes a screenshot and then it reads the screenshot vi by vision.
So this is what I do for my, my Zoom upload thing, right? Because I, I have paper club sessions that I need to upload to Zoom and I want it to automatically. Uh, title them and do show notes and everything. So it just take screenshots and try to try its best. Yeah. It wouldn’t probably benefit from transcribing, which it’s doing by, it’s operating by Pure Vision now, but it’s good enough.
Felix: Yeah.
swyx: And then I, uh, I do have to call, uh, out to Nano Banana to do images. So unless you guys do images for me, uh, I have to call other people your images.
Felix: We’re aware. We’re aware. It’s, it’s just like so fun for me because like, this is the thing that I’m increasingly doing, like increasingly curious about cloud’s, creativity and like figuring out what is great Claude’s approach is like some problem.
swyx: Yeah. Vision for everything is, is like the, the superpower, right? Like, you know, and computer use, you guys were the first to do computer use, right. And when it was launched, I was very unimpressed. I was like, it’s slow, it’s unreliable, it’s wild. How much better? ‘cause it is one year ago.
Felix: Yeah, I know. Like it was barely usable.
Yeah. I, I remember it was very usable, but is it wild how much better things have gotten? Yeah.
swyx: Yeah.
Felix: Over that one year
swyx: we went to the anthropic office because you, uh, for the launch event for computer use. Like there was like this hackathon. Yeah. And like nobody hack on computer use.
Felix: But I did see, I, I I don’t know if you’re okay with me saying that, but I did see briefly that you do have like a, like an automate Mac, SMCB server installed.
Right. Uhhuh, you use that ever.
swyx: What? Sorry? Which one? Where?
Felix: Um, if you go to your settings.
swyx: Oh, settings. Okay. Uh, where, sorry, this one?
Felix: Yeah.
swyx: Yeah.
Felix: Um, I noticed that in your connectors,
swyx: Uhhuh. Uh, I probably said it at one time, but I don’t use it actively.
Felix: Oh, okay. The
swyx: a max automated. Yeah. Yeah. So, so I, yeah, this one I really wanted to like, just automate everything in my thing.
I didn’t find, I didn’t find it super reliable.
Felix: Okay.
swyx: Why?
Felix: No, no, no question at all.
swyx: Cloud is much better writing Apple Script and executing its own Apple Script than relying on these, uh, third party tools.
Felix: Yeah.
swyx: Uh, so I’ve increased, I, I initially installed Im CP and like all these other fcps that people built, and, but now I don’t use any of them anymore.
Like just, just let cloud write its own thing.
Felix: Yeah. It’s
swyx: gonna be more custom made. We keep going up the stack,
Felix: but if using computer uses like a fairly interesting area to me, and it’s like also interesting in the sense that I don’t think we’re far away from, I don’t think we’re far away from clapping, very effective, but like using your computer and not just it’s theoretical computer.
Alessio: Mm-hmm. What’s the relationship between the user and the computer? Like, uh, there, there were some tweets about how huge some of the VMs, the Claude Cowork creates ours, like 12, 15 gigabytes and people complain. Yeah. But at some point it’s like, if you’re using the computer, you’re taking action on, it’s, it’s just your computer.
And I’m just looking at it, you know, it’s like, I, I think that’s why people like the idea of like the Mac mini and the open claw or whatever on it because it’s like, it got its own home. You know? It is doing its thing, I’m doing my thing. I think there’s some kind of like, not like risk condition, but it’s like, okay, if I kickstart this task now I can’t really use the computer.
Felix: Yeah.
Alessio: You know, because car coworkers doing things on it and it’s kind of awkward, like, yeah. I’m not sure.
Felix: I, I do think it’s a super interesting area because I, I can maybe tell you like some of the things I thought about that I think are actually a bad idea. So when, when we initially started working on cowork, I, I did have some dreams about, well, would it look like for cloud of its own cursor?
Could be cool, right? Like it’s a computer, we can write code, we can touch everything. Like who says that computers need to have one cursor? We could do a second cursor, but that actually breaks down quite a bit. Even if you go and like present cool dreams to both Apple and Microsoft, you’re like, wouldn’t it be cool if, um, it breaks down quite a bit?
‘cause so many of our models on a computer are built around this idea of like, there’s only one thing working on it. Yeah, there’s like a foreground app, a background app, cloud and Chrome can work in the background, but that’s like within one application. But the operating system layer, that is a lot harder to implement.
So I’m, I’m still grappling with what, what does it mean for cloud to actually act on your computer. It’s the right format for cloud to have its own computer that you set up. And maybe every now and then you like zoom in and you play with it. Or is the right format for Claude to just like, wait until you are.
Stepping away for a little bit and take over while you’re gone. Or it’s the right move for cloud. Just like if it’s on computer in the cloud, and like whatever you want cloud to do, you have to set up yourself. Right. There’s like a, there’s like a number of different options. Um, this is the thing I think about a lot, like what is the relationship between you and your computer and you and your data on their computer?
Because how intimate that relationship is kind of depends on the tool and Right. The thing that you’re current looking at, right? Like we’re quite comfortable sharing some things, very uncomfortable, sharing other things. And I think whatever product is gonna be successful, we’ll have to deal with those, like, with those different things.
But you probably, even if Claude was capable of making a determination, would you want Claude to make that determination in the first place? It’s tricky, Barry, because it’s like, it’s more than just privacy. It’s like almost intimacy and it’s like tricky to reason about in a way that will make everyone comfortable.
Alessio: Yeah, I could see. You know, a virtual box, like actual virtual box app where like you run the VM and then you have like a screen within the screen, you know, you can put it in the background, but then you can like jump in the screen and like you,
Felix: that’s not a bad idea. Yeah.
Alessio: You know, like, I mean I used it, you know, people used to do it virtualizing like C Linux in a Windows machine.
Felix: Yeah.
Alessio: And like you would just jump in and then you would jump out. But it’s like, it’s not like a dual boot. It’s like within the thing. The problem is that you need twice the amount of ram, twice the amount of, you know, it’s like, it’s kind of taxing on the machine. But I think that would be cool. Kinda like see, you know, the little quad window.
I can see desktop look cute. It is clicking around things
swyx: I was gonna bring up. He’s the original machine and the machine guy, because he has the uh, windows. Windows 95 project. Where’s, where’s the Windows 85 project at?
Felix: It’s probably somewhere in my GI guitar,
swyx: right? No, no, no, no, no. It is like the first thing you see is this one.
Nice. Yeah,
Felix: yeah,
swyx: exactly.
Felix: That was honestly a very fun project though. Like, obviously I didn’t, I, I should say this, just so that No, it’s the wrong impression. I did not write the actual, the actual, obviously I didn’t build Windows only five because I was a child, but also I did not build the actual engine that is capable of like simulating an X 86 processor and JavaScript and m um, that’s a tool called V 86, which is very cool and everyone should try.
But this came out of a, this came out of like a debate we had at work where people were like, they often are in the into debating the merits of electron and whether or not we should be building software in JavaScript, yes or no. And I still am very upset that I can run all of Windows 95 in JavaScript.
And launch Microsoft Excel inside the virtualized JavaScript Windows only five machine, and do things that pro, I can do that entire chain faster than I can do a lot of other things in like traditional SaaS applications. Mm-hmm. Uh, this is sort of like a, like a performance rampage that I went on. So I’m mostly built this as a joke for some of my colleagues at Slack.
This took, took like one night. Um, what, but then that I, it was, it was not hard to do. It was all the hard work is in V 86. Yeah. Like if, go to the repo, it’s gonna say like, 99% of his work is done by, by um, a guy who goes after the, by the name. Copy. His name is Fabian.
swyx: Yeah.
Felix: Um,
swyx: cool. I think you’re, you’re kind of back on the Windows grind ‘cause you’re building out the Windows support.
Uh, I thought there was some really cool technical stories to tell. Uh, and it gives people an appreciation of like, well here’s how hard it is and here’s how important here, how, how you invested the sandbox. So maybe this is like a good opportunity to talk about something in the details.
Felix: Oh yeah, the, the VM honestly is like so cool.
There’s a lot of things we dislike about the vm, right? Like there, there’s a lot of things that are real trade offs and you want to know why you making those trade offs. Um, and you’re right, like a lot of people write me like, Hey, how, how come cloud is taking up 10 gigabytes? I could say on the point, it’s not actually taking up 10 gigabytes.
It’s just like a way that macros displays bites is like wrong, but the way we actually ride it to disc is by we collapse the empty space and the image, so it’s not actually taking up 10 gigs. But that’s a technical differentiation. That’s probably not gonna matter to, like,
swyx: to me, the the, the outcome is it takes too long to start.
Yeah. It’s like 30 seconds sometimes. So I don’t know. Oh, it should be faster than that. Whatever it be te about this feels like 30.
Felix: Yeah. Like even either way, like whatever it is, it’s going to be, it’s going to be slower than just running Log Ultra on your computer. Right. So the trade offs are real, but what we’re doing on Windows, we’re using the Windows, windows, uh, host compute system.
It’s the same thing that WSL two runs on, like the Windows subsystem for Linux that I think a lot of developers appreciate quite a bit. Yeah. Um, and it’s, it’s pretty cool because we sort of like have to separate out which system space the virtual machine runs in, in who gets to talk the virtual machine because obviously you give this virtual machine a decent amount of power.
How do we optimize not just the connection between the two systems, but also how do we make sure that random other application doesn’t get to talk to Clot inside the vm?
swyx: Hmm.
Felix: We do some pretty interesting things. Um, last week we started writing a new networking service. A networking driver. That optimizes how Claw talks to the internet.
If your company’s doing like weird internet things like pack inspection and like, like, you know, taking your part as a cell and inside your company, I think there was probably like a very small, easy version to build of cowork that is much simpler but also breaks on most com most users, computers. And this one is quite nice because it works on most users computers.
Um, and the default example I always go for is I, I really want this to be highly effective on like a, on like a machine that most people pick up. And that machine will probably not have Python, it will not have no j And even if I just take away those two things, cloud is going to be so much less effective from
swyx: your computer.
So what do you do? You don’t even, I mean, may maybe require people to install Node in Python.
Felix: Oh, like, you mean for like a, what does the feature look like without a vm?
swyx: No, no, no. So, so like, like you said, right? Let’s say a target machine is whatever’s a default spec, windows laptop.
Felix: We do this, which is quite cool.
So on, on, uh, mes, we use the, um, apple virtualization framework, which is pretty solid, optimized, like it’s good stuff, and instead simple a p call, right?
swyx: It’s
Felix: like super simple.
swyx: I, I saw the code recently and I’m like, that’s it. What the f**k
Felix: would you, once you start like shipping production code on it, you start adding like all of these edge cases, your new
swyx: Oh
Felix: yeah, it ends up being a little longer, but, um, I think Apple really cooked with a virtualization framework and it’s very, very good.
It is very fast, it’s very reliable. And same on Windows. The, the host compute system. I think WSL two as well is maybe one of the diamonds within Windows. It’s like one of the few things that developers universally rave about is very, very cool. And like hooking into the same subsystem makes a lot easier for us to say We don’t really care how locked down your computer is.
Maybe it’s like your employer’s computer and your employer has decided that you get to install nothing.
Alessio: Mm-hmm.
Felix: Not trusted, but it’s true in a lot of environments, right? Like even at Anthropic, um, our IT department controls what kinda stuff you install, just like a pretty common experience for many companies.
Um, and this gives it departments a decent amount of, like, it makes their job so much easier because we can say you can separate out cloud’s computer from the user’s computer. And then for cloud’s computer, where you probably care about its data loss, you care about like a potentially hostile actor, you care about maybe data being exfiltrated.
And once you control the network and the file system layer, you don’t really care necessarily anymore. That cloud might be writing super useful Python scripts. What worries you about the fact is that like once you install Python, now anyone can do anything on a computer. Once you put that in the vm, that risk really goes down.
swyx: Yeah.
Felix: So that’s why we jumped through all of these hoops.
swyx: Yeah. I think you, you had a different, uh, tweet about this. Um, but it, it’s, it’s almost like people have also approved exhaustion. Like, it’s like you can’t approve every single commands. Like sometimes by, by default, some of the theis, I think even early called code, uh, we have to approve every single command.
Yeah. And, and like it’s so, so there’s this sort of dichotomy between either approve every step or dangerously get permissions.
Felix: Yeah.
swyx: And actually sandboxing is like, kind of like the middle ground.
Felix: Yeah. I do think, I do think it, it’s maybe on us as like the AI industry to come up something better than, oh, this is super safe as long as it doesn’t do anything right.
Right. But if you want this to be useful, then you have to like approve every single step of the way. And like, computer use is a good example. The only way to make computer use on your host, like super safe, like really super safe is probably if you approve every single action, right. Like models, like, I would like to type the word.
You’re like, okay, that seems fine. I know, I know. Which, like cursor is focused. Yeah. It’s not
swyx: automation if you don’t delegate.
Felix: Yeah, exactly. You need to like properly delegate. You need to be able to like delegate and walk away and trust that this thing is not gonna like mess dramatically. And I don’t even think we need to build perfect systems.
I don’t think we need to wait for like a hundred percent model alignment. We can rely on the same Swiss cheese model we’ve used in the industry for a long time. But I do think we need to like universally maybe eventually invest more. And that’s what we’re doing. We need to invest more in systems where we can say, you do not need to approve everything.
swyx: Speaking of Swiss cheese model, he just wrote a thing about this.
Felix: Oh cool.
swyx: Yeah. Uh, yeah. Um, yeah. Super cool. I mean, yeah, it’s, it’s weird how like, I guess usually I think safety and security is kind of like a boring word to, to engineers. They’re like, just gimme be unsafe, gimme unsecure. But, um, I think.
Achieving the right thing. Like you are going after a consumer slash prosumer.
Felix: Yeah. Yeah. Talking both kind of like both. I think I, I also want to capture people who would’ve no trouble using clock code like yourself, right?
swyx: Yeah. Yeah.
Felix: But still find it maybe just convenient, easier. You’re like, oh cool.
That’s like the list on the right. I can edit it. Those things are just easier to do if you have
swyx: to. But this is like clearly the knowledge work side. Yeah. Claude Code will clearly capture the development workflow. But like I, I, I do think like you have to sweat this like safety and security details in order for people to trust it.
And like the even Claude and Chrome, like having the whatever API uses to do the background thing.
Felix: Yeah.
swyx: Um, that’s the only reason I use it is because otherwise I would have to just get a separate machine.
Felix: Yeah.
swyx: And just run it, run to the, and that sounds like
Felix: super annoying.
swyx: Yeah. I mean, like currently doing it, but,
Felix: and I think, I think also as developers, um, maybe we’re, we are more risk tolerant, but we’re also just like accepting we are more risk tolerant, but I think we also just have.
I don’t wanna say arrogance, but like sort of the trust that if like the really bad thing happens, we can probably fix it.
swyx: I just tell Claude to like, check with me before doing any irreversible action. Like sending an email or doing permanently. Yeah, it’s good enough.
Felix: But like, not even Claude, I mean like simple things such as NPM install, right?
Like we’re all running NPM install with full user permissions and if it wants to like read SSH, it well crazy that that is the default kind of why. Yeah, I know. I agree. I agree. Fine. Like I’m obviously doing it every single day. No, right. Like, uh, and I think obviously NPM and GitHub too have like done a pretty good job maybe over the last couple months to like clean house and come up with like more specific tokens.
But generally speaking, I think as engineers we’ve always been a little bit more risk tolerant. And if you do a little bit of introspection and you ask yourself, is that how we should be doing things, you might not always come up with the right answer. And I think for models too, like my approach, like I’m not gonna, the the safest thing is to do nothing.
We do want products that are quite capable, but to the extent possible, I don’t wanna ask you, are you okay with the script? Because I kind of believe that once it starts becoming a part of your workflow, you’re probably not either, either you don’t have the skill to understand whether or not the python, the script is safe or you’re not gonna read it anyway.
swyx: Cool. I guess a, a couple partying questions. Uh, what’s the future of clockwork?
Felix: I think we’re still, we’re still such early days. We’re gonna keep shipping things that we’re gonna keep shipping, things that, um, we’re gonna keep iterating on this thing like pretty quickly, but, which I mean, you can sort of continue to expect that every single week there’s gonna be like a small new feature, if not a big new feature.
Um, I’m going to continue probably to double down on your computer and like making you effective in your computer and making cloud effective in your computer. Um, we’re starting to grapple, as we talked about today, grapple more with a question of like, what does it mean? What does your computer mean? Does it have to be the one in front of you or like a VM on your computer or like a computer somewhere else?
And then the third thing that I’m quite excited about is. We’re continuing to go off this hill climbing on slowly taking users who are used to asking questions and getting an answer to slowly teaching them to like step more and more away. And that claw take over like bigger and bigger tasks and work both in time as well as in like scope.
And I think you can probably see most of the, our investments on our feature releases to like work on both of those things, like the ability to do more on your computer and then the ability to do more independently for longer.
swyx: Does remote control work for Claude Cowork yet? No. Right.
Felix: Excellent question.
swyx: Coming soon. I mean, that’s an obvious thing if you want to keep betting on the, on your computer, but I, to me like. You know, we, we talk about like, people are not ready this year. Like the, there’s, there’s no wall. It’s, it’s accelerating to me like what will be we be doing differently at the end of this year that, you know, we are maybe not even thinking about this, uh, at the start of this year.
Right. Like, I’m just trying to look ahead as to like, what, what’s like a good use case that you’re, that we sort of aim towards? So for, for example, for the machine learning scientists, it’s always, okay, well I want AI scientists, I can automate, automate machine learning, but like for, for knowledge work, I mean, I can already, you know, get it to sign up for Google Cloud to mean as a GI.
Felix: Yeah. ‘
swyx: cause Google cuts are, but like, what, what is, what’s beyond that? I don’t know.
Felix: I think it’s basically the idea that like you still had to tell her to build your script, right? He was still kind of involved.
swyx: Yes.
Felix: In maybe a way that felt kind of magical to you, but like, maybe to me on the other side is the person building this product still feels kind of heavy handed.
I see so much process that I’m like, oh, lemme take that away from you. Okay. But like, how do I just go, I will continues to go or continue to go like further and further up the stack. Make your life easier and easier.
swyx: Oh, here’s one. Right?
Felix: Yeah.
swyx: Watch, uh, I, you know, I don’t care about my own privacy or whatever, or I trust cloud, I trust philanthropic.
So just watch everything I do on a normal day-to-day basis. At the end of the day, tell me what you is called co workable.
Felix: Yeah. I
swyx: dunno.
Felix: I think the funny thing about a lot of these products is that like, for good reason, I don’t enjoy, I, I don’t, throughout my entire career, I’ve never like teased too much what I’m working on because I think you should just like, yeah.
Release it. Yeah. Build the base and release it, and then talk about it. Like I’m, I’m not a big fan of the like vague posting my own work ahead of time.
swyx: Yeah.
Felix: But the thing that is like always so fascinating to me is like, both of you all multiple times a day, you’ve like mentioned things and I’m like, yeah, that is obviously like very obvious
swyx: Okay.
Felix: That someone should be working on those things. Um, and I think we’re still in the space where if you look at cowork. The things that we will be releasing will probably not be a big surprise to either of you. You’re gonna be like, yeah, obviously that’s valuable obviously that we’re working on those things.
swyx: Yeah.
Yeah.
Felix: And obviously that’s good and useful. And the more I hit those points, the more our features fit into that category, I think the better it is for us because then we don’t end up building things that are too hyper specialized to difficult harness style.
swyx: Yeah. I think the hyper specialized thing is very important.
It keeps you like general purpose. It, it means you’re not thinking too small. Maybe I don’t, I don’t know what the, the word is.
Felix: Yeah, yeah, exactly. It’s like the whole concept that like at no point if we release, you know, there’s no Claude Code for no jazz applications that use React and 10 Stack. I know any of those two things.
And like if it’s anything else, I know several startups like that. I think that’s pretty, like, I’m not a vc, I’m not an investor. It’s like hard for me to predict where the markets go. But in terms of the building box that I’m interested in, the electron is probably by far the most popular thing I ever built.
And, um, electron itself is like. Very abstractable and generalizable. Right? Like so many apps run in it. And I think it would’ve been hard for me to predict how many apps actually end up using Electron.
swyx: Yeah.
Felix: Um, and what would’ve been even less useful for me to predict this in what those apps do. I distinctly remember a bloom coming out of being like, that is cool.
Like you are a camera in a little circle in the corner. That is pretty smart.
swyx: That’s an app. Yeah. Yeah.
Felix: Or at least was, I’m not sure if it still is. It was for a while. Or like one password has so many interesting things. Right. It, it’s, it’s, it’s a level of the stack that I’m quite comfortable with. And whenever I give other engineers, advisors actually that layer that I think is most valuable to invest in because the tools of that layer are not that good.
But that’s where you get the most leverage
swyx: for like,
Felix: the future in general.
swyx: Just quick tangent on Electron. ‘cause I always wonder this, uh, have you looked at Tori?
Felix: I have, yeah.
swyx: What’s your take? Uh, you know, look, my, my my, my view is like most things should be Tori by default, unless you really need the full power of electron, but.
Felix: Yeah, I can give like my take on, I can give my big take. Why do we ship an entire version of chromium inside the thing, right? Like why do we do that? And, um, people ask me this question a lot because it’s like very counterintuitive. Wouldn’t it be much easier to use the web use that are on the operating system?
Wouldn’t it be much easier not to have to do that? And the answer is yes. And like obviously I did that once upon a time. I did that there was a version of the Slack app that used just the operating system that use Wait, did you, did you start the Slack app? I would, well, team effort and
swyx: Yeah, but I was, I was there.
We built the Slack app.
Felix: Yeah. It’s crazy. Um, I mean obviously you get the electron guy to do it, but, well, but this is an interesting point. Like, by the time, by the time I joined Slack, they already had an app that was built with something at the time called Met Gap. It was a little bit like the same app gap thing for mobile.
It just used the operating systems. Web views. Um, and that didn’t work for like so many reasons. Um, and they were like, all right, maybe we need like bigger guns. We need to like take more control of the rendering stack. And there’s, there’s a few things I always mention here. Um, I think if you’re building a small app, just going with the operating systems web view is perfectly fine.
If you’re building an app, maybe that doesn’t have too many users who will like cry bloody murder. If it doesn’t work, that is fine. The reason to go with your own embedded rendering engine is because, and this is still true in 2026, the operating system render engines are not that good. They’re just not that good.
Both Microsoft and Apple are trying to move away from that. They so far really haven’t, the only way to upgrade those is to upgrade your operating system. So if you are, say Slack and you have critical rendering bug in WK WebU and some of the other WebU options, your only recourse is to tell your customer, oh, sorry, you’re too poor.
You didn’t bother the, its MacBook. Unacceptable.
swyx: Mm-hmm.
Felix: Unacceptable to user, unacceptable to user developer. So you sort of need to like go down the stack and like find the best rendering engine, then put it in your app. Why chromium, even though it’s very big chromium is by far the best thing. Like I, I often like to remind people the unreal engine, you wanna render some text.
They use chromium. Like chromium is part of the unreal engine for same purposes. Chromium is very, very good. I think it’s like one of the marvels of engineering. It’s very hard for, we’re in San Francisco right now where we’re recording. Most of the people in the city are web developers. It’s hard for me to like overstate how magical it is.
They run seat like rendering a YouTube video dynamically. Negotiating a bit rate, figuring out what to do about your extremely broken hardware driver. Actually, this is a fun thing. Um, okay, you can enter Chrome call on Wack Wack GPU. Okay? And if you scroll down a little bit, these are all the enabled workarounds because something is going wrong on your computer.
If you’re doing this on a Windows computer with like A GPU, that is not the most popular GPO, it will be much longer. And all of these are usually just there to make sure that if I say as a developer, I want a red pixel to appear here, that that actually happens. Chrome is such a marvel because of works on all the machines that user might throw you and it’s gonna work fairly reliably.
And if it doesn’t, they will probably fix it within 24 hours.
swyx: I see. So this is the super operating system, right? That that works everywhere.
Felix: Yeah.
swyx: Right. Okay. Yeah.
Felix: So a lot of the magic of Electron is honestly just that it makes it very easy for you to ch chromium in a way that serves you exactly in your use cases.
Elect, uh, exactly.
swyx: Our next interview is with Morgan Dreesen.
Felix: Yeah.
swyx: Who had the phrase like, desktop OSS are just poorly deep, uh, poor implications of the, the actual os, which is Chrome, which like actually works everywhere. And this is this, this is the platform where you ship apps.
Felix: I, I think the wild thing is that like as engineers, we so often sort of assume that the platform, like the layer below us is like super stable.
Mm-hmm. And then you talk to those people and they’re like, ah, we are also just like guessing. Um, uh, and I had like a distinct moment at Slack where one of our customers at Slack was Nvidia, and for a while I really put GPU developers on this pedestal in my head. And I do think they’re still probably much smarter than I am.
But I was like hardware engineers who built the chips, who then like built the drivers. Their work must be so much harder than mine. They must be very good. And we had like one bug in Slack where like if you had a YouTube video in Slack, it wouldn’t quite render why. Like it would have these weird artifacts.
And, um, that ended up being a chromium bug. And I ended up on this like giant thread. So I got to see a lot of the source code. And they also are just like common to do. We don’t know why this is weird, but if you flip this bit, things work. You know, this is just like happening with every layer of the stack.
Maybe the, uh, you know, the,
swyx: the end of year a GI prediction is that clock can build chromium. You see, you see you, you laugh now. But yeah, like, you know, someday
Felix: it’s, it’s sounding, it could get pretty good. Like it used to be completely useless. Um, mostly just like overwhelmed, both with how hyper specialized tools are inside the chromium repo.
Like for, for a long time. Chrome has like sort of reinvent all the tools because none of them are capable of ending Chrome. I think the EGI moment I am kind of waiting for is at what point are we gonna say Electron is probably no longer necessary because you can just build fully native apps. The Swifty?
Yeah. Like not just in Swift because this is one thing, like it’s pretty easy if you, I think our current models are quite capable of taking an electron app and replicating it Swift, are they gonna be capable of like building an app that is actually more performant, which is less memory? All of that stuff, um, is gonna go into the same hyper optimization that developers have done for like a long time.
We’re not quite there yet. Work and like point even our best models at a thing and say, just replicate this, a native code. Make no mistakes. Ultra think. Right? We’re not quite there yet. Um, ultra
swyx: think is bad
Felix: today. Think is back. Yes. Okay.
swyx: Or we’ll get an ultra think for like days,
Felix: just a pretty long time before,
swyx: but he worked on Ultra think for days.
Yeah. Why he just, it’s just. Front,
Alessio: I’ll let it, the
Felix: more goes into
swyx: it. Yeah. Okay.
Alessio: Another question I had is like coworks. So if I have my Claude Cowork, like what’s kinda like the multiplayer mode? I think sub agents is like single player Split up the context.
Felix: Yeah.
Alessio: And the multiplayer cowork is like, my colleague is some file on their machine that I wanna know about or I wanna know how their task is going to then update my thing.
Like is that interesting? Is that something that makes sense for you to build or for like
Felix: It’s like super interesting to me it, it almost goes back to like some of the scaffolding room. Like okay, are we gonna be end up, are we, will we end up building scaffolding that will just go away? And like a question I have here is at what point do we just assign these things, like their own Gmail account?
We just give them their like Slack handle and then they will just like use the same tools we humans use to interact with each other. You mentioned our finance people, they’ve been working pretty hard on very good office integrations. And I think for a while we’ve been like, we built so much tech around cloud, leaving useful comments inside a Google Doc, and now it just does, it just like leaves a comment in your Google Doc and that’s how you interact with it.
Maybe like the similar thing where I still have open questions around what is the best interaction mode? Is it for us to build something super custom for cowork agents to talk to each other? Or is it okay, let’s just jump straight to the finish line and say, well, we’re just gonna give this thing, if you use Slack at work, we’re just gonna give this thing a Slack handle.
And that’s going to be the way, it’s like multiplayer capable.
Alessio: They communicate with each other. Yeah. Yeah. Like, you know, as a, as a fun project, I build this thing called piq, which basically takes any repo and the PI agent, uh, coding agent, it puts it in a VPS, and then there’s a public web hook where anybody can submit a coding task.
Oh. And then there’s a dashboard in which you review the task and then piq pi, pi, uh, queue.
Yeah. You basically get all these like tasks, anybody can submit a task.
Felix: Mm-hmm.
Alessio: And to me it’s almost like in the organization of the future, it’s like the sales people are talking to the engineering team that is talking to the marketing team, to the product team, and all these coworker are going to like queue up decisions for other people to approve in a way.
Felix: Yeah.
Alessio: You know, and I’m kind of curious what that looks like and like how do you, how do I give my cowork the ability to build a proof task without asking me
Felix: Yeah.
Alessio: And how to decide which one I need to review. Yeah. You know, because for some of these things it’s like, you know, you wanna change the color of something that’s kinda like a branding decision.
Or another one is like, hey, your thing is just broken. It’s like, this is like how you fix it. Yeah. And Claude can actually review whether or not that prompt matches what he’s trying to do today. Everything is still very, it’s like multiplayer within the single player, you know? Yeah. I guess spin up many of them, but like, how do I get multiple people to hand off to each other things using their particular context?
Felix: Yeah. And for both of your coworkers to like talk to each other. Right,
Alessio: right. Yeah. Hey, we got an episode today. Can you like, have you, you know, or
Felix: Yeah. This is like a, uh, I know we’re like running out of time here, but like we, we previously talked about sharing skills and I did have this question of like, what if your cowork would just like ask the other coworks if they have a skill for this task?
Doesn’t matter. These could do.
swyx: Right. Like, okay, so skill transfer.
Felix: Yeah, like,
swyx: um, and again, that’s, maybe
Felix: this maybe goes back into the territory of like building something very powerful and building something creepy often goes hand in hand. Um, because I could tell from the reaction that my fellow engineers said that this is probably not what we’re gonna do, but like.
We have Bluetooth le right? Like I, this computer can figure out that it’s sitting right next to this computer. So you’re probably working on the same thing. Um, well, you see that in cowork, probably not. But, um, there’s like, I think really creative solutions to problems that we really haven’t tried yet.
Yeah,
Alessio: yeah, yeah. Yeah.
swyx: Excellent. I guess the, the last thing is, uh, philanthropic labs. Uh, I always have this mental model of a model lab versus, uh, agent lab. And this is basically Anthropics internal agent lab, which co Claude Code, uh, is now under, right? It’s part of the whole org.
Felix: I mean, people are so fungible, right?
Like,
swyx: okay, this is just, I, I don’t know how, I don’t know real. This is, I don’t know.
Felix: No, it’s a real team. It’s a very, um, the, the last team is primarily working though on things that you don’t see in public yet. Um, they’re trying like really wild out there, ideas that seem quite improbable. Um, the mad science
swyx: thing.
But you, you’re, are you officially under this thing or
Felix: No? We’re, where is the Claude Code is, but now Claude Code is like a fairly big group where. I actually know many people we are like, like I remember yesterday coming into our weekly COVID meeting. I was like, woo,
Alessio: this is hot.
Felix: There’s a lot of people here.
Um, but we still have a labs team and we actually made the labs team a lot bigger. Mike just joined the labs team as a, as an ic, which I think is very cool and very fun. But they’re, they’re working on things that you have not seen yet that are extremely out there and probably half broken. Right? Like the sort of the idea of a lab team is that it should only work on things that make really no sense for anyone else to work on.
swyx: Okay. Well, looking for exciting things from there, but thank you so much. I know we’re out of time, but uh, appreciate your joining us. I appreciate co cowork, everyone go use it. Uh, it is the closest I’ve felt to a I this year. That’s so nice you to say. Thank you very much. Yeah. Thank you for your time. Yeah.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe