1 00:00:00,080 --> 00:00:05,519 A fun story that you recently shared on X as well is how you were part of the team that 2 00:00:05,520 --> 00:00:11,279 built this internal Google bot that was Chad GPT but a year before Chad GPT >> that caught 3 00:00:11,280 --> 00:00:12,479 up like you know wildfires. 4 00:00:12,480 --> 00:00:17,759 It felt more than a research [music] project >> for Codeex you built it in Rust and at 5 00:00:17,760 --> 00:00:20,559 the time the model was not on distribution for Rust. 6 00:00:20,560 --> 00:00:25,438 >> Turns out it was quite clear that Rust as a language would actually be quite good 7 00:00:25,439 --> 00:00:28,719 for agents fairly quickly if we decided to put some effort into it. 8 00:00:28,720 --> 00:00:31,358 When someone joins the Codeex team, what do you tell them? 9 00:00:31,359 --> 00:00:32,718 How do things get done here? 10 00:00:32,719 --> 00:00:35,119 >> The thing that they hear the most about when they have a question is like, 11 00:00:35,120 --> 00:00:36,238 "Have you asked Codex?" 12 00:00:36,239 --> 00:00:40,639 It still surprises new starters that you can basically ask it anything. 13 00:00:40,640 --> 00:00:43,839 >> Building was the fun part, but then maintenance was the painful. 14 00:00:43,840 --> 00:00:47,759 >> So maintenance is really sort of like a tax that you pay over time just to keep things 15 00:00:47,760 --> 00:00:51,359 running. But where I think it changes is like a lot of it is just going to be automated. 16 00:00:51,360 --> 00:00:53,198 >> So you still have the concept of code review. 17 00:00:53,199 --> 00:00:55,198 >> The role of code review is changing. 18 00:00:55,199 --> 00:00:58,399 And the role of code review now is like I think 19 00:01:01,359 --> 00:01:04,238 Codex is one of the [music] most popular AI coding harnesses today. 20 00:01:04,239 --> 00:01:05,919 But how did it all start? 21 00:01:05,920 --> 00:01:09,519 Many of you will know today's guest Tibo [music] from his generous and pretty frequent 22 00:01:09,520 --> 00:01:11,069 Codex usage resets. 23 00:01:11,119 --> 00:01:15,118 He was also there when Codex as a product started and has led the broader Codex team 24 00:01:15,119 --> 00:01:20,559 since. Today we cover how Codex started and why it was built in Rust and made open source. 25 00:01:20,560 --> 00:01:23,919 How coderies are changing inside the Codex team and open AAI. 26 00:01:23,920 --> 00:01:27,519 What it means when maintenance and rearchitecting are getting ridiculously cheap. 27 00:01:27,520 --> 00:01:31,919 what the merge of codeex into chat GPC looked like [music] and the many underappreciated 28 00:01:31,920 --> 00:01:33,679 engineering challenges of this project. 29 00:01:33,680 --> 00:01:37,839 If you want to understand how teams inside of OpenAI plan review and ship software, 30 00:01:37,840 --> 00:01:39,709 this episode is for you. 31 00:01:39,759 --> 00:01:43,199 This episode is presented by Turbopuffer, a ridiculously scalable, 32 00:01:43,200 --> 00:01:47,118 fast and cheap hybrid search engine built on top of object storage by an engineering 33 00:01:47,119 --> 00:01:49,679 team that I've really grown to like after spending time with them. 34 00:01:49,680 --> 00:01:53,359 Turbopuffer is the tool that companies like Entropic, Notion Cognition, 35 00:01:53,360 --> 00:01:57,999 and Harvey all use to connect their AI products to massive amounts of unstructured data. 36 00:01:58,000 --> 00:02:00,270 When I've talked with engineers who use Turbopuffer, 37 00:02:00,320 --> 00:02:04,478 the theme that always comes up is reliability and performance at scale. 38 00:02:04,479 --> 00:02:07,999 The reasons for this have everything to do with Turbopuffer's architecture. 39 00:02:08,000 --> 00:02:14,269 Turbopuffer uses only object storage for state and MVME SSDs with memory cache for compute. 40 00:02:14,319 --> 00:02:17,310 Data in Turbopuffer is organized into namespaces. 41 00:02:17,360 --> 00:02:22,479 You can think of a namespace as a database table or a search index or an S3 prefix depending 42 00:02:22,480 --> 00:02:24,189 on the world you come from. 43 00:02:24,239 --> 00:02:26,159 When a namespace is not being queried, 44 00:02:26,160 --> 00:02:29,999 it stays on cheap object storage with no associated compute cost. 45 00:02:30,000 --> 00:02:34,159 When a namespace is active, Turbopuffer pulls it up into hot caching tiers, 46 00:02:34,160 --> 00:02:36,350 so queries are very fast. 47 00:02:36,400 --> 00:02:40,909 This design fundamentally makes it effortless to scale to hundreds of millions of namespaces. 48 00:02:40,959 --> 00:02:43,199 If you're building a multi-tenant AI product, 49 00:02:43,200 --> 00:02:48,030 every user and their agent can have their own dedicated search index without any overhead. 50 00:02:48,080 --> 00:02:53,070 And each namespace can hold hundreds of millions of documents without any special configuration. 51 00:02:53,120 --> 00:02:55,759 You can scale Turbopuffer virtually without limit. 52 00:02:55,760 --> 00:02:59,518 And the performance, reliability, and operating model all stay the same. 53 00:02:59,519 --> 00:03:03,679 If you need to connect AI to lots of data, Turbopuffer should be your first choice. 54 00:03:03,680 --> 00:03:05,439 Check it out at turbopuffer.com/pragmatic. 55 00:03:07,040 --> 00:03:08,559 Tibo, welcome to the podcast. 56 00:03:08,560 --> 00:03:09,679 So good to have you here. 57 00:03:09,680 --> 00:03:10,958 Thank you for having me. 58 00:03:10,959 --> 00:03:11,919 It's so good to see you again. 59 00:03:11,920 --> 00:03:12,878 >> It's good to do this. 60 00:03:12,879 --> 00:03:14,318 Last time we did it in person. 61 00:03:14,319 --> 00:03:15,999 Now, now we're doing our video. 62 00:03:16,000 --> 00:03:20,878 First, I I wanted to ask you, how did you get into tech? 63 00:03:20,879 --> 00:03:24,990 When did you first know that you want to work with computers? 64 00:03:25,040 --> 00:03:25,839 >> It's a good question. 65 00:03:25,840 --> 00:03:28,878 It was a long long time ago. 66 00:03:28,879 --> 00:03:36,590 Um my my parents actually decided to move out of Brussels where I was born and 67 00:03:36,640 --> 00:03:42,158 just just thought it was great to um just buy a small house and refurbish it. 68 00:03:42,159 --> 00:03:47,439 But it was in the middle of the middle of a village with not much going on. 69 00:03:47,440 --> 00:03:51,150 I think there was like roughly 200 people living there. 70 00:03:51,200 --> 00:03:56,318 Not many that I felt like I wanted to talk to or you know could make friends with. 71 00:03:56,319 --> 00:04:01,119 And so I kind of got stuck uh this is like very early like 8 8 years old. 72 00:04:01,120 --> 00:04:04,958 I kind of got stuck cuz like you know computers and you know it's like early days of 73 00:04:04,959 --> 00:04:10,079 like the the for me the internet and you know that was my way to learn about things and 74 00:04:10,080 --> 00:04:13,119 so just the rest is just like you know came from that. 75 00:04:13,120 --> 00:04:17,599 Um I sort of like I owe it to my parents to you know have moved into the middle of nowhere 76 00:04:17,600 --> 00:04:20,798 and then you know I had no choice but to get interested in computers. 77 00:04:20,799 --> 00:04:24,879 Once you you finished high school like you went on and went to university, right? 78 00:04:24,880 --> 00:04:26,750 Actually studying it properly. 79 00:04:26,800 --> 00:04:31,709 >> Yes. Uh I I studied mathematics, applied mathematics at university. 80 00:04:31,759 --> 00:04:34,559 I I went there quite quite early. 81 00:04:34,560 --> 00:04:37,599 Um and so I I graduated early as well. 82 00:04:37,600 --> 00:04:42,478 Like I thought for a long time um that I would actually not make it and that I would 83 00:04:42,479 --> 00:04:49,519 drop out. I had like small companies and small consulting business like uh like while 84 00:04:49,520 --> 00:04:52,478 I was studying uh I was like working for banks. 85 00:04:52,479 --> 00:04:58,350 I was working for I was like um very interested in supply chain and applied mathematics 86 00:04:58,400 --> 00:05:01,758 problems and I sort of like selling that and learning a lot through that. 87 00:05:01,759 --> 00:05:04,879 Eventually ended up in the startup world in Belgium. 88 00:05:04,880 --> 00:05:10,239 I did that for for a little while and then moved to London to work uh initially at Google 89 00:05:10,240 --> 00:05:15,679 and then Deep Mind and then now you know moved uh to be here at OpenAI like this California. 90 00:05:15,680 --> 00:05:16,879 I love the California weather. 91 00:05:16,880 --> 00:05:18,159 We can talk about that. 92 00:05:18,160 --> 00:05:19,519 Uh it's been very good. 93 00:05:19,520 --> 00:05:23,599 >> Right after university you started you you've founded a startup right? 94 00:05:23,600 --> 00:05:27,038 You had the startup bug in you or the entrepreneur entrepreneurial bug. 95 00:05:27,039 --> 00:05:33,279 >> Yeah. So this this startup was all about uh pharmaceutical uh supply chain um looking 96 00:05:33,280 --> 00:05:38,319 at the supply chain for uh clinical trials and like try to optimize and decide like hey 97 00:05:38,320 --> 00:05:41,679 you know should you produce more medicine where should you send it where should dispatch 98 00:05:41,680 --> 00:05:46,910 it like how do you avoid waste and through that making clinical trials more efficient 99 00:05:46,960 --> 00:05:53,760 and this was using traditional like nonML techniques um like traditional like more optimization 100 00:05:54,160 --> 00:05:59,599 solving Monte Carlo simulations these kinds things stochastic multi-stage optimization 101 00:05:59,600 --> 00:06:05,119 problem really and we also applied it on steel industry and we applied it to electrical 102 00:06:05,120 --> 00:06:09,839 grid as well in Europe um it's like anything that sort of had the shape of like an optimization 103 00:06:09,840 --> 00:06:13,758 problem we sort of like get interested in and you know to this day like this this this 104 00:06:13,759 --> 00:06:17,839 company still exists and I think they do some of the most interesting work uh still but 105 00:06:17,840 --> 00:06:21,918 it's changing a lot uh you know with with modern AI for sure >> but it's interesting 106 00:06:21,919 --> 00:06:25,599 because you kind of said like oh yeah that wasn't ML it was just the traditional stuff 107 00:06:25,600 --> 00:06:29,439 and then you go into like Monte Carlo simulation and optimization and this algorithm 108 00:06:29,440 --> 00:06:33,679 I get a sense that you kind of just went deep right that it was like okay like here's 109 00:06:33,680 --> 00:06:37,839 a problem space like how can I use mathematics stuff that I learned stuff that I didn't 110 00:06:37,840 --> 00:06:41,839 learn to just go deeper and deeper do I sense that correctly >> yeah that's that's why 111 00:06:41,840 --> 00:06:47,439 I was obsessed with applied mathematics is just really this idea of you have theoretical 112 00:06:47,440 --> 00:06:51,758 mathematics or you have theoretical science and physics and like there you just you you 113 00:06:51,759 --> 00:06:56,478 do it because there's something to be discovered and something beautiful about it and 114 00:06:56,479 --> 00:07:00,079 it's all about patterns and pushing the frontier but you don't necessarily always know 115 00:07:00,080 --> 00:07:03,999 like how you're going to apply it and then there was like the real world right this like 116 00:07:04,000 --> 00:07:08,159 you know there's all these cool problems that just lie around and I was like very interested 117 00:07:08,160 --> 00:07:12,399 in seeing like you know how can I make the world better and so like how do I apply like 118 00:07:12,400 --> 00:07:16,799 you know sophisticated mathematics you know to just optimize the world around me and 119 00:07:16,800 --> 00:07:20,318 that was like a lot of the thesis behind that startup >> yeah and then after a startup 120 00:07:20,319 --> 00:07:25,598 you ended up at Google and first at Google London it was in 2015 and I remember 2015 121 00:07:25,599 --> 00:07:29,359 Google is a really really competitive place to get into like maybe as competitive as 122 00:07:29,360 --> 00:07:33,038 open AI is today in terms of the industry or terms of prestige. 123 00:07:33,039 --> 00:07:36,719 You worked on maps initially and then you moved over to deep mind. 124 00:07:36,720 --> 00:07:41,198 Can you talk a little bit what what you worked on and and then why did you move on from 125 00:07:41,199 --> 00:07:45,119 a already really interesting space that you clearly loved you know like optimization 126 00:07:45,120 --> 00:07:46,559 logistics and all these things? 127 00:07:46,560 --> 00:07:49,359 >> Yes. I I didn't I didn't start on Google Maps. 128 00:07:49,360 --> 00:07:55,230 I started on a project that was meant to make the web faster. 129 00:07:55,280 --> 00:08:00,159 uh and make to me to uh make websites faster especially on mobile. 130 00:08:00,160 --> 00:08:05,119 At the time you know Google was kind of so like seeing the transition from desktop to 131 00:08:05,120 --> 00:08:09,198 mobile and like more and more traffic going to like you know mobile phones and so like 132 00:08:09,199 --> 00:08:10,159 wanted to get ahead of that. 133 00:08:10,160 --> 00:08:13,918 So funded like a number of a number of initiatives and projects. 134 00:08:13,919 --> 00:08:15,839 Um I was working on one of them. 135 00:08:15,840 --> 00:08:21,839 This was like really really fun because it was a small group um within actually the ads 136 00:08:21,840 --> 00:08:26,478 organization. It was meant to sort of like you know offset the the loss uh for the ad 137 00:08:26,479 --> 00:08:31,439 revenue loss because of the shift of traffic to mobile and worked on it for roughly two 138 00:08:31,440 --> 00:08:36,639 years. Um and then it was cancelled and although it was like the most fun I've had on, 139 00:08:36,640 --> 00:08:38,958 you know, solving hard technical uh challenges. 140 00:08:38,959 --> 00:08:42,958 I learned a lot from not having product market fit, not having the right users, 141 00:08:42,959 --> 00:08:44,639 not having the right feedback loop, 142 00:08:44,640 --> 00:08:48,398 not trusting your product manager when they say the project is going well when in fact 143 00:08:48,399 --> 00:08:49,439 it's not going well at all. 144 00:08:49,440 --> 00:08:53,679 And then you know one day it's just like this VP flew in uh from California and then 145 00:08:53,680 --> 00:08:57,439 it was just like oh yeah it's like you know we're canceling this project um you know 146 00:08:57,440 --> 00:09:01,759 unfortunately you only have you know hundreds of users and this is clearly not Google 147 00:09:01,760 --> 00:09:07,359 scale and then uh it's unbelievable but people were surprised um and I think there's 148 00:09:07,360 --> 00:09:12,159 a lesson there um that I that I carry with me of course is you know just always always 149 00:09:12,160 --> 00:09:17,359 question always go to like always you know deeply think about the impact that you're 150 00:09:17,360 --> 00:09:20,909 having but also like the importance of the overall project that you're contributing. 151 00:09:20,959 --> 00:09:23,039 And then I moved into Google Maps. 152 00:09:23,040 --> 00:09:24,319 Google Maps was super fun. 153 00:09:24,320 --> 00:09:25,439 Worked on reviews. 154 00:09:25,440 --> 00:09:28,398 And then after roughly a year, I couldn't ignore like Deep Mind. 155 00:09:28,399 --> 00:09:31,518 It was just it was this special place. 156 00:09:31,519 --> 00:09:33,789 Uh headquartered in London. 157 00:09:33,839 --> 00:09:35,199 So many great things were happening. 158 00:09:35,200 --> 00:09:36,958 This was like really the early days, you know, 159 00:09:36,959 --> 00:09:42,639 with rumblings of um things like AlphaGo and they just seemed to be doing extraordinary 160 00:09:42,640 --> 00:09:43,518 things and you know, 161 00:09:43,519 --> 00:09:46,879 just really tackling the very very hardest problems that you can tackle. 162 00:09:46,880 --> 00:09:51,199 And like with my background I was obviously drawn to that started there like I worked 163 00:09:51,200 --> 00:09:54,879 on uh a lot of like the research infrastructure research tooling. 164 00:09:54,880 --> 00:09:58,479 This is a theme that I carried on for almost a decade and it's like this is very much 165 00:09:58,480 --> 00:10:04,319 also the like how I approach things is how can I build tooling and products that help 166 00:10:04,320 --> 00:10:07,759 make others more efficient and bring a lot of utility to them. 167 00:10:07,760 --> 00:10:13,199 Initially I was doing this for research and then like over time you know I got like into 168 00:10:13,200 --> 00:10:17,199 thinking about things in a much more like more general and general and general way you 169 00:10:17,200 --> 00:10:19,759 know eventually like you know ending up where where I am now. 170 00:10:19,760 --> 00:10:24,799 >> Yeah. And a fun story that you recently shared on X as well is how you were part of 171 00:10:24,800 --> 00:10:30,159 the team that built this internal Google bot that was you know if you want to say similar 172 00:10:30,160 --> 00:10:35,439 to Chad GPT but a year before Chad GPT can you can you talk about that? 173 00:10:35,440 --> 00:10:37,119 That's a that that is a new story. 174 00:10:37,120 --> 00:10:38,398 I haven't heard it before. 175 00:10:38,399 --> 00:10:39,518 >> This was part of deep mind. 176 00:10:39,519 --> 00:10:40,879 There were like multiple efforts as well. 177 00:10:40,880 --> 00:10:43,039 There was like brain as well that was separate at the time. 178 00:10:43,040 --> 00:10:45,599 They had their own efforts on large language models, 179 00:10:45,600 --> 00:10:47,950 but it was definitely something that was being explored. 180 00:10:48,000 --> 00:10:50,479 It was not the main thrust of of deep mind. 181 00:10:50,480 --> 00:10:55,599 Deep mind was like very much worried um and and busy like thinking about grand challenges 182 00:10:55,600 --> 00:11:00,909 and games and you know thinking about RL not in the language sense. 183 00:11:00,959 --> 00:11:06,559 And so there was like this group um that was pushing on large language models and you 184 00:11:06,560 --> 00:11:11,119 know thinking about you know think what what if what if large text corpuses are everything. 185 00:11:11,120 --> 00:11:17,119 What if uh you just pushed language to its maximum and you just scaled language models 186 00:11:17,120 --> 00:11:20,239 like you know would that be enough to get to general intelligence? 187 00:11:20,240 --> 00:11:25,679 That was like a hot debate at the time and then one one group decided to just really 188 00:11:25,680 --> 00:11:30,398 push on that and then it it felt really natural like you know as I was building tooling 189 00:11:30,399 --> 00:11:33,679 you know with others for for research is like you know obviously you're like hey you 190 00:11:33,680 --> 00:11:37,199 know what can we do with this model like how do we present it you know to the researcher 191 00:11:37,200 --> 00:11:41,119 like how can they sort of like you know debug the inputs outputs and eventually you sort 192 00:11:41,120 --> 00:11:43,599 of like end up with you know like a chat system. 193 00:11:43,600 --> 00:11:44,879 So we built that internally. 194 00:11:44,880 --> 00:11:45,999 We had a lot of fun. 195 00:11:46,000 --> 00:11:49,439 Uh initially the models were like you know kind of like almost like a little bit absurd 196 00:11:49,440 --> 00:11:51,310 like you know not very coherent. 197 00:11:51,360 --> 00:11:56,719 Uh not super useful but it was a lot of fun um to sort of like tinker with them that 198 00:11:56,720 --> 00:11:59,759 caught up like you know like wildfires like you know this application is just sort of 199 00:11:59,760 --> 00:12:04,958 like you know everyone uh was kind of like u sharing little conversations within deep 200 00:12:04,959 --> 00:12:10,398 mind. It felt more than like a research project or like a research a project for researchers. 201 00:12:10,399 --> 00:12:14,159 And so then there was this desire over time to like launch it as an external product. 202 00:12:14,160 --> 00:12:16,479 But Deepine was just like not set up, you know, 203 00:12:16,480 --> 00:12:19,838 there was like the the right way to launch products at Google. 204 00:12:19,839 --> 00:12:22,638 There's like, you know, the whole machinery of like, you know, how you do that. 205 00:12:22,639 --> 00:12:25,599 Um, you know, the whole like blessed production stack. 206 00:12:25,600 --> 00:12:29,439 You obviously very very optimized over the years to do things well. 207 00:12:29,440 --> 00:12:33,119 Um, but also very very hard as an environment to truly innovate. 208 00:12:33,120 --> 00:12:37,759 And then I I I wanted to ask what made you you know look around or or or maybe even consider 209 00:12:37,760 --> 00:12:42,239 open AI but I feel you partially answered this question just just putting myself back 210 00:12:42,240 --> 00:12:47,119 into your shoes like you're you know if it's it's 2024 or 2023 you're inside of Google 211 00:12:47,120 --> 00:12:51,439 who are publishing amazing papers doing really good research you're doing super fun stuff 212 00:12:51,440 --> 00:12:55,679 right that pushing the limits of what what's been done before it's inside a company where 213 00:12:55,680 --> 00:12:59,919 you already moved you know for people who are feeling kind of comfortable or good about 214 00:12:59,920 --> 00:13:04,078 where they are right now which I imagine you must have been like what made you still 215 00:13:04,079 --> 00:13:07,278 explore all right like what else might be there? 216 00:13:07,279 --> 00:13:12,990 Yeah, I was I was very comfortable uh at my it's it's a good place, 217 00:13:13,040 --> 00:13:18,719 but really I had a I had a desire to, you know, meet great people, 218 00:13:18,720 --> 00:13:22,559 but also join a mission that I truly believed in and that, you know, 219 00:13:22,560 --> 00:13:27,359 I felt like the people were true to the mission and cared deeply about impacting the 220 00:13:27,360 --> 00:13:32,479 world in in a very in a deeply positive way, but also in a direct way, 221 00:13:32,480 --> 00:13:34,479 not not being like, oh yeah, it's just like, you know, 222 00:13:34,480 --> 00:13:37,919 we just do this work over here and then it's like it's the job of someone else to figure 223 00:13:37,920 --> 00:13:40,479 out you know how to how to make this useful. 224 00:13:40,480 --> 00:13:43,999 It's like I wanted to join a group where you know like all the parameters were sort of 225 00:13:44,000 --> 00:13:48,078 like considered together where you know research and product were like really co-designing. 226 00:13:48,079 --> 00:13:49,679 Open was just like crushing it. 227 00:13:49,680 --> 00:13:52,879 Uh I thought chatbt was like you know taking off. 228 00:13:52,880 --> 00:13:57,198 I was like I was uh I met a couple people from OpenAI and then I was like wait what you 229 00:13:57,199 --> 00:14:04,159 know you only have like 20 people working on Chad like that is that is an insanely small 230 00:14:04,160 --> 00:14:07,759 number that that must be like extremely empowering like you know how does that work? 231 00:14:07,760 --> 00:14:13,518 How do you manage to maintain you know a product with that level of scale um and with 232 00:14:13,519 --> 00:14:17,599 that level of autonomy with you know only only 20 engineers and then you know as I kind 233 00:14:17,600 --> 00:14:21,518 of dug and dug and dug and it's like it was just a an amazing group of people amazing 234 00:14:21,519 --> 00:14:26,719 mission you know super talented super driven and like it was it was like drew me in and 235 00:14:26,720 --> 00:14:32,319 then I joined pre-ereasoning uh efforts immediately like typical openi fashions like 236 00:14:32,320 --> 00:14:36,078 I joined it was like oh yeah you know like there's this thing going on like you know 237 00:14:36,079 --> 00:14:39,439 we're going to launch reasoning models like you know it's like some new paradigm and 238 00:14:39,440 --> 00:14:43,439 then you know start sprinting on that and like you know like a month later like the company 239 00:14:43,440 --> 00:14:49,039 launched 01 uh 01 01 preview and that was exhilarating to be part of I wanted to be part 240 00:14:49,040 --> 00:14:55,119 of like a place that moves fast cares about impact would be in tune with the world and 241 00:14:55,120 --> 00:14:59,759 you know just really listen um and sort of like that's also to me like you know what 242 00:14:59,760 --> 00:15:03,999 I've carried with me like when when building codecs when building products is like having 243 00:15:04,000 --> 00:15:09,518 a community listen to the community just really focus on like a really intense feedback 244 00:15:09,519 --> 00:15:14,559 loop uh and then building something that is just like you know you just really want to 245 00:15:14,560 --> 00:15:18,398 care about it and like you know care about the utility of it that it provides to the 246 00:15:18,399 --> 00:15:21,599 world >> and then of course you started to work pretty quickly on codeex. 247 00:15:21,600 --> 00:15:23,469 So you joined in 2024. 248 00:15:23,519 --> 00:15:29,599 Can you take us back what the thinking back there when you joined was about AI or LMS 249 00:15:29,600 --> 00:15:35,119 and and code? I I know there was this ASWE um effort back then. 250 00:15:35,120 --> 00:15:38,719 The we talked about it in the deep dive as well that we did in the pragmatic engineer 251 00:15:38,720 --> 00:15:41,119 the autonomous software engineer >> AS3. 252 00:15:41,120 --> 00:15:44,239 Yeah, that's what it's it was pronounced internally. 253 00:15:44,240 --> 00:15:49,119 Um we we don't we don't >> have an A3 effort anymore like you know it's it's it's codeex. 254 00:15:49,120 --> 00:15:54,479 Um but really for me it was I I joined I started building infrastructure for research 255 00:15:54,480 --> 00:16:01,679 like my a lot of what I did before was large scale um data storage analysis 256 00:16:01,680 --> 00:16:04,078 and then tools to understand training runs. 257 00:16:04,079 --> 00:16:07,278 I I I did a lot of different things over my years. 258 00:16:07,279 --> 00:16:13,439 Um, but it was always about building for others and making them faster and just really 259 00:16:13,440 --> 00:16:14,479 caring about, you know, 260 00:16:14,480 --> 00:16:18,879 fundamentally doing that well and then through tooling and infrastructure making new 261 00:16:18,880 --> 00:16:19,599 things possible. 262 00:16:19,600 --> 00:16:21,439 And so when I joined OpenAI, 263 00:16:21,440 --> 00:16:26,479 I was like with the same idea and then with the re with with one preview and like you 264 00:16:26,480 --> 00:16:28,398 know some some of some some of the later models, 265 00:16:28,399 --> 00:16:33,550 it was very clear that we had to use the models themselves to help us go faster. 266 00:16:33,600 --> 00:16:37,758 And so I just really got obsessed with this idea of what were the limitations, 267 00:16:37,759 --> 00:16:41,440 how were we going to use those models for research itself. 268 00:16:41,680 --> 00:16:44,879 So got together with um other folks in research. 269 00:16:44,880 --> 00:16:46,398 We started training models. 270 00:16:46,399 --> 00:16:48,239 We started building little agents. 271 00:16:48,240 --> 00:16:51,518 Those were truly the precursor to to to Codex. 272 00:16:51,519 --> 00:16:57,039 And like this was like we were training internal models to be very uh proficient on the 273 00:16:57,040 --> 00:17:03,278 Python codebase of OpenAI and then uh very proficient with you know having like good 274 00:17:03,279 --> 00:17:07,999 good taste in architecture, good taste in you know like code style. 275 00:17:08,000 --> 00:17:12,318 It was like Python only and then the idea was like you know we would sort of like use 276 00:17:12,319 --> 00:17:16,879 that to build infrastructure very quickly and you know help researchers code faster as 277 00:17:16,880 --> 00:17:21,198 well. uh and then you know and then we would move faster and then over time when you 278 00:17:21,199 --> 00:17:25,359 just kind like push that and simplify it to its core you're making a lot of uh you know 279 00:17:25,360 --> 00:17:29,759 we found like you could make a lot of progress very quickly and then learn very quickly 280 00:17:29,760 --> 00:17:36,190 and then Greg and Sam are you know people with they're immensely supportive and also 281 00:17:36,240 --> 00:17:41,359 uh Greg was very adamant that you know we would we would not just focus on ourselves 282 00:17:41,360 --> 00:17:46,798 but we would also focus on benefiting uh the world and so he just sort of encouraged 283 00:17:46,799 --> 00:17:51,199 that we would be thinking about this not just as a tool for OpenAI itself but also as 284 00:17:51,200 --> 00:17:56,399 something that we would actually make into a product and this is when uh we merged this 285 00:17:56,400 --> 00:18:01,519 research effort with this AS3 effort um and we started building one thing and then that 286 00:18:01,520 --> 00:18:05,918 led to a sprint which was like the initial cloud codecs that we launched which didn't 287 00:18:05,919 --> 00:18:10,639 really have PMF because it was like a little bit too high friction and then we also launched 288 00:18:10,640 --> 00:18:16,479 the codeex CLI and we continued to push but it was always this idea of >> hey how do 289 00:18:16,480 --> 00:18:18,319 we get models to really help here. 290 00:18:18,320 --> 00:18:21,839 You mentioned that first you started to build this model to train on the Python code 291 00:18:21,840 --> 00:18:26,639 and and actually help build in for better but then you made this interesting decision 292 00:18:26,640 --> 00:18:33,199 where for codeex you built it in Rust and at the time the the model was not on distribution 293 00:18:33,200 --> 00:18:38,319 for Rust right it wasn't as good as in Rust and it was in Python or Typescript why did 294 00:18:38,320 --> 00:18:42,399 you make that kind of a decision was was it kind of like did you expect that it'll catch 295 00:18:42,400 --> 00:18:46,910 up or or you figure that performance is more important or because it was very counterintuitive 296 00:18:46,960 --> 00:18:50,558 most of the other harnesses built were actually not built in Rust. 297 00:18:50,559 --> 00:18:53,759 They were built on distribution on TypeScript or Python or something else. 298 00:18:53,760 --> 00:19:01,199 >> Yes. From first principles like we very early on we were thinking about 299 00:19:01,200 --> 00:19:04,910 the product interface and the agent as different things. 300 00:19:04,960 --> 00:19:10,160 So it was very important um to build 301 00:19:10,720 --> 00:19:17,519 the core of the agent in a way that was robust, that was secure as well, 302 00:19:17,520 --> 00:19:24,749 that was um you know engineered for efficiency and scale and having 303 00:19:24,799 --> 00:19:30,319 worked through projects over the years that go from hey this is a fun thing to like, 304 00:19:30,320 --> 00:19:36,239 hey, we need to scale this to the scale of like the largest data center um is the ear 305 00:19:36,240 --> 00:19:41,759 decisions early on are like really turn out to be quite important as long as you don't 306 00:19:41,760 --> 00:19:45,839 sacrifice too much of the velocity and so it's like it's a it's a trade-off but we had 307 00:19:45,840 --> 00:19:51,279 very prolific and amazing Rust developers our internal models were not bad at Rust um 308 00:19:51,280 --> 00:19:55,678 and then you get a lot of uh validation as well at compile time it's like you know statically 309 00:19:55,679 --> 00:20:00,319 verified and all these things and that is great for agents too so turns out you know 310 00:20:00,320 --> 00:20:05,678 it was quite clear that you know Rust as a language would actually be quite good for 311 00:20:05,679 --> 00:20:09,519 agents fairly quickly if we decided to put some effort into it. 312 00:20:09,520 --> 00:20:13,759 But primarily we were focused on correctness and we were focused on efficiency as well. 313 00:20:13,760 --> 00:20:18,319 >> Interesting. So you're saying you know it's it's worth in your case it was worth thinking 314 00:20:18,320 --> 00:20:21,759 ahead of where you want this thing to be and for example thing like a language choice. 315 00:20:21,760 --> 00:20:25,839 Obviously with agents you can rewrite a bunch of stuff and easier than in the past, 316 00:20:25,840 --> 00:20:31,439 but it's still like you can say save yourself reworking by putting in the right I guess 317 00:20:31,440 --> 00:20:35,999 scaffolding or or or well the you know the the baseline of of what you're building on 318 00:20:36,000 --> 00:20:40,158 right >> I think we could have been successful if we had written it in Typescript or 319 00:20:40,159 --> 00:20:43,439 you know maybe even Python and then it would have fine and then you know we would have 320 00:20:43,440 --> 00:20:49,918 rewritten it at some point but having a very clean separation between the agent itself 321 00:20:49,919 --> 00:20:53,070 which can exist irrespective of the product. 322 00:20:53,120 --> 00:20:58,158 Um it was a very important principle and if you write everything in the same codebase 323 00:20:58,159 --> 00:21:02,959 in the same language it's like inevitably you're going to be a little bit sloppy and 324 00:21:02,960 --> 00:21:06,719 um you're going to intertwine things more than you should and then it's going to prevent 325 00:21:06,720 --> 00:21:08,830 further innovation after that. 326 00:21:08,880 --> 00:21:12,558 And so that was that was very important like the rust boundary in a sense like was very 327 00:21:12,559 --> 00:21:13,678 useful for that. 328 00:21:13,679 --> 00:21:18,670 One interesting decision that you made which is unique across all of the major labs is 329 00:21:18,720 --> 00:21:23,279 having this built-in open source right the CLI is open source the SDK and the app server 330 00:21:23,280 --> 00:21:27,918 are all open source when and why did you decide that it's not a given especially you 331 00:21:27,919 --> 00:21:32,479 know there used to be jokes about open AI having things closed but this this is actually 332 00:21:32,480 --> 00:21:37,359 the opposite where like this is open whereas like some competitors would would ship closed 333 00:21:37,360 --> 00:21:41,038 source harnesses which again I I I think it's very easy to understand why you would want 334 00:21:41,039 --> 00:21:45,759 something closed source why did you want it open source There was something really cool 335 00:21:45,760 --> 00:21:51,759 about the idea of having the code open source because fundamentally what you're building 336 00:21:51,760 --> 00:21:54,158 is you're building a coding agent. 337 00:21:54,159 --> 00:21:59,439 And so we were sort of like thinking about well if you have that you know you're obviously 338 00:21:59,440 --> 00:22:04,879 going to point it at itself and you know maybe you know you can build a community of 339 00:22:04,880 --> 00:22:08,558 you know contributors that use it to improve it and then you know you can learn a lot 340 00:22:08,559 --> 00:22:11,759 from that. Also, it felt at the time is like, you know, 341 00:22:11,760 --> 00:22:15,038 very clear to us that if we were going to be successful, 342 00:22:15,039 --> 00:22:20,079 open source itself would change and the role of code itself would change and so being 343 00:22:20,080 --> 00:22:23,279 part of that community seemed important instead of divorced from it. 344 00:22:23,280 --> 00:22:23,918 I think, you know, 345 00:22:23,919 --> 00:22:28,509 it's it's it's hard to solve problems if you don't sort of like witness them yourself. 346 00:22:28,559 --> 00:22:32,639 Uh, and then the the other thing was just it still feels like early, 347 00:22:32,640 --> 00:22:34,639 but it was very early at the time. 348 00:22:34,640 --> 00:22:39,678 Um, it felt like we would have some ideas for how to solve things well. 349 00:22:39,679 --> 00:22:42,479 Um, and we were co-designing these, you know, 350 00:22:42,480 --> 00:22:47,839 with with with with the training and and and and the research and it's it's all about 351 00:22:47,840 --> 00:22:52,350 expressing like the capabilities of model in like the most flexible and the best way, 352 00:22:52,400 --> 00:22:58,079 but also we didn't have all the answers and sort of being very open about, hey, 353 00:22:58,080 --> 00:23:00,959 this is what a good harness looks like. 354 00:23:00,960 --> 00:23:02,399 This is how we think about it. 355 00:23:02,400 --> 00:23:06,399 We did like a couple of like very technical like deep dives and blog posts and we talked 356 00:23:06,400 --> 00:23:11,279 about it a lot and we thought you know hey it's just like the world is vast out there's 357 00:23:11,280 --> 00:23:14,639 like you know crazy smart people it's like you know we're we're going to get inspired 358 00:23:14,640 --> 00:23:21,678 by other open source project as well and so let's just make this a level playing field 359 00:23:21,679 --> 00:23:27,119 and sort of like encourage a lot of tinkering um and exploration at this stage. 360 00:23:27,120 --> 00:23:32,239 Now this has been now you know like a a year later a year and a half later which is a 361 00:23:32,240 --> 00:23:38,030 very long time in right now in this AI time frame but looking back or taking the experience 362 00:23:38,080 --> 00:23:41,918 what are the benefits you've seen the kind of engineering benefits the engineering team's 363 00:23:41,919 --> 00:23:46,479 benefits from being open source and just honestly what are things that are kind of hard 364 00:23:46,480 --> 00:23:50,879 about being open source right like there must be downsides like just try trying to get 365 00:23:50,880 --> 00:23:55,199 an honest take on both sides >> yeah there there there are definitely downsides it it 366 00:23:55,200 --> 00:23:57,199 it comes at a cost Right. 367 00:23:57,200 --> 00:24:01,439 Um the the benefits are it's almost something to build in the open. 368 00:24:01,440 --> 00:24:05,918 Uh it's awesome to have like a small a small repo as well. 369 00:24:05,919 --> 00:24:09,199 Like whenever we hire uh someone and they join the Codex team, 370 00:24:09,200 --> 00:24:11,278 it's like they've seen the repo before. 371 00:24:11,279 --> 00:24:12,719 They've they've looked at PRs. 372 00:24:12,720 --> 00:24:14,158 They're like >> onboarding is done. 373 00:24:14,159 --> 00:24:15,278 >> You know, it's it's it's done. 374 00:24:15,279 --> 00:24:18,558 Yeah. It's like in onboarding is just like you use Codex to look at the repo, 375 00:24:18,559 --> 00:24:19,918 you know, with you and you ask them questions, 376 00:24:19,919 --> 00:24:23,999 but it's like it's not it's not a secret issue that you can get productive right away. 377 00:24:24,000 --> 00:24:25,918 We get a lot of good contributions. 378 00:24:25,919 --> 00:24:30,719 Although we get like you know an a tsunami of like random stuff as well. 379 00:24:30,720 --> 00:24:33,519 >> Obviously you and everyone else right open source is changing. 380 00:24:33,520 --> 00:24:35,678 I think this is one of the examples. 381 00:24:35,679 --> 00:24:36,158 >> That's right. 382 00:24:36,159 --> 00:24:40,158 And then to to me it just and and to a lot of the team it just brings a lot of energy 383 00:24:40,159 --> 00:24:45,038 to just be part of the community and like be directly contributing um not just saying 384 00:24:45,039 --> 00:24:48,319 that we care about the community but actually doing things that you know you can see 385 00:24:48,320 --> 00:24:50,239 it's it's costing us effort right. 386 00:24:50,240 --> 00:24:52,430 U we don't have to do it. 387 00:24:52,480 --> 00:24:56,558 the the downsides are, you know, it's it's separate from the rest of our code. 388 00:24:56,559 --> 00:25:00,399 Um so, you know, sometimes we have to draw like artificial boundaries and, 389 00:25:00,400 --> 00:25:02,430 you know, work across multiple repos. 390 00:25:02,480 --> 00:25:06,879 Um when we're working on something particularly exciting, um and uh you know, 391 00:25:06,880 --> 00:25:10,719 we're building it in the open then, you know, at times we find that uh you know, 392 00:25:10,720 --> 00:25:14,079 others copy it, you know, before we have the time to release it. 393 00:25:14,080 --> 00:25:17,359 And it's like it's just a little bit sad. 394 00:25:17,360 --> 00:25:20,479 Um, but also it's like it's part of the game, you know. 395 00:25:20,480 --> 00:25:22,479 It's it's like you're building in the open. 396 00:25:22,480 --> 00:25:22,959 It's like, you know, 397 00:25:22,960 --> 00:25:25,278 that's that's sort of like the contract that you signed is like, you know, 398 00:25:25,279 --> 00:25:26,239 you can copy it. 399 00:25:26,240 --> 00:25:28,479 Uh, we have a very permissive license as well, 400 00:25:28,480 --> 00:25:31,999 but it does sting a little bit when you're working on something and you're like, 401 00:25:32,000 --> 00:25:35,759 you know, and then uh the the third thing is just just like everyone else is like, 402 00:25:35,760 --> 00:25:39,278 you know, we are overwhelmed with, you know, random contributions and, you know, 403 00:25:39,279 --> 00:25:41,759 we have to deal with that additional tax. 404 00:25:41,760 --> 00:25:46,399 Um, but then that pushes us to, you know, also like try and solve for it, right? 405 00:25:46,400 --> 00:25:47,359 which I think is good. 406 00:25:47,360 --> 00:25:50,639 >> And on top of the open source, one thing that surprised me about Codeex, 407 00:25:50,640 --> 00:25:55,759 and I didn't even know about it until recently, it's not tied to the OpenAI models, 408 00:25:55,760 --> 00:25:59,278 you can use other models with Codeex, you know, 409 00:25:59,279 --> 00:26:01,678 like putting myself if in a vendor's shoe, 410 00:26:01,679 --> 00:26:05,038 it it might not be very obvious because again, 411 00:26:05,039 --> 00:26:09,678 all the other vendors I look at when they do a a CLI, it's kind of use it with our models. 412 00:26:09,679 --> 00:26:15,759 again what made you decide to be this permissive about you know using or allowing to 413 00:26:15,760 --> 00:26:18,879 use your harness with with other models? 414 00:26:18,880 --> 00:26:23,200 >> It it felt quite natural if if you are 415 00:26:24,000 --> 00:26:31,038 part of this community and building an excellent coding harness is like why would you 416 00:26:31,039 --> 00:26:32,910 couple it to your model? 417 00:26:32,960 --> 00:26:36,558 That that that felt like sort of like quite disappointing to make that decision. 418 00:26:36,559 --> 00:26:38,479 So it didn't it didn't feel right. 419 00:26:38,480 --> 00:26:42,959 Um and in general it's like I think you know it's like I kind of try to make decisions 420 00:26:42,960 --> 00:26:47,599 that I'm like yes you know just like I can just sort of like explain it you know it is 421 00:26:47,600 --> 00:26:51,599 correct. It's the same reasoning with you know it is open source in the first place. 422 00:26:51,600 --> 00:26:57,119 It would have been trivial for anyone to fork it and then add support for another thing. 423 00:26:57,120 --> 00:27:00,719 But then but then you're just encouraging people to just like you know go and use that 424 00:27:00,720 --> 00:27:05,439 fork and then now suddenly you have overhead and the only reason you have a fork is because 425 00:27:05,440 --> 00:27:09,918 you know you wanted to change like 10 lines of code to add support for like another model 426 00:27:09,919 --> 00:27:11,759 provider that feels very silly. 427 00:27:11,760 --> 00:27:14,670 So like you know why not just support it in the first place. 428 00:27:14,720 --> 00:27:19,950 The other thing is we benefit a lot from like being able to just give optionality. 429 00:27:20,000 --> 00:27:22,558 So, you know, it's like maybe today, you know, 430 00:27:22,559 --> 00:27:26,639 you you you love using OpenAI models um and you know, 431 00:27:26,640 --> 00:27:28,719 you're super productive with them, 432 00:27:28,720 --> 00:27:32,079 but like tomorrow there's a new model that comes out, you want to try that. 433 00:27:32,080 --> 00:27:37,038 Why force you to go and completely change your setup just to try a new model and then 434 00:27:37,039 --> 00:27:39,999 we benefit from the feedback that we didn't get uh which is like, you know, 435 00:27:40,000 --> 00:27:41,759 maybe there's something that you liked about that model. 436 00:27:41,760 --> 00:27:43,278 Maybe it actually didn't work well. 437 00:27:43,279 --> 00:27:49,278 But it's sort of like being nice to our users and to the community is like you know feels 438 00:27:49,279 --> 00:27:51,199 like the right thing to do here. 439 00:27:51,200 --> 00:27:55,678 Um and then you know we also you we also try like other models right so and you know 440 00:27:55,679 --> 00:28:00,558 we try them in the same harness and you know it's just all all good and then this is 441 00:28:00,559 --> 00:28:06,640 all also often like um this optionality is very important to companies that we work with. 442 00:28:07,600 --> 00:28:10,079 This is something that, you know, we we absolutely lean into. 443 00:28:10,080 --> 00:28:12,558 >> This last point, I think, you know, as any serious company, 444 00:28:12,559 --> 00:28:16,830 you want to have optionality and you want to use a tool that gives you that optionality. 445 00:28:16,880 --> 00:28:20,639 >> But I kind of appreciate it cuz I feels to me like it's kind of honest like look like 446 00:28:20,640 --> 00:28:25,599 it forces the whole company to be to compete the best in everywhere in the model layer 447 00:28:25,600 --> 00:28:29,519 and the harness layer with open source with with choosable models and it kind of like 448 00:28:29,520 --> 00:28:32,158 doesn't doesn't allow you to like kick back and say like, "All right, we're done. 449 00:28:32,159 --> 00:28:34,798 We we can we can we can hang back for a little bit for now." 450 00:28:34,799 --> 00:28:38,719 >> Yeah. I want us to win users by having, you know, the best models, 451 00:28:38,720 --> 00:28:41,678 the most efficient models, the best product, and then, you know, 452 00:28:41,679 --> 00:28:44,319 if we do all of these things, it's like we're going to have a good time. 453 00:28:44,320 --> 00:28:47,599 If we sort of like force you to use the product because, you know, this one thing, 454 00:28:47,600 --> 00:28:50,719 it's just like then I don't think that will attract, you know, 455 00:28:50,720 --> 00:28:53,359 the the very best people to work on this product either. 456 00:28:53,360 --> 00:28:56,239 And like, you know, it's like we we're doing our best work here. 457 00:28:56,240 --> 00:28:58,349 We care a lot about the experience. 458 00:28:58,399 --> 00:29:00,158 It should feel delightful, you know, 459 00:29:00,159 --> 00:29:04,430 like it doesn't irrespect of the model that powers it, it should feel delightful. 460 00:29:04,480 --> 00:29:07,839 I love the idea of winning based on merit, not based on lockin. 461 00:29:07,840 --> 00:29:10,719 And this is a perfect time to mention our season sponsor, Entire, 462 00:29:10,720 --> 00:29:12,719 who also play by the same rules. 463 00:29:12,720 --> 00:29:17,150 Like it or not, Git is becoming a bottleneck for modern agent heavy software development. 464 00:29:17,200 --> 00:29:19,038 Devs are creating more code with agents. 465 00:29:19,039 --> 00:29:20,798 These agents are pushing more code. 466 00:29:20,799 --> 00:29:22,719 Many devs are running more parallel agents. 467 00:29:22,720 --> 00:29:24,639 These are pushing even more code. 468 00:29:24,640 --> 00:29:28,430 GitHub is clearly struggling to keep up and has frequent outages. 469 00:29:28,480 --> 00:29:29,599 So what's the solution? 470 00:29:29,600 --> 00:29:35,199 Entire was founded by GitHub's last coza and he rebuilt git hosting for the agentic era 471 00:29:35,200 --> 00:29:39,599 from scratch. Entire was built to be very fast and to have your repos regionally close 472 00:29:39,600 --> 00:29:44,349 to you to reduce latency allowing for fleets of agents to push in parallel. 473 00:29:44,399 --> 00:29:45,759 Some numbers they published. 474 00:29:45,760 --> 00:29:48,879 Entire can handle 418 pushes per second. 475 00:29:48,880 --> 00:29:52,798 That's up to 89 times faster than every competitor on the market. 476 00:29:52,799 --> 00:29:53,918 When GitHub is down, 477 00:29:53,919 --> 00:29:57,519 you can still keep working and you don't even need to migrate from GitHub. 478 00:29:57,520 --> 00:30:00,558 You just sign up to entire and the platform mirrors your repo. 479 00:30:00,559 --> 00:30:01,918 And one more neat thing, 480 00:30:01,919 --> 00:30:06,509 have you ever wondered what prompt resulted in this specific code being generated? 481 00:30:06,559 --> 00:30:10,239 I find that the prompt and conversation with the agent carries more information than 482 00:30:10,240 --> 00:30:12,158 the PR itself, at least for me. 483 00:30:12,159 --> 00:30:16,558 Entire captures all the prompt history with your agent right in the repo easy to check 484 00:30:16,559 --> 00:30:19,678 back and has a pretty innovative UI to show all of this. 485 00:30:19,679 --> 00:30:22,719 If you're looking for Git hosting that works even when GitHub is down, 486 00:30:22,720 --> 00:30:25,999 head to entire.io/pragmatic, io/pragmatic, install the CLI, 487 00:30:26,000 --> 00:30:27,759 and mirror your repo with a click. 488 00:30:27,760 --> 00:30:29,119 I've already done it. 489 00:30:29,120 --> 00:30:32,989 Oh, and did I mention that it works with any agent and it's open source? 490 00:30:33,039 --> 00:30:36,190 I'd also like to mention our season sponsor, Anticys. 491 00:30:36,240 --> 00:30:40,430 Tibo talked about how the experience of the software you use should feel delightful. 492 00:30:40,480 --> 00:30:42,830 Delightful includes no annoying bugs. 493 00:30:42,880 --> 00:30:46,719 But when you're using agents to write your code, how do you avoid chipping bugs? 494 00:30:46,720 --> 00:30:50,319 Reviewing every line of code is becoming a challenge with the amount of code that agents 495 00:30:50,320 --> 00:30:54,109 generate, which is why antithesis goes well beyond code review. 496 00:30:54,159 --> 00:30:57,710 Antithesis runs your whole system in a hostile simulation. 497 00:30:57,760 --> 00:31:01,278 This simulation includes both targeted testing and fuss testing. 498 00:31:01,279 --> 00:31:05,359 By running this simulation, it finds every bug before your users do. 499 00:31:05,360 --> 00:31:09,439 And because the simulation is fully deterministic, it doesn't only find bugs, 500 00:31:09,440 --> 00:31:11,918 it gives you a perfect reproduction of every issue, 501 00:31:11,919 --> 00:31:14,430 which makes it much easier to fix issues. 502 00:31:14,480 --> 00:31:18,398 The first thing I thought when I heard about antithesis is that automated bug discovery 503 00:31:18,399 --> 00:31:21,759 and fully deterministic testing sounds like science fiction, 504 00:31:21,760 --> 00:31:24,719 but it's actually hardcore engineering under the hood. 505 00:31:24,720 --> 00:31:26,158 Jane Street, Fly.io, 506 00:31:26,159 --> 00:31:30,319 and the Etscd community ship agent written code with full confidence because they know 507 00:31:30,320 --> 00:31:32,430 it's been verified by antithesis. 508 00:31:32,480 --> 00:31:35,759 To see more case studies and details, head to antithesis.com/pragmatic. 509 00:31:36,880 --> 00:31:41,950 And with this, let's get back to Tibo and why competition between tools is great. 510 00:31:42,000 --> 00:31:45,710 Yeah. And I think as as an engineer like I always see that whenever there's competition 511 00:31:45,760 --> 00:31:50,319 as someone who's using tools it's it's always amazing like I remember like when Microsoft 512 00:31:50,320 --> 00:31:55,678 had with Jet Brains the IDU wars and then there's the clouds battling with each other 513 00:31:55,679 --> 00:31:59,359 with all the features and now of course we have the harnesses we have the models and 514 00:31:59,360 --> 00:32:03,599 as a as a as a user it's great because now we have more more choice they just develop 515 00:32:03,600 --> 00:32:06,239 faster I guess our voice gets heard a bit better. 516 00:32:06,240 --> 00:32:07,999 So it's it's great to hear. 517 00:32:08,000 --> 00:32:09,199 Speaking of the harness, 518 00:32:09,200 --> 00:32:13,359 can you tell me how it works today in the sense of like when I 519 00:32:13,679 --> 00:32:16,239 start a codeex task, 520 00:32:16,720 --> 00:32:19,038 does it run always on my machine? 521 00:32:19,039 --> 00:32:20,910 Does it choose the cloud? 522 00:32:20,960 --> 00:32:22,670 Does it use a sandbox? 523 00:32:22,720 --> 00:32:27,678 And how do I control this or or know this or how much should I know about this as as 524 00:32:27,679 --> 00:32:29,469 an engineer? >> Yes. 525 00:32:29,519 --> 00:32:34,989 So, by default, it runs uh in uh it runs sandboxed. 526 00:32:35,039 --> 00:32:41,599 Um it everything that if there is like an an a command that should run with additional 527 00:32:41,600 --> 00:32:47,599 permissions outside of the sandbox, it will ask uh you as a user for permission, 528 00:32:47,760 --> 00:32:52,639 but everything every tool execution happens within the sandbox by default and it runs 529 00:32:52,640 --> 00:32:56,509 entirely on the machine um your local machine. 530 00:32:56,559 --> 00:33:02,430 And this has been the case for you know more than a year now. 531 00:33:02,480 --> 00:33:08,639 But it is something that is evolving and shifting where like you can select to run this 532 00:33:08,640 --> 00:33:14,558 um in the cloud which then runs in like a managed VM uh where it's the same VM that you 533 00:33:14,559 --> 00:33:20,398 get through chatk work um and you can you can sort of like inspect it but like it runs 534 00:33:20,399 --> 00:33:26,749 in a kata container it's like a secure environment and so everything runs inside of that 535 00:33:26,799 --> 00:33:30,798 VM and it doesn't run on your machine and then the only thing that happens on your machine 536 00:33:30,799 --> 00:33:36,479 is like the the your input and then the streaming back of the output and so that you 537 00:33:36,480 --> 00:33:40,719 know obviously then uh is like much nicer on your CPU and and your machine and you can 538 00:33:40,720 --> 00:33:47,839 scale much much um much more and this is like just a step um it's going to be 539 00:33:47,840 --> 00:33:53,199 much more seamless in the future um to like you know use cloud machines and then you 540 00:33:53,200 --> 00:33:58,190 know maybe have a combination of like partial execution on your laptop partial execution 541 00:33:58,240 --> 00:34:03,599 on on on cloud machines and really The thing that we're thinking about that is very natural 542 00:34:03,600 --> 00:34:08,589 is as models just get better and uh more capable. 543 00:34:08,639 --> 00:34:14,559 They can leverage so much more compute and many more resources than are available on 544 00:34:14,560 --> 00:34:20,319 your local machine and so it would be a constraint at some point to just limit execution 545 00:34:20,320 --> 00:34:21,358 on your local machine. 546 00:34:21,359 --> 00:34:24,479 >> Yeah. One thing that is great about it running locally, 547 00:34:24,480 --> 00:34:27,118 and I think the reason I I love it when it runs locally. 548 00:34:27,119 --> 00:34:30,158 Of course, it's a pain because if I'm doing some work, it's like, you know, 549 00:34:30,159 --> 00:34:32,799 I have several agents, it's it's eating CPU. 550 00:34:32,800 --> 00:34:37,519 If I want to close my laptop, I I I cannot kind of leave it like half open, right? 551 00:34:37,520 --> 00:34:39,918 When I was in one of the offices of an AI company, 552 00:34:39,919 --> 00:34:42,398 I I had it half open and they're like, "Are you running agents?" 553 00:34:42,399 --> 00:34:43,678 I'm like, "Yeah, I have one running." 554 00:34:43,679 --> 00:34:44,719 He's like, "I get it." 555 00:34:44,720 --> 00:34:47,358 >> But the reason the reason I do it because I have my local tools, 556 00:34:47,359 --> 00:34:49,999 I have my local Postgress database. 557 00:34:50,000 --> 00:34:52,398 I have my my this this and that. 558 00:34:52,399 --> 00:34:55,999 How are you thinking about the cloud is amazing but it doesn't have this setup or it's 559 00:34:56,000 --> 00:34:57,439 just a pain to set it up. 560 00:34:57,440 --> 00:35:02,399 are you thinking or or are you experimenting with you know making the these setups and 561 00:35:02,400 --> 00:35:06,719 I'm kind of reminded of a topic that we talked about prea which is cloud development 562 00:35:06,720 --> 00:35:12,399 environments and like 2022 23 they're hot and then we talked about AI more but >> yes 563 00:35:12,400 --> 00:35:18,559 I think outside of large tech companies like cloud dev boxes really never took off because 564 00:35:18,560 --> 00:35:23,358 there's a very big upfront cost uh and then you need to pay like a maintenance cost as 565 00:35:23,359 --> 00:35:27,999 well and you just you know don't benefit from it as like a a solo developer or like a 566 00:35:28,000 --> 00:35:32,399 small team with the level of capabilities that we have in agents now is like the setup 567 00:35:32,400 --> 00:35:33,519 almost is free, right? 568 00:35:33,520 --> 00:35:37,919 So like this the setup cost and this maintenance cost is like if if your agent is capable 569 00:35:37,920 --> 00:35:40,159 of doing it, you know, it should just do it for you. 570 00:35:40,160 --> 00:35:42,078 So for example, if you're saying like hey, you know, 571 00:35:42,079 --> 00:35:49,118 I have like I have my local um SQLite or I have a local server and MCPS and 572 00:35:49,119 --> 00:35:54,078 whatnot. It's like how hard is it to actually configure exactly the same setup and keep 573 00:35:54,079 --> 00:35:56,990 it in sync on a cloud dev box? 574 00:35:57,040 --> 00:35:59,919 Well, maybe it's not that hard if the model just does it for you. 575 00:35:59,920 --> 00:36:03,519 And so I think we're going to see a resurgence of, you know, 576 00:36:03,520 --> 00:36:09,439 fully cloud orchestrated uh machines which then frees you from your laptop, right? 577 00:36:09,440 --> 00:36:13,358 It's like one thing that we've seen a ton of success with with chat work is like it's 578 00:36:13,359 --> 00:36:14,879 just available on your mobile. 579 00:36:14,880 --> 00:36:18,879 I start my day just dictating a bunch of tasks into it. 580 00:36:18,880 --> 00:36:20,959 uh next to the coffee and it just does it. 581 00:36:20,960 --> 00:36:22,159 It has access to my calendar. 582 00:36:22,160 --> 00:36:24,159 It has access to my my email. 583 00:36:24,160 --> 00:36:28,799 Uh it has access to Slack and it's just so awesome to just be able to walk around and 584 00:36:28,800 --> 00:36:34,719 you know get stuff done without having to you know carry my laptop everywhere and I think 585 00:36:34,720 --> 00:36:38,319 it's the same things like you know we shipped like Codex remote where you know execution 586 00:36:38,320 --> 00:36:42,879 is like still happening on your laptop but it would be wonderful if you know you didn't 587 00:36:42,880 --> 00:36:44,319 have to keep your laptop open. 588 00:36:44,320 --> 00:36:48,799 Can you tell me a bit on how in the past how did you improve codecs? 589 00:36:48,800 --> 00:36:53,679 Because I remember when I first used codeex this was one of the early versions you know 590 00:36:53,680 --> 00:36:58,078 like you could talk to it did stuff but for example I said like all right make this change 591 00:36:58,079 --> 00:37:02,479 and it did that change and I had unit tests and it didn't run it and then later a few 592 00:37:02,480 --> 00:37:05,950 months later I don't know exactly when it just started to run it automatically. 593 00:37:06,000 --> 00:37:10,399 were these things did you improve the you know the the script that runs you know the 594 00:37:10,400 --> 00:37:14,639 instructions I'm not sure how exactly you call the you know the the bootstrapping script 595 00:37:14,640 --> 00:37:19,919 or whatever that is is it improving the model like as a dev how can I imagine you making 596 00:37:19,920 --> 00:37:24,799 each version better between the harness and then between the model and like what's the 597 00:37:24,800 --> 00:37:31,679 connection between the two >> yeah this is a good question so the uh the harness in a 598 00:37:31,680 --> 00:37:36,399 sense is always a little bit ahead of the model >> oh really how so? 599 00:37:36,400 --> 00:37:40,479 >> Oh, what what I mean by that is that you you have the model, 600 00:37:40,480 --> 00:37:41,759 it's capable of certain things, 601 00:37:41,760 --> 00:37:48,239 but then you set it up with like a couple of crutches so that it can actually do the 602 00:37:48,240 --> 00:37:55,279 thing um to a level of reliability and um in in in a way um that is like 603 00:37:55,280 --> 00:37:59,630 efficient and also with the behavior that you expect as a user. 604 00:37:59,680 --> 00:38:01,679 And sort of like that's the role of the harness, right? 605 00:38:01,680 --> 00:38:06,399 is like you know provide guard rails like safety, make it more efficient, 606 00:38:06,400 --> 00:38:07,759 make it more like steerable, 607 00:38:07,760 --> 00:38:12,559 controllable and then the harness usually is also responsible for you know what we call 608 00:38:12,560 --> 00:38:19,710 like the the developer um message which is sort of like infected in the context 609 00:38:19,760 --> 00:38:22,350 uh at the start of of each turn. 610 00:38:22,400 --> 00:38:27,358 And so that affects obviously like the the purpose of that to affect like the behavior 611 00:38:27,359 --> 00:38:30,479 of the agent throughout uh throughout the turn. 612 00:38:30,480 --> 00:38:36,239 A lot of what you have is like the result of of the harness and the model initially like 613 00:38:36,240 --> 00:38:38,159 maybe you're like oh it doesn't run tests. 614 00:38:38,160 --> 00:38:42,078 So you know you have to remind it to run tests and then you know we train a battle model 615 00:38:42,079 --> 00:38:48,559 that is uh just you know capable of like better reflecting on what is it that you really 616 00:38:48,560 --> 00:38:52,639 want when you ask for something uh and then you know you don't actually have to tell 617 00:38:52,640 --> 00:38:58,078 it anymore. So over time what we see is like the system uh the the developer message 618 00:38:58,079 --> 00:39:02,239 shrinks and then the harness also shrinks >> inside of the codeex team. 619 00:39:02,240 --> 00:39:03,679 Do you have specific goals? 620 00:39:03,680 --> 00:39:09,519 Do you say like all right now the the codeex as a hardness and and model to combine is 621 00:39:09,520 --> 00:39:14,239 not very good at this or it's kind of doing silly mistakes or here or how can I imagine 622 00:39:14,240 --> 00:39:20,159 how as the engineering team how you're working on the next you know version of of codeex 623 00:39:20,160 --> 00:39:24,479 is because the thing that I don't really get as a as a dev is like okay there's a model 624 00:39:24,480 --> 00:39:27,679 which to me is this magical thing which will get better of course I'm sure you have some 625 00:39:27,680 --> 00:39:31,039 feedback channels but you also have the harness which is the tools that you're building 626 00:39:31,040 --> 00:39:34,479 like that's probably what the team is responsible for how do set even your goals right 627 00:39:34,480 --> 00:39:36,879 like in traditional software you'll be like we will build this feature and you build 628 00:39:36,880 --> 00:39:40,719 that feature because you know how to do it but it feels a bit more fuzzy to me this this 629 00:39:40,720 --> 00:39:41,679 development process. 630 00:39:41,680 --> 00:39:47,309 >> Yeah it is and it's why we we co-design you know most things and it's it's a process 631 00:39:47,359 --> 00:39:52,959 where it's a collaboration between research and the engineering team like primarily building 632 00:39:52,960 --> 00:39:55,870 the the core the core agent harness. 633 00:39:55,920 --> 00:40:01,118 It's always uh it's always a question of like okay we see today that you know we are 634 00:40:01,119 --> 00:40:05,439 very good at this but we're not very good at this and you know we have a desire to do 635 00:40:05,440 --> 00:40:09,039 like another thing because it would be a very cool products feature and then you know 636 00:40:09,040 --> 00:40:13,598 we always like look at it it's like okay this should this be like a harness change or 637 00:40:13,599 --> 00:40:17,358 should this be a model change and if it's a model change like how soon can we have it 638 00:40:17,359 --> 00:40:20,879 can we have it in a month can we have it in you know three months six months and we sort 639 00:40:20,880 --> 00:40:25,999 of like work through that and then depending on you know how soon we can just fix it 640 00:40:26,000 --> 00:40:28,319 in the model at which level of training. 641 00:40:28,320 --> 00:40:32,639 Then we might decide to not even do something in the harness at all and not and just 642 00:40:32,640 --> 00:40:35,039 wait for for for the model to to solve it. 643 00:40:35,040 --> 00:40:36,719 You know, it's it's agents all the way, right? 644 00:40:36,720 --> 00:40:40,479 So we use agents to analyze like a lot of the feedback to like, you know, 645 00:40:40,480 --> 00:40:41,838 come up with themes, you know, 646 00:40:41,839 --> 00:40:45,870 to just help us have these conversations and decide on priorities. 647 00:40:45,920 --> 00:40:48,078 But we analyze it across all of coding. 648 00:40:48,079 --> 00:40:52,719 We analyze it across like all of like know the other domains like finance, coms, 649 00:40:52,720 --> 00:40:53,439 marketing, you know, 650 00:40:53,440 --> 00:40:57,999 all the things where our users are using these agents nowadays and there's like you know 651 00:40:58,000 --> 00:41:01,999 subcategories within those and then we roughly know like you know how well we perform 652 00:41:02,000 --> 00:41:05,759 and then we're always pushing the frontier and there's a thing that is interesting is 653 00:41:05,760 --> 00:41:09,919 like as we make you know as our pre-training model gets better as we make the overall 654 00:41:09,920 --> 00:41:14,318 model better like the whole thing lifts up but then there are sometimes things that we 655 00:41:14,319 --> 00:41:15,598 pay a little bit more attention to. 656 00:41:15,599 --> 00:41:18,159 You mentioned you know he analyzes agents all all the way. 657 00:41:18,160 --> 00:41:25,440 Can we talk about the the software development life cycle on codeex in the sense of whenever 658 00:41:25,599 --> 00:41:30,159 a new engineer joins a team any team it's like okay how are things done here and you 659 00:41:30,160 --> 00:41:34,399 know back preai it would have been you join the company like Uber or Google and they 660 00:41:34,400 --> 00:41:38,559 would tell you that cool the way it works is we have an idea or the PM has an idea we 661 00:41:38,560 --> 00:41:43,118 make a plan we get together we do some estimations we break up the work we code the work 662 00:41:43,119 --> 00:41:48,078 we do tests we do code reviews we release we do feature flags and then you know we we 663 00:41:48,079 --> 00:41:52,639 were on call that that you know that's how it used to be when someone joins the codeex 664 00:41:52,640 --> 00:41:57,039 team you know they they've clearly been contributing to the open source part but what 665 00:41:57,040 --> 00:42:01,598 what do you tell them how do things get get done here if they're like a brand total newbie 666 00:42:01,599 --> 00:42:06,318 >> I introduce them to great people um and then the thing that they hear the most about 667 00:42:06,319 --> 00:42:12,399 like when they have a question is like have you asked Codex um and Codex like is just 668 00:42:12,400 --> 00:42:16,959 by default at openi is like plugged into everything so it has access to slack it has 669 00:42:16,960 --> 00:42:19,789 access to all the documents, access to all the code, 670 00:42:19,839 --> 00:42:25,679 and it still surprises uh new starters that you can basically ask it anything. 671 00:42:25,680 --> 00:42:30,879 Uh and it will very often just like come up with like a really good response. 672 00:42:30,880 --> 00:42:37,039 Um and so the easiest way to understand the state of a project or who's working on something 673 00:42:37,040 --> 00:42:41,390 or why a decision was made is like critics knows about it all internally. 674 00:42:41,440 --> 00:42:43,919 >> Um and so you just you just use all of that. 675 00:42:43,920 --> 00:42:48,430 We do a lot of work in you know for that reason we do a lot of work in public channels. 676 00:42:48,480 --> 00:42:53,999 Um we open up uh documents with like you know fairly broad uh permissions and so that 677 00:42:54,000 --> 00:42:57,439 you know everyone has access to this information as well and so that you know your agent 678 00:42:57,440 --> 00:43:00,479 can go through things and like you know reason through things and then you know we have 679 00:43:00,480 --> 00:43:05,999 a couple of other things that are just really uh very helpful for uh team productivity 680 00:43:06,000 --> 00:43:09,999 and team collaboration that we haven't released yet but are are going to come like some 681 00:43:10,000 --> 00:43:11,039 of it at def day. 682 00:43:11,040 --> 00:43:15,039 All of that just sort of like makes you very grounded and in tune with the rest of the 683 00:43:15,040 --> 00:43:17,439 team. Um, and allows you to like, you know, 684 00:43:17,440 --> 00:43:20,959 just very very quickly like understand the state of things and and and produce things 685 00:43:20,960 --> 00:43:25,838 yourself. The general recommendation is just like, hey, care about the user, uh, 686 00:43:25,839 --> 00:43:30,559 care about the coherence of the product, care about the models and where they're going. 687 00:43:30,560 --> 00:43:34,719 Uh, if you're doing something and you know you're building like this 10,000 lines of 688 00:43:34,720 --> 00:43:37,199 code crutch to work around the model flaws, like you know, 689 00:43:37,200 --> 00:43:38,318 you're probably doing the wrong thing. 690 00:43:38,319 --> 00:43:39,950 So, we have a set of principles. 691 00:43:40,000 --> 00:43:46,239 Um but it's just really um sort of like a team culture and ethos at this point and you 692 00:43:46,240 --> 00:43:50,078 know it's just very much so like carries on you know when people join it's just like 693 00:43:50,079 --> 00:43:53,598 through the rest of the team just like you know sort like teaching the ropes and then 694 00:43:53,599 --> 00:43:58,239 when I have an idea I think it's a good idea I I talk it through with codeex maybe I 695 00:43:58,240 --> 00:44:01,199 talk it through with some my colleagues like here's a cool new feature I'm going to build 696 00:44:01,200 --> 00:44:06,879 as my first first contribution or first major contribution at to codeex how do I go about 697 00:44:06,880 --> 00:44:10,719 that obviously I I code it down with codeex I obviously test it and make sure that it 698 00:44:10,720 --> 00:44:15,919 works from there on what's the process do you still have the concept of code review or 699 00:44:15,920 --> 00:44:20,959 AI code review of verification of rolling out of verifying of stage rolled out you know 700 00:44:20,960 --> 00:44:25,838 the things because codex itself it goes out to millions of people like I just crossed 701 00:44:25,839 --> 00:44:31,118 a big 20 million active user mark but if it's chat GPT then it also goes out to like 702 00:44:31,119 --> 00:44:36,959 even a lot bigger number of people >> yes um but it's it's surprisingly like a similar 703 00:44:36,960 --> 00:44:41,919 process whether you ship on codeex or chat even though Chibd goes out to like a billion 704 00:44:41,920 --> 00:44:48,959 a billion you know active users and growing you can ship a PR um you know you can make 705 00:44:48,960 --> 00:44:52,479 a change and you know get it shipped like the next day or like even the same day um and 706 00:44:52,480 --> 00:44:55,920 it just goes out to a billion users and it's fine we 707 00:44:56,240 --> 00:45:01,999 just really instill a sense of ownership and care so you're like people are very empowered 708 00:45:02,000 --> 00:45:05,759 to make changes even large changes the general thing that is being asked is like sort 709 00:45:05,760 --> 00:45:10,399 of like evidence that it's going to be wellreceived D evidence that is like a worthy 710 00:45:10,400 --> 00:45:15,039 addition uh evidence that you know it's like it is worth maintaining over time but also 711 00:45:15,040 --> 00:45:19,519 like the cost of maintenance is like just really as you know gotten done significantly 712 00:45:19,520 --> 00:45:23,519 as well. So we we we think about these things slightly differently than you know say 713 00:45:23,520 --> 00:45:25,358 like two years ago or 3 years ago. 714 00:45:25,359 --> 00:45:27,759 The other thing as well is like you know we automate as much as possible. 715 00:45:27,760 --> 00:45:32,399 So like a lot of like the process of like code review and deploys and you know catching 716 00:45:32,400 --> 00:45:35,710 regressions is like you know all of that is like pretty much automated. 717 00:45:35,760 --> 00:45:41,759 Uh and so like you know you get to just focus on just really the idea and you know how 718 00:45:41,760 --> 00:45:46,078 it's going to help our users and you care about you know the coherence of it all and 719 00:45:46,079 --> 00:45:50,559 so like the overall power of the agent um and making things better and like we don't 720 00:45:50,560 --> 00:45:54,318 we have a long long list of things that you know we sort of like aspire to do and haven't 721 00:45:54,319 --> 00:45:58,239 gotten to yet. And then there's like the sort of like the northstar direction um which 722 00:45:58,240 --> 00:46:05,039 is a delightful simple to use personal AGI that you know knows everything about you like 723 00:46:05,040 --> 00:46:08,959 you know that it needs to know has access to the right resources can take like you know 724 00:46:08,960 --> 00:46:12,479 sometimes risky actions on your behalf but then you know you get like the push notification 725 00:46:12,480 --> 00:46:15,759 and then you know you can verify that and it's like a thing that you know you deeply 726 00:46:15,760 --> 00:46:20,078 understand as a user but also it knows about your schedule it knows about your goals 727 00:46:20,079 --> 00:46:23,999 it can be proactive and it should be like extremely natural it should be something that 728 00:46:24,000 --> 00:46:27,199 you can control through like natural language, voice, you know, 729 00:46:27,200 --> 00:46:28,879 like maybe it should understand, you know, 730 00:46:28,880 --> 00:46:30,879 your emotions like if it has like a camera feed, 731 00:46:30,880 --> 00:46:32,799 it should be the most natural thing on earth. 732 00:46:32,800 --> 00:46:36,959 It's like it should not be like a thing with 10, you know, different buttons and configurations. 733 00:46:36,960 --> 00:46:39,838 It's like AGI should be simple to use. 734 00:46:39,839 --> 00:46:43,279 Now, >> you kind of mentioned just briefly the review the code review, 735 00:46:43,280 --> 00:46:45,598 but I wanted to go back to it. 736 00:46:45,599 --> 00:46:49,679 you you worked at Google on a on a product used by you know like hundreds of millions 737 00:46:49,680 --> 00:46:54,318 which is Google maps and Google is very well known for their culture of very strict code 738 00:46:54,319 --> 00:46:56,639 reviews. They have I think two layers of code reviews. 739 00:46:56,640 --> 00:46:59,999 There's a language correctness review and they they've taken I think they've really perfected 740 00:47:00,000 --> 00:47:04,318 it across the industry for for a long time and they do believe that it it works and they 741 00:47:04,319 --> 00:47:08,078 they use it. How do you think that part is changing specifically the human review? 742 00:47:08,079 --> 00:47:12,239 Because for a very long time until maybe a year or two ago, 743 00:47:12,240 --> 00:47:16,559 I would have said you code review has all these benefits that knowledge sharing the second 744 00:47:16,560 --> 00:47:20,399 pair of eyes removing the bus factor because now someone else understands and when that 745 00:47:20,400 --> 00:47:25,199 person is out that they can jump in conversations are happening about architecture not 746 00:47:25,200 --> 00:47:32,318 just not just the code but now there's you know there's a lot more code uh and 747 00:47:32,319 --> 00:47:34,750 what was the value of code review? 748 00:47:34,800 --> 00:47:36,750 What in what cases? 749 00:47:36,800 --> 00:47:40,590 And so on your team, because you guys are so ahead of this, 750 00:47:40,640 --> 00:47:45,199 where do you see humans still being or developers being involved in the review stage 751 00:47:45,200 --> 00:47:48,078 valuable? And and where is it fine? 752 00:47:48,079 --> 00:47:52,240 Did you find it find it fine to uh hand it off to an agent? 753 00:47:52,480 --> 00:47:55,439 >> Yeah, the the the role of code review is changing. 754 00:47:55,440 --> 00:48:00,639 One one of the early projects that I did on Codex was like working with research on developing 755 00:48:00,640 --> 00:48:04,559 a code review model um that was 756 00:48:05,119 --> 00:48:11,630 going to be to a level where it can spot mistakes in logic and reasoning. 757 00:48:11,680 --> 00:48:17,279 uh to a degree where it would require humans like you know multiple like potentially 758 00:48:17,280 --> 00:48:21,358 multiple hours to capture the same level of mistake because it requires like really digging 759 00:48:21,359 --> 00:48:25,519 like you know three four levels deep into like the dependencies and like you know understand 760 00:48:25,520 --> 00:48:29,199 that maybe the documentation actually was wrong and like the implementation of like this 761 00:48:29,200 --> 00:48:32,078 third party dependencies like different from what you expected and so therefore your 762 00:48:32,079 --> 00:48:36,558 invarants are not upheld um and these things it's just like you know unless you're an 763 00:48:36,559 --> 00:48:41,039 expert in that library you wouldn't know uh and therefore you have a bug and so we developed 764 00:48:41,040 --> 00:48:44,959 like these uh code review models and you know we we we released them and now they're 765 00:48:44,960 --> 00:48:50,479 like the same level of like capability and like ability to spot these mistakes by doing 766 00:48:50,480 --> 00:48:54,399 like you know deep verification are like just part of the mainline models like when we 767 00:48:54,400 --> 00:48:58,558 benchmark them it's like they're like super human in code review and this is not just 768 00:48:58,559 --> 00:48:59,519 true for correctness. 769 00:48:59,520 --> 00:49:03,679 This is also true for security for example where you're capable of like reasoning across 770 00:49:03,680 --> 00:49:06,879 like you know very very complex things and then you know coming up with like hey you 771 00:49:06,880 --> 00:49:11,439 know you have a critical security vulnerability here which is now mandatory across like 772 00:49:11,440 --> 00:49:16,479 all of OpenAI pull requests like we block pull requests from merging if you know we flag 773 00:49:16,480 --> 00:49:21,439 them with like a a security issue and this is like all automatic and the role of code 774 00:49:21,440 --> 00:49:25,358 review now is like I think it was always about correctness it was always about you know 775 00:49:25,359 --> 00:49:29,519 ensuring that things worked but it was also sort of like a little ritual for information 776 00:49:29,520 --> 00:49:32,799 exchange and you know bringing people on the same page and like you know encouraging 777 00:49:32,800 --> 00:49:37,199 like a discussion which ideally would have happened before but sometimes it just only 778 00:49:37,200 --> 00:49:40,639 happens like around the code because once it merged it just actually runs in production 779 00:49:40,640 --> 00:49:45,039 it's doing stuff >> and then you have to maintain it so there's like this social aspect 780 00:49:45,040 --> 00:49:49,598 to it as well it's I think all of it is changing like the correctness the cyber the the 781 00:49:49,599 --> 00:49:54,399 security is like I think that will be automated really what we see and I see is there's 782 00:49:54,400 --> 00:50:00,879 a sort of um really discussion around the intent that takes place around the poll request. 783 00:50:00,880 --> 00:50:02,799 It's like what are you even trying to do? 784 00:50:02,800 --> 00:50:04,959 Um and is that a right thing to attempt to do? 785 00:50:04,960 --> 00:50:07,439 I think you can have that discussion outside of the poll request. 786 00:50:07,440 --> 00:50:08,879 It doesn't have to be around code. 787 00:50:08,880 --> 00:50:13,358 >> So, so maybe this helps crystallize the you know like where a discussion needs to 788 00:50:13,359 --> 00:50:17,598 happen versus versus where we we did it because maybe we didn't have the type the type 789 00:50:17,599 --> 00:50:19,039 of tooling that we have right now. 790 00:50:19,040 --> 00:50:22,879 Yeah, I think this is going to change and it was like a forcing function because you 791 00:50:22,880 --> 00:50:25,759 know you have to have that discussion where like it's good to have that discussion before 792 00:50:25,760 --> 00:50:31,439 you merge it and it becomes production code but I I think there are other ways to have 793 00:50:31,440 --> 00:50:35,279 these discussions and you know design things together and make sure that the intent is 794 00:50:35,280 --> 00:50:39,679 good uh and then the code doesn't matter as much >> and it's interesting because when 795 00:50:39,680 --> 00:50:43,439 I think back of all my code reviews like of course I have like memories where like it 796 00:50:43,440 --> 00:50:47,279 was great we we had a good discussion or I learned something really interesting but a 797 00:50:47,280 --> 00:50:50,318 bunch of times honestly It was such a pain in the ass. 798 00:50:50,319 --> 00:50:52,399 Like I I was trying to get my stuff. 799 00:50:52,400 --> 00:50:54,959 You're paying. Hey, could you remove my code? 800 00:50:54,960 --> 00:50:56,558 And like, no, right now I'm busy. 801 00:50:56,559 --> 00:50:58,479 No, I really need this to unblock me. 802 00:50:58,480 --> 00:50:59,759 And then you context switch. 803 00:50:59,760 --> 00:51:03,598 And then I feel it's always been like good and bad, right? 804 00:51:03,599 --> 00:51:08,078 So I I feel whatever we do there will be always upsides and and downside, 805 00:51:08,079 --> 00:51:09,279 but there now they're just moving. 806 00:51:09,280 --> 00:51:15,439 So I guess one upside is as an engineer you might have to not give your attention to 807 00:51:15,440 --> 00:51:18,719 just kind of basic stuff that doesn't need your input per se. 808 00:51:18,720 --> 00:51:25,118 Yes, it saves time and I think progressively what we're going to see is also like you 809 00:51:25,119 --> 00:51:29,919 have an agreement on you know the box and the overall contract of what it's supposed 810 00:51:29,920 --> 00:51:33,999 to do and then you know what is inside the box as long as you have like strict guarantees 811 00:51:34,000 --> 00:51:39,838 in terms of resource utilization, data access, um security, these kinds of things. 812 00:51:39,839 --> 00:51:43,999 It's like what happens inside the box is you know it could be literally anything. 813 00:51:44,000 --> 00:51:47,759 it's like don't really need to care and like really what you need to agree on is like 814 00:51:47,760 --> 00:51:51,598 what does the box actually do and what are the invariants that must be satisfied and 815 00:51:51,599 --> 00:51:55,598 I think that is then worthy you know having like a really good conversation on you know 816 00:51:55,599 --> 00:52:00,558 maybe assisted by by your favorite uh agent but then once you have that and you have 817 00:52:00,559 --> 00:52:03,999 that understanding it's just like changing anything within the box is like you know doesn't 818 00:52:04,000 --> 00:52:07,759 require for discussion and it's like you know just really preserves your attention >> 819 00:52:07,760 --> 00:52:12,318 the cost of maintenance has gone down you know maintenance is always such a hot topic 820 00:52:12,319 --> 00:52:15,919 whenever we build thing uh inside of all these companies like Google, Uber, 821 00:52:15,920 --> 00:52:20,239 even startups like building was the fun part but then maintenance was the painful and 822 00:52:20,240 --> 00:52:25,439 that's when we learned like okay it was not we're building it etc inside of codeex and 823 00:52:25,440 --> 00:52:31,199 open AAI what do you see maintenance becoming cheaper changing in terms of instead of 824 00:52:31,200 --> 00:52:36,479 what you're building what the ambition is the I guess custom tooling those kind of things 825 00:52:36,480 --> 00:52:40,719 >> maintenance is really like sort of like a tax that you pay over time just to keep 826 00:52:40,720 --> 00:52:43,999 things running and it's It's it's it's always been necessary. 827 00:52:44,000 --> 00:52:45,199 It will continue to be necessary, 828 00:52:45,200 --> 00:52:48,558 but where I think it changes is like a lot of it is just going to be automated. 829 00:52:48,559 --> 00:52:50,318 So, you know, it's like okay, 830 00:52:50,319 --> 00:52:54,639 you have you have this third party dependencies like you need to upgrade the version 831 00:52:54,640 --> 00:52:57,120 number. It's like a like 832 00:52:57,441 --> 00:53:03,118 [laughter] you can fully automate this you know um if you have good change log and you 833 00:53:03,119 --> 00:53:06,239 know and the code is well documented and like you know and and the model can just like 834 00:53:06,240 --> 00:53:10,318 reason through it like you know you can just like blast through your codebase do it uh 835 00:53:10,319 --> 00:53:14,479 in a couple of hours and you know previously you would have like punted on it because 836 00:53:14,480 --> 00:53:18,639 it's not the most fun thing to do but it's actually really important for your business. 837 00:53:18,640 --> 00:53:22,239 So it's like really important for your project, you know, especially for security vulnerabilities. 838 00:53:22,240 --> 00:53:23,919 You want to stay up to date, right? 839 00:53:23,920 --> 00:53:26,640 You want to apply, you know, all these patches. 840 00:53:26,960 --> 00:53:29,199 Um I think that's just going to be fully automated. 841 00:53:29,200 --> 00:53:33,519 So a large part of like maintenance, it just kind of comes for free, right? 842 00:53:33,520 --> 00:53:38,558 And then um I think it's it's awesome to also think about before like you know when you 843 00:53:38,559 --> 00:53:43,279 wanted to just completely react re you have to do like a new architecture because you're 844 00:53:43,280 --> 00:53:47,519 trying to make space for like a new you know different kind of trade-offs or you have 845 00:53:47,520 --> 00:53:51,759 a new understanding of like the workload or you're trying to fit a new feature and like 846 00:53:51,760 --> 00:53:55,439 suddenly you realize like your current system is just very limiting and you need to completely 847 00:53:55,440 --> 00:53:59,358 rearchitecture it that was like a really really costly endeavor right so you know like 848 00:53:59,359 --> 00:54:04,239 sometimes like multiple years and I think this is also like super super accelerated now. 849 00:54:04,240 --> 00:54:09,999 So you like the cost of mistakes uh you know I would say like you know is going down 850 00:54:10,000 --> 00:54:14,558 but then at the same time the good old rules I would say of software engineer of like 851 00:54:14,559 --> 00:54:17,999 you know having good abstractions like really help like you know is going back to this 852 00:54:18,000 --> 00:54:21,838 like having the box with invariance like you know if you sort of like draw the right 853 00:54:21,839 --> 00:54:25,439 shape you're going to be able to change things much more quickly within the box and like 854 00:54:25,440 --> 00:54:29,920 not affect the rest of the of the services or the rest of your infrastructure 855 00:54:30,240 --> 00:54:34,558 and I think it's important it's important to design for very quick iteration and I remember 856 00:54:34,559 --> 00:54:39,999 when I talked with Peter Shamberger that was before he joined OpenAI but about open claw 857 00:54:40,000 --> 00:54:43,598 and how he thinks about it like you know he told me that he doesn't read the code but 858 00:54:43,599 --> 00:54:47,118 he kept thinking about like I could see that he's holding the architecture in his head 859 00:54:47,119 --> 00:54:51,199 and he was telling me how he rearchitects a lot and he thinks about how to make it modular 860 00:54:51,200 --> 00:54:56,879 how to allow 100 contributors to each build their thing without stepping on each other's 861 00:54:56,880 --> 00:55:03,999 toes. So I'm hearing what you're saying that this this care this this planning this 862 00:55:04,000 --> 00:55:08,639 this structuring has become maybe just a lot more important to like which which which 863 00:55:08,640 --> 00:55:12,159 was which which was something back in the day you know it was like the architect or the 864 00:55:12,160 --> 00:55:16,799 staff engineer or experienced folks were doing this thing and other engineers around 865 00:55:16,800 --> 00:55:20,509 them were kind of building this you know smaller parts but it sounds like now all engineers 866 00:55:20,559 --> 00:55:24,719 need to be aware of when you're building your software right and plan for it >> yeah 867 00:55:24,720 --> 00:55:28,318 and and the GP models are getting better and better at this as well of like you know 868 00:55:28,319 --> 00:55:31,999 thinking about long-term maintenance and like good architecture and like this is like 869 00:55:32,000 --> 00:55:34,078 a natural sort of like next step right. 870 00:55:34,079 --> 00:55:37,838 It's like not just about code quality in the sense of like oh is this code clean within 871 00:55:37,839 --> 00:55:41,598 this file but like you know is this like is the architecture actually correct to reduce 872 00:55:41,599 --> 00:55:46,318 maintenance burden over time and like you know make space for like future u product or 873 00:55:46,319 --> 00:55:50,159 feature extensions or changes and just really this act of like you know engineering over 874 00:55:50,160 --> 00:55:54,639 time that's uh kind of like something that models are starting to become capable of like 875 00:55:54,640 --> 00:56:00,319 thinking about very well I think it's just kind of fascinating to understand that the 876 00:56:00,480 --> 00:56:03,679 software that we're building is just going through the life cycle much much faster. 877 00:56:03,680 --> 00:56:07,358 Right? Like you know before you had you know you were you were scaling you were starting 878 00:56:07,359 --> 00:56:11,519 it you know maybe as like a small team of you know yourself maybe a couple of engineers 879 00:56:11,520 --> 00:56:15,919 and then you would add engineers like slowly and then you know maybe after a year you 880 00:56:15,920 --> 00:56:19,679 know it's like if it's very very successful you would have 50 engineers on it or like 881 00:56:19,680 --> 00:56:20,959 a 100 engineers on it. 882 00:56:20,960 --> 00:56:22,558 You would have time to see it coming. 883 00:56:22,559 --> 00:56:26,318 you would have time to see like you know the humans on board and you sort of like you 884 00:56:26,319 --> 00:56:29,838 know you can think about the documentation all of that stuff but now it's just sort of 885 00:56:29,839 --> 00:56:33,679 like that explosion of like you know suddenly you have like a 100 agents contributing 886 00:56:33,680 --> 00:56:37,999 to this thing is like you know that can happen like you know in a weekend and so you 887 00:56:38,000 --> 00:56:42,639 know you're just going through it at you know major major speed compared to before >> 888 00:56:42,640 --> 00:56:46,479 okay but how do you and and the folks at OpenAI like deal with this like does it not 889 00:56:46,480 --> 00:56:51,039 mess with your mind like you know what I mean in the sense of like you you you've been 890 00:56:51,040 --> 00:56:56,959 in this business for quite some time now like like decades or or well over and there 891 00:56:56,960 --> 00:57:01,519 was a pace that we kind of got used to and obviously is it's now a lot faster but how 892 00:57:01,520 --> 00:57:06,558 do you get your head around the fact that a it's faster b the stuff that you've been 893 00:57:06,559 --> 00:57:11,598 doing a year ago right now you're not doing because now the model is is good at it and 894 00:57:11,599 --> 00:57:14,399 you know like how do you kind of reconcile that because there I'm sure there's stuff 895 00:57:14,400 --> 00:57:17,919 that you've been really good at uh related to software that now you can hand off to the 896 00:57:17,920 --> 00:57:22,110 agent do you not get a little bit of sting you know we talked about it's stinging for 897 00:57:22,160 --> 00:57:23,838 your features to be implemented open source. 898 00:57:23,839 --> 00:57:27,358 But it can also sting that I've been really good at like I don't know refactoring or 899 00:57:27,359 --> 00:57:31,679 or or right now it might be architecture but maybe the model will be really good at that 900 00:57:31,680 --> 00:57:35,999 and now I'm like uh okay damn like I'm glad but also like uh it would have been nice 901 00:57:36,000 --> 00:57:36,879 for me to do that. 902 00:57:36,880 --> 00:57:39,838 >> Yeah, I think there's like a craft aspect to it. 903 00:57:39,839 --> 00:57:44,879 Um which occasionally I still you know pull up an editor and like write some code. 904 00:57:44,880 --> 00:57:50,318 Um, and it's just like it it it it feels nice and and and it's sort of like I have fond 905 00:57:50,319 --> 00:57:55,679 memories of like late nights sitting [laughter] in uh in Vim and you know, 906 00:57:55,680 --> 00:57:58,318 just like >> cranking it out, >> you know, 907 00:57:58,319 --> 00:58:02,959 drinking um Coke Zero and uh yeah, 908 00:58:02,960 --> 00:58:05,919 just not having to think about anything else other than like the problem in front of 909 00:58:05,920 --> 00:58:13,199 me. But really, I think it's um it's all about being in the flow and and solving problems. 910 00:58:13,200 --> 00:58:17,838 And what I find is like you know folks here and also like everyone I talk to is just 911 00:58:17,839 --> 00:58:24,029 like adapting very quickly and I think if you if you have a mindset where it's all about 912 00:58:24,079 --> 00:58:28,078 code is a tool to solve problems and you can solve so many more problems. 913 00:58:28,079 --> 00:58:32,479 It's like before you wanted to benchmark something and you weren't quite sure where you 914 00:58:32,480 --> 00:58:33,279 were going to net out. 915 00:58:33,280 --> 00:58:35,199 It's like you can just do it. 916 00:58:35,200 --> 00:58:38,639 it's it's going to take you like no more than 30 seconds, you know, 917 00:58:38,640 --> 00:58:40,558 to launch something in the background and, you know, 918 00:58:40,559 --> 00:58:43,439 get proper numbers and be able to do like a better trade-off. 919 00:58:43,440 --> 00:58:48,749 It makes you it should make you a better engineer if you just really care about, 920 00:58:48,799 --> 00:58:51,679 you know, the outcome and the system working well. 921 00:58:51,680 --> 00:58:55,919 And so what it allows us to do at OpenAI, it allows us to run, you know, 922 00:58:55,920 --> 00:58:57,759 our inference much more efficiently. 923 00:58:57,760 --> 00:59:01,279 It allows us to, you know, get like much more like effective compute and, you know, 924 00:59:01,280 --> 00:59:02,239 deploy that to the world. 925 00:59:02,240 --> 00:59:06,959 And so like everyone's just like very focused on that and solving important problems 926 00:59:06,960 --> 00:59:08,798 at the speed that was not possible before. 927 00:59:08,799 --> 00:59:12,959 And like I I haven't yet, you know, encountered someone who's like, "Oh, 928 00:59:12,960 --> 00:59:14,558 that's not that's not good. 929 00:59:14,559 --> 00:59:15,759 Um that's not fun." 930 00:59:15,760 --> 00:59:19,118 >> Do I understand correctly that it sounds like if you have ambitious problems, 931 00:59:19,119 --> 00:59:23,439 if you have way more problems than what you can solve today or tomorrow or the next week, 932 00:59:23,440 --> 00:59:26,639 sounds like this is not really a problem because when you know you get more efficient 933 00:59:26,640 --> 00:59:28,558 somewhere, you keep going. 934 00:59:28,559 --> 00:59:30,318 Which which is a lot of startups, right? 935 00:59:30,319 --> 00:59:34,399 like startups are always way more ambitious than than what they're able to do. 936 00:59:34,400 --> 00:59:37,199 >> I don't we're not we're not out of problems for sure, right? 937 00:59:37,200 --> 00:59:39,519 So, and and I don't think we will be for a while. 938 00:59:39,520 --> 00:59:45,039 Um we have a long long road ahead of us in terms of like mathematical breakthroughs, 939 00:59:45,040 --> 00:59:46,719 scientific breakthroughs, you know, 940 00:59:46,720 --> 00:59:51,919 making the world a better place like just really building for humans and solving the 941 00:59:51,920 --> 00:59:56,798 most important problems that everyone is is facing and just doing it in a a deeply human 942 00:59:56,799 --> 01:00:00,269 way. that's that's what we're here for. 943 01:00:00,319 --> 01:00:02,879 Also, just going back to coding and like you know these late nights, 944 01:00:02,880 --> 01:00:07,199 it's like I think there's like it's also like maybe like a glamorous version of it. 945 01:00:07,200 --> 01:00:11,789 Just like I also had very a lot of late nights where I was trying to refactor something. 946 01:00:11,839 --> 01:00:15,759 Um and you know just like I would be like three hours deep into the refactor and then 947 01:00:15,760 --> 01:00:18,639 realize like actually this is a dead end. 948 01:00:18,640 --> 01:00:22,479 Uh and I must restart from scratch and it was like very frustrating. 949 01:00:22,480 --> 01:00:26,798 Um, and so it's like there was like there are like these very very fun times, 950 01:00:26,799 --> 01:00:30,910 but there's also the time where it's like it doesn't compile and you're just like, 951 01:00:30,960 --> 01:00:32,639 why is this not compiling yet? 952 01:00:32,640 --> 01:00:36,798 Like >> I'm sure you had the time where you you go you go later, 953 01:00:36,799 --> 01:00:37,838 it's now super late, 954 01:00:37,839 --> 01:00:41,358 you need to go to bed cuz you need to you need to get some sleep and then you can't really 955 01:00:41,359 --> 01:00:46,430 sleep and you have this thing where like you you have some some task that is halfway 956 01:00:46,480 --> 01:00:48,639 and it upsets you sometimes. 957 01:00:48,640 --> 01:00:51,118 I remember dreaming about the code as well. 958 01:00:51,119 --> 01:00:56,639 And I guess one thing I don't really have these days when I'm working on my uh software 959 01:00:56,640 --> 01:01:01,759 for my business is I don't really have something that is halfway cuz I can just tell 960 01:01:01,760 --> 01:01:05,279 it do this and then I can leave it at a state where it's kind of like you know done either 961 01:01:05,280 --> 01:01:08,558 finished it's either working or it's I have proof that it's failed. 962 01:01:08,559 --> 01:01:10,959 But it's interesting because you know everything's sped up right. 963 01:01:10,960 --> 01:01:15,919 >> Yeah. I may maybe like I I do have like what what a lot of people do and I do myself 964 01:01:15,920 --> 01:01:19,679 is like you know I have like sometimes like bigger questions that I'm asking myself like 965 01:01:19,680 --> 01:01:24,078 and I you know from conversations I've had during the day or like I haven't yet you know 966 01:01:24,079 --> 01:01:29,279 just like had the time to just look into it and so I you know I will send off codecs 967 01:01:29,280 --> 01:01:33,519 to just like look at it overnight and then I'm very excited to then wake up and look 968 01:01:33,520 --> 01:01:37,118 at the results and so you know it's always like an exciting morning. 969 01:01:37,119 --> 01:01:39,999 Well, I feel there's an art to doing longunning tasks. 970 01:01:40,000 --> 01:01:43,999 And of course, you can use the slashgoal which will go and and and run. 971 01:01:44,000 --> 01:01:47,039 You know, that's also something that was recently added like a few months ago, right? 972 01:01:47,040 --> 01:01:48,798 The /go goal command to codeex. 973 01:01:48,799 --> 01:01:52,558 >> Yeah. And back to, you know, maybe like the harness is a crutch, right? 974 01:01:52,559 --> 01:01:59,838 Is uh uh slash goal was like necessary to allow like no to keep the the model 975 01:01:59,839 --> 01:02:04,399 like on track on like a singular goal for a very long uh period of time. 976 01:02:04,400 --> 01:02:08,639 And it's like it allows the model to literally run for days or or weeks if it's like 977 01:02:08,640 --> 01:02:10,318 a really really hard problem. 978 01:02:10,319 --> 01:02:14,196 But with the new generation of models like what we're seeing is like you know you don't 979 01:02:14,197 --> 01:02:15,439 [clears throat] need SL goal anymore. 980 01:02:15,440 --> 01:02:16,719 You don't need a harness around it. 981 01:02:16,720 --> 01:02:20,318 You can just tell the model like you know hey go and work for a week and you know it 982 01:02:20,319 --> 01:02:21,439 will actually do it. 983 01:02:21,440 --> 01:02:24,879 >> Speaking of hard problems and the fact that you're not out of them. 984 01:02:24,880 --> 01:02:30,479 One of the interesting things that you shipped from the outside it I would say it it 985 01:02:30,480 --> 01:02:35,199 was you know as an engineer it was moderately interesting uh is the what you call the 986 01:02:35,200 --> 01:02:40,399 merge which is codeex appeared inside of chat GPC and the reason I say that as engineers 987 01:02:40,400 --> 01:02:43,999 it's kind of moderately interesting because we've been using codeex like yeah it's there 988 01:02:44,000 --> 01:02:48,078 you can now open it in the chat GPT app great like I just went there and I just immediately 989 01:02:48,079 --> 01:02:53,679 went to Codex cuz I don't I don't really use Chad GBT in the app per se but I talk with 990 01:02:53,680 --> 01:02:57,919 uh folks at Open AAI and people in your team and you know they were telling me like there 991 01:02:57,920 --> 01:03:01,679 was a lot of preparation going on a lot of engineering challenges. 992 01:03:01,680 --> 01:03:04,879 Can you give a sense of how big this project was, 993 01:03:04,880 --> 01:03:09,679 what you needed to do and why was it difficult to pull off and and how you know how did 994 01:03:09,680 --> 01:03:13,919 codeex and other tools help you get it done in in ways that would have been hard before 995 01:03:13,920 --> 01:03:19,519 cuz since you've launched the merge the the numbers that you keep sharing of how many 996 01:03:19,520 --> 01:03:23,118 people use codeex it's like it's going up way faster than before. 997 01:03:23,119 --> 01:03:26,798 So I assume I assume there's a big scale uh problem you've solved here. 998 01:03:26,799 --> 01:03:28,799 >> A lot of things were 999 01:03:29,200 --> 01:03:32,799 the challenging with the merge is first of all 1000 01:03:33,760 --> 01:03:35,118 completely different stacks. 1001 01:03:35,119 --> 01:03:37,519 Chbt is like fully 1002 01:03:37,920 --> 01:03:44,350 uh managed cloud-based like you know you run everything on our uh on our systems. 1003 01:03:44,400 --> 01:03:49,679 We we store things like traditional traditional way of like building things. 1004 01:03:49,680 --> 01:03:53,470 Um built for scale, built for for efficiency. 1005 01:03:53,520 --> 01:03:55,519 Codex fully local. 1006 01:03:55,520 --> 01:04:01,118 And so the merge is just really like how do you get the same uh the same benefits and 1007 01:04:01,119 --> 01:04:05,838 the same capabilities from this local coding agent and then build a product around it 1008 01:04:05,839 --> 01:04:11,038 and build it in a way where it can benefit like a much much broader pool uh of people 1009 01:04:11,039 --> 01:04:17,439 which is which is also why you know all of us joined OpenAI is like to benefit like this 1010 01:04:17,440 --> 01:04:22,399 very very broad population across the world and so it was like a very exciting journey 1011 01:04:22,400 --> 01:04:28,719 of like figuring out like how do we build a cloud version of this that in essence is 1012 01:04:28,720 --> 01:04:33,279 capable of like very very much the same things um but is also built in a way where you 1013 01:04:33,280 --> 01:04:38,399 know we can serve it to like tens and hundred millions of hundreds of millions of users 1014 01:04:38,400 --> 01:04:42,318 in a way that is still like you know efficient so that we can include it all the way 1015 01:04:42,319 --> 01:04:49,789 into the plus plan work is essentially like running uh the full codeex harness 1016 01:04:49,839 --> 01:04:55,118 in uh a cloud like together with like a a cloud computer it's It's a very powerful machine 1017 01:04:55,119 --> 01:04:59,038 actually like people have sort of picked up on it and showed like you know what you can 1018 01:04:59,039 --> 01:05:03,309 do like you know if if you are creative with the prompt is that you know you can get 1019 01:05:03,359 --> 01:05:09,549 um you can get chatbt to like you know train another model in there you know >> wow [laughter] 1020 01:05:09,599 --> 01:05:14,159 >> there are some pretty wild things uh you know you can get it to install blender and 1021 01:05:14,160 --> 01:05:19,199 you know do like 3D modeling it's like very permissive it has like internet access it's 1022 01:05:19,200 --> 01:05:23,838 like a powerful machine and then codex just works on it and this is like what we ship 1023 01:05:23,839 --> 01:05:25,199 through cas work. 1024 01:05:25,200 --> 01:05:27,439 a lot of system uh challenges. 1025 01:05:27,440 --> 01:05:29,439 We the team did it very quickly. 1026 01:05:29,440 --> 01:05:35,038 Obviously like Codex helped you know to make it more efficient look at and and build 1027 01:05:35,039 --> 01:05:36,880 a lot of the infrastructure 1028 01:05:37,280 --> 01:05:41,838 and then you know help resolve a lot of the little differences as well that you know 1029 01:05:41,839 --> 01:05:47,919 had been occurring between between Codex and CHBT like merging plugins architecture um 1030 01:05:47,920 --> 01:05:51,679 you know merging library and like so like really really working towards like unified 1031 01:05:51,680 --> 01:05:55,118 system which is really the goal is like you shouldn't feel like you know you can do something 1032 01:05:55,119 --> 01:05:59,598 in Codex that you can't do in TAB or vice versa like what we're trying to build is like 1033 01:05:59,599 --> 01:06:04,078 one unified product that gives you access uh to the same intelligence but in the way 1034 01:06:04,079 --> 01:06:05,710 that you want to use it. 1035 01:06:05,760 --> 01:06:10,479 And so it was very fun as well because Codex throughout the whole journey also acted 1036 01:06:10,480 --> 01:06:15,279 as a journalist to sort of like document all the steps and the debates and the discussions 1037 01:06:15,280 --> 01:06:16,558 that the teams were having. 1038 01:06:16,559 --> 01:06:20,639 And it was very animated debate you know of how we should do it and how we should name 1039 01:06:20,640 --> 01:06:25,519 the thing and you know when to introduce it in what way and like what to merge into what. 1040 01:06:25,520 --> 01:06:29,919 there were like many different permutations considered and so there's like a very fun 1041 01:06:29,920 --> 01:06:34,558 journalistic element to it where we like we have a full recounting that Codex did over 1042 01:06:34,559 --> 01:06:42,159 time uh and uh yeah it's just it's kind of become known as well as like the toggle arc 1043 01:06:42,160 --> 01:06:47,199 of uh OpenAI where you know we introduced like the work toggle um which there was also 1044 01:06:47,200 --> 01:06:50,159 a lot of debate around of like you know whether this was like the right thing and then 1045 01:06:50,160 --> 01:06:55,358 you know just like we kind of grew to to just really like it um but over time we're going 1046 01:06:55,359 --> 01:06:56,479 to merge things further. 1047 01:06:56,480 --> 01:07:01,118 So it's like we're really headed into this direction of like full unification and you 1048 01:07:01,119 --> 01:07:06,159 know we kind of view this as like um a temporary state uh where you know you have like 1049 01:07:06,160 --> 01:07:12,159 you have better stronger capabilities when you're in work mode but over time we're bringing 1050 01:07:12,160 --> 01:07:16,670 this you know all all the way to like you know everyone that uses CHP. 1051 01:07:16,720 --> 01:07:19,390 And how do you personally use codeex? 1052 01:07:19,440 --> 01:07:22,639 Like what's your what's what's your working setup in terms of agents, 1053 01:07:22,640 --> 01:07:26,479 in terms of task, in terms of what you what what you manage uh with it. 1054 01:07:26,480 --> 01:07:27,679 And related to this, 1055 01:07:27,680 --> 01:07:32,639 I asked Peter Stainberger what I should ask about you and he said like you I I you need 1056 01:07:32,640 --> 01:07:38,078 to ask him how do you deal with the fact that you're involved in all these projects? 1057 01:07:38,079 --> 01:07:42,640 Uh your calendar is like Tetris, but usually you show up pretty cheerful. 1058 01:07:43,680 --> 01:07:45,679 >> My calendar is fine. 1059 01:07:45,680 --> 01:07:52,719 Um um and it's just I am capable of doing so many more 1060 01:07:52,720 --> 01:07:59,439 things nowadays because I have the technology like CEX and I actually shifted a lot of 1061 01:07:59,440 --> 01:08:06,558 like my my my work on uh on mobile using tab work where whenever I have 1062 01:08:06,559 --> 01:08:09,919 something that I want to take note of I just like fire that off. 1063 01:08:09,920 --> 01:08:11,598 I use dictation a lot. 1064 01:08:11,599 --> 01:08:13,679 Um, whenever I have a question, 1065 01:08:13,680 --> 01:08:17,519 instead of like writing it down to look into later or delegating to someone, 1066 01:08:17,520 --> 01:08:21,278 I just like fire it off in charge of your work and I get like a report. 1067 01:08:21,279 --> 01:08:27,599 It has like a whole bunch of like custom skills and um custom uh instructions where it's 1068 01:08:27,600 --> 01:08:31,599 now like very very tailored to like you know produce the kinds of reports and slide decks 1069 01:08:31,600 --> 01:08:36,559 and uh code explorations you know in the style that I can consume effectively. 1070 01:08:36,799 --> 01:08:39,919 And so every time I'm like between meetings or like you know you'll kind of like see 1071 01:08:39,920 --> 01:08:41,999 me like you know I was just like dictating to my phone. 1072 01:08:42,000 --> 01:08:46,559 As I said before it's just like we do a lot of work in public channels. 1073 01:08:46,560 --> 01:08:48,559 We have like a lot in Slack. 1074 01:08:48,560 --> 01:08:51,919 We have a lot in in in notion and Google Docs as well. 1075 01:08:51,920 --> 01:08:56,158 And so there's pretty much like there's no question really that I feel I cannot ask that 1076 01:08:56,159 --> 01:08:59,758 you know Codex will be able to sort of like do at least a first pass of thinking through 1077 01:08:59,759 --> 01:09:06,238 whether it is like public sentiment on a feature um looking at production logs for you 1078 01:09:06,239 --> 01:09:09,758 know how much usage we have on a certain thing making a list of things that we should 1079 01:09:09,759 --> 01:09:13,599 deprecate because they're not getting traction uh understanding what a certain team is 1080 01:09:13,600 --> 01:09:17,838 up to. It's like any question I have I can get an answer to like you know within 30 minutes. 1081 01:09:17,839 --> 01:09:18,959 And so that's how I use it. 1082 01:09:18,960 --> 01:09:20,158 I use it for everything. 1083 01:09:20,159 --> 01:09:24,718 It's like my personal uh agent in in like all the ways. 1084 01:09:24,719 --> 01:09:28,158 And then oftentimes on on weekends as well, 1085 01:09:28,159 --> 01:09:32,798 I do some like code explorations or like I build some prototypes and I have fun like 1086 01:09:32,799 --> 01:09:35,919 sort of like imagining the future of the product in some ways. 1087 01:09:35,920 --> 01:09:38,399 And I do that with others on on on on the teams. 1088 01:09:38,400 --> 01:09:39,919 It's not always the same team. 1089 01:09:39,920 --> 01:09:46,318 And it's just like in one day I can build things that I sort of like I had it in my system, 1090 01:09:46,319 --> 01:09:50,399 right? It's like it's like I woke up one day I was just like we should explore what it 1091 01:09:50,400 --> 01:09:54,879 means to build this and then I can just sort of express all of that and get like something 1092 01:09:54,880 --> 01:09:59,759 in front of people in a day so that they can think through it and criticize it and hopefully 1093 01:09:59,760 --> 01:10:00,639 get inspired by it. 1094 01:10:00,640 --> 01:10:05,039 It's like by no means you know we need to ship it but it's more like okay I flush it 1095 01:10:05,040 --> 01:10:08,079 out of my system and then you know I go on and like you know do other things. 1096 01:10:08,080 --> 01:10:13,198 So it's just like so I know it's such a magical time and it's like so empowering. 1097 01:10:13,199 --> 01:10:17,759 And as closing, what would your advice be for a software engineer/ AAI engineer, 1098 01:10:17,760 --> 01:10:24,158 someone who builds software who would want to get the skill set and the experience to 1099 01:10:24,159 --> 01:10:26,879 have the opportunity to work at a place like the Codeex team, 1100 01:10:26,880 --> 01:10:28,718 like OpenAI or like an AI startup. 1101 01:10:28,719 --> 01:10:34,158 So like you know just become this really great builder with with these tools because 1102 01:10:34,159 --> 01:10:38,030 the question that comes up is often like should I start with the theory? 1103 01:10:38,080 --> 01:10:39,359 How important are the basics? 1104 01:10:39,360 --> 01:10:41,870 Should I just get really good at using the tools? 1105 01:10:41,920 --> 01:10:47,119 >> Yeah. I think there are two things that are important is a deep deep curiosity for 1106 01:10:47,120 --> 01:10:51,759 how things work um and an ability to like you know train yourself to understand things 1107 01:10:51,760 --> 01:10:56,959 very quickly and so it's it's it is the case that things will continue to change but 1108 01:10:56,960 --> 01:11:01,119 people that do extremely well at OpenAI are like you know people that just sort of like 1109 01:11:01,120 --> 01:11:07,359 are able to like gro uh a system quickly and like you know also dive into like a new 1110 01:11:07,360 --> 01:11:09,999 code base and sort of like know make sense of it. 1111 01:11:10,000 --> 01:11:14,158 Um but obviously like all of that is helped with agents um nowadays, right? 1112 01:11:14,159 --> 01:11:17,439 So just like there's so much information that you need to absorb and like you know being 1113 01:11:17,440 --> 01:11:21,759 able to understand and reason through it and a lot of that is asking good questions really 1114 01:11:21,760 --> 01:11:25,919 about you know how do things work and just like going into like the five W's which I 1115 01:11:25,920 --> 01:11:29,279 think you know you can just kind of keep digging and digging and digging and you know 1116 01:11:29,280 --> 01:11:31,359 you're learning very very fast through that. 1117 01:11:31,360 --> 01:11:36,238 The other thing is being in tune with the community or you know the people that you're 1118 01:11:36,239 --> 01:11:37,839 trying to solve a problem for. 1119 01:11:37,840 --> 01:11:40,158 It's like not everything is like solving a direct problem. 1120 01:11:40,159 --> 01:11:42,799 Sometimes you're solving a problem that will be useful, you know, 1121 01:11:42,800 --> 01:11:46,559 to like another group of people in the pursuit of like solving a problem for humans. 1122 01:11:46,560 --> 01:11:52,079 But just being crisp about the taste or the needs or the requirements uh and being able 1123 01:11:52,080 --> 01:11:56,399 to think clearly and like you know exercising through this clarity of thought feels really 1124 01:11:56,400 --> 01:11:56,959 important to me. 1125 01:11:56,960 --> 01:11:58,959 Like if you can't explain what you're trying to achieve, 1126 01:11:58,960 --> 01:12:02,639 if you can't explain your intent, if you don't have a tie to a community, 1127 01:12:02,640 --> 01:12:07,678 if you don't have the taste, it's it's um it's going to be much harder to do great work. 1128 01:12:07,679 --> 01:12:09,999 >> Awesome, TB. Well, thanks a bunch for this conversation. 1129 01:12:10,000 --> 01:12:10,799 This was awesome. 1130 01:12:10,800 --> 01:12:12,238 >> Thanks for having me. 1131 01:12:12,239 --> 01:12:14,079 >> I've always wanted to get together with Tibo, 1132 01:12:14,080 --> 01:12:16,510 and I'm glad that we finally made it happen. 1133 01:12:16,560 --> 01:12:19,999 I appreciated how Tibo talked about not just the upsides of open source, 1134 01:12:20,000 --> 01:12:21,470 but also the downsides. 1135 01:12:21,520 --> 01:12:25,359 most notably how competitors can copy features you [music] are just working on in the 1136 01:12:25,360 --> 01:12:30,479 open right now and then ship it right before release and just how [music] much this stings. 1137 01:12:30,480 --> 01:12:33,759 Plus, you get a lot of low-quality contributions that you still need to somehow deal 1138 01:12:33,760 --> 01:12:37,678 with. Another interesting one was Tibo was saying how the harness [music] is always a 1139 01:12:37,679 --> 01:12:38,959 step ahead of the model. 1140 01:12:38,960 --> 01:12:39,599 From the inside, 1141 01:12:39,600 --> 01:12:43,659 the Codex team see their job as building clutches for the model with the harness, 1142 01:12:43,660 --> 01:12:46,158 [music] the tools, and the setup instruction. 1143 01:12:46,159 --> 01:12:50,479 And then the next version of the model will be trained to need fewer of these clutches. 1144 01:12:50,480 --> 01:12:51,839 I'll be honest, as a dev, 1145 01:12:51,840 --> 01:12:56,479 this sounds a little demotivating that the stuff I build in the next version of the model, 1146 01:12:56,480 --> 01:12:59,439 it'll [music] just know and we can get rid of it. 1147 01:12:59,440 --> 01:13:03,759 Plus, I do suspect that it's not just about building these clutches, 1148 01:13:03,760 --> 01:13:06,479 but also building tools that models will use. 1149 01:13:06,480 --> 01:13:10,799 And it's not like the next version of the model will reinvent an MCB protocol or scales 1150 01:13:10,800 --> 01:13:12,799 or plugins. At least I hope not. 1151 01:13:12,800 --> 01:13:17,519 I also enjoyed hearing what the merge merging chat GPC and codeex look like from the 1152 01:13:17,520 --> 01:13:21,649 inside. It was merging a previously fully local coding agent, Codeex, 1153 01:13:21,650 --> 01:13:25,919 [music] into a managed cloud-based stack and doing it efficient enough so that it can 1154 01:13:25,920 --> 01:13:31,759 be included in OpenAX $20 per month plan when $20 is not all that much in terms of compute 1155 01:13:31,760 --> 01:13:36,399 purchase. It was pretty amusing to hear how CODC itself acted as a journalist of the 1156 01:13:36,400 --> 01:13:40,158 whole project [music] as it was present in all the Slack conversations and all the documents 1157 01:13:40,159 --> 01:13:43,198 and so it could capture all the important debates and decisions. 1158 01:13:43,199 --> 01:13:43,920 I'm not going to lie, 1159 01:13:43,921 --> 01:13:49,198 [music] this part felt a little bit of a big brother feel to it where the AI is always 1160 01:13:49,199 --> 01:13:52,435 watching, but it could well become the new normal in startups in the future. 1161 01:13:52,436 --> 01:13:55,119 [music] I've not yet decided how I feel about this. 1162 01:13:55,120 --> 01:13:58,830 And finally, I appreciated Tibo's advice for engineers to succeed. 1163 01:13:58,880 --> 01:14:01,678 Be curious, [music] understand symptoms quickly, 1164 01:14:01,679 --> 01:14:04,319 and be in tune with the group you are building for. 1165 01:14:04,320 --> 01:14:08,238 It's reassuring to hear from Tibo as well how much the fundamentals [music] still matter. 1166 01:14:08,239 --> 01:14:11,359 Do check out the show notes below for deep dives on how codecs, clock code, 1167 01:14:11,360 --> 01:14:14,238 and cursor were built and other related topics. 1168 01:14:14,239 --> 01:14:15,519 If you like what you heard, 1169 01:14:15,520 --> 01:14:17,919 please hit a rating on a podcast player that [music] you're using. 1170 01:14:17,920 --> 01:14:20,238 It means a lot to me and to the show. 1171 01:14:20,239 --> 01:14:22,720 Thanks, and I'll see you in the next