1
00:00:00,400 --> 00:00:01,919
So, hello folks.

2
00:00:01,920 --> 00:00:03,918
I've got another treat for you today.

3
00:00:03,919 --> 00:00:06,878
Last time on this kind of podcasty thing, I suppose,

4
00:00:06,879 --> 00:00:09,598
we had Uncle Bob and we talked about software quality.

5
00:00:09,599 --> 00:00:10,718
We talked about agents.

6
00:00:10,719 --> 00:00:12,589
We talked about lots of cool stuff.

7
00:00:12,639 --> 00:00:17,358
Now, we have uh an incredible guest, someone who I'm delighted to welcome on,

8
00:00:17,359 --> 00:00:22,590
who's been exploding on Twitter recently about software factories,

9
00:00:22,640 --> 00:00:25,759
um about increasing the quality of your work,

10
00:00:25,760 --> 00:00:30,559
about increasing your velocity and climbing the trust ladder with agents so that you

11
00:00:30,560 --> 00:00:32,238
can ship more and more and more.

12
00:00:32,239 --> 00:00:34,590
And it is potato.

13
00:00:34,640 --> 00:00:36,910
Welcome. Thank you so much for joining.

14
00:00:36,960 --> 00:00:37,759
>> Thanks for having me.

15
00:00:37,760 --> 00:00:39,519
Yeah, very excited to be here.

16
00:00:39,520 --> 00:00:42,398
Yeah, big fan of yours >> and a huge fan of yours.

17
00:00:42,399 --> 00:00:45,679
I [laughter] think people have been talking about this like it's like the meeting of

18
00:00:45,680 --> 00:00:47,390
the skill minds,

19
00:00:47,440 --> 00:00:51,599
the skill Mount Olympus or something because both of us have very popular skill libraries.

20
00:00:51,600 --> 00:00:54,159
Um I've not, as I was saying before we started,

21
00:00:54,160 --> 00:00:59,358
I've not used a ton of yours and like I want to get all of the juice out of your brain

22
00:00:59,359 --> 00:01:02,669
so that I can go and use it properly and use it better.

23
00:01:02,719 --> 00:01:07,599
And I think where I want to start with this is you gave a talk um pretty recently like

24
00:01:07,600 --> 00:01:13,118
um about 10 days ago and posted on X which went absolutely nuts as about how I shipped

25
00:01:13,119 --> 00:01:20,319
2,500 PRs last month to production got about 3 million views or something on X and I

26
00:01:20,320 --> 00:01:24,719
watched it and I loved it and I recommended it and I kind of want to run this as almost

27
00:01:24,720 --> 00:01:31,118
like a Q&A of that talk basically of giving you because it just I just had tons of questions

28
00:01:31,119 --> 00:01:33,118
about it and I wanted to dive into it.

29
00:01:33,119 --> 00:01:39,118
And I think where I want to start is you talk about a trust ladder with agents where

30
00:01:39,119 --> 00:01:41,599
you as you trust agents more,

31
00:01:41,600 --> 00:01:47,438
you can get them to do better and better things and or scale them to up to use more and

32
00:01:47,439 --> 00:01:53,999
more agents. So what is your story of how you climbed the trust ladder and how did that

33
00:01:54,000 --> 00:01:58,480
work when like you got SpaceX and >> started climbing more and more?

34
00:01:58,640 --> 00:02:03,919
So I think this the the journey sort of began even before I joined cursor uh which is

35
00:02:03,920 --> 00:02:08,880
now SpaceX AI. Uh [snorts] so the story is um

36
00:02:09,599 --> 00:02:13,620
after Meta so I I used to work at Meta on the React team.

37
00:02:13,621 --> 00:02:18,559
[snorts] Uh I took a month off uh because I was feeling kind of burnt out and of course

38
00:02:18,560 --> 00:02:20,238
when what what do you do when you're burnt out?

39
00:02:20,239 --> 00:02:22,350
You go and start a new side project.

40
00:02:22,400 --> 00:02:24,559
Um and so I started a side project.

41
00:02:24,560 --> 00:02:28,479
you know, I was uh of course using AI to to write code.

42
00:02:28,480 --> 00:02:31,279
Uh [snorts] but then I started to realize uh you know,

43
00:02:31,280 --> 00:02:35,839
I was spending like so many hours just micromanaging one agent, right?

44
00:02:35,840 --> 00:02:40,479
And you know, at the time, this was back in February, maybe February,

45
00:02:40,480 --> 00:02:43,039
early February or January, you know,

46
00:02:43,040 --> 00:02:45,359
people were really obsessed with this idea of like orchestration.

47
00:02:45,360 --> 00:02:48,719
This was like, you know, before, you know, things like cursor, you know,

48
00:02:48,720 --> 00:02:51,710
like the agents window was had become popular.

49
00:02:51,760 --> 00:02:56,559
So people were still in like like 2 land you know in their terminal and they were all

50
00:02:56,560 --> 00:03:00,799
talking about okay here you know I built a custom orchestrator right and so of course

51
00:03:00,800 --> 00:03:05,919
I had I was a bit nerd sniped by that [snorts] and you know as I was building my toy

52
00:03:05,920 --> 00:03:12,079
project uh I got nerd sniped by oh how do I make my AI coding setup more efficient and

53
00:03:12,080 --> 00:03:17,759
so you know I I kind of started the journey there where I just you know took a step back

54
00:03:17,760 --> 00:03:22,878
and realized you know I was spending all this time micromanaging a single agent you know

55
00:03:22,879 --> 00:03:27,518
I was [snorts] creating skills and I was like finding it quite difficult to measure the

56
00:03:27,519 --> 00:03:33,598
output or the the result the impact of the skill as well so I was kind of flying blind

57
00:03:33,599 --> 00:03:41,549
but I was you know iterating really fast um and um so that project eventually

58
00:03:41,599 --> 00:03:47,279
sort of became the basis of PAC even though I didn't know it at the time um and a lot

59
00:03:47,280 --> 00:03:51,839
of some tricks I had learned like building that early set of skills.

60
00:03:51,840 --> 00:03:55,359
Actually, it's still open source if you want to if anybody wants to take a look.

61
00:03:55,360 --> 00:03:57,920
It's on my GitHub like potato

62
00:03:58,560 --> 00:04:01,359
noodle n o d l e.

63
00:04:01,360 --> 00:04:04,971
Um, and in there you will see some skills and a brain directory.

64
00:04:04,972 --> 00:04:09,438
[snorts] And so I was really interested in this idea of how do I, you know,

65
00:04:09,439 --> 00:04:14,798
extract my own ability, if that makes sense, and give it to the agent, right?

66
00:04:14,799 --> 00:04:18,958
cuz I was I I realized that you know all I was trying to do was trying to teach the agent

67
00:04:18,959 --> 00:04:23,950
to write code more like me you know do do you do do workflows more like me.

68
00:04:24,000 --> 00:04:28,480
So you know the skills were like an entry point to doing that.

69
00:04:28,960 --> 00:04:34,638
Um and then you know after I joined cursor uh I was starting to work on the agents window

70
00:04:34,639 --> 00:04:37,790
and uh it had a lot of performance issues.

71
00:04:37,840 --> 00:04:40,239
Uh it was it was it was pretty laggy.

72
00:04:40,240 --> 00:04:44,478
Uh and so since I had experience working in React, I was asked like, "Hey,

73
00:04:44,479 --> 00:04:47,310
do you want to come and help out uh with the agents window?"

74
00:04:47,360 --> 00:04:54,719
Um and so the the this beginning of the cursor journey was very manual.

75
00:04:54,753 --> 00:05:00,799
[snorts] Uh I was deep in like looking at like flame graphs and heap snapshots and trying

76
00:05:00,800 --> 00:05:03,790
to see like why exactly is the app so slow.

77
00:05:03,840 --> 00:05:08,589
Uh but then coming back to the same realization like you know I was sort of the bottleneck.

78
00:05:08,639 --> 00:05:09,999
I was doing everything manually.

79
00:05:10,000 --> 00:05:12,399
I was sort of the meat proxy in a way, right?

80
00:05:12,400 --> 00:05:16,270
I was the meat proxy between my agent and Chrome DevTools.

81
00:05:16,320 --> 00:05:18,719
Uh and I was like really annoyed by that.

82
00:05:18,720 --> 00:05:20,638
>> And what month of the year is that?

83
00:05:20,639 --> 00:05:22,799
Let's say where are we in the timeline?

84
00:05:22,800 --> 00:05:25,279
>> Uh so I joined Cursor in March.

85
00:05:25,280 --> 00:05:30,719
So this was like early early April probably early April is when [snorts] you know uh

86
00:05:30,720 --> 00:05:32,638
I joined and I didn't have any skills, right?

87
00:05:32,639 --> 00:05:37,198
I had I I sort of abandoned my personal skills because I didn't think they'd be relevant

88
00:05:37,199 --> 00:05:43,520
anymore. Uh but then working on the agents window uh and [snorts] now working on grockbot

89
00:05:43,759 --> 00:05:49,279
uh I sort of realized that a lot of the lessons I had learned from those skill time building

90
00:05:49,280 --> 00:05:55,360
the the initial set of skills were very relevant especially around things like verification

91
00:05:55,840 --> 00:06:02,239
uh you know being very rigorous in your work um [clears throat] because I think from

92
00:06:02,240 --> 00:06:07,200
my experience even the the frontier ones tend to

93
00:06:07,680 --> 00:06:09,149
tend to take shortcuts.

94
00:06:09,199 --> 00:06:12,830
Uh they tend to do the easy thing.

95
00:06:12,880 --> 00:06:18,399
Uh so uh a lot of the skills that I've built have been around how do I make the easy

96
00:06:18,400 --> 00:06:19,679
thing the right thing?

97
00:06:19,680 --> 00:06:22,399
You know, how do I make that the best thing?

98
00:06:22,639 --> 00:06:29,439
>> The idea of sort of distilling your expertise and turning what you do every day into

99
00:06:29,440 --> 00:06:32,239
processes, that's something that feels super familiar to me.

100
00:06:32,240 --> 00:06:34,879
That's exactly what I've been doing with the skills.

101
00:06:34,880 --> 00:06:39,918
And I suppose there's something in that which is a lot of people think domain expertise

102
00:06:39,919 --> 00:06:46,559
is getting less useful now as people uh start to rely more on AI where what do you think

103
00:06:46,560 --> 00:06:51,439
about that just as a sort of vibe check before we start talking [laughter] >> I actually

104
00:06:51,440 --> 00:06:57,679
feel like domain expertise is is more important than ever you know uh I think I wrote

105
00:06:57,680 --> 00:07:03,918
this on my ex at some point but you know at times I sometimes think of you know AI as

106
00:07:03,919 --> 00:07:08,638
is like especially as the models get smarter and smarter and more capable and the frontier

107
00:07:08,639 --> 00:07:15,519
models are just getting so good like I [snorts] love Opus 5.5 by the way um uh you know

108
00:07:15,520 --> 00:07:20,318
as the models get really really really good it almost becomes like the bottleneck is

109
00:07:20,319 --> 00:07:26,879
no longer the agent right it becomes your ability to express your intent and your goals

110
00:07:26,880 --> 00:07:33,598
in a clear way that [snorts] the agent can understand and actually carry out and That's

111
00:07:33,599 --> 00:07:38,719
why I think you know like people with a lot of domain expertise are extremely have a

112
00:07:38,720 --> 00:07:43,758
have a huge advantage in my opinion especially if you're a little bit like you know tech

113
00:07:43,759 --> 00:07:49,359
technoc curious you know so I I think of people like you know like uh like a doctor or

114
00:07:49,360 --> 00:07:55,070
a lawyer or you know someone who who has a deep expertise in a particular non-engineering

115
00:07:55,120 --> 00:07:59,439
domain and [snorts] if they're actually just a little bit techsavvy and they can figure

116
00:07:59,440 --> 00:08:04,239
out how to use agents they can actually build really really great products, right?

117
00:08:04,240 --> 00:08:09,198
If they if they have a clear enough vision in their head and they can articulate it in

118
00:08:09,199 --> 00:08:12,318
a way that the agent can build it, you know,

119
00:08:12,319 --> 00:08:18,799
I think that that is really the the bottleneck these days is is like the transfer of

120
00:08:18,800 --> 00:08:23,040
your intent, right, and your vision to the agent.

121
00:08:23,440 --> 00:08:27,598
>> Yeah. I've been obsessed with language basically since agents um dropped.

122
00:08:27,599 --> 00:08:33,199
are just obsessed 100% and thinking constantly about the the composition of words,

123
00:08:33,200 --> 00:08:39,278
how I can make things sharper, what um what might be hidden in the phrases that I'm using.

124
00:08:39,279 --> 00:08:45,039
And it's and finding what I love is when you find a word that the agent then hooks on

125
00:08:45,040 --> 00:08:48,159
to and then goes, "Okay, I'm going to reinforce that word.

126
00:08:48,160 --> 00:08:50,559
I'm going to reuse that in my thinking traces."

127
00:08:50,560 --> 00:08:53,919
You know, I found that with um TDD was an early example of that.

128
00:08:53,920 --> 00:08:56,879
a lot of chat about TDD recently of like, you know, people say,

129
00:08:56,880 --> 00:08:58,479
should you use TDD with agents?

130
00:08:58,480 --> 00:09:02,559
Doesn't matter. What you're doing is you're getting the agent to think about TDD,

131
00:09:02,560 --> 00:09:03,838
getting it to write tests,

132
00:09:03,839 --> 00:09:07,039
getting it to prioritize things in a different way than it did before.

133
00:09:07,040 --> 00:09:10,110
And that's why sort of grilling, I think, works effectively.

134
00:09:10,160 --> 00:09:11,359
Grilling is a >> Yeah.

135
00:09:11,360 --> 00:09:14,398
Yeah. It it draws those words out of you, right?

136
00:09:14,399 --> 00:09:20,479
or or at least it helps the agent understand your thinking so that they can propose those

137
00:09:20,480 --> 00:09:23,710
words to you and you can pick up and say yes exactly that.

138
00:09:23,760 --> 00:09:27,199
>> Uh I've actually copied some of the the tips that you've shared as well where you

139
00:09:27,200 --> 00:09:31,119
know one of my favorite ones that you've shared recently or or not or like maybe in the

140
00:09:31,120 --> 00:09:37,070
past couple weeks is [snorts] about uh reducing or eliminating tautological tests.

141
00:09:37,120 --> 00:09:41,518
Like one of my pet peeves of agents is like all of the useless tests that they write.

142
00:09:41,519 --> 00:09:45,999
And so, you know, that was one thing where, you know, the word tutology, right,

143
00:09:46,000 --> 00:09:47,439
is is is I guess, you know,

144
00:09:47,440 --> 00:09:52,110
not many people necessarily know that if if especially if English isn't your first language,

145
00:09:52,160 --> 00:09:54,879
but there's a lot of meaning to that word.

146
00:09:54,880 --> 00:09:57,919
And it's like it's almost like compressed, right?

147
00:09:57,920 --> 00:10:02,430
Like you compress a lot of intent and meaning into words.

148
00:10:02,480 --> 00:10:04,479
And so I I I totally agree with you.

149
00:10:04,480 --> 00:10:09,199
I think language I've always been interested in language actually uh like programming

150
00:10:09,200 --> 00:10:14,398
languages natural human languages and how they came to be and it's so interesting that

151
00:10:14,399 --> 00:10:20,398
now with agents it's sort of like this meeting of natural language with programming language

152
00:10:20,399 --> 00:10:24,239
but it's all it's all language out of the hood it's all communication >> totally I did

153
00:10:24,240 --> 00:10:27,999
a drama degree right so you know I've been thinking about language and Shakespeare and

154
00:10:28,000 --> 00:10:33,119
stuff for a long time [laughter] and so this all feels very familiar >> um so okay there's

155
00:10:33,120 --> 00:10:39,278
sort before we get into like because I think the thing I want from you is like software

156
00:10:39,279 --> 00:10:40,398
factory stuff, right?

157
00:10:40,399 --> 00:10:42,078
Software factory is the big buzzword.

158
00:10:42,079 --> 00:10:44,239
Software factory is the thing that I'm thinking about too.

159
00:10:44,240 --> 00:10:47,070
I'm sort of releasing a course in that direction too.

160
00:10:47,120 --> 00:10:54,319
>> And it's this sort of scaling yourself up to un unrealistic numbers of PRs basically

161
00:10:54,320 --> 00:10:59,119
or PR numbers that sound ridiculous to people who don't understand how this works.

162
00:10:59,120 --> 00:11:03,919
So where I want to get to is sort of from people who are doing kind of like one to five

163
00:11:03,920 --> 00:11:05,838
agents today up to, you know,

164
00:11:05,839 --> 00:11:09,310
hundreds of agents running at once and how that sort of functions.

165
00:11:09,360 --> 00:11:14,719
And so I'd love to hear about your metaphor of the Michelin Kitchen instead of the software

166
00:11:14,720 --> 00:11:18,559
factory because I think that says a bit about the way you think about this stuff.

167
00:11:18,560 --> 00:11:19,760
>> Yes.

168
00:11:20,079 --> 00:11:24,430
Yeah. I I I've I've never really liked the term software factory.

169
00:11:24,480 --> 00:11:29,679
Not because you know it's not accurate but I think I think the a lot of people when they

170
00:11:29,680 --> 00:11:36,239
think factory right they don't necessarily equate that with quality or craft right things

171
00:11:36,240 --> 00:11:41,119
which are very important to me and a lot of people and technologists who work you know

172
00:11:41,120 --> 00:11:46,990
building products we care about the user experience we care about the things we're building.

173
00:11:47,040 --> 00:11:51,439
So while so while I do think software factory is an apt term,

174
00:11:51,440 --> 00:11:57,039
it also I guess maybe conjures up negative, you know, maybe sometimes negative connotations.

175
00:11:57,040 --> 00:12:02,398
So Michelin Kitchen is the thing that I've sort of landed on where it's much more I feel

176
00:12:02,399 --> 00:12:06,958
like it's much more aspirational and uh I like the metaphor a lot cuz you know I like

177
00:12:06,959 --> 00:12:11,359
food. I'm called potato of course [snorts] and I like cooking [laughter] and I see a

178
00:12:11,360 --> 00:12:16,559
lot of parallels right like with food right when you're cooking a meal for yourself for

179
00:12:16,560 --> 00:12:20,799
example [clears throat] it's both utilitarian like you're trying to just feed yourself

180
00:12:20,800 --> 00:12:26,239
right and and survive uh but it can actually be transformed into art right and that's

181
00:12:26,240 --> 00:12:32,638
what what a Michelin starred chef or even just a chef or a cook can do with food is take

182
00:12:32,639 --> 00:12:37,919
something very ordinary and turn it into a delicious meal that you know takes you back

183
00:12:37,920 --> 00:12:40,589
to your childhood days or something like that.

184
00:12:40,639 --> 00:12:47,278
Um and so it almost like mirrors that trust letter that I talk about where uh you can

185
00:12:47,279 --> 00:12:50,479
sort of imagine your own journey as a home cook, right?

186
00:12:50,480 --> 00:12:54,510
Uh as a home cook, you are doing all of the food, the cooking yourself.

187
00:12:54,560 --> 00:12:59,039
You cut all the vegetables, you do all the prep work, you do all the cleanup,

188
00:12:59,040 --> 00:13:04,830
you know, you are the one man or one woman show really.

189
00:13:04,880 --> 00:13:08,879
Um, and it's an interesting thought experiment like, okay,

190
00:13:08,880 --> 00:13:12,479
if you were to cook a meal and then you add people, right,

191
00:13:12,480 --> 00:13:16,559
your your your partner trying to your brother, your sister,

192
00:13:16,560 --> 00:13:18,879
and now suddenly you have your whole family in the kitchen.

193
00:13:18,880 --> 00:13:21,439
I think most people would get very stressed by that, right?

194
00:13:21,440 --> 00:13:24,799
The thought of, oh, so many people are just mocking around in my kitchen.

195
00:13:24,800 --> 00:13:27,439
They have no no idea where all the utensils are.

196
00:13:27,440 --> 00:13:29,950
>> I have a max capacity of one person in the kitchen.

197
00:13:30,000 --> 00:13:30,719
Yeah, absolutely.

198
00:13:30,720 --> 00:13:35,278
So, I feel like that that's really apt because when you ask yourself that question of

199
00:13:35,279 --> 00:13:38,398
how do I go from being a solo cook, right,

200
00:13:38,399 --> 00:13:44,159
to having an army or even not not even an army but a few sue chefs, right,

201
00:13:44,160 --> 00:13:45,518
that that are helping me in the kitchen.

202
00:13:45,519 --> 00:13:50,239
How do I think about dividing the work in a way that makes sense?

203
00:13:50,240 --> 00:13:52,719
You know, I'm not dividing work just for the sake of it,

204
00:13:52,720 --> 00:13:57,599
but in a way that actually makes the sum the to the better than, you know,

205
00:13:57,600 --> 00:13:59,630
the total of its parts.

206
00:13:59,680 --> 00:14:05,119
And so the Michelin kitchen metaphor to me like works really well in that regard because

207
00:14:05,120 --> 00:14:11,119
you know as a chef you're you know if you become a chef you're in a position where you're

208
00:14:11,120 --> 00:14:16,398
not necessarily cooking all the food yourself anymore but you are thinking you're almost

209
00:14:16,399 --> 00:14:21,439
like the tech lead right for the kitchen where uh you know chefs have to think about

210
00:14:21,440 --> 00:14:25,278
you know not just cooking but they have to basically organize the whole kitchen and they're

211
00:14:25,279 --> 00:14:29,919
like the CEO of the kitchen they have to think about when do you order ingredients,

212
00:14:29,920 --> 00:14:33,919
how do you store them, how do you prepare them, when do they have to be prepared,

213
00:14:33,920 --> 00:14:35,919
you know, it's a whole it's a whole job, right?

214
00:14:35,920 --> 00:14:37,550
That's not just cooking.

215
00:14:37,600 --> 00:14:44,078
Um, and I think that again it mirrors so much of how engineers write code today where

216
00:14:44,079 --> 00:14:46,078
you are not writing the code yourself anymore.

217
00:14:46,079 --> 00:14:47,439
You have agents, right?

218
00:14:47,440 --> 00:14:51,759
But you as the human are still responsible for the final outcome, right?

219
00:14:51,760 --> 00:14:54,879
your name still is associated with the work that you do,

220
00:14:54,880 --> 00:15:01,439
your reputation and you know so how you set up your kitchen right and how you set up

221
00:15:01,440 --> 00:15:07,599
your skills your environment your codebase I think are ultimately the new ingredients

222
00:15:07,600 --> 00:15:11,839
that go into um building product

223
00:15:12,560 --> 00:15:16,719
>> yeah I think what I love about your approach is the amount of focus that you put into

224
00:15:16,720 --> 00:15:21,119
the environment that the agent operates in right because I think a lot of people they

225
00:15:21,120 --> 00:15:23,439
think, right, the agent is good.

226
00:15:23,440 --> 00:15:26,319
I'm probably not going to be able to make it better.

227
00:15:26,320 --> 00:15:31,119
Let's just trust what these magic model people have put into the harness and the model

228
00:15:31,120 --> 00:15:33,919
combination. Uh, there's nothing I can really do,

229
00:15:33,920 --> 00:15:37,999
like I can't mess about with claw codes internals or something or whatever you're using.

230
00:15:38,000 --> 00:15:43,518
Um, but what I love about your approach, and it's something I advocate for too,

231
00:15:43,519 --> 00:15:46,958
is that you can change the environment the agent operates in, right?

232
00:15:46,959 --> 00:15:53,198
you can make changes in the codebase and also give it tools for verification as well

233
00:15:53,199 --> 00:15:55,599
and allow it to verify its own work.

234
00:15:55,600 --> 00:16:01,390
So the thing I I loved about watching that talk is the amount of focus you put in verification

235
00:16:01,440 --> 00:16:05,119
and like that is the lever that you can start to generate trust.

236
00:16:05,120 --> 00:16:07,919
Can you talk about that and what that concretely looks like?

237
00:16:07,920 --> 00:16:12,270
Let's start like looking at practical ways that people can improve their own processes,

238
00:16:12,320 --> 00:16:13,839
their own kitchens.

239
00:16:14,480 --> 00:16:19,599
Yeah, I've I've I've said this a lot actually that you know even if you don't use PAC

240
00:16:19,600 --> 00:16:25,518
or you know your skills I think that the single most important skill that should be in

241
00:16:25,519 --> 00:16:32,078
your toolkit is verification because without verification and for for by the way for

242
00:16:32,079 --> 00:16:36,078
those watching who don't know what that means it's this idea that you can give you can

243
00:16:36,079 --> 00:16:43,838
sort of give your agent uh hands and eyes in a way that's the the analogy I where

244
00:16:43,839 --> 00:16:47,039
the agent is able to run the code, right?

245
00:16:47,040 --> 00:16:52,319
And actually >> [snorts] >> uh interact with it like a normal human user would and also

246
00:16:52,320 --> 00:16:57,360
do things like you know debug it, you know, take traces and snapshots.

247
00:16:57,600 --> 00:17:01,919
Um and uh funnily enough like that was actually the first skill I built when I joined

248
00:17:01,920 --> 00:17:06,318
Cursor. uh that gave me a lot of that was that was the thing that actually started to

249
00:17:06,319 --> 00:17:10,879
let me ascend the trust ladder a little bit in a way that some of the other skills I

250
00:17:10,880 --> 00:17:17,519
had looked at or built had not really let me do because no matter how good you know some

251
00:17:17,520 --> 00:17:22,879
of the other skills were like the how skill, the why skill, the unsop skill were,

252
00:17:22,880 --> 00:17:29,070
I was still relying on me right as the proxy between my agent and the output.

253
00:17:29,120 --> 00:17:33,279
So that you know if the agent can't actually see the result of its work there's no way

254
00:17:33,280 --> 00:17:37,999
it can actually iterate right and so this is where people start to talk about loops this

255
00:17:38,000 --> 00:17:45,519
idea of a loop and really I think the term loop you know seems kind of uh almost abstract

256
00:17:45,520 --> 00:17:49,839
like people like what what what is a loop what is an agent loop but really to me like

257
00:17:49,840 --> 00:17:55,519
the most important part of a loop that allows it to be a loop is the verification part

258
00:17:55,520 --> 00:18:02,239
because the agent is able to to verify by its own work and uh you know that takes you

259
00:18:02,240 --> 00:18:06,879
out of the equation where now I can actually do something like so the very one of the

260
00:18:06,880 --> 00:18:10,879
very first use cases I had for verification was you know like the performance work that

261
00:18:10,880 --> 00:18:15,359
I was doing on cursors agent window and I want I wanted to get to a point where I could

262
00:18:15,360 --> 00:18:21,038
do something called hill climbing uh which is a term that I I think the labs uh talk

263
00:18:21,039 --> 00:18:25,999
about a lot which is this idea that you know you have some kind of rubric or a way to

264
00:18:26,000 --> 00:18:30,159
judge or score something And now because you have a loop,

265
00:18:30,160 --> 00:18:34,670
you can have an agent continually try to make improvements to that.

266
00:18:34,720 --> 00:18:36,239
Uh I think Carpathy,

267
00:18:36,240 --> 00:18:41,599
Andre Carpathy also famously released uh something called auto research that has a lot

268
00:18:41,600 --> 00:18:43,119
of these ideas.

269
00:18:43,679 --> 00:18:49,839
Um but yeah, verification I would say is probably the most important skill in PAC uh

270
00:18:49,840 --> 00:18:52,910
and many [clears throat] other you know tool sets.

271
00:18:52,960 --> 00:18:57,359
Uh and I think it's the most important thing to focus on.

272
00:18:57,360 --> 00:19:03,790
So a lot of the a lot of I spent a lot of time actually you know tuning the verification

273
00:19:03,840 --> 00:19:10,399
the creative verification skill um and also internally the the we have so many verification

274
00:19:10,400 --> 00:19:17,038
skills now like every app that cursor has or spaceexai has has a uh verification skill

275
00:19:17,039 --> 00:19:21,678
that is automaintained as well [clears throat] >> uh and it's become critical infrastructure

276
00:19:21,679 --> 00:19:26,399
for our team because everybody uses it >> and you went pretty far with that too right

277
00:19:26,400 --> 00:19:31,119
like you had a um in your talk I saw that you actually built a custom CLI for that too.

278
00:19:31,120 --> 00:19:32,558
So what does that CLI do?

279
00:19:32,559 --> 00:19:35,999
Like how does it [clears throat] execute things and why did you I mean that's proof of

280
00:19:36,000 --> 00:19:39,440
how deep you're going right of how much you're pushing that.

281
00:19:39,760 --> 00:19:45,200
>> Yeah. So this is actually a tip I learned early on where

282
00:19:45,840 --> 00:19:52,558
um I guess you know back in January or or late last year the thing that people were concerned

283
00:19:52,559 --> 00:19:54,319
about was context window, right?

284
00:19:54,320 --> 00:19:59,839
That was the big the big topic at the time was how do I you know manage the context window

285
00:19:59,840 --> 00:20:04,879
because you know compaction summarization wasn't really that good yet [snorts] and people

286
00:20:04,880 --> 00:20:09,519
were always people had this there was almost this meme in the community that you know

287
00:20:09,520 --> 00:20:15,839
once your agent summarized or compacted once it would become sort of stupid right for

288
00:20:15,840 --> 00:20:17,390
the rest of your session.

289
00:20:17,440 --> 00:20:20,480
So there was a lot of thinking around like

290
00:20:20,799 --> 00:20:25,839
you know being very efficient with your context usage and so that was actually the inspiration

291
00:20:25,840 --> 00:20:32,190
for some of the uh the CLI work inside of the verification skills.

292
00:20:32,240 --> 00:20:38,959
I guess now it's less so about context because uh you know agents are much better or

293
00:20:38,960 --> 00:20:41,999
harnesses have gotten a lot better with summarization.

294
00:20:42,400 --> 00:20:49,950
Um, I still think there's some benefits to, you know, uh, having a clean context window.

295
00:20:50,000 --> 00:20:55,359
Uh, so the CLI is really just more of a way for me to take the deterministic parts of

296
00:20:55,360 --> 00:21:02,558
what the skill does and encode that into a script or CLI to reduce to kind of take away

297
00:21:02,559 --> 00:21:09,599
the judgment that would otherwise unnecessarily be used because with judgment so

298
00:21:09,600 --> 00:21:15,119
I also think of you know agents and skills in sort of like it's like a gradient

299
00:21:15,760 --> 00:21:21,119
you have some parts of the work that are entirely judge measurement based right you know

300
00:21:21,120 --> 00:21:25,359
something that requires thought you know putting together multiple pieces of context

301
00:21:25,360 --> 00:21:29,999
thinking um and then you have the more deterministic parts like I don't know if you wanted

302
00:21:30,000 --> 00:21:36,719
to uh refactor some code right from one pattern to another that's very mechanical right

303
00:21:36,720 --> 00:21:42,639
you don't you don't need an agent to think about it and come up with it in a novel way

304
00:21:42,640 --> 00:21:47,279
each time right and so that that was really the inspiration for the CLI and you'll see

305
00:21:47,280 --> 00:21:52,798
this in a lot of the other skills that I built is like I try to extract out the deterministic

306
00:21:52,799 --> 00:21:58,158
parts and turn that into code and just leave only the parts that actually require judgment

307
00:21:58,159 --> 00:22:02,959
to the agent. So in a way I think of the seal as kind of like your wrapper, right?

308
00:22:02,960 --> 00:22:07,119
It's a wrapper with some light instructions around how to use these custom tools that

309
00:22:07,120 --> 00:22:09,070
are inside of the skill.

310
00:22:09,120 --> 00:22:13,519
Um but yeah, I don't think the CL is really that interesting in its own really.

311
00:22:13,520 --> 00:22:15,839
It's not like a novel piece of software.

312
00:22:15,840 --> 00:22:20,639
It's just something that interacts with like Playright and the Chrome DevTools protocol

313
00:22:20,640 --> 00:22:22,558
and calls a bunch of APIs.

314
00:22:22,559 --> 00:22:24,079
It's like it's just a bunch of glue.

315
00:22:24,080 --> 00:22:27,439
>> No, it's fascinating because it's a way of hiding information from the skill, right?

316
00:22:27,440 --> 00:22:31,519
It's a way of conserving the skill, keeping the skill quite small, I imagine,

317
00:22:31,520 --> 00:22:36,079
and then you're able to delegate more of the complicated deterministic stuff into a script

318
00:22:36,080 --> 00:22:37,359
within the skill.

319
00:22:37,360 --> 00:22:43,199
So it's almost you're compressing information and making the agent do more consistent

320
00:22:43,200 --> 00:22:45,550
things more consistently.

321
00:22:45,600 --> 00:22:47,230
>> Yeah, >> that's fascinating.

322
00:22:47,280 --> 00:22:51,279
And it it [clears throat] also helps I guess if you care about context window it it does

323
00:22:51,280 --> 00:22:58,319
help because now the agent doesn't need to uh you know re reinvent uh things cuz uh one

324
00:22:58,320 --> 00:23:02,080
thing I had noticed early on when we didn't have a CLI

325
00:23:02,480 --> 00:23:07,839
was that uh well the the agent would try to verify it work but it would basically rebuild

326
00:23:07,840 --> 00:23:12,079
the world each time and then every agent did it differently and I was starting to notice

327
00:23:12,080 --> 00:23:16,558
like that's very inefficient right I was wasting it it was actually not just about context

328
00:23:16,559 --> 00:23:18,239
usage but also speed, right?

329
00:23:18,240 --> 00:23:22,158
Like because now an agent had to actually go off and write the scripts or the CLI and

330
00:23:22,159 --> 00:23:26,639
test it and you know and it doesn't work and the last agent did it and it worked but

331
00:23:26,640 --> 00:23:27,519
it discarded it.

332
00:23:27,520 --> 00:23:32,079
So it was just very obvious at that point like I should just turn this into a CLI and

333
00:23:32,080 --> 00:23:38,399
put that inside of the skill uh so that every agent that uses it now benefits from that

334
00:23:38,400 --> 00:23:44,079
same piece. Um but I I also think like you know it's a good push for people to think

335
00:23:44,080 --> 00:23:49,520
about is how much of your skills and rules could actually be deterministic.

336
00:23:50,480 --> 00:23:55,599
Um that's like another core thing or or one of my core principles that I like to think

337
00:23:55,600 --> 00:24:03,440
about is yeah how do I uh make very efficient use of determinism and non-determinism

338
00:24:04,080 --> 00:24:09,519
and you know let Asians shine at the non-deterministic parts right because that's what

339
00:24:09,520 --> 00:24:11,150
they're trained to do.

340
00:24:11,200 --> 00:24:16,798
Um and the other parts which are much more mechanical or you know straightforward can

341
00:24:16,799 --> 00:24:18,910
be just pure determinism.

342
00:24:18,960 --> 00:24:22,830
Um, and you'll see this as well for things like doing migrations.

343
00:24:22,880 --> 00:24:28,109
Um, which is another big thing that I've I've talked about is, you know,

344
00:24:28,159 --> 00:24:33,439
going from one technology to another, especially one that is better for agents, right?

345
00:24:33,440 --> 00:24:36,239
And a lot of how you can do that migration is, I think,

346
00:24:36,240 --> 00:24:41,550
through things like scripts and CLIs, like the deterministic parts like code mods,

347
00:24:41,600 --> 00:24:48,079
you know, like crawling the abstract syntax tree and transforming code literally mechanically,

348
00:24:48,080 --> 00:24:51,600
right? like a script does it for you instead of the agent.

349
00:24:52,400 --> 00:24:53,439
>> Totally makes sense.

350
00:24:53,440 --> 00:25:00,558
I I mean I think what there's another thing there which is you're taking stuff away from

351
00:25:00,559 --> 00:25:06,239
the agent and you're kind of putting it in the environment too a little bit which is

352
00:25:06,240 --> 00:25:10,399
let's say you have a a thing that you notice the agent always gets wrong.

353
00:25:10,400 --> 00:25:14,749
You want to make that um just impossible within the environment.

354
00:25:14,799 --> 00:25:18,798
And that sort of comes down to code quality as well.

355
00:25:18,799 --> 00:25:24,639
I mean, I talk about a lot like having a what a good codebase means, right?

356
00:25:24,640 --> 00:25:26,079
What is a good codebase?

357
00:25:26,080 --> 00:25:30,239
And there's a definition I like which is a a good codebase is a codebase that's easy

358
00:25:30,240 --> 00:25:32,239
to make changes in, right?

359
00:25:32,240 --> 00:25:36,319
Easy to um change stuff without things screwing up.

360
00:25:36,320 --> 00:25:41,199
And that means that you have a lot of guard rails that you have a lot of um the agent

361
00:25:41,200 --> 00:25:44,158
or the human is constrained to very narrow paths.

362
00:25:44,159 --> 00:25:46,749
And that's again something you talk about in your talk.

363
00:25:46,799 --> 00:25:52,319
>> And you talk about this not only on the kind of sort of automated checks side of things.

364
00:25:52,320 --> 00:25:56,830
So linting and type checking blah blah blah but also in the way you design abstractions.

365
00:25:56,880 --> 00:26:01,790
And you guys even I think built a framework uh for your agent to work in too.

366
00:26:01,840 --> 00:26:07,038
>> I think what I'd love to hear is you obviously think of that as very important, right?

367
00:26:07,039 --> 00:26:11,918
And that's how important is that compared to other things you could be doing like building

368
00:26:11,919 --> 00:26:14,080
features or shipping work.

369
00:26:14,480 --> 00:26:19,839
Yeah, I think that's a um I almost feel like the new job of the engineer is really to

370
00:26:19,840 --> 00:26:22,190
to spend time on the environment.

371
00:26:22,240 --> 00:26:26,350
Um I almost actually wrote a tweet about this yesterday, but I but I didn't.

372
00:26:26,400 --> 00:26:28,160
But I think that

373
00:26:29,760 --> 00:26:33,278
I think that you you know if you if you haven't really spent time, you know,

374
00:26:33,279 --> 00:26:36,670
building trust in your agents and building skills and tools,

375
00:26:36,720 --> 00:26:41,119
you can get stuck in this mode where you're very low on that trust ladder, right?

376
00:26:41,120 --> 00:26:43,599
you don't have a lot of trust in your agents work.

377
00:26:43,600 --> 00:26:49,199
And so the only way to cope in that when you're in that situation is just to kind of

378
00:26:49,200 --> 00:26:51,759
lock in and micromanage your agents.

379
00:26:51,760 --> 00:26:53,390
And that's very time consuming.

380
00:26:53,440 --> 00:26:54,798
And when you're stuck in that mode,

381
00:26:54,799 --> 00:26:58,350
you don't really have the luxury to think about,

382
00:26:58,400 --> 00:27:04,989
you know, uh higher level things like like making yourself more productive.

383
00:27:05,039 --> 00:27:10,959
In the same way that uh I guess analogy would be like if you've never taken the time

384
00:27:10,960 --> 00:27:15,918
to learn like your tools right as a developer when you were writing code yourself and

385
00:27:15,919 --> 00:27:19,550
you know you've never heard of VS Code, you've never heard of Vim,

386
00:27:19,600 --> 00:27:24,079
you only knew about Notepad [laughter] uh and you had hadn't even heard about Git.

387
00:27:24,080 --> 00:27:25,038
That's sort of the analogy.

388
00:27:25,039 --> 00:27:29,599
It's like you you haven't spent the time sharpening your own knives, right?

389
00:27:29,600 --> 00:27:34,800
And so, of course, if you have a dull knife, then everything's going to take a long time.

390
00:27:34,960 --> 00:27:39,519
Um, and you're going to be you're just going to be and and especially if you know deadlines

391
00:27:39,520 --> 00:27:43,678
are looming, then you don't have the now you're stuck in this rut, right?

392
00:27:43,679 --> 00:27:48,639
Where where you you you don't have sharp knives, you don't have good tools,

393
00:27:48,640 --> 00:27:50,639
but you're under all this pressure to ship, right?

394
00:27:50,640 --> 00:27:53,470
And so, all you can do is just focus on that.

395
00:27:53,520 --> 00:27:54,959
But I do think that, you know,

396
00:27:54,960 --> 00:27:59,710
if you can find yourself the time to actually spend time thinking about your setup,

397
00:27:59,760 --> 00:28:02,639
it's again going back to the cooking, you know,

398
00:28:02,640 --> 00:28:06,879
analogy, it's like uh, you know, if you, for example,

399
00:28:06,880 --> 00:28:11,519
if if cutting cutting cutting the garlic is like super slow, right?

400
00:28:11,520 --> 00:28:13,038
There are garlic mashers, right?

401
00:28:13,039 --> 00:28:16,879
You can you can buy and you put it in the thing and you like squeeze it out, right?

402
00:28:16,880 --> 00:28:18,109
It's super fast.

403
00:28:18,159 --> 00:28:21,999
Uh, machines and tools were invented for a reason, right?

404
00:28:22,000 --> 00:28:27,599
And so if you're operating a Michelin kitchen and your your your cooks had no tools,

405
00:28:27,600 --> 00:28:31,599
then of course everything's going to be extremely inefficient, very very, you know,

406
00:28:31,600 --> 00:28:35,359
every every every cook is going to make something up of their own.

407
00:28:35,360 --> 00:28:41,439
So I think the tools and the determinism to me are you know taking that part away and

408
00:28:41,440 --> 00:28:44,079
and just like you said about constraints as well.

409
00:28:44,080 --> 00:28:47,280
It's the constraints are are to me as well like

410
00:28:48,240 --> 00:28:54,319
uh actually a slight tangent on that is uh I think we should talk about TypeScript cuz

411
00:28:54,320 --> 00:28:58,479
like we we actually both share like a background in Typescript where you know you obviously

412
00:28:58,480 --> 00:29:02,158
have done a lot of work with TypeScript and total TypeScript and you know you're a leader

413
00:29:02,159 --> 00:29:08,479
in that space and I uh had adopted TypeScript pretty early and I had given like a talk

414
00:29:08,480 --> 00:29:14,079
or two at Typescript conf uh many years ago and so one of the the talk that I did actually

415
00:29:14,080 --> 00:29:18,959
was about type systems and constraining the constraining types.

416
00:29:18,960 --> 00:29:23,519
Like one of my most favorite things about Typescript is actually type narrowing, right?

417
00:29:23,520 --> 00:29:26,558
This idea that you go from a very broad type, right?

418
00:29:26,559 --> 00:29:32,959
That could be anything and then you through type guards and you know type narrowing and

419
00:29:32,960 --> 00:29:36,959
you know runtime checks you can actually narrow the space and say like oh this isn't

420
00:29:36,960 --> 00:29:40,158
just a string this is a very special type of string.

421
00:29:40,159 --> 00:29:41,119
It's a constant, right?

422
00:29:41,120 --> 00:29:46,879
like I but I I determine that through the type system and in a way it's like uh there's

423
00:29:46,880 --> 00:29:52,639
a lot of parallels I think to that with constraints in your codebase where is it you're

424
00:29:52,640 --> 00:29:58,319
you're constraining the space right if you if you think about category theory as well

425
00:29:58,320 --> 00:30:04,560
you know you're constraining the the number of possible types right that can can exist

426
00:30:05,039 --> 00:30:10,398
and you're saying there's only one type right and for for us like that framework that

427
00:30:10,399 --> 00:30:12,270
I'm called Dune.

428
00:30:12,320 --> 00:30:14,479
Uh it's not an open source framework.

429
00:30:14,480 --> 00:30:17,038
It's the the way I describe it to people.

430
00:30:17,039 --> 00:30:22,190
It's it's kind of like a internal Nex.js for our Electron apps.

431
00:30:22,240 --> 00:30:27,439
Uh but it comes with a lot of really really restrictive lit rules and the codebase is

432
00:30:27,440 --> 00:30:31,119
designed in a way that there's really only one way to do something.

433
00:30:31,120 --> 00:30:35,200
So we make use a we make use of a lot of conventional

434
00:30:35,520 --> 00:30:40,270
patterns. So like features all go into a specific directory.

435
00:30:40,320 --> 00:30:42,158
Well, every feature has its own directory.

436
00:30:42,159 --> 00:30:43,599
As an example, you know,

437
00:30:43,600 --> 00:30:48,239
there's like a a thing that discovers features like through a registry and like crawling

438
00:30:48,240 --> 00:30:50,509
the codebase and stuff like that.

439
00:30:50,559 --> 00:30:57,759
But this conventional pattern and the lint rules make for an environment where it's actually

440
00:30:57,760 --> 00:31:00,720
very hard to write bad code.

441
00:31:01,520 --> 00:31:08,989
And that sort of frees up the it both frees up your own mental uh you know

442
00:31:09,039 --> 00:31:14,319
capacity as well as the agent sort of doesn't have to think about that anymore where

443
00:31:14,320 --> 00:31:17,999
it's just like oh there's only there's I should just if I want to add a new feature it

444
00:31:18,000 --> 00:31:21,759
just goes in the feature the new feature directory and all the code goes in there and

445
00:31:21,760 --> 00:31:26,639
I'm not going to append to a god file right that was really actually the inspiration

446
00:31:26,640 --> 00:31:33,599
for those feature directories is the very first couple of versions of Grockbot were composed

447
00:31:33,600 --> 00:31:39,918
of like eight god files which were like at least 10,000 lines long if not longer and

448
00:31:39,919 --> 00:31:43,280
so I kind of had to break it up into smaller pieces.

449
00:31:43,840 --> 00:31:48,158
Uh but it was just observing you know actually that's another important part is observing

450
00:31:48,159 --> 00:31:53,599
how agents fail and then every time you see a mistake every time you see something that

451
00:31:53,600 --> 00:31:58,319
could be done better you think you step back and think how do I turn this into a lint

452
00:31:58,320 --> 00:32:02,158
rule? How do I make it so that the code base makes this impossible?

453
00:32:02,159 --> 00:32:07,119
Right? And it comes back to me for my you know my background learning Typescript and

454
00:32:07,120 --> 00:32:14,158
types uh type systems is how do I constrain the space so that you know I

455
00:32:14,159 --> 00:32:18,239
know precisely what I'm working with and I think yeah there's a lot of parallels there.

456
00:32:18,240 --> 00:32:19,359
>> Totally makes sense.

457
00:32:19,360 --> 00:32:23,678
And don't I mean it's funny that you mentioned TypeScript and Goth files in the same

458
00:32:23,679 --> 00:32:28,640
sentence because Typescript famously has a 25,000line type uh file.

459
00:32:29,302 --> 00:32:32,398
>> [laughter] >> Although I don't know if they've rewritten that and go as they probably

460
00:32:32,399 --> 00:32:33,439
have, haven't they?

461
00:32:33,440 --> 00:32:34,640
Um,

462
00:32:34,960 --> 00:32:37,839
okay. So, environment is important.

463
00:32:37,840 --> 00:32:42,239
You should watch your agent like a hawk to make sure that any mistakes it makes.

464
00:32:42,240 --> 00:32:44,719
You turn them into things in the environment.

465
00:32:44,720 --> 00:32:48,558
And the benefit of the environment is you're not overloading your agent, right,

466
00:32:48,559 --> 00:32:51,439
in terms of rules, in terms of things it has to remember.

467
00:32:51,440 --> 00:32:52,639
It's just in the environment.

468
00:32:52,640 --> 00:32:57,839
And so it stumbles into the rules and exactly um you know bounces off them and hits them

469
00:32:57,840 --> 00:32:59,070
at the right moment.

470
00:32:59,120 --> 00:33:02,719
>> So okay, we still haven't talked about the 2,500 PRs.

471
00:33:02,720 --> 00:33:03,759
Where do those come from?

472
00:33:03,760 --> 00:33:08,079
Like how do you you've built your trust ladder, you've worked on your environment,

473
00:33:08,080 --> 00:33:12,270
and you understand, okay, um I now want to scale up.

474
00:33:12,320 --> 00:33:15,629
So what are the mechanics of that scaling?

475
00:33:15,679 --> 00:33:19,040
Are you um initiating 2500

476
00:33:19,360 --> 00:33:21,119
like chats per month?

477
00:33:21,120 --> 00:33:21,999
That can't be right.

478
00:33:22,000 --> 00:33:27,119
So there must be are there any kind of automated triggers that trigger stuff in your

479
00:33:27,120 --> 00:33:32,509
repo? Like how do you get the software factory kind of triggering work by itself?

480
00:33:32,559 --> 00:33:33,759
>> Right.

481
00:33:34,240 --> 00:33:41,199
Um I'll definitely say that the prerequisite to you know something like a very high volume

482
00:33:41,200 --> 00:33:43,200
of of pull requests

483
00:33:43,600 --> 00:33:45,599
um is the environment.

484
00:33:45,600 --> 00:33:49,278
you know, the the stuff we just talked about where I definitely would not have been able

485
00:33:49,279 --> 00:33:53,278
to do this if I had not spent the time, you know, thinking about the kitchen,

486
00:33:53,279 --> 00:33:56,430
right, and the knives and the tools for my Asians.

487
00:33:56,480 --> 00:33:58,158
And so, in a way,

488
00:33:58,159 --> 00:34:04,109
I I think of this as I've spent the time building one kitchen and one restaurant.

489
00:34:04,159 --> 00:34:10,270
And now I I'm in a position where I don't actually have to be there anymore because the

490
00:34:10,320 --> 00:34:12,959
environment, you know, that the same analogy, right?

491
00:34:12,960 --> 00:34:14,102
It works really well.

492
00:34:14,103 --> 00:34:14,559
[laughter] Yeah.

493
00:34:14,560 --> 00:34:16,559
You open chain of restaurants, right?

494
00:34:16,560 --> 00:34:17,678
That's >> Yeah. Exactly.

495
00:34:17,679 --> 00:34:20,959
Yeah. Exactly. It's like you're Gordon Ramsay and you know,

496
00:34:20,960 --> 00:34:26,239
you you've taught your executive chef like all the tricks of the of coming up with great

497
00:34:26,240 --> 00:34:29,439
menu. Uh and like the kitchen is set up really well.

498
00:34:29,440 --> 00:34:33,439
Everything's just perfect and you're now in a position where you can open your second

499
00:34:33,440 --> 00:34:34,990
your third restaurant.

500
00:34:35,040 --> 00:34:39,439
And [snorts] I guess I I sort of see each project that I work on,

501
00:34:39,440 --> 00:34:42,878
like each big chat is sort of like a restaurant, right?

502
00:34:42,879 --> 00:34:48,559
and and I'm I'm I have multiple of them operating at the same time and I'm sort of like

503
00:34:48,560 --> 00:34:54,158
helicoptering between them sometimes some more than others depending on how in the loop

504
00:34:54,159 --> 00:35:00,479
I am but yeah definitely I think there's there there are external triggers and context

505
00:35:00,480 --> 00:35:05,120
that those projects don't have that

506
00:35:05,680 --> 00:35:10,799
for a long time I was the proxy for that so uh the best example I have is like you know

507
00:35:10,800 --> 00:35:15,838
you have a project that's working on a feature uh or you're trying to fix a bug and you're

508
00:35:15,839 --> 00:35:16,879
getting bug reports,

509
00:35:16,880 --> 00:35:22,959
but the bug reports are going to things like Slack or linear or X, right?

510
00:35:22,960 --> 00:35:27,919
And these are external systems that aren't connected to your inner loop.

511
00:35:27,920 --> 00:35:32,000
So, I like to talk about this outer loop and the inner loop.

512
00:35:32,320 --> 00:35:35,838
Uh I don't know if I'm using the definition correctly but to me my inner loop is like

513
00:35:35,839 --> 00:35:43,279
basically my engineers my agent engineers working on the code to an building towards

514
00:35:43,280 --> 00:35:49,999
an intent or snapshot of my intent right and the thing about that is that the snapshot

515
00:35:50,000 --> 00:35:55,519
can go stale right new information comes to light that I then have to be the proxy of

516
00:35:55,520 --> 00:36:00,959
and you know transfer that context to my agent so you know if you if you don't have these

517
00:36:00,960 --> 00:36:04,190
triggers pulling information back into the interloop,

518
00:36:04,240 --> 00:36:07,200
then you sort of have to play that role

519
00:36:07,520 --> 00:36:09,118
where you're you're off, you know,

520
00:36:09,119 --> 00:36:14,078
in Slack or X or or whatever and you're gathering context, right?

521
00:36:14,079 --> 00:36:18,639
You're getting context about bug reports, about feature requests, about, you know,

522
00:36:18,640 --> 00:36:20,319
something someone said about, you know,

523
00:36:20,320 --> 00:36:25,230
our backend infrastructure has some limitation, you know, all that information,

524
00:36:25,280 --> 00:36:28,590
you have to f that across to your agent.

525
00:36:28,640 --> 00:36:33,838
So that's where I think like tools like Grogbot are really good because they help you

526
00:36:33,839 --> 00:36:36,239
automate the outer loop as well.

527
00:36:36,240 --> 00:36:38,319
And when you connect those two loops,

528
00:36:38,320 --> 00:36:43,519
it's very very powerful because now all of a sudden your agents have the ability to get

529
00:36:43,520 --> 00:36:46,479
context for this for themselves, right?

530
00:36:46,480 --> 00:36:48,000
If for example

531
00:36:48,880 --> 00:36:54,430
uh you know either through just as a simple example like maybe you have the Slack MCP,

532
00:36:54,480 --> 00:36:57,519
right? Or you have uh your own harness, right,

533
00:36:57,520 --> 00:37:02,190
that you've built a Slack subscription into for a particular Slack channel.

534
00:37:02,240 --> 00:37:06,799
Now all of a sudden you can tell your agents, okay, subscribe to the Slack channel.

535
00:37:06,800 --> 00:37:11,919
Every time there's a uh, you know, bug report about something,

536
00:37:11,920 --> 00:37:14,319
go off and triage that thing, right?

537
00:37:14,320 --> 00:37:16,239
Go reproduce the issue, right?

538
00:37:16,240 --> 00:37:20,879
Using the verification skills that we've already spent time building and all of those

539
00:37:20,880 --> 00:37:23,999
other skills that we've set up so that I have a lot of trust, right?

540
00:37:24,000 --> 00:37:28,879
I have a lot of trust that these agents can actually go off and understand the bug,

541
00:37:28,880 --> 00:37:35,200
you know, uh verify that the bug actually still exists on main and it wasn't

542
00:37:35,599 --> 00:37:41,358
something about you maybe the users setup or their data or maybe I don't know they didn't

543
00:37:41,359 --> 00:37:49,069
install a dependency or something like that like uh basically I think uh creating that

544
00:37:49,119 --> 00:37:53,519
yeah creating those two loops and connecting them is really a very important part of

545
00:37:53,520 --> 00:37:54,959
the job these days.

546
00:37:54,960 --> 00:37:58,720
Um, especially if you are thinking about how to scale yourself.

547
00:37:58,960 --> 00:38:04,959
So, a big theme here is really just like always thinking about like what where am I the

548
00:38:04,960 --> 00:38:06,719
bottleneck in this process?

549
00:38:06,720 --> 00:38:09,999
Why do my agents need me, you know, to answer this question?

550
00:38:10,000 --> 00:38:11,919
I I always like to think about that.

551
00:38:11,920 --> 00:38:16,559
And so I try to think about how do I actually get the agent to answer its own question,

552
00:38:16,560 --> 00:38:21,230
right? But not by hallucinating, not by guessing, but actually real data.

553
00:38:21,280 --> 00:38:24,959
And you know, a lot of people talk about this idea of a company brain, right,

554
00:38:24,960 --> 00:38:26,559
or a context graph.

555
00:38:26,560 --> 00:38:28,960
I feel like those terms are unnecessarily

556
00:38:29,280 --> 00:38:33,069
complex uh or even abstract.

557
00:38:33,119 --> 00:38:39,199
To me, it's just about um how do I take information that my agent needs that I would

558
00:38:39,200 --> 00:38:44,799
otherwise have to go and pass it myself and just teach it how to do it, right?

559
00:38:44,800 --> 00:38:47,358
And that removes me from the equation.

560
00:38:47,359 --> 00:38:53,999
And so how I arrive at 2,000 or however many PRs is the fact that I have all these loops

561
00:38:54,000 --> 00:39:00,479
set up, right? And so uh it allows me to open chain restaurants, right?

562
00:39:00,480 --> 00:39:03,230
I can I can really parallels myself.

563
00:39:03,280 --> 00:39:06,159
So yeah, I'm not sitting there creating 2,500 chats, right?

564
00:39:06,160 --> 00:39:12,159
Of course, it's really like these projects are um actually cursor has a new feature called

565
00:39:12,160 --> 00:39:15,309
projects which are these like coordinator agents.

566
00:39:15,359 --> 00:39:21,519
Um and so the coordination co coordinator agents are really good at sort of delegating

567
00:39:21,520 --> 00:39:26,479
and not doing work of their own but they manage and supervise like almost a list of tasks

568
00:39:26,480 --> 00:39:28,719
and they spawn sub agents to go and do them.

569
00:39:28,720 --> 00:39:33,118
And so I'm just constantly feeding context or teaching the agents how to get their own

570
00:39:33,119 --> 00:39:36,670
context and then they're going off and doing the work for me.

571
00:39:36,720 --> 00:39:41,199
Uh and really the big the last thing I'll say to this is like the big unlock for me for

572
00:39:41,200 --> 00:39:47,999
getting to 2,000 PRs is starting from the question and working backwards of how do I

573
00:39:48,000 --> 00:39:51,630
get to the point where my agent can merge its own code?

574
00:39:51,680 --> 00:39:52,880
Because

575
00:39:53,359 --> 00:39:56,719
the obvious thing people ask me when when they when I tell them, "Oh,

576
00:39:56,720 --> 00:39:59,679
I shipped 2,000 and 2,500 pull requests last month."

577
00:39:59,680 --> 00:40:01,919
They'll be like, "How did you review that?"

578
00:40:01,920 --> 00:40:04,078
Right? That that's a lot of PRs to review.

579
00:40:04,079 --> 00:40:05,679
Like your team must hate you.

580
00:40:05,680 --> 00:40:07,358
>> Do do you mind if we go there in a second?

581
00:40:07,359 --> 00:40:09,519
Because a good question about that.

582
00:40:09,520 --> 00:40:10,399
>> Yeah. Yeah. Yeah.

583
00:40:10,400 --> 00:40:12,078
>> I want to like this analogy is great.

584
00:40:12,079 --> 00:40:17,759
I want to like deepen it a bit which is before if you're like manually initiating all

585
00:40:17,760 --> 00:40:22,239
those chats it's like you're bringing the orders to your chefs manually right whereas

586
00:40:22,240 --> 00:40:26,799
if you've got an agent sort of like doing the expo then you're able to sort of run it

587
00:40:26,800 --> 00:40:31,519
yourself itself what is what does that concretely look like then you've got these sort

588
00:40:31,520 --> 00:40:36,639
of grock bots that are um subscribing to channels pulling in Slack messages and you it

589
00:40:36,640 --> 00:40:42,479
sounds like have a couple of coordinator agents or like chief of staff agents that like

590
00:40:42,480 --> 00:40:46,879
monitor that or something like when you look at your computer to manage your agents,

591
00:40:46,880 --> 00:40:48,960
what does it look like?

592
00:40:49,760 --> 00:40:54,078
>> Yeah. So, [clears throat] so uh this is I guess somewhat confusing but we're working

593
00:40:54,079 --> 00:41:00,559
on you know simplifying and unifying but so uh there's graphbot uh which or you know

594
00:41:00,560 --> 00:41:04,719
you can use other tools of course as well but I I largely think of these tools as like

595
00:41:04,720 --> 00:41:05,598
your outer loop.

596
00:41:05,599 --> 00:41:10,639
These are tools like you know Grabbot that have connectors right these are connectors

597
00:41:10,640 --> 00:41:15,519
I guess they a lot of people call them personal agents um but they're connectors to things

598
00:41:15,520 --> 00:41:22,670
like your email your calendar slack uh plaid I don't know like all these different services

599
00:41:22,720 --> 00:41:29,759
and they are a great source of pulling context in to your work so the same way

600
00:41:29,760 --> 00:41:35,358
that a human like you know if I were if I was a manager and I was leading a team of engineers

601
00:41:35,359 --> 00:41:39,199
years. Um, you know, like when I used to work in Netflix,

602
00:41:39,200 --> 00:41:43,519
one of the biggest things that managers would talk about was this idea of context not

603
00:41:43,520 --> 00:41:46,559
control, which funnily enough, you know,

604
00:41:46,560 --> 00:41:51,549
has so much uh has so much uh carry over to the agents world.

605
00:41:51,599 --> 00:41:53,919
Uh, of you know, you you know,

606
00:41:53,920 --> 00:41:57,919
you you of course can drive to an outcome you want by control, right?

607
00:41:57,920 --> 00:42:02,479
Like by micromanaging, but what you want is to provide context instead, right?

608
00:42:02,480 --> 00:42:03,679
like teach the agent,

609
00:42:03,680 --> 00:42:08,479
teach your engineers how to be self-sufficient and then you don't have to micromanage

610
00:42:08,480 --> 00:42:11,999
them. >> Um, and so I see a lot of parallels there.

611
00:42:12,000 --> 00:42:13,549
Uh, but yeah, graphbot.

612
00:42:13,599 --> 00:42:17,999
So, concretely, I have some graph bots that look at my Slack channels,

613
00:42:18,000 --> 00:42:23,039
look at my X, uh, or my emails, uh, or linear,

614
00:42:23,040 --> 00:42:26,269
and they're just constantly they have routines that subscribe.

615
00:42:26,319 --> 00:42:31,279
So they're constantly watching and I have I I'll tell them things like you know uh I'll

616
00:42:31,280 --> 00:42:36,879
watch for issues with uh bugs in the graphbot desktop app as an example.

617
00:42:36,880 --> 00:42:41,279
Uh and whenever you find that send it to my cursor project.

618
00:42:41,280 --> 00:42:45,069
So one of the really cool things about grabbot is it connects to cursor.

619
00:42:45,119 --> 00:42:50,590
So cursor has uh like I I just mentioned this new feature called projects.

620
00:42:50,640 --> 00:42:56,399
And a project is really a uh again like a you get a coordinator agent that's in the cloud.

621
00:42:56,400 --> 00:43:00,959
It has its own computer and all it really does is like it's a manager of agents.

622
00:43:00,960 --> 00:43:02,559
It's like your executive chef, right?

623
00:43:02,560 --> 00:43:04,239
Your your chief of staff.

624
00:43:04,240 --> 00:43:06,190
It doesn't do the work itself.

625
00:43:06,240 --> 00:43:13,630
It delegates and orchestrates and manages the work of other sub agents to you know

626
00:43:13,680 --> 00:43:18,110
that report to your chief your chief uh of staff.

627
00:43:18,160 --> 00:43:25,069
And it basically is responsible for driving the work forward and managing things and

628
00:43:25,119 --> 00:43:27,118
uh passing context to them.

629
00:43:27,119 --> 00:43:28,719
>> So if you get a sudden burst of issues,

630
00:43:28,720 --> 00:43:33,039
let's say you get 30 issues at once in one payload or something or very quickly the coordinator

631
00:43:33,040 --> 00:43:34,719
agent can figure it out and delegate.

632
00:43:34,720 --> 00:43:35,199
>> Yeah, exactly.

633
00:43:35,200 --> 00:43:40,559
It gets like uh you know 30 the 30 or so payloads and spawns a sub agent or a single

634
00:43:40,560 --> 00:43:41,439
coordinator agent.

635
00:43:41,440 --> 00:43:46,159
It can actually do a bunch of different topologies of agents and it will sort of figure

636
00:43:46,160 --> 00:43:53,358
out the best way to uh you know efficiently distribute the tasks to your team

637
00:43:53,359 --> 00:44:00,559
of agents. Um so I use uh cursor projects a lot um and I also use

638
00:44:00,560 --> 00:44:05,919
grapot a lot and cursor projects are my inner loop and grabbot is my outer loop.

639
00:44:05,920 --> 00:44:09,760
Grabbot takes all the context, external context,

640
00:44:09,920 --> 00:44:13,950
gives it to the projects because it can actually just send messages to those projects,

641
00:44:14,000 --> 00:44:15,999
right? You don't even have to open cursor.

642
00:44:16,000 --> 00:44:21,199
You can just tell your grabbot, okay, create a project, right, for these series of tasks.

643
00:44:21,200 --> 00:44:22,639
They're all related, right?

644
00:44:22,640 --> 00:44:24,479
Maybe as an example, you know,

645
00:44:24,480 --> 00:44:29,598
you've had a uh [clears throat] a big burst of issues that are all about performance,

646
00:44:29,599 --> 00:44:34,078
right? Your app is slow uh and they're all connected, right?

647
00:44:34,079 --> 00:44:37,598
Maybe some of them even have a similar fix, right?

648
00:44:37,599 --> 00:44:41,759
But and you can certainly go off and just spawn one agent per task,

649
00:44:41,760 --> 00:44:45,118
but then you've lost that sort of thread between them, right?

650
00:44:45,119 --> 00:44:49,598
And and you may duplicate work or you may not really think about the higher level problem.

651
00:44:49,599 --> 00:44:52,479
You know, sometimes when you you you solve bugs, you know,

652
00:44:52,480 --> 00:44:56,799
it helps to have multiple bug reports that are are slightly different because it helps

653
00:44:56,800 --> 00:44:58,959
you really, you know, zoom out and see actually, you know,

654
00:44:58,960 --> 00:45:02,159
the problem when I looked at this one report, I thought the bug was here,

655
00:45:02,160 --> 00:45:07,679
but actually when when I see the other multitude of bugs is actually up here, >> right?

656
00:45:07,680 --> 00:45:08,399
>> Yeah. Got you.

657
00:45:08,400 --> 00:45:11,279
So that that's why you have so many agents in that loop then, right?

658
00:45:11,280 --> 00:45:15,039
Because it's not just you have um like you have a bug report comes in,

659
00:45:15,040 --> 00:45:17,679
you spawn a single agent to look at that bug report.

660
00:45:17,680 --> 00:45:22,479
that a that single agent will be duplicating work with other um other agents, right?

661
00:45:22,480 --> 00:45:24,879
Because if there are multiple bug reports coming in through the same thing,

662
00:45:24,880 --> 00:45:26,318
that can be duplicated work.

663
00:45:26,319 --> 00:45:27,920
>> Yeah. >> Fascinating.

664
00:45:28,106 --> 00:45:29,870
[snorts] >> That's really fascinating.

665
00:45:29,920 --> 00:45:35,199
Okay. And so this just this endless series of triggers um coming from real users reporting

666
00:45:35,200 --> 00:45:41,519
real reports um builds up this sort of and accelerates the factory sort of adds more

667
00:45:41,520 --> 00:45:43,838
orders in. Other than bug reports,

668
00:45:43,839 --> 00:45:50,080
are there any other sources that you use for like um accelerating for pushing these PRs?

669
00:45:51,599 --> 00:45:56,879
>> Uh well, funnily enough, it's some of it comes from uh reading the code, too.

670
00:45:56,880 --> 00:46:03,759
So, I guess I have sort of uh well, so to clarify that, you know, the 2,500 PRs,

671
00:46:03,760 --> 00:46:07,199
they're not obviously like 2,500 features, right?

672
00:46:07,200 --> 00:46:13,759
they are a lot of the work actually is spent on gardening like another term that I really

673
00:46:13,760 --> 00:46:17,039
love. Uh where

674
00:46:17,520 --> 00:46:22,318
so I guess this is more important when you have a big team of engineers human engineers

675
00:46:22,319 --> 00:46:26,959
that you work with where and also this goes back a little bit to what I was talking about

676
00:46:26,960 --> 00:46:27,759
with the environment.

677
00:46:27,760 --> 00:46:30,959
You know, setting up a really good environment that doesn't just help you and your agents,

678
00:46:30,960 --> 00:46:32,799
but everybody on your team, right?

679
00:46:32,800 --> 00:46:37,279
Think of a new hire who doesn't have a lot of context on all of your engineering practices

680
00:46:37,280 --> 00:46:39,150
joining your your team.

681
00:46:39,200 --> 00:46:43,279
And if you have a really good environment, they can be productive from day one, right?

682
00:46:43,280 --> 00:46:47,439
They can they don't have to like, you know, make open a bunch of lowquality PRs.

683
00:46:47,440 --> 00:46:52,110
They can start, you know, they can start just turning out really good code.

684
00:46:52,160 --> 00:46:53,360
Um,

685
00:46:54,160 --> 00:46:58,799
and uh, yeah, I think I I sort of lost my train of thought.

686
00:46:58,800 --> 00:46:59,919
>> I've got a I've got a followup,

687
00:46:59,920 --> 00:47:05,679
which is what's what are the mechanics of like triggering >> like how when when do you

688
00:47:05,680 --> 00:47:07,920
trigger a a agent

689
00:47:08,720 --> 00:47:10,078
to go and look at the code, right?

690
00:47:10,079 --> 00:47:11,759
Because some people might say, "Oh,

691
00:47:11,760 --> 00:47:16,639
let's just do that every hour or something or like on a chron job or what's >> Oh, yeah.

692
00:47:16,640 --> 00:47:18,350
Yeah. Yeah. Yeah.

693
00:47:18,400 --> 00:47:21,999
Uh I saw some of your recent tweets as well about like you know the some of the tweets

694
00:47:22,000 --> 00:47:25,950
you've been doing which are great for setting up your routines.

695
00:47:26,000 --> 00:47:29,838
Uh I have some routines like that as well.

696
00:47:29,839 --> 00:47:35,039
Um so uh one of them is uh

697
00:47:35,520 --> 00:47:41,118
like looking through just another simple example is you know [clears throat] React has

698
00:47:41,119 --> 00:47:42,590
a lot of foot guns.

699
00:47:42,640 --> 00:47:45,519
Um, so, uh, as as I'm sure you're aware.

700
00:47:45,520 --> 00:47:49,390
And so I have an agent that's just constantly looking for band patterns.

701
00:47:49,440 --> 00:47:53,598
And the interesting thing about that one is that I don't actually tell it to fix the

702
00:47:53,599 --> 00:47:56,830
issue first. I tell it to append it to a document.

703
00:47:56,880 --> 00:48:00,719
And then every couple of days I look at it and I see actually these are all the same

704
00:48:00,720 --> 00:48:06,269
thing, you know, and so that gives me, you know, you almost want like a buffer, a queue.

705
00:48:06,319 --> 00:48:10,879
Sometimes that's actually more effective than just spawning off a couple of like a lot

706
00:48:10,880 --> 00:48:15,519
of sub agents to fix every single thing because when you are [clears throat] in kind

707
00:48:15,520 --> 00:48:20,879
of pure execution mode and just trying to like you know f uh you know execute on the

708
00:48:20,880 --> 00:48:25,069
orders that are coming in very fast you sometimes miss the big picture.

709
00:48:25,119 --> 00:48:29,118
So sometimes having a buffer forces you to think about the big picture because you you

710
00:48:29,119 --> 00:48:36,318
you have these artifacts and things that you can look at as a human um and sort of

711
00:48:36,319 --> 00:48:41,439
use your own human judgment to or I guess you can use an agent to do that as well.

712
00:48:41,440 --> 00:48:46,078
But you give the [clears throat] agent and yourself a way to identify patterns,

713
00:48:46,079 --> 00:48:52,078
right, that you might otherwise miss if you're just only solving each bug at a time.

714
00:48:52,079 --> 00:48:56,879
And that's also really the benefit of having something like a chief of staff agent is

715
00:48:56,880 --> 00:49:03,840
uh it can see the forest right uh in addition to actually doing the execution.

716
00:49:04,480 --> 00:49:08,399
>> Fascinating. That's I mean my brain is exploding a bit there with the sort of chief

717
00:49:08,400 --> 00:49:10,159
of staff at the software factory.

718
00:49:10,160 --> 00:49:13,519
I might have to change some of the course that I'm filming next week.

719
00:49:13,520 --> 00:49:16,078
That's [laughter] >> uh all right.

720
00:49:16,079 --> 00:49:17,999
Let's talk about let's talk about review, right?

721
00:49:18,000 --> 00:49:20,159
because this is the reply that you get, you know,

722
00:49:20,160 --> 00:49:27,470
is >> did you read did you taste all 2500 of those dishes as they swept past you?

723
00:49:27,520 --> 00:49:32,559
>> And I assume the answer is a variety is a version of no.

724
00:49:34,319 --> 00:49:38,239
>> Yeah, I think you you you don't want to be in a position where you're not tasting

725
00:49:38,240 --> 00:49:39,870
your food ever again.

726
00:49:39,920 --> 00:49:41,919
Uh but you also, you know, for scale,

727
00:49:41,920 --> 00:49:45,358
you cannot be tasting every single dish that comes out of your kitchen,

728
00:49:45,359 --> 00:49:47,118
especially if you have multiple restaurants.

729
00:49:47,119 --> 00:49:52,879
So it becomes more about sampling right and thinking about the processes in the same

730
00:49:52,880 --> 00:49:58,479
way that you know if I guess maybe this is where the the the factory analogy is a bit

731
00:49:58,480 --> 00:50:04,719
more apt is you know as a quality supervisor on a factory you you can't look at every

732
00:50:04,720 --> 00:50:09,919
single item you sample right you take you you you go in there every day and you look

733
00:50:09,920 --> 00:50:14,239
at the quality of the pull requests you look at the code that the agents are writing

734
00:50:14,240 --> 00:50:21,069
and you scrutinize it very rig rigorously and you think about all the inefficiencies,

735
00:50:21,119 --> 00:50:26,159
the bad patterns that the agents are doing and then you think about how to course correct

736
00:50:26,160 --> 00:50:27,679
the environment, right?

737
00:50:27,680 --> 00:50:29,789
Not not that single agent.

738
00:50:29,839 --> 00:50:35,118
Uh because if maybe if it if it was a one-off incident, it's fine.

739
00:50:35,119 --> 00:50:37,950
You know that maybe there's nothing to fix there.

740
00:50:38,000 --> 00:50:42,399
But if you actually notice that multiple agents are are having the same issue, right?

741
00:50:42,400 --> 00:50:44,078
They're taking the same shortcut.

742
00:50:44,079 --> 00:50:47,870
they're they're propagating the same workaround everywhere.

743
00:50:47,920 --> 00:50:53,679
Uh that's a sign that you should go off and think about how to uh amend your kitchen

744
00:50:53,680 --> 00:50:55,199
or your factory, right?

745
00:50:55,200 --> 00:50:58,799
Like thinking about your skills, your constraints, your lints,

746
00:50:58,800 --> 00:51:05,679
your type systems um and setting or adjusting it so that that problem doesn't happen

747
00:51:05,680 --> 00:51:07,838
again. And when you do that enough times,

748
00:51:07,839 --> 00:51:13,358
then you get to a place where the codebase is again like the environment is so constrained

749
00:51:13,359 --> 00:51:19,358
and so it guides you so well that you can just you can just step away, right?

750
00:51:19,359 --> 00:51:20,159
That's the dream.

751
00:51:20,160 --> 00:51:25,150
And I'll I'll definitely say it um it's very hard to get to this point.

752
00:51:25,200 --> 00:51:26,959
I don't want to sell this as like, you know,

753
00:51:26,960 --> 00:51:30,399
something that you can just do easily by using PAC.

754
00:51:30,400 --> 00:51:35,598
Like it takes a lot of time and effort to think about your code and where you see your

755
00:51:35,599 --> 00:51:38,959
agents failing and thinking very thoughtfully,

756
00:51:38,960 --> 00:51:45,679
intentionally and setting up guard rails and constraints so that they do the right thing

757
00:51:45,680 --> 00:51:51,279
by default. >> And you're not [clears throat] like if to go back to the software factory

758
00:51:51,280 --> 00:51:53,838
analogy, this isn't a dark factory, right?

759
00:51:53,839 --> 00:51:55,519
This the lights are on, right?

760
00:51:55,520 --> 00:51:56,078
>> It kind of is.

761
00:51:56,079 --> 00:51:57,679
Yeah, actually. >> Is it?

762
00:51:57,680 --> 00:52:02,479
>> Yeah. Well, it's dark in the sense that so um it's dark in the sense that well I think

763
00:52:02,480 --> 00:52:08,399
if my agents are merging their own pull requests it's sort of become dark where I go

764
00:52:08,400 --> 00:52:11,120
to sleep my agents now work 24/7

765
00:52:11,599 --> 00:52:16,959
uh I have I have the equivalent of like more than 10 chiefs of staff right each working

766
00:52:16,960 --> 00:52:21,598
on a different area like for example I have one that's working on performance of the

767
00:52:21,599 --> 00:52:27,759
Grockbot desktop app I have one that's working on uh fixing bugs that users report I

768
00:52:27,760 --> 00:52:32,479
have one that's exploring rewriting it in a different language just for fun,

769
00:52:32,480 --> 00:52:33,999
you know, like what if what if, you know,

770
00:52:34,000 --> 00:52:37,199
just reimagining what what it would be if it was like a native app.

771
00:52:37,200 --> 00:52:38,479
It's just a toy.

772
00:52:38,480 --> 00:52:40,879
Um, but the idea is like yeah,

773
00:52:40,880 --> 00:52:44,670
I uh I when you spend the time setting up your environment,

774
00:52:44,720 --> 00:52:48,959
I've gotten to a point where I review the pull request after it's landed, right?

775
00:52:48,960 --> 00:52:55,199
I I tell my agents full autopilot is is something that you can do in in PAC and that

776
00:52:55,200 --> 00:53:00,959
will trigger off this very intense rigorous verification loop where it will spawn a bunch

777
00:53:00,960 --> 00:53:06,558
of verifier agents for every pull request and it will fuzz right fuzzing meaning that

778
00:53:06,559 --> 00:53:08,239
it will actually run the application.

779
00:53:08,240 --> 00:53:11,358
It's going to click around and try to use it like a real human.

780
00:53:11,359 --> 00:53:12,399
look for regressions,

781
00:53:12,400 --> 00:53:19,039
look for bugs in your implementation and um it will try to find issues with the thing

782
00:53:19,040 --> 00:53:21,199
and then it will fix it itself.

783
00:53:21,200 --> 00:53:26,000
It'll do that again and eventually get the PR to a state where it can land.

784
00:53:26,800 --> 00:53:30,558
Uh so it does it does it is quite token intensive.

785
00:53:30,559 --> 00:53:32,430
You can tune this of course.

786
00:53:32,480 --> 00:53:37,279
Uh so you know instead of like 10 verifier agents you might do like one, right?

787
00:53:37,280 --> 00:53:40,479
Or you just tell the agent to verify it's done work.

788
00:53:40,480 --> 00:53:46,078
But yeah, the key thing is the verification part is really the key piece that gives me

789
00:53:46,079 --> 00:53:47,680
a lot of confidence

790
00:53:48,000 --> 00:53:50,479
that I guess verification plus the environment, right?

791
00:53:50,480 --> 00:53:54,719
It's these the combination of these two things that allow me to step away and say agents

792
00:53:54,720 --> 00:53:56,159
go off and merge your thing.

793
00:53:56,160 --> 00:54:01,199
I'll review it in the morning by looking at my commit history >> and if I see problems,

794
00:54:01,200 --> 00:54:03,358
I go and course correct, >> right?

795
00:54:03,359 --> 00:54:08,990
And I'll go and revert or modify, add new link rules and whatever.

796
00:54:09,040 --> 00:54:13,838
Um, so it does it does take time to get to that point, but once you get it, oh,

797
00:54:13,839 --> 00:54:16,029
it's so it feels so magical.

798
00:54:16,079 --> 00:54:21,358
Uh, I I I tell people like I'm sleeping so much better now because, you know,

799
00:54:21,359 --> 00:54:26,078
it took it the very first day I turned on the sort of dark factory was very scary because

800
00:54:26,079 --> 00:54:28,399
I was like, "Ooh, what if I call the SE, right?

801
00:54:28,400 --> 00:54:30,830
What if I break something overnight?"

802
00:54:30,880 --> 00:54:36,318
>> Uh, and it took a lot of it took a lot of uh bravery, I think, to do that,

803
00:54:36,319 --> 00:54:37,919
but >> somehow I did it.

804
00:54:37,920 --> 00:54:43,759
And yeah, now I'm in a place where my Asians are are merging their own code while I sleep.

805
00:54:43,760 --> 00:54:46,078
>> It sounds like >> I think it's dark in that sense.

806
00:54:46,079 --> 00:54:47,759
>> Yes, it's dark sometimes, right?

807
00:54:47,760 --> 00:54:48,719
You do sometimes.

808
00:54:48,720 --> 00:54:49,838
>> That's true. That's true.

809
00:54:49,839 --> 00:54:54,318
>> Because I think of a dark factory is like almost like if you take the original definition

810
00:54:54,319 --> 00:54:58,318
of Kapathy's vibe coding, right, which is the code almost doesn't exist.

811
00:54:58,319 --> 00:55:00,318
You forget that code might be a thing.

812
00:55:00,319 --> 00:55:03,039
I think your approach is totally different from that,

813
00:55:03,040 --> 00:55:06,879
which is that code and the environment is essential.

814
00:55:06,880 --> 00:55:10,239
And if the code in the environment are bad, then you will get bad outputs.

815
00:55:10,240 --> 00:55:11,919
Garbage in, garbage out.

816
00:55:11,920 --> 00:55:15,759
So I I I think this is a this is a different thing.

817
00:55:15,760 --> 00:55:21,439
It's like, you know, the I don't know, maybe there's a dimmer switch or something, right?

818
00:55:21,440 --> 00:55:23,838
Like, you know, some parts of dark, some parts were light.

819
00:55:23,839 --> 00:55:28,959
This is why the maybe the the kitchen is a better analogy >> the restaurant because you

820
00:55:28,960 --> 00:55:32,000
know even as a as a as a

821
00:55:32,319 --> 00:55:36,959
as a [clears throat] restaurant restaurant you still might go to your restaurants every

822
00:55:36,960 --> 00:55:41,439
now and then to take take a peek in taste the food right >> uh I think that's >> the

823
00:55:41,440 --> 00:55:45,439
idea of sampling instead of blocking I think is really important >> I think what would

824
00:55:45,440 --> 00:55:50,399
you say to people who are in I guess you're obviously in a pretty security conscious

825
00:55:50,400 --> 00:55:55,679
environment where you're working very security conscious >> mh [clears throat] Um maybe

826
00:55:55,680 --> 00:56:02,399
there are folks working in like um medical applications or law or finance or something.

827
00:56:02,400 --> 00:56:07,519
I think of the like some PRs are kind of like two-way doors which is you can merge it

828
00:56:07,520 --> 00:56:08,719
and then revert it, right?

829
00:56:08,720 --> 00:56:10,159
It's cheap back through.

830
00:56:10,160 --> 00:56:12,399
But there are some PRs that are one-way doors, right?

831
00:56:12,400 --> 00:56:18,318
That will cause data loss of some kind that will >> do something that can't be easily

832
00:56:18,319 --> 00:56:23,519
walked back. How do you deal with situations where most of your PRs, let's say,

833
00:56:23,520 --> 00:56:24,719
are one-way doors?

834
00:56:24,720 --> 00:56:28,590
Like, is this something you just wouldn't recommend or like what do you think?

835
00:56:28,640 --> 00:56:30,479
>> Yeah, I think that's a really good question.

836
00:56:30,480 --> 00:56:32,160
I think that

837
00:56:32,559 --> 00:56:37,360
it all comes back to me to the quality of the verification that you're able to

838
00:56:37,920 --> 00:56:41,150
um get out of your agent.

839
00:56:41,200 --> 00:56:45,440
And I think for domains where the work is verifiable,

840
00:56:46,079 --> 00:56:47,439
this is easier, right?

841
00:56:47,440 --> 00:56:52,720
and the the oneway doors become two-way doors in a sense.

842
00:56:53,040 --> 00:56:57,279
But I guess I don't know if you're working on something that is like extremely

843
00:56:57,599 --> 00:57:02,719
is very hard to verify programmatically then I think yeah you're definitely in a position

844
00:57:02,720 --> 00:57:05,870
where it's very hard to get to that point.

845
00:57:05,920 --> 00:57:11,999
Um so I do think like yeah verifiability of the domain is an important aspect to be able

846
00:57:12,000 --> 00:57:16,798
to do this. Um and software engineering is just one of those things where it's quite

847
00:57:16,799 --> 00:57:22,798
verifiable in in a lot of cases maybe not totally um you know like other domains like

848
00:57:22,799 --> 00:57:29,598
mathematics I think are another example of not all of it of course but some aspects of

849
00:57:29,599 --> 00:57:31,920
mathematics can be verifiable

850
00:57:32,480 --> 00:57:39,039
if you write a proof for example um and so yeah I think it's a great question that I

851
00:57:39,040 --> 00:57:42,959
don't really have the answer to and I think that this is something the industry and us

852
00:57:42,960 --> 00:57:49,199
as engineers will have to figure out is you know my sort of uh hope and prediction for

853
00:57:49,200 --> 00:57:55,279
the future is that we'll see more and more interesting new agentoriented programming

854
00:57:55,280 --> 00:57:59,199
languages and one of the most fascinating ones that I've seen so far is this one called

855
00:57:59,200 --> 00:58:06,830
bend bend d bend um and that language is one where it kind of

856
00:58:06,880 --> 00:58:13,279
marries programming with proofs right there used to be a time, you know,

857
00:58:13,280 --> 00:58:17,069
where you actually had to write your proofs in a different language.

858
00:58:17,119 --> 00:58:18,078
And proofs, by the way,

859
00:58:18,079 --> 00:58:24,029
for those uh who who aren't familiar is this idea of uh that you can sort of formally

860
00:58:24,079 --> 00:58:28,719
verify that some code is correct mathematically, right?

861
00:58:28,720 --> 00:58:33,759
Especially if you've written your code in a very functional programming way.

862
00:58:34,720 --> 00:58:39,549
uh but for the longest time you had to do that in a separate language like lean or tla+

863
00:58:39,599 --> 00:58:45,759
or uh I'm blanking on some of the other other examples but uh like languages like that

864
00:58:45,760 --> 00:58:50,479
where you would construct the mathematical proof and then use a solver essentially to

865
00:58:50,480 --> 00:58:51,680
det

866
00:58:52,000 --> 00:58:57,600
that that you've covered all the cases you don't have like a race condition or or whatever.

867
00:58:58,240 --> 00:59:01,039
So yeah, I think

868
00:59:02,079 --> 00:59:03,838
trying to sum up the question, I think yeah,

869
00:59:03,839 --> 00:59:08,558
if you are in a position where you can figure out how your agents can truly verify the

870
00:59:08,559 --> 00:59:13,919
work in a way that gives you confidence, you can actually, you know,

871
00:59:13,920 --> 00:59:17,519
uh have the PRs merge cuz if it compiles, right,

872
00:59:17,520 --> 00:59:22,798
if it if the proofs show you that it's correct, then why wouldn't you just merge it?

873
00:59:22,799 --> 00:59:27,759
Um but of course, yeah, not all the means are verifiable.

874
00:59:27,920 --> 00:59:30,080
Yeah, it's a tough one.

875
00:59:30,240 --> 00:59:35,999
Um, okay. I think we've got to think about wrapping up because we are nearly on the hour.

876
00:59:36,000 --> 00:59:37,439
Have you Have you got something after this?

877
00:59:37,440 --> 00:59:39,679
I mean, I've got something before I give my son dinner,

878
00:59:39,680 --> 00:59:42,479
but >> I I can go a bit longer after you.

879
00:59:42,480 --> 00:59:44,159
>> Okay, let's let's go five minutes longer then.

880
00:59:44,160 --> 00:59:48,480
Um I think I just want to have one more question which is

881
00:59:49,119 --> 00:59:54,799
I think I want to ask how you see Pstack

882
00:59:55,280 --> 01:00:00,639
and how you see skills in general like in terms of we talked about this before we went

883
01:00:00,640 --> 01:00:07,439
on air which is like people think of as like my skills versus your skills and how do

884
01:00:07,440 --> 01:00:09,199
you combine frameworks together?

885
01:00:09,200 --> 01:00:12,078
How do you use Pstack with my stuff?

886
01:00:12,079 --> 01:00:18,879
like what should you take from each one and because I think I see skills as sort of just

887
01:00:18,880 --> 01:00:25,439
derived from process basically like they're just processes turned into words and I would

888
01:00:25,440 --> 01:00:26,960
love to know

889
01:00:27,440 --> 01:00:31,838
how you recommend people take Pstack and take my stuff as well and turn it into their

890
01:00:31,839 --> 01:00:33,200
own processes.

891
01:00:35,040 --> 01:00:39,069
I think you shared a tip actually today that I thought was actually very relevant,

892
01:00:39,119 --> 01:00:43,358
which is this idea that you go off and look at your previous transcripts, right?

893
01:00:43,359 --> 01:00:46,239
And you sort of mine for information of, you know,

894
01:00:46,240 --> 01:00:50,318
your own look through your own your own prompts, right,

895
01:00:50,319 --> 01:00:56,639
to the agents where you correct them where you have to constantly intervene and uh you

896
01:00:56,640 --> 01:01:00,719
know take that higher level learning and turn that into a a reusable skill, right?

897
01:01:00,720 --> 01:01:03,950
So that agents stop repeating that mistake.

898
01:01:04,000 --> 01:01:09,439
I think that uh PAC and your skills are very complimementaryary.

899
01:01:09,440 --> 01:01:13,519
I I totally agree with you that they're like a skill is really much just process.

900
01:01:13,520 --> 01:01:17,759
I mean it's just at the end of the day a skill is just English or or language.

901
01:01:17,760 --> 01:01:19,309
It's just markdown.

902
01:01:19,359 --> 01:01:23,118
>> Um and I think you can you can definitely weave them,

903
01:01:23,119 --> 01:01:26,480
combine them in a way that makes sense to you.

904
01:01:26,640 --> 01:01:33,630
But I do think that uh everyone should have their own set of knives, right?

905
01:01:33,680 --> 01:01:37,598
I I keep going back to the the the cooking analogy, but it's so apt because like,

906
01:01:37,599 --> 01:01:40,879
you know, every chef when they go to a different job, right,

907
01:01:40,880 --> 01:01:44,078
when they go to a different restaurant, they carry they bring their knives with them.

908
01:01:44,079 --> 01:01:45,999
The tools go with them, right?

909
01:01:46,000 --> 01:01:49,470
And so trust to me is really about trust in your own tools.

910
01:01:49,520 --> 01:01:53,118
And when you spend the time sharpening them and understanding them really,

911
01:01:53,119 --> 01:01:54,558
really well, you can do great things.

912
01:01:54,559 --> 01:01:58,639
And everybody's skills and tool set is going to look different.

913
01:01:58,640 --> 01:02:02,239
you know, someone might find a lot of success combining, you know,

914
01:02:02,240 --> 01:02:05,039
like your grill me with docs, uh,

915
01:02:05,040 --> 01:02:09,710
or wayfinder skill with some of the execution skills in Pstack as an example.

916
01:02:09,760 --> 01:02:13,679
Some people might use more of your skills, some people might use more of my skills.

917
01:02:13,680 --> 01:02:15,358
I think at the end end of the day,

918
01:02:15,359 --> 01:02:19,919
it really just comes back to how much do you trust, you know, me and Matt, right?

919
01:02:19,920 --> 01:02:23,199
Like if you if you trust us both, of course, use our skills,

920
01:02:23,200 --> 01:02:26,350
but I also encourage you to, you know, look at your own transcripts.

921
01:02:26,400 --> 01:02:30,399
Um um tell the agent to look through, you know,

922
01:02:30,400 --> 01:02:33,519
some of the all of the patterns that you've used,

923
01:02:33,520 --> 01:02:36,639
the the times you've had to intervene, you know,

924
01:02:36,640 --> 01:02:40,990
suggest turning them into lint rules or new skills, right?

925
01:02:41,040 --> 01:02:46,639
The the past chats I I often say is like a a treasure trove of context because that,

926
01:02:46,640 --> 01:02:51,759
you know, it's it's like the process materialized, right?

927
01:02:51,760 --> 01:02:52,879
Like it's the real process.

928
01:02:52,880 --> 01:02:55,358
It's not an abstract idea in your head.

929
01:02:55,359 --> 01:02:59,759
it's the actual thing right and you can actually see how it happened in practice and

930
01:02:59,760 --> 01:03:03,679
extract so much information from that and there's so much so that I actually turn I have

931
01:03:03,680 --> 01:03:10,240
a skill in pac called recall which is exactly that um where this was a pattern where

932
01:03:10,799 --> 01:03:15,118
you know I was working in I was working on a similar problem so specifically I was working

933
01:03:15,119 --> 01:03:20,798
on virtualization for the cursor application and there were a lot of bugs and so you

934
01:03:20,799 --> 01:03:23,838
know every time I started a new chat I was like ah this there's so much good context

935
01:03:23,839 --> 01:03:27,038
from the last one So, you know, I want to bring it over to the new chat.

936
01:03:27,039 --> 01:03:27,759
How do I do that?

937
01:03:27,760 --> 01:03:30,639
And that's where the transcript came, uh, you know,

938
01:03:30,640 --> 01:03:32,318
looking at the past transcript came about.

939
01:03:32,319 --> 01:03:37,439
And then recall was just a way for me to collapse collapse and compress that workflow

940
01:03:37,440 --> 01:03:42,318
into a skill. So that I didn't have to just say I didn't have to write a long essay every

941
01:03:42,319 --> 01:03:43,919
time. Go look at all these chats, right?

942
01:03:43,920 --> 01:03:45,679
And, you know, blah blah blah.

943
01:03:45,680 --> 01:03:47,439
So, I I largely think of skills,

944
01:03:47,440 --> 01:03:52,269
especially as agents get more capable as really encoding workflows.

945
01:03:52,319 --> 01:03:56,639
you know skills from last year were really more about like almost like implementation

946
01:03:56,640 --> 01:04:02,558
details like here are the exact script commands you know you should use right I think

947
01:04:02,559 --> 01:04:07,679
with the latest models you can just delete those parts and just really focus on the workflow

948
01:04:07,680 --> 01:04:12,720
right until it it's more the skill becomes more like a series of steps

949
01:04:13,280 --> 01:04:19,118
a series of your process uh and I think over time we'll see that skills get smaller and

950
01:04:19,119 --> 01:04:25,358
smaller you know more compact Um, and yeah, they're very compatible.

951
01:04:25,359 --> 01:04:29,519
Or you can, you know, if you want, why not read our skills, right,

952
01:04:29,520 --> 01:04:32,318
and com and combine them in your of your own, right?

953
01:04:32,319 --> 01:04:36,239
Combine Wfinder with potato mode and make your own custom mode, right?

954
01:04:36,240 --> 01:04:42,430
Like like skills are the the thing I love about skills that is are that they're so malleable.

955
01:04:42,480 --> 01:04:43,679
You can do anything you want.

956
01:04:43,680 --> 01:04:44,910
It's just language.

957
01:04:44,960 --> 01:04:46,879
>> Absolutely. There's nothing magical in them, right?

958
01:04:46,880 --> 01:04:47,759
They're just words.

959
01:04:47,760 --> 01:04:50,078
And >> Exactly. If if there is any magic in them,

960
01:04:50,079 --> 01:04:57,519
it's just the words chosen and the phrases used and the thinking that's been done to

961
01:04:57,520 --> 01:05:02,509
turn those like take abstract process and turn them into language.

962
01:05:02,559 --> 01:05:05,279
And once that thinking has been done, then it's just there.

963
01:05:05,280 --> 01:05:08,078
It's available. It's on the surface and you just nick it.

964
01:05:08,079 --> 01:05:10,639
Um Lauren, thank you so much.

965
01:05:10,640 --> 01:05:11,759
This has been glorious.

966
01:05:11,760 --> 01:05:12,959
>> Yeah, this has been super fun.

967
01:05:12,960 --> 01:05:15,439
I really enjoyed talking to you.

968
01:05:15,440 --> 01:05:17,598
Hope we can do it again.

969
01:05:17,599 --> 01:05:18,798
I'd love to do it again.

970
01:05:18,799 --> 01:05:19,759
I'd love to do it again.

971
01:05:19,760 --> 01:05:23,870
Absolutely. Um yeah, we'll check in in uh >> I don't know.

972
01:05:23,920 --> 01:05:25,838
Yeah, Monday. Let's do it.

973
01:05:25,839 --> 01:05:27,470
>> Yeah, [laughter] let's do it.

974
01:05:27,520 --> 01:05:29,118
Part two. >> Well, thank you so much.

975
01:05:29,119 --> 01:05:30,318
I'm going to close the stream here.

976
01:05:30,319 --> 01:05:33,118
Laura and I will uh chat a little bit and stay here.

977
01:05:33,119 --> 01:05:34,719
But thank you guys so much for watching.

978
01:05:34,720 --> 01:05:36,080
The glorious.