1
00:00:00,080 --> 00:00:01,599
met Larry and Sergey from Google.

2
00:00:01,600 --> 00:00:03,039
Somehow one of them looked me up,

3
00:00:03,040 --> 00:00:05,519
asked me to come interview at Google >> when Google was three years old.

4
00:00:05,520 --> 00:00:06,079
>> Three years old.

5
00:00:06,080 --> 00:00:07,918
And uh I said, "No, >> you did not."

6
00:00:07,919 --> 00:00:09,279
>> I've always been a very prolific coder.

7
00:00:09,280 --> 00:00:10,879
I look back at my GitHub output.

8
00:00:10,880 --> 00:00:13,919
It's like peak years, maybe 100,000 lines of code in a year.

9
00:00:13,920 --> 00:00:16,399
PreI, right? This is when you're back doing this manually.

10
00:00:16,400 --> 00:00:17,118
I kind of look at that,

11
00:00:17,119 --> 00:00:20,079
it's like there's kind of a max that you can hold in your head at a time.

12
00:00:20,080 --> 00:00:21,599
We did your coding in classes.

13
00:00:21,600 --> 00:00:22,719
That was kind of a big breakthrough.

14
00:00:22,720 --> 00:00:24,159
You have nine chunks of data,

15
00:00:24,160 --> 00:00:26,559
but any five of those chunks can be used to reconstruct it.

16
00:00:26,560 --> 00:00:29,598
And what this means is you can lose any four copies and you can still reconstruct your

17
00:00:29,599 --> 00:00:34,799
data. >> What do you think good software engineering looks today compared to four years

18
00:00:34,800 --> 00:00:38,800
ago before we had a >> I think the ambition has to increase.

19
00:00:42,000 --> 00:00:45,849
He built a image editor while at college designed the original storage system behind

20
00:00:45,850 --> 00:00:49,759
[music] Gmail and built many more large complex and widely used systems.

21
00:00:49,760 --> 00:00:52,799
This is Peter Mattis, co-founder and CTO of Cockroach Labs.

22
00:00:52,800 --> 00:00:55,759
Before we sat down to talk, Peter told me, "For the last 30 years,

23
00:00:55,760 --> 00:00:59,550
I've always been a prolific coder, but my current output is a bit insane.

24
00:00:59,600 --> 00:01:02,959
And this isn't [music] vioded junk, but database worthy, highquality,

25
00:01:02,960 --> 00:01:06,158
high performance code thanks to working with strong coding models.

26
00:01:06,159 --> 00:01:09,919
Today, we cover why B trees are so important when building databases,

27
00:01:09,920 --> 00:01:13,999
and why Peter kept reaching for the data structure over and over throughout his career.

28
00:01:14,000 --> 00:01:17,790
How he wrote 100,000 lines of production code per year pre-ai,

29
00:01:17,840 --> 00:01:22,239
stop coding between 2022 and 2024, and why he is back now.

30
00:01:22,240 --> 00:01:26,319
why he thinks AI agents are lazy about testing and how this is an [music] easier fix

31
00:01:26,320 --> 00:01:27,839
than it looks and many more.

32
00:01:27,840 --> 00:01:31,359
If you're interested in distributed databases, distributed storage systems,

33
00:01:31,360 --> 00:01:35,839
or knowing how Peter manages to use AI to produce unusually highquality and productionready

34
00:01:35,840 --> 00:01:37,839
code, this episode is for you.

35
00:01:37,840 --> 00:01:41,679
This episode is presented by Turbopuffer, a ridiculously scalable, fast,

36
00:01:41,680 --> 00:01:45,599
and cheap hybrid search engine built on top of object storage by an engineering team

37
00:01:45,600 --> 00:01:47,599
I've really grown to like after spending time with them.

38
00:01:47,600 --> 00:01:50,559
The Turbo engineering team is doing something really, really cool.

39
00:01:50,560 --> 00:01:54,239
They're completely redesigned their storage architecture from first principles to make

40
00:01:54,240 --> 00:01:57,039
storage faster, cheaper, and more reliable at scale.

41
00:01:57,040 --> 00:01:58,398
If you follow Turbuffer,

42
00:01:58,399 --> 00:02:02,190
you know that their storage architecture was a massive part of their early success.

43
00:02:02,240 --> 00:02:04,959
Redesigning a winning architecture is a big deal.

44
00:02:04,960 --> 00:02:07,839
It's one thing for your query plans to pass unit tests.

45
00:02:07,840 --> 00:02:11,840
It's another to bring each query plan to performance par

46
00:02:12,160 --> 00:02:15,279
while maintaining correctness and reliability in production.

47
00:02:15,280 --> 00:02:16,479
Here's a cool part.

48
00:02:16,480 --> 00:02:18,639
Turbopuffer is documenting the whole thing.

49
00:02:18,640 --> 00:02:22,318
Their new storage architecture, which they're calling Tuff V3,

50
00:02:22,319 --> 00:02:25,999
demands some hardcore systems engineering, and they're building it in public,

51
00:02:26,000 --> 00:02:29,519
sharing design decisions and benchmark results as they ship.

52
00:02:29,520 --> 00:02:33,199
They're keeping a worklog of their journey, and the first post just dropped today.

53
00:02:33,200 --> 00:02:35,200
Follow along at turbopuffer.com/v3.

54
00:02:36,400 --> 00:02:38,079
That is turbuffer.com/v3.

55
00:02:39,760 --> 00:02:41,279
Peter, welcome to the podcast.

56
00:02:41,280 --> 00:02:42,238
>> Oh, I'm happy to be here.

57
00:02:42,239 --> 00:02:43,439
This is awesome.

58
00:02:43,440 --> 00:02:46,559
>> I I wanted to get into How did you get into tech originally?

59
00:02:46,560 --> 00:02:49,279
When did you figure out computers are interesting?

60
00:02:49,280 --> 00:02:52,318
>> I figured that out in kind of elementary school, high school.

61
00:02:52,319 --> 00:02:55,439
Gaming was a little bit of a gateway drug for me as a software engineer as many other

62
00:02:55,440 --> 00:03:00,878
people. Uh I remember like early on my mom did programming at IBM at some point.

63
00:03:00,879 --> 00:03:02,399
I'm never even quite sure what she did,

64
00:03:02,400 --> 00:03:05,999
but we had computers always around our house like Apple 2 Plus, Apple 2GS.

65
00:03:06,000 --> 00:03:07,439
I'm from that era.

66
00:03:07,440 --> 00:03:12,399
um on up and you know go to the li uh the bookstore I'd find a book on basic or magazine

67
00:03:12,400 --> 00:03:15,759
type in programs no idea what it was doing but you know just kind of like I was addicted

68
00:03:15,760 --> 00:03:19,919
like you could produce put stuff into these computers and you get interesting stuff out

69
00:03:19,920 --> 00:03:23,279
and then I got to college and I was like I didn't think there was any money in computers

70
00:03:23,280 --> 00:03:27,039
I didn't know anything about it I started as a mechanical engineer following the footsteps

71
00:03:27,040 --> 00:03:31,439
of my dad >> you started mechanical engineering as as your specialization >> as my major

72
00:03:31,440 --> 00:03:35,119
yeah and I got in there and I was like doing like I'd done some computer stuff before

73
00:03:35,120 --> 00:03:39,039
it was a real foolish move to do this But first semester of doing these homework assignments

74
00:03:39,040 --> 00:03:42,878
in mechanical engineering and they were god awful like six pages for a single problem

75
00:03:42,879 --> 00:03:48,559
and I happened to take a CS course at the same time and it was so easy and then everybody

76
00:03:48,560 --> 00:03:49,439
almost was failing it.

77
00:03:49,440 --> 00:03:51,839
I'm like I'm in the wrong I'm in the wrong field.

78
00:03:51,840 --> 00:03:53,759
Let me switch. >> And and then you switched.

79
00:03:53,760 --> 00:03:54,399
>> Then I switched.

80
00:03:54,400 --> 00:03:58,399
Yeah. >> What was the first software that you built either at college it must have been

81
00:03:58,400 --> 00:04:01,999
at college that you were like all right this is a piece of software that I'm kind of

82
00:04:02,000 --> 00:04:03,518
proud of. That's a complete piece of software.

83
00:04:03,519 --> 00:04:07,039
Well, I mean the big thing that I did along with uh my roommate in college,

84
00:04:07,040 --> 00:04:08,399
we had this course.

85
00:04:08,400 --> 00:04:09,999
Was it a compiler's course?

86
00:04:10,000 --> 00:04:11,119
I'm not even quite sure anymore.

87
00:04:11,120 --> 00:04:12,479
This is like 30 years ago.

88
00:04:12,480 --> 00:04:13,919
And we were kind of bored with it.

89
00:04:13,920 --> 00:04:16,559
So, we wanted to do something like kind of fun on the side.

90
00:04:16,560 --> 00:04:20,078
And I'd uh done journalism in high school, my sen junior senior year,

91
00:04:20,079 --> 00:04:23,519
and I knew stuff about kind of computer graphics and wanted to do something like Adobe

92
00:04:23,520 --> 00:04:27,999
Photoshop. So, started just kicking the tires and we built up this program that a lot

93
00:04:28,000 --> 00:04:32,478
of people know of called the And along with the I did a lot of the the graphics library,

94
00:04:32,479 --> 00:04:36,559
GTK. This has since evolved just massively since then.

95
00:04:36,560 --> 00:04:39,599
>> It's kind of interesting cuz after college, I kind of stepped away from it.

96
00:04:39,600 --> 00:04:42,638
Didn't really stay involved much past my first year out of college,

97
00:04:42,639 --> 00:04:44,478
but definitely people still know me.

98
00:04:44,479 --> 00:04:47,839
And it uh it led to other some interesting uh events in my career.

99
00:04:47,840 --> 00:04:51,039
And and it it was like starting it was literally just you saying, "All right,

100
00:04:51,040 --> 00:04:52,478
I want to do something like Photoshop.

101
00:04:52,479 --> 00:04:53,198
How hard could it be?"

102
00:04:53,199 --> 00:04:56,879
And then you just this wasn't through what you learned in college, right?

103
00:04:56,880 --> 00:05:00,879
This was like you figuring out how to build, you know,

104
00:05:00,880 --> 00:05:05,519
like a graphical I I guess engine, rendering, drawing, data structures,

105
00:05:05,520 --> 00:05:06,239
all of that stuff, right?

106
00:05:06,240 --> 00:05:06,799
>> All that stuff.

107
00:05:06,800 --> 00:05:07,599
Figured it all out.

108
00:05:07,600 --> 00:05:09,999
I remember trying to look at some papers back then.

109
00:05:10,000 --> 00:05:11,599
My roommate was looking at papers.

110
00:05:11,600 --> 00:05:13,279
We're just figuring it all out.

111
00:05:13,280 --> 00:05:17,119
And it's one of these things you almost kind of need to be a little bit naive to start

112
00:05:17,120 --> 00:05:19,279
anything like that because if you know how hard it's going to be,

113
00:05:19,280 --> 00:05:20,239
you would never start it.

114
00:05:20,240 --> 00:05:23,198
So, I think there's this like you have to have this level of like if you're really truly

115
00:05:23,199 --> 00:05:25,359
wise about the effort, you like never get into it.

116
00:05:25,360 --> 00:05:27,679
But then when as you get going, it just keeps on snowballing.

117
00:05:27,680 --> 00:05:31,279
It gets far further and further along and then at some point you're like, "Wow,

118
00:05:31,280 --> 00:05:32,239
this is really awesome."

119
00:05:32,240 --> 00:05:36,319
But there's an interesting little tidbit associated with our work on the which is kind

120
00:05:36,320 --> 00:05:37,038
of a little bit known.

121
00:05:37,039 --> 00:05:38,638
Um I think we've talked about this before,

122
00:05:38,639 --> 00:05:42,478
but it's just kind of fascinating where we were uh getting to the point where we're like,

123
00:05:42,479 --> 00:05:44,159
we should release this to the public.

124
00:05:44,160 --> 00:05:47,839
And back then there was Usenet news groups where people would post stuff and there's

125
00:05:47,840 --> 00:05:48,638
one on graphics.

126
00:05:48,639 --> 00:05:52,478
And I remember like just a couple weeks before we were going to uh release our first

127
00:05:52,479 --> 00:05:54,959
version of the someone else came on there were like,

128
00:05:54,960 --> 00:05:58,638
I've been working on this graphics program and it did everything the did,

129
00:05:58,639 --> 00:06:00,159
every last thing and then some.

130
00:06:00,160 --> 00:06:02,559
And we're just like, well, that sucks.

131
00:06:02,560 --> 00:06:04,799
I guess we'll just keep on working on it's been fun.

132
00:06:04,800 --> 00:06:08,910
And then we release the Never heard from this other guy again.

133
00:06:08,960 --> 00:06:10,799
And I just took a little lesson with that.

134
00:06:10,800 --> 00:06:12,799
There's always going to be someone else working on your idea.

135
00:06:12,800 --> 00:06:15,038
You can't get dissuaded if they pre-announce it.

136
00:06:15,039 --> 00:06:16,399
Nothing ever comes of it.

137
00:06:16,400 --> 00:06:20,079
And a lot of the marketing behind some of that stuff that you might hear is like I mean

138
00:06:20,080 --> 00:06:22,478
I don't know if we like we stole his thunder.

139
00:06:22,479 --> 00:06:23,439
I'm not even sure what happened.

140
00:06:23,440 --> 00:06:25,439
He never traced down what happened.

141
00:06:25,440 --> 00:06:27,279
But there's a real lesson there.

142
00:06:27,280 --> 00:06:31,918
So there there there's a possible future where you read this announcement of someone

143
00:06:31,919 --> 00:06:35,359
saying I'm going to build all of this thing and you go like uh you know someone did it

144
00:06:35,360 --> 00:06:38,638
and you kind of go back and just do something else and the game never happens.

145
00:06:38,639 --> 00:06:39,198
>> That's right.

146
00:06:39,199 --> 00:06:40,318
>> Wow. >> Yeah.

147
00:06:40,319 --> 00:06:42,799
I I guess especially with today with startups, you know,

148
00:06:42,800 --> 00:06:45,359
this sounds like just do your thing.

149
00:06:45,360 --> 00:06:46,959
Put it out there at the very least, right?

150
00:06:46,960 --> 00:06:49,999
>> And the advice I give people is like you might have a unique idea,

151
00:06:50,000 --> 00:06:53,359
but most likely there's like a dozen people out in the world who've had the same idea.

152
00:06:53,360 --> 00:06:55,038
And there might be a couple of them working on it,

153
00:06:55,039 --> 00:06:56,399
but a lot of people don't even work on it.

154
00:06:56,400 --> 00:06:59,198
They're just like whatever they can't get going on their idea.

155
00:06:59,199 --> 00:07:03,038
So, you know, I wouldn't be concerned at all if like you hear someone else working on

156
00:07:03,039 --> 00:07:04,799
your same idea. That's probably the case.

157
00:07:04,800 --> 00:07:07,519
You know, we're working on some cool stuff now at my current company.

158
00:07:07,520 --> 00:07:10,318
I guarantee you there's other competitors out there working on the same thing.

159
00:07:10,319 --> 00:07:11,758
And just know it's like it's a competition.

160
00:07:11,759 --> 00:07:14,318
You got to enjoy that aspect of it, not be afraid of it.

161
00:07:14,319 --> 00:07:17,119
>> And what were some fun things that the game led to?

162
00:07:17,120 --> 00:07:17,839
>> Well, you know,

163
00:07:17,840 --> 00:07:21,679
um I got kind of tired with graphics and that's why I kind of moved away from it after

164
00:07:21,680 --> 00:07:26,799
college. I got into storage systems and I bounced around uh uh you know I worked at one

165
00:07:26,800 --> 00:07:31,598
of the early search engines in me and then I I got over to another startup and um I met

166
00:07:31,599 --> 00:07:36,239
through that time period in that first startup I was doing um met Larry and Sergey from

167
00:07:36,240 --> 00:07:40,159
Google because it turns out that Larry I think it was maybe Sergey I'm not quite sure

168
00:07:40,160 --> 00:07:45,038
the very first version of the Google logo was done in the so they knew of us somehow

169
00:07:45,039 --> 00:07:49,198
one of them you know looked me up asked me to come interview at Google I did back in

170
00:07:49,199 --> 00:07:52,159
2001 and uh >> this is when Google was 3 years old.

171
00:07:52,160 --> 00:07:53,198
>> Yeah. 3 years old.

172
00:07:53,199 --> 00:07:54,830
And uh I said no.

173
00:07:54,880 --> 00:07:56,239
[laughter] >> You did not.

174
00:07:56,240 --> 00:07:57,279
>> I said no. Yeah.

175
00:07:57,280 --> 00:07:59,038
No. Here's the the calculation in my mind.

176
00:07:59,039 --> 00:08:00,399
I was like Google's down in Mountain View.

177
00:08:00,400 --> 00:08:03,439
I was living up in San Francisco and I didn't want to do the commute.

178
00:08:03,440 --> 00:08:07,918
So I uh went and worked at another startup for a year and uh that one after a year I

179
00:08:07,919 --> 00:08:09,279
like I saw it wasn't going anywhere.

180
00:08:09,280 --> 00:08:11,758
And um they called me back up and they said, "Hey, do you want to interview again?"

181
00:08:11,759 --> 00:08:12,799
I was like, "Sure."

182
00:08:12,800 --> 00:08:15,198
Like we're not going to be able to offer you the same stock options you got before.

183
00:08:15,199 --> 00:08:15,839
I'm like, "Okay."

184
00:08:15,840 --> 00:08:18,878
I can't remember what it was and I still can't remember what they offered me the first

185
00:08:18,879 --> 00:08:22,239
time. Um, but I probably would have made a lot more money if I had taken that first.

186
00:08:22,240 --> 00:08:23,038
But I did all right.

187
00:08:23,039 --> 00:08:25,598
I'm not like complaining, but it's one of these things like, yeah, yeah,

188
00:08:25,599 --> 00:08:26,559
now I think back about that.

189
00:08:26,560 --> 00:08:28,638
I'm like, oh, a lot of good stuff came out of it.

190
00:08:28,639 --> 00:08:30,878
Some point along the line, Red Hat was going public.

191
00:08:30,879 --> 00:08:34,398
They actually offered friends and family stock during their IPO to a lot of people.

192
00:08:34,399 --> 00:08:35,199
We got offered friends.

193
00:08:35,200 --> 00:08:36,398
I made a little bit of money off that.

194
00:08:36,399 --> 00:08:38,718
Not a lot, but it was like just after college.

195
00:08:38,719 --> 00:08:41,038
It was actually quite significant at the time, you know.

196
00:08:41,039 --> 00:08:44,158
So, I mean there's a number of good things that kind of came out of that that >> and

197
00:08:44,159 --> 00:08:48,479
I I also use the as a you know I was looking for like oh I can't afford Photoshop and

198
00:08:48,480 --> 00:08:50,958
here's the and it did so many things.

199
00:08:50,959 --> 00:08:55,359
So like I'm sure there's so many people like you had a positive impact of being able

200
00:08:55,360 --> 00:09:00,079
to use this thing and and the nice thing that I what I really liked about it is it was

201
00:09:00,080 --> 00:09:03,838
free but I wasn't like stealing any software from anyone.

202
00:09:03,839 --> 00:09:04,879
You see what I mean?

203
00:09:04,880 --> 00:09:08,958
This was at a time where free software was not as common as today.

204
00:09:08,959 --> 00:09:12,319
Free and open source was not as mainstream as it is today.

205
00:09:12,320 --> 00:09:15,119
>> Yeah. Yeah. >> And then the other cool thing I've heard from a lot of people because

206
00:09:15,120 --> 00:09:18,398
people still like I mentioned this like oh I've used it and then some people you know

207
00:09:18,399 --> 00:09:21,518
software engineers like I learned how to program by looking at your code and I'm just

208
00:09:21,519 --> 00:09:25,039
like wow that was the code I wrote 30 years ago and I wasn't nearly as good as a software

209
00:09:25,040 --> 00:09:25,999
engineer as I am now.

210
00:09:26,000 --> 00:09:29,919
>> And then you got into Google second time around you you you said yes.

211
00:09:29,920 --> 00:09:31,838
What did you start to work on?

212
00:09:31,839 --> 00:09:36,559
>> Yeah. Yeah. Well I got in there they said like hey you know this was early days 2002

213
00:09:36,560 --> 00:09:39,278
was April 1st auspicious date.

214
00:09:39,279 --> 00:09:41,039
Um your sorry date.

215
00:09:41,040 --> 00:09:42,799
Yeah. Uh April 1st, 2002.

216
00:09:42,800 --> 00:09:45,199
And I got in there and they're like, "Oh, well, you know,

217
00:09:45,200 --> 00:09:48,879
we're going to actually build uh email uh Google email."

218
00:09:48,880 --> 00:09:50,398
And it wasn't called Gmail at the time.

219
00:09:50,399 --> 00:09:51,439
There's a code name internally.

220
00:09:51,440 --> 00:09:52,639
It's called Caribou.

221
00:09:52,640 --> 00:09:53,759
>> Caribou. Yeah.

222
00:09:53,760 --> 00:09:55,919
Um and got in there, started working on it,

223
00:09:55,920 --> 00:10:00,239
and I was kind of tasked with working on the the backend threading and message storage

224
00:10:00,240 --> 00:10:01,518
and indexing system.

225
00:10:01,519 --> 00:10:06,159
And worked on that, you know, pretty hardcore for the first year and a half.

226
00:10:06,160 --> 00:10:09,919
Actually, I was on maybe it was three years um on it through the launch which actually

227
00:10:09,920 --> 00:10:12,349
happened to be April 1st, 2004.

228
00:10:12,399 --> 00:10:13,039
Do you remember this?

229
00:10:13,040 --> 00:10:16,319
There was a uh it was like considering April Fool joke.

230
00:10:16,320 --> 00:10:17,119
Yeah, I remember.

231
00:10:17,120 --> 00:10:18,319
So, correct me if I'm wrong,

232
00:10:18,320 --> 00:10:24,559
but the the launch said Gmail with 1 GBTE of storage or or something infinite storage

233
00:10:24,560 --> 00:10:25,999
and like star one gigabyte.

234
00:10:26,000 --> 00:10:26,799
I'm not sure what it was,

235
00:10:26,800 --> 00:10:31,518
but back then most email providers would give you about 10 megabyte of storage,

236
00:10:31,519 --> 00:10:32,639
the free email providers,

237
00:10:32,640 --> 00:10:38,159
and then you could pay for maybe 50 megabytes or 100 megabytes, but that was really expensive.

238
00:10:38,160 --> 00:10:42,879
This launch, it looked like an April Fool's joke cuz who could possibly give you a free

239
00:10:42,880 --> 00:10:47,679
email service with 20 to 50 times or 100 times more storage?

240
00:10:47,680 --> 00:10:48,799
>> We were talking about this internally.

241
00:10:48,800 --> 00:10:51,359
It was like kind of a shock and awe campaign for the industry.

242
00:10:51,360 --> 00:10:55,679
I I think it was actually four megabytes for like Hotmail or Yahi Mail and then you had

243
00:10:55,680 --> 00:11:00,830
this and not only that we had uh so much more storage but it was indexed really fast.

244
00:11:00,880 --> 00:11:04,190
So you know you could do a search and it would come back almost instantaneously.

245
00:11:04,240 --> 00:11:07,759
But c can you tell me can you tell me internally like when the project started okay we're

246
00:11:07,760 --> 00:11:12,719
going to do email for Google how did you and the team arrived to the point of like okay

247
00:11:12,720 --> 00:11:17,359
we will offer all this storage and we'll do it fast because this was at a time if you

248
00:11:17,360 --> 00:11:22,078
can take us back but what I remember is hard drives were still expensive they were relatively

249
00:11:22,079 --> 00:11:26,479
slow we're talking HDD we're not talking necessarily about SSD if I remember but if you

250
00:11:26,480 --> 00:11:30,639
can take us c can you take us back of what what it was like what the constraints were

251
00:11:30,640 --> 00:11:35,599
like and then how you innovated like to to actually like do something that's never been

252
00:11:35,600 --> 00:11:36,319
done before. >> Yeah.

253
00:11:36,320 --> 00:11:37,439
Yeah. So, at the time,

254
00:11:37,440 --> 00:11:41,919
Google internally had this large distributed file system called GFS, Google file system.

255
00:11:41,920 --> 00:11:45,518
Yeah. And uh we were like basically looking at numbers and being like, "Yeah,

256
00:11:45,519 --> 00:11:46,879
we think we can build on top of this."

257
00:11:46,880 --> 00:11:50,479
They had a lot of knowledge about how to do um search and retrieval.

258
00:11:50,480 --> 00:11:53,838
It ended up being that like some of the existing systems they had for search and retrieval,

259
00:11:53,839 --> 00:11:56,719
they started building the prototype on there and then that was completely rewritten.

260
00:11:56,720 --> 00:11:59,199
That was actually what I got involved in because I got in there and there was already

261
00:11:59,200 --> 00:12:01,359
a prototype and then it was completely rewritten.

262
00:12:01,360 --> 00:12:03,999
So, I was on that, you know, uh, threading side.

263
00:12:04,000 --> 00:12:07,439
We decided to have message threading right from the get-go, which is also kind of innovative.

264
00:12:07,440 --> 00:12:11,278
That wasn't common in email systems >> and and spreading to have the message thread,

265
00:12:11,279 --> 00:12:14,879
which I assume needed data structures on on the back and it needed like, you know,

266
00:12:14,880 --> 00:12:18,879
storage and and figuring out read intensity, those kind of things.

267
00:12:18,880 --> 00:12:21,838
>> Yeah. You know, there's uh some B trees involved in that.

268
00:12:21,839 --> 00:12:24,879
I think that might have been the second time I implement B trees and I think I've implemented

269
00:12:24,880 --> 00:12:26,559
them like a dozen times now.

270
00:12:26,560 --> 00:12:29,999
>> How do B trees relate to email threads?

271
00:12:30,000 --> 00:12:33,039
uh it's just like you have to you know have some storage there where you have a thread

272
00:12:33,040 --> 00:12:37,359
ID a message comes in you have to look up it was using the search index to match actually

273
00:12:37,360 --> 00:12:41,039
the the subject or the message ID into the thread and there are some other things taking

274
00:12:41,040 --> 00:12:44,958
place in the B tree as well because we were keeping track of the unread counts of of

275
00:12:44,959 --> 00:12:48,719
threads and and whatnot so you know I can't even remember all the details now this is

276
00:12:48,720 --> 00:12:53,518
ancient history this is 2004 what are we in 2026 22 years ago so it's like left my memory

277
00:12:53,519 --> 00:12:57,199
but there there was definitely be trees there was also this you know inverted index taking

278
00:12:57,200 --> 00:12:59,999
place there all this code has since been completely rear end.

279
00:13:00,000 --> 00:13:01,838
>> What were the economics like in the team?

280
00:13:01,839 --> 00:13:06,559
You must have done the economics of being able to offer free email which of course I'm

281
00:13:06,560 --> 00:13:11,599
sure you did some maths of like how it could be subsidized but to like not make a terrible

282
00:13:11,600 --> 00:13:12,879
terrible loss. >> Yeah.

283
00:13:12,880 --> 00:13:13,599
Yeah. Yeah. Yeah.

284
00:13:13,600 --> 00:13:18,398
No, I mean we were kind of uh kicking around ideas and um you know one of the ideas that

285
00:13:18,399 --> 00:13:22,319
had come up at some point was hey maybe we can put ads on this and it was one of these

286
00:13:22,320 --> 00:13:24,239
crazy things. So I didn't wasn't involved in this.

287
00:13:24,240 --> 00:13:29,759
This is Paul Bukite went on to do some cool stuff at uh friend feed and Facebook and

288
00:13:29,760 --> 00:13:34,239
ended up I think a partner Y cominator at some point but he just like one night he's

289
00:13:34,240 --> 00:13:38,239
like no I think I can just take some of our existing ad functionality and incorporate

290
00:13:38,240 --> 00:13:42,319
it in there and it just went like gang busters and there's a huge business that you know

291
00:13:42,320 --> 00:13:44,479
kind of grew up from that which is kind of incredible.

292
00:13:44,480 --> 00:13:47,838
So I mean I think there's a lot of things that it's just like you kind of have a sense

293
00:13:47,839 --> 00:13:51,599
of what can be done like oh we think this is going to cost you know per user a couple

294
00:13:51,600 --> 00:13:52,479
dollars per year.

295
00:13:52,480 --> 00:13:53,679
how do we monetize that?

296
00:13:53,680 --> 00:13:55,359
And you know, we didn't want to charge for it.

297
00:13:55,360 --> 00:13:58,719
We eventually they did charge for it because you have the whole Google Workspace stuff,

298
00:13:58,720 --> 00:14:01,679
but at the time it was more like, ah, can we do this for free?

299
00:14:01,680 --> 00:14:03,439
Oh, it looks like it can be economical.

300
00:14:03,440 --> 00:14:06,799
And uh it it took a little bit of a a leap of faith.

301
00:14:06,800 --> 00:14:08,559
And do you remember the launch?

302
00:14:08,560 --> 00:14:12,398
Uh the reason I'm asking because I remember that it was an invite- based system.

303
00:14:12,399 --> 00:14:17,599
like not everyone could get in and I assume that must have been to control the expected

304
00:14:17,600 --> 00:14:21,198
demand cuz again you were you offering something for free that was paid before and it

305
00:14:21,199 --> 00:14:24,349
was kind of pretty obvious that there would be massive demand.

306
00:14:24,399 --> 00:14:31,119
>> How did you think about it kind of monitor demand decide how many people to on board?

307
00:14:31,120 --> 00:14:33,599
>> Honestly this is a team effort and that I wasn't involved in that.

308
00:14:33,600 --> 00:14:37,759
I mean I do remember being worried about like the load that would happen and then someone

309
00:14:37,760 --> 00:14:41,599
came up the idea of like hey we should do an invite based system and this also had like

310
00:14:41,600 --> 00:14:45,759
it played dual roles so it kind of constrained the growth but it also kind of drummed

311
00:14:45,760 --> 00:14:48,319
up this excitement oh can get me a Gmail invite.

312
00:14:48,320 --> 00:14:51,439
I remember people asking me at the time I was like yeah I can get you as many Gmail invites

313
00:14:51,440 --> 00:14:56,319
as you want. >> And then after after after you built the the system where did you move

314
00:14:56,320 --> 00:14:57,838
on to what was your next project?

315
00:14:57,839 --> 00:14:59,039
Was was it the build system?

316
00:14:59,040 --> 00:15:02,078
Well, there was the build system and that was kind of all muddled in my mind because

317
00:15:02,079 --> 00:15:05,999
I was kind of doing that part-time when I was doing um Gmail as well at some point.

318
00:15:06,000 --> 00:15:08,638
You know, Google actually had this large monor repo.

319
00:15:08,639 --> 00:15:11,838
I think they actually still have the monitor reporter and they started out with the Google

320
00:15:11,839 --> 00:15:13,919
one um repo. That was before my time.

321
00:15:13,920 --> 00:15:15,278
Then they moved on to Google 2.

322
00:15:15,279 --> 00:15:19,359
That was what was there when I got to Google and at some point we saw the the strains

323
00:15:19,360 --> 00:15:22,559
of Google 2. And Google 2 was just one single monolithic make file.

324
00:15:22,560 --> 00:15:26,479
I think it actually had some submake files but it was this like really large unwieldy

325
00:15:26,480 --> 00:15:29,679
make file. someone came to me and was like, "Uh, I think we could do something."

326
00:15:29,680 --> 00:15:32,319
I'd had some interest in the build systems and I, you know,

327
00:15:32,320 --> 00:15:34,958
kind of put the uh foundation in for Google 3.

328
00:15:34,959 --> 00:15:36,159
There was a bunch of people involved.

329
00:15:36,160 --> 00:15:38,398
I was kind of doing like kind of the initial work on Google 3.

330
00:15:38,399 --> 00:15:43,599
And Google 3 uh the initial insight was like, hey, make files kind of suck to write.

331
00:15:43,600 --> 00:15:47,359
We introduced this thing called build files and I decided to do it.

332
00:15:47,360 --> 00:15:49,278
It's just this stripped down kind of Python language,

333
00:15:49,279 --> 00:15:51,198
but it was still Python at that point.

334
00:15:51,199 --> 00:15:54,479
And what Gconfig spit out at the end was a monolithic make file,

335
00:15:54,480 --> 00:15:55,599
but one you didn't have to write.

336
00:15:55,600 --> 00:16:01,599
Over time this evolved um it became uh blaze internally um there's some other systems

337
00:16:01,600 --> 00:16:04,398
are associated with now I don't even know how complicated it's got I haven't seen it

338
00:16:04,399 --> 00:16:08,159
for a long long time um but then that became basil externally it became buck there's

339
00:16:08,160 --> 00:16:11,679
some other you know people went to Facebook and they like that was awesome at Google

340
00:16:11,680 --> 00:16:16,638
>> what was the reason that make make files or make didn't really work was was it were

341
00:16:16,639 --> 00:16:22,270
we talking about build performance are we talking about maintainability or readability

342
00:16:22,320 --> 00:16:27,469
>> yeah so like I think make itself is like It's okay in declaring dependencies,

343
00:16:27,519 --> 00:16:31,359
but it's a little bit like kind of assembly language and you didn't want to write,

344
00:16:31,360 --> 00:16:33,119
you know, all your dependencies in assembly language.

345
00:16:33,120 --> 00:16:34,479
And if you didn't know what you were doing,

346
00:16:34,480 --> 00:16:37,198
it was easy to make a mistake and miss dependencies and whatnot.

347
00:16:37,199 --> 00:16:40,879
And so the the kind of my the thought behind the build files is like, hey,

348
00:16:40,880 --> 00:16:44,159
you need to express these dependencies but at a higher level and just like kind of cleaner

349
00:16:44,160 --> 00:16:45,439
semantics associated with it.

350
00:16:45,440 --> 00:16:48,638
And from that you can compile down into the assembly language.

351
00:16:48,639 --> 00:16:51,039
And then eventually folks were like, oh, you don't have to compile down to that.

352
00:16:51,040 --> 00:16:55,599
we can just kind of implement you know the dependency kind of update engine directly

353
00:16:55,600 --> 00:16:58,479
>> and then that's where the performance improvements were able to come in.

354
00:16:58,480 --> 00:17:03,119
So like because at Google scale like when you have a large repo in general like my understanding

355
00:17:03,120 --> 00:17:09,198
is that the reason Basil and Buck are so popular for large code bases is it can help

356
00:17:09,199 --> 00:17:10,879
you improve your build performance.

357
00:17:10,880 --> 00:17:14,798
It gives you a lot more le lovers to play around with from caching from being smart about

358
00:17:14,799 --> 00:17:17,119
cash generation to obviously just raw performance.

359
00:17:17,120 --> 00:17:18,399
>> Yeah, that's exactly it.

360
00:17:18,400 --> 00:17:23,869
Yeah. So you kind of just dabbled and like okay I'll I'll make this built file did that

361
00:17:23,919 --> 00:17:26,558
people took that over but what was your next main focus?

362
00:17:26,559 --> 00:17:28,959
Uh well so I mentioned GFS earlier,

363
00:17:28,960 --> 00:17:33,119
Google file system and at some point we realized that there was some limitations in GFS

364
00:17:33,120 --> 00:17:37,038
scalability bottlenecks because I've been working on the storage system for Gmail.

365
00:17:37,039 --> 00:17:39,359
Um I actually dabbled in another storage system, you know,

366
00:17:39,360 --> 00:17:41,599
kind of a research thing that never went anywhere.

367
00:17:41,600 --> 00:17:43,119
Um but because we're working on that,

368
00:17:43,120 --> 00:17:46,479
um got invited to participate in like the founding team of Colossus,

369
00:17:46,480 --> 00:17:48,319
which was the successor to GFS.

370
00:17:48,320 --> 00:17:50,319
And as far as I know, Colossus still exists.

371
00:17:50,320 --> 00:17:52,399
It's gone through multiple iterations at Google.

372
00:17:52,400 --> 00:17:55,839
Um but it's like the second generation, you know, distributed file system.

373
00:17:55,840 --> 00:17:57,950
So what is Colossus?

374
00:17:58,000 --> 00:18:00,479
>> Yeah. So when I say a distributed file system,

375
00:18:00,480 --> 00:18:03,678
externally you might think of something like S3 kind of blob storage,

376
00:18:03,679 --> 00:18:05,999
it had a flat name space kind of like S3,

377
00:18:06,000 --> 00:18:09,599
you give names and there's like a minor hierarchy there, but it's like very limited.

378
00:18:09,600 --> 00:18:11,839
Um but it's not like a PZIX file system.

379
00:18:11,840 --> 00:18:13,678
So you don't have the full directory hierarchy.

380
00:18:13,679 --> 00:18:15,599
You didn't even have the full like kind of permission system.

381
00:18:15,600 --> 00:18:18,959
A lot of that stuff got it added later, but the files are not stored on your local machine.

382
00:18:18,960 --> 00:18:22,558
There is a fleet, you know, kind of a service out there that has all the files.

383
00:18:22,559 --> 00:18:27,599
they're writing it down to their hard drives or now SSDs and your client can access that

384
00:18:27,600 --> 00:18:28,639
and it's all replicated.

385
00:18:28,640 --> 00:18:31,519
So if there's any crashes and whatnot, you're not losing your data.

386
00:18:31,520 --> 00:18:34,798
I don't know when um at some point S3 got eraser coding.

387
00:18:34,799 --> 00:18:36,399
We did eraser coding in Colossus.

388
00:18:36,400 --> 00:18:37,599
That was kind of a big breakthrough.

389
00:18:37,600 --> 00:18:40,399
>> What which coding >> we we used Re Solomon.

390
00:18:40,400 --> 00:18:43,359
Um so for the audience who's not familiar with eraser coding,

391
00:18:43,360 --> 00:18:45,629
you might think of like I want to have replicas.

392
00:18:45,679 --> 00:18:50,159
Um and there's a this thing in hard disks called RAID where it's like I don't actually

393
00:18:50,160 --> 00:18:51,439
have to have full replicas.

394
00:18:51,440 --> 00:18:55,759
I can actually you know if you take a plus a and b you can exor them together and then

395
00:18:55,760 --> 00:18:59,359
you can have this kind of third kind of version and there's more and more complicated

396
00:18:59,360 --> 00:19:02,879
versions of that read solomon is kind of you know I think there's actually better codes

397
00:19:02,880 --> 00:19:06,879
now but it's like one of the known ways to known ways yeah and we we had to you know

398
00:19:06,880 --> 00:19:10,479
kind of pioneer internally like oh how are we going to actually make this work in distributed

399
00:19:10,480 --> 00:19:15,950
file system >> were you focused on on latency on on being able to store data more efficiently

400
00:19:16,000 --> 00:19:21,439
>> storing it more efficiently so in GFS and there's varying costes where it's Well,

401
00:19:21,440 --> 00:19:24,319
you're not storing two copies or one copy of your data or two copies,

402
00:19:24,320 --> 00:19:27,759
you're storing three triplication, three times as much storage you're having to use.

403
00:19:27,760 --> 00:19:30,079
And with read Solomon, you can get that down quite a bit lower.

404
00:19:30,080 --> 00:19:34,239
I can't remember offhand exactly what our um what we used for read solid,

405
00:19:34,240 --> 00:19:36,319
but I think it was like essentially 2x.

406
00:19:36,320 --> 00:19:38,639
But you you get that with also the redundancy, too.

407
00:19:38,640 --> 00:19:41,038
So, it's like it's smaller and the redundancy is higher.

408
00:19:41,039 --> 00:19:44,558
>> So, cuz cuz I guess the the na the naive thing if if you're saying all right,

409
00:19:44,559 --> 00:19:46,798
I I want my data to be replicated at three places.

410
00:19:46,799 --> 00:19:50,319
You take three nodes, three machines, physical machines, and you say like copy one,

411
00:19:50,320 --> 00:19:52,558
copy one, copy one, I have it in three places.

412
00:19:52,559 --> 00:19:54,959
Great. If one explodes, I still have two.

413
00:19:54,960 --> 00:19:59,119
Wonderful. And then you're you're saying that the you know the algorithm here is you

414
00:19:59,120 --> 00:20:01,599
could take not 3x the data but 2x the data,

415
00:20:01,600 --> 00:20:05,678
split it smartly across machines or or maybe you could take it lower and you still have

416
00:20:05,679 --> 00:20:07,439
the thing where like oh one of them explodes,

417
00:20:07,440 --> 00:20:11,199
I still have all my data because it's it's split in a >> that's exactly right.

418
00:20:11,200 --> 00:20:15,199
and like kind of the mental model you know if you just want to understand at a high level

419
00:20:15,200 --> 00:20:19,519
which is essentially you might want to say like I want to have eight replicas of this

420
00:20:19,520 --> 00:20:23,279
data or I think I can't remember offh hand I think S3 might use nine they've actually

421
00:20:23,280 --> 00:20:28,158
talked about this publicly you have kind of nine chunks of data but any five of those

422
00:20:28,159 --> 00:20:31,839
chunks can be used to reconstruct it and what this means is you can lose any four copies

423
00:20:31,840 --> 00:20:35,839
and you can still reconstruct your data um and oftentimes it's more like you know the

424
00:20:35,840 --> 00:20:39,999
the first five chunks are exact replicas and the the other four are kind of parody ones

425
00:20:40,000 --> 00:20:44,079
I've kind of forgot some of details escape my mind but it's along this line >> but but

426
00:20:44,080 --> 00:20:47,999
when you come up with an algorithm you can then prove that this algorithm will work right

427
00:20:48,000 --> 00:20:52,639
like this is a little bit like I know in software engineering like maths and algorithms

428
00:20:52,640 --> 00:20:56,079
is a bit out of fashion but in this case this is really important because once you once

429
00:20:56,080 --> 00:21:00,399
you can prove it that this algorithm works it will work >> it will work and um you know

430
00:21:00,400 --> 00:21:05,599
it's like the math behind here is like kawa fields over like GF2 something like that

431
00:21:05,600 --> 00:21:09,839
I don't even know that the I never actually understood the full uh math behind it I always

432
00:21:09,840 --> 00:21:12,079
regretted not doing or math in college,

433
00:21:12,080 --> 00:21:14,879
but you didn't have to like read and saw them and proved how this worked.

434
00:21:14,880 --> 00:21:18,639
I think it's like back in the 1970s something associated with like communication network.

435
00:21:18,640 --> 00:21:23,918
So you just take that and uh you know kind of use that expertise but leverage that and

436
00:21:23,919 --> 00:21:27,119
have to do all the engineering behind it to make it work in a storage system.

437
00:21:27,120 --> 00:21:30,749
>> And then when building a distributed storage system like Colossus,

438
00:21:30,799 --> 00:21:32,719
what were other things beyond okay,

439
00:21:32,720 --> 00:21:35,999
you want to store data in a resilient but efficient way,

440
00:21:36,000 --> 00:21:38,719
what were other things that problems that you needed to solve?

441
00:21:38,720 --> 00:21:44,910
I'm I'm thinking things potentially like sharding and resharding or or or or or metadata

442
00:21:44,960 --> 00:21:46,639
uh being important those kind of things.

443
00:21:46,640 --> 00:21:50,479
>> Uh particularly for the scale that Google wanted to operate at GFS kind of had this

444
00:21:50,480 --> 00:21:51,279
scalability limit.

445
00:21:51,280 --> 00:21:54,798
I believe it was like you know you could have a thousand machines in a GFS cluster and

446
00:21:54,799 --> 00:21:59,038
it kind of >> which I guess sounds big until until like today it's kind of ridiculously

447
00:21:59,039 --> 00:22:01,600
small right >> it sounds big for

448
00:22:02,000 --> 00:22:06,319
and then Google was like no we need to have the scale up to 10,000 machines and there

449
00:22:06,320 --> 00:22:10,079
were some bottlenecks there was this GFS master as a single node it was a bit of a bottleneck

450
00:22:10,080 --> 00:22:13,599
we're like ah we need to have a distributed master to store the metadata for all the

451
00:22:13,600 --> 00:22:19,359
objects and Google at the time happened to have this system called bigtable and so colossus

452
00:22:19,360 --> 00:22:23,359
stored this metadata inside big table and one of the things that I'm kind of proud and

453
00:22:23,360 --> 00:22:27,439
like kind of like also you know a little bit embarrassed by uh one of the design choices

454
00:22:27,440 --> 00:22:32,879
I I went down but actually worked out is we wanted to use big table for the metadata

455
00:22:32,880 --> 00:22:37,999
for classes >> and the metadata just for for those of us not as into distributed system

456
00:22:38,000 --> 00:22:41,279
what is the metadata in a distributed file system >> yeah it's like the uh the names

457
00:22:41,280 --> 00:22:45,199
of the files and for each of the files the files are broken into chunks what were they

458
00:22:45,200 --> 00:22:49,038
64 megaby chunks and then you have to have the list of chunks for each file >> and then

459
00:22:49,039 --> 00:22:52,558
you you have to um periodically you know the master has to be scanning over this and

460
00:22:52,559 --> 00:22:56,558
doing repair um repair work and you know but there's more metadata than that but that's

461
00:22:56,559 --> 00:23:01,678
it in a nutshell >> so we wanted Colossus to have this you know kind of scalable you

462
00:23:01,679 --> 00:23:07,038
know service big table to store it metadata the biggest user of GFS is bigtable so we

463
00:23:07,039 --> 00:23:11,550
want big table to work on top of Colossus >> so wrap your head around the circle dependency

464
00:23:11,600 --> 00:23:17,038
>> yeah well uh there's a bootstrapping thing right >> bootstrapping so yes >> yeah which

465
00:23:17,039 --> 00:23:20,399
one starts up like you need to mock something somewhere right Yeah.

466
00:23:20,400 --> 00:23:23,678
No, I mean the way it actually worked at the time and they've since replaced this,

467
00:23:23,679 --> 00:23:25,678
you know, because this is like the way to get started,

468
00:23:25,679 --> 00:23:28,158
leverage what you have and eventually you kind of get rid of it.

469
00:23:28,159 --> 00:23:32,959
But there is the uh kind of a foundational big table that big table didn't use Colossus.

470
00:23:32,960 --> 00:23:36,399
There's the Colossus using the big table and then there's normal big table sitting on

471
00:23:36,400 --> 00:23:39,279
top of Colossus and it all worked for years.

472
00:23:39,280 --> 00:23:41,038
So I don't even know when they got rid of that.

473
00:23:41,039 --> 00:23:42,079
They got rid of it at some point,

474
00:23:42,080 --> 00:23:48,079
but uh that was >> But I I I guess sounds like you can make like hacks that go really

475
00:23:48,080 --> 00:23:50,558
long knowing that they're hacks and they get you off the ground, right?

476
00:23:50,559 --> 00:23:51,599
>> They get you off the ground, right?

477
00:23:51,600 --> 00:23:52,959
Cuz if we had to like, you know,

478
00:23:52,960 --> 00:23:56,798
implement that kind of big table layer from the get-go or it just delayed how long it

479
00:23:56,799 --> 00:23:58,639
took to get, you know, Colossus built.

480
00:23:58,640 --> 00:24:00,800
One of the things

481
00:24:01,120 --> 00:24:06,639
that strike me about a system like Colossus is it promises or this was internal to Google

482
00:24:06,640 --> 00:24:12,270
but but even distributed file system that are are external they will promise high throughput

483
00:24:12,320 --> 00:24:17,999
high availability and low latency and to me it's always a bit conflicting of like well

484
00:24:18,000 --> 00:24:22,639
it's pretty easy to I guess build a distributed file system with my limited knowledge

485
00:24:22,640 --> 00:24:26,558
I could probably do something where I have either high throughput but high latency because

486
00:24:26,559 --> 00:24:29,199
whenever I write something I write it out all the replicas.

487
00:24:29,200 --> 00:24:32,319
You you already like like mentioned one one technique of of doing it,

488
00:24:32,320 --> 00:24:36,959
but how did you kind of reconcile like how how did you get like low latency while you

489
00:24:36,960 --> 00:24:40,479
have high throughput while you also have replication going on on the file system?

490
00:24:40,480 --> 00:24:42,158
>> Well, I mean the these distributed file systems,

491
00:24:42,159 --> 00:24:45,519
I mean the latency isn't super low in particular when they're when they're running on

492
00:24:45,520 --> 00:24:49,359
hard disks which Colossus was doing at the time which S3 does.

493
00:24:49,360 --> 00:24:50,558
You actually notice the latency.

494
00:24:50,559 --> 00:24:52,558
So like S3 is a high performance system.

495
00:24:52,559 --> 00:24:57,439
Google um GCS the the competitor from Google which is built on top of Colossus it's high

496
00:24:57,440 --> 00:25:01,599
performance has incredible throughput but the latencies are like 20 to 30 milliseconds

497
00:25:01,600 --> 00:25:06,639
per first read and that is bounded by your hard disk latency if you put it on SSD it

498
00:25:06,640 --> 00:25:11,439
gets down to closer to SSD latencies but not actually kind of the state of art SSD latencies

499
00:25:11,440 --> 00:25:14,639
which is kind of this crazy thing that's been happening in our industry it's just like

500
00:25:14,640 --> 00:25:19,119
how much faster the hardware has been getting reading from a hard drive maybe 5 to 10

501
00:25:19,120 --> 00:25:25,759
milliseconds nowadays I never touch hard drives reading from an SSD over NVMe 30 microscs,

502
00:25:25,760 --> 00:25:28,798
50 microsconds. So that's a microsconds.

503
00:25:28,799 --> 00:25:30,959
There's 1,000 microscs in 1 millisecond.

504
00:25:30,960 --> 00:25:32,879
So we're talking like a huge huge difference.

505
00:25:32,880 --> 00:25:35,918
>> Well, there are now startups or infrastructure companies that are starting to take

506
00:25:35,919 --> 00:25:39,918
advantage of the fact that they can they can have an MVME layer and and they they they

507
00:25:39,919 --> 00:25:42,798
they pull things up either predictively or not.

508
00:25:42,799 --> 00:25:45,918
But but as you say, like when the physical reality changes,

509
00:25:45,919 --> 00:25:48,959
you can build systems on top of it that that should take advantage of it.

510
00:25:48,960 --> 00:25:49,839
>> Yeah, absolutely.

511
00:25:49,840 --> 00:25:52,879
He knows thought that the discs are so much faster with SSDs.

512
00:25:52,880 --> 00:25:54,558
The networks are so much faster.

513
00:25:54,559 --> 00:26:00,158
I mean, just kind of crazy how fast like the intrazone latencies are in a Google or Amazon

514
00:26:00,159 --> 00:26:03,038
center. And it wasn't like that when we were building Colossus, you know,

515
00:26:03,039 --> 00:26:04,479
I can't even remember what the numbers were,

516
00:26:04,480 --> 00:26:08,319
but milliseconds to do a network round trip and now it's down in, you know,

517
00:26:08,320 --> 00:26:10,399
100 microsconds within a zone.

518
00:26:10,400 --> 00:26:12,879
I mean, I just look at these things, I'm like, holy crap.

519
00:26:12,880 --> 00:26:14,959
You know, the hardware guys have really done a good job.

520
00:26:14,960 --> 00:26:19,439
>> Yeah. Some sometimes I feel that our software should be feel way more snappy and there

521
00:26:19,440 --> 00:26:23,599
are some snappy software but sometimes I almost wonder if we're getting too complacent

522
00:26:23,600 --> 00:26:27,119
with all the abstractions or we're not even doing this like napkin math.

523
00:26:27,120 --> 00:26:31,359
Simon Erikson at Turbopuffer talks about this napkin math where like like you was like

524
00:26:31,360 --> 00:26:36,639
all right here's the theoretical limitation of of the hardware read from SSD might be

525
00:26:36,640 --> 00:26:42,079
I don't know 30 m microsconds and then like how can I build a system that is as close

526
00:26:42,080 --> 00:26:45,678
to this as possible as opposed to the other way around saying okay like you know a human

527
00:26:45,679 --> 00:26:49,918
will notice like 20 mill like 100 milliseconds let's build around that.

528
00:26:49,919 --> 00:26:52,319
>> Yeah. Yeah. And sometimes like when you're architecting something,

529
00:26:52,320 --> 00:26:55,599
you need to think about the human kind of perceptible latencies,

530
00:26:55,600 --> 00:26:58,079
but oftentimes when you're dealing at the storage system layer, you know,

531
00:26:58,080 --> 00:27:01,119
I know Simon working on Turboper, they're doing great stuff over there.

532
00:27:01,120 --> 00:27:03,918
You have to think about the machine scale and the machine speed,

533
00:27:03,919 --> 00:27:05,999
which is a lot faster than human perception.

534
00:27:06,000 --> 00:27:10,399
Like a human can tolerate maybe 100 milliseconds of delay or if you're playing a game,

535
00:27:10,400 --> 00:27:13,790
maybe you need to have like, you know, frame rates of like every four milliseconds,

536
00:27:13,840 --> 00:27:16,319
but the machine wants much much faster than that.

537
00:27:16,320 --> 00:27:17,519
Someone says napkin math.

538
00:27:17,520 --> 00:27:21,678
I call this speed of light numbers and like sometimes it's literally the speed of light

539
00:27:21,679 --> 00:27:23,038
bottlenecking you.

540
00:27:23,039 --> 00:27:27,439
The cross zone latencies between zones, cross region latencies is speed of light in fiber.

541
00:27:27,440 --> 00:27:29,119
You want to hear a kind of crazy fact?

542
00:27:29,120 --> 00:27:30,479
>> I I love hearing crazy facts.

543
00:27:30,480 --> 00:27:35,038
>> The fastest way to send um packive data across the globe is to send into space.

544
00:27:35,039 --> 00:27:39,519
>> Is it because the speed of light is faster in vacuum?

545
00:27:39,520 --> 00:27:40,399
>> Quite a bit faster.

546
00:27:40,400 --> 00:27:42,430
>> Or well, it's not full vacuum.

547
00:27:42,480 --> 00:27:45,759
No way. So So you cover a higher distance.

548
00:27:45,760 --> 00:27:47,999
>> Yeah. Well, you actually uh I believe the the way to do this,

549
00:27:48,000 --> 00:27:50,879
you send it straight up and then you with Starlink, you send it straight up,

550
00:27:50,880 --> 00:27:53,278
you bounce it around between Starlink, you send it down to the other side.

551
00:27:53,279 --> 00:27:55,950
So, you want to get it out of the atmosphere as quickly as possible.

552
00:27:56,000 --> 00:27:57,759
>> But now, if we're if we're talking speed of light,

553
00:27:57,760 --> 00:28:00,239
this will also go there as like a digital transformation happening.

554
00:28:00,240 --> 00:28:04,398
So, you would need to calculate how long it takes for that system to to process and do

555
00:28:04,399 --> 00:28:07,759
it. And and but you're saying that even if if you do this like really well,

556
00:28:07,760 --> 00:28:12,239
it will be faster than beaming it through an optical c and an optical cable slows down

557
00:28:12,240 --> 00:28:13,439
the speed of light, right?

558
00:28:13,440 --> 00:28:13,999
>> Yeah. Yeah. Yeah.

559
00:28:14,000 --> 00:28:16,239
Yeah, the speed of light is is only the speed of light in vacuum.

560
00:28:16,240 --> 00:28:17,759
In every other medium, it's slower.

561
00:28:17,760 --> 00:28:21,918
>> I was talking I did a deep dive on uh the hedge fund industry and they didn't tell

562
00:28:21,919 --> 00:28:25,519
me exactly. They said that they do use satellites and microwaves and some of these things.

563
00:28:25,520 --> 00:28:28,079
They they will not tell you because you know this is their thing.

564
00:28:28,080 --> 00:28:33,918
But I had a suspicion that they might have found a faster way and I think this is like

565
00:28:33,919 --> 00:28:37,678
somewhat wellknown but the the details they're not going to get into because again that

566
00:28:37,679 --> 00:28:40,719
but like Yeah. So so they're probably bouncing stuff in space.

567
00:28:40,720 --> 00:28:41,678
>> Yeah. Yeah. So, I mean,

568
00:28:41,679 --> 00:28:45,199
one of the things that they do in the high frequency train is like between New York and

569
00:28:45,200 --> 00:28:49,119
Chicago, it's actually not far enough distance-wise to make it worthwhile to send it

570
00:28:49,120 --> 00:28:51,278
into space. So, they were doing microwave beams there.

571
00:28:51,279 --> 00:28:53,199
Yeah. >> But, I mean, if you really want to get faster,

572
00:28:53,200 --> 00:28:57,038
you need to build like a vacuum tube between them and like send a maybe they're doing

573
00:28:57,039 --> 00:28:57,519
that. I don't know.

574
00:28:57,520 --> 00:28:58,319
Maybe they're doing that.

575
00:28:58,320 --> 00:28:59,038
>> Yeah. But, I mean,

576
00:28:59,039 --> 00:29:02,558
the thing that I think is like just kind of awesome about performance nowadays is I mean,

577
00:29:02,559 --> 00:29:06,639
there's so many layers you have to be paying attention to in terms of performance that

578
00:29:06,640 --> 00:29:09,038
the rabbit hole just goes so deep there.

579
00:29:09,039 --> 00:29:12,079
you're almost certainly running on multi-threaded systems, right?

580
00:29:12,080 --> 00:29:14,398
And you're like, "Oh, I need to have a multi-threaded program.

581
00:29:14,399 --> 00:29:16,319
I need to have synchronization in there."

582
00:29:16,320 --> 00:29:19,629
Well, you get the best performance if you just kind of avoid the synchronization.

583
00:29:19,679 --> 00:29:22,398
And part of this is lock free programming, but part of it is arranging.

584
00:29:22,399 --> 00:29:23,519
So, you don't need locks at all.

585
00:29:23,520 --> 00:29:25,599
You need to carry about your processor caches.

586
00:29:25,600 --> 00:29:29,439
And there's this whole like kind of uh setup of caches above the CPU.

587
00:29:29,440 --> 00:29:33,150
Like you can think of registers as a cache, and you have L1, L2, L3,

588
00:29:33,200 --> 00:29:35,038
um even your memory and onto disk.

589
00:29:35,039 --> 00:29:38,079
Um and there's just you have to pay attention to all those levels.

590
00:29:38,080 --> 00:29:41,038
And if you do, your performance gets way better.

591
00:29:41,039 --> 00:29:44,398
And if you ignore it and you're like, "Oh, I'm not worrying about kind of kind of,

592
00:29:44,399 --> 00:29:46,239
you know, kind of the cache accesses.

593
00:29:46,240 --> 00:29:50,879
I'm just accessing data all over your program will just be way way slower."

594
00:29:50,880 --> 00:29:53,359
And some folks pay a ton of attention to this.

595
00:29:53,360 --> 00:29:55,999
The high frequency trading, I mean, they do this all day long.

596
00:29:56,000 --> 00:29:59,038
And and in so many other places, like no one pays attention to it,

597
00:29:59,039 --> 00:30:03,278
and you get this kind of gradual degradation in the the performance of the hardware or

598
00:30:03,279 --> 00:30:06,158
the software. I did want to talk a bit more about low-level stuff,

599
00:30:06,159 --> 00:30:07,918
but not about the the speed of light,

600
00:30:07,919 --> 00:30:12,270
but low-level data structures and programming language features.

601
00:30:12,320 --> 00:30:15,599
You made some contributions to the standard library, right?

602
00:30:15,600 --> 00:30:18,239
>> Well, I've done a couple um not quite standard libraries.

603
00:30:18,240 --> 00:30:21,839
So, uh I mean I just like have been always fascinated by data structures.

604
00:30:21,840 --> 00:30:23,199
It's kind of awesome.

605
00:30:23,200 --> 00:30:26,798
I mean, I I think just algorithms in general, you just like sorting algorithms,

606
00:30:26,799 --> 00:30:27,918
they're just kind of awesome.

607
00:30:27,919 --> 00:30:31,439
You could probably explain, you know, insertion sort and you know, kind of everything.

608
00:30:31,440 --> 00:30:33,999
Could could you explain quick sort as well?

609
00:30:34,000 --> 00:30:35,119
>> Quicks sort's a little bit harder.

610
00:30:35,120 --> 00:30:36,879
I >> that's the thing.

611
00:30:36,880 --> 00:30:38,798
>> Yeah. And then >> No, no, but it's it's a smart one.

612
00:30:38,799 --> 00:30:42,558
>> Yeah. And then you get to these levels of like, you know, it's like, oh, wait,

613
00:30:42,559 --> 00:30:44,239
someone really smart came up with this.

614
00:30:44,240 --> 00:30:48,719
So, one of the things I worked on just as a little bit of a side at Google at some point

615
00:30:48,720 --> 00:30:51,519
was a colleague came to me and he was like, you know what,

616
00:30:51,520 --> 00:30:54,479
we're using the STL map structure all over the place.

617
00:30:54,480 --> 00:30:57,278
And the STL map structure is a balanced binary tree.

618
00:30:57,279 --> 00:31:01,199
I can't remember if it was red black trees or one of these other bouncing algorithms

619
00:31:01,200 --> 00:31:03,918
that many CS college students implement.

620
00:31:03,919 --> 00:31:06,798
And he came to me, he's like, I think we could do better because, you know,

621
00:31:06,799 --> 00:31:08,479
there's actually a cache problem here.

622
00:31:08,480 --> 00:31:12,079
Every time you every node you're traversing down, you're going to a different cache line.

623
00:31:12,080 --> 00:31:15,278
And he was thinking about using something else, a skip list to do this.

624
00:31:15,279 --> 00:31:19,439
Um, which, um, that's another awesome awesome data structure everybody should kind of

625
00:31:19,440 --> 00:31:22,558
look at. But at some point, I was like, actually, this feels more like a B tree.

626
00:31:22,559 --> 00:31:28,398
So I'd implemented B trees a couple times before and figured out like how to implement

627
00:31:28,399 --> 00:31:32,158
a B tree that implemented almost all the semantics of the STL map.

628
00:31:32,159 --> 00:31:37,518
It couldn't quite do it perfectly and the reason is uh when you insert into a B tree

629
00:31:37,519 --> 00:31:40,639
node you have to shift stuff around so you don't get pointer stability.

630
00:31:40,640 --> 00:31:44,319
This is just kind of fundamental but if you can you know you don't need that for your

631
00:31:44,320 --> 00:31:46,479
use case you actually can pack more data in.

632
00:31:46,480 --> 00:31:50,398
So the thing about B tree like the the the real easy way to describe this is like you

633
00:31:50,399 --> 00:31:54,879
just have a small list of items like eight items that the the best way to store that

634
00:31:54,880 --> 00:31:58,879
if you want to you know kind of fast access in sorted order is just to sort the items

635
00:31:58,880 --> 00:32:00,319
right literally not to have a tree at all.

636
00:32:00,320 --> 00:32:03,839
Yeah. for like eight items >> just have it in a very simple list >> very very just an

637
00:32:03,840 --> 00:32:07,278
array sort the array and then you can either do a linear scan over it you can do a binary

638
00:32:07,279 --> 00:32:10,879
search and oftentimes linear scan is faster and then you think about that like well if

639
00:32:10,880 --> 00:32:14,879
I want to store more than eight items I can just have one node that has eight items and

640
00:32:14,880 --> 00:32:17,999
then I have another node and then you have a parent node that connects them together

641
00:32:18,000 --> 00:32:21,678
and that is essentially like the you know you build it you think about building it bottom

642
00:32:21,679 --> 00:32:26,558
up you start with just one node of eight items oh I need to insert the the ninth um thing

643
00:32:26,559 --> 00:32:30,719
I'll split into two chunks and the two chunks one will have four the other will five

644
00:32:30,720 --> 00:32:34,239
and then you have a a parent node that points to them and then you just kind of recurse

645
00:32:34,240 --> 00:32:36,558
on that. That's the B tree algorithm in a nutshell.

646
00:32:36,559 --> 00:32:37,518
Everybody go implement it.

647
00:32:37,519 --> 00:32:40,479
Actually, nobody should implement this anymore because nowadays we we have something

648
00:32:40,480 --> 00:32:44,639
else that will implement this and all the optimizations because there's a crap ton of

649
00:32:44,640 --> 00:32:46,879
optimizations that you can do on a P tree.

650
00:32:46,880 --> 00:32:50,639
But but I just want to go back to this like there was already an existing implementation

651
00:32:50,640 --> 00:32:56,239
for maps and then so your your colleague looked at the code and said we I think we can

652
00:32:56,240 --> 00:33:00,319
do better. I what I want to figure out is like in my mind again someone sitting outside

653
00:33:00,320 --> 00:33:07,518
of you know I'm not involved in in how some of the these libraries are or data structures

654
00:33:07,519 --> 00:33:11,759
are built. I always thought and again this is might be naive but really smart people

655
00:33:11,760 --> 00:33:15,439
sit down they kind of look at the state-of-the-art they implement it and there's no way

656
00:33:15,440 --> 00:33:16,158
it can be faster.

657
00:33:16,159 --> 00:33:20,798
In fact I've had arguments in the past saying like oh let's let's write a faster sorting

658
00:33:20,799 --> 00:33:22,959
thing. Like it's surely it is the fastest.

659
00:33:22,960 --> 00:33:27,439
But if if you could bring us a little bit of like how like you you've been inside how

660
00:33:27,440 --> 00:33:33,119
it act how it happens and how other people like yourself and and your colleague can say

661
00:33:33,120 --> 00:33:35,518
like oh what what if we what if we try something else?

662
00:33:35,519 --> 00:33:40,479
>> Yeah. Yeah. So I mean my recollection here is he was working on this kind of the big

663
00:33:40,480 --> 00:33:44,479
um internal system I think it was called Gaia that actually had the mapping from you

664
00:33:44,480 --> 00:33:49,439
know you log in you have your user ID and you have to look this up and they were storing

665
00:33:49,440 --> 00:33:54,639
you know all all like this the map from user ID and email to whatnot to the metadata

666
00:33:54,640 --> 00:33:58,398
about the user >> in STL maps and you just noticed like well there's a lot of memory

667
00:33:58,399 --> 00:34:01,599
usage here and it shows up on profiles and then we're like well what can we do to do

668
00:34:01,600 --> 00:34:05,199
better and that was kind of the genesis of it and you know he happened to be working

669
00:34:05,200 --> 00:34:08,878
on it me happened and they were working with me and like we just started kind of noodling

670
00:34:08,879 --> 00:34:11,838
on this problem like oh can we do something better and it's not one of these things like

671
00:34:11,839 --> 00:34:16,078
I think now with Google they have a whole team working on their kind of internal libraries

672
00:34:16,079 --> 00:34:20,239
at the time it was more of like you know everybody working on their own systems and contributing

673
00:34:20,240 --> 00:34:24,719
to a shared base but but I guess it still goes back to what you were just saying of like

674
00:34:24,720 --> 00:34:29,358
just go down the layers try to understand and if something just doesn't add up like suddenly

675
00:34:29,359 --> 00:34:33,439
like oh there's this big explosion of memory usage like you know just ask the questions

676
00:34:33,440 --> 00:34:37,199
why why is this and if you're able to or you're happen to be like, oh,

677
00:34:37,200 --> 00:34:38,158
can we do something about it?

678
00:34:38,159 --> 00:34:40,878
Right. >> Yeah. And and one of the things that, you know, he observed early on,

679
00:34:40,879 --> 00:34:45,118
I think part of one of the things was it was like a map from integer ID to something

680
00:34:45,119 --> 00:34:47,759
else. And like if you look at red black tree,

681
00:34:47,760 --> 00:34:51,678
every node you have your kind of value that you're storing in the map and then you have

682
00:34:51,679 --> 00:34:56,319
two pointers. You might have an integer ID that's like four bytes and then two waste.

683
00:34:56,320 --> 00:34:59,039
>> Yeah. And you look at that, you're like, "Oo, that seems like a lot of overhead."

684
00:34:59,040 --> 00:35:00,399
And like >> just you could use a bit.

685
00:35:00,400 --> 00:35:04,399
Well, the beach tree actually has a lot better um it has better spatial locality and

686
00:35:04,400 --> 00:35:05,519
that's what made it faster.

687
00:35:05,520 --> 00:35:09,519
Um but it was actually smaller as well at the same time because you had less pointers

688
00:35:09,520 --> 00:35:11,838
involved. >> You also contributed to Go, right?

689
00:35:11,839 --> 00:35:13,279
>> Yeah. Yeah. Well, that came later.

690
00:35:13,280 --> 00:35:14,799
>> Yeah. Yeah, it came a lot later.

691
00:35:14,800 --> 00:35:15,838
Can we talk about that?

692
00:35:15,839 --> 00:35:18,879
>> Yeah. I just one of these other things, you know, I I pay attention to like,

693
00:35:18,880 --> 00:35:22,319
you know, when there's research papers coming out about new data structures and like

694
00:35:22,320 --> 00:35:27,358
hashts. Hashts are like the the one of the earliest things you learn about in college

695
00:35:27,359 --> 00:35:28,239
in data structures.

696
00:35:28,240 --> 00:35:33,519
like how do I map keys to values where the ordering is unimportant that's when hashts

697
00:35:33,520 --> 00:35:37,279
come in and there's like you know the the very earliest ways to do this I've implemented

698
00:35:37,280 --> 00:35:41,759
hashts multiple times is like you take your key and it might be a string and you you

699
00:35:41,760 --> 00:35:45,439
put it through a function and it spits out an integer and then you map that into an array

700
00:35:45,440 --> 00:35:49,118
of buckets and if multiple things map to the same bucket you have to have a link link

701
00:35:49,119 --> 00:35:52,639
list >> you have a link list yeah this is a naive implementation >> naive implementation

702
00:35:52,640 --> 00:35:57,358
used quite frequently and over time people discovered like a lot better ways to do hashts

703
00:35:57,359 --> 00:35:59,759
there's very Like that's called chaining of your hash.

704
00:35:59,760 --> 00:36:03,598
There's another technique called open addressing where instead of actually having a link

705
00:36:03,599 --> 00:36:08,319
list, you just uh kind of hash it again and move on or kind of walk down to subsequent

706
00:36:08,320 --> 00:36:12,159
buckets to find out like or sub subsequent slots to find out where you should be.

707
00:36:12,160 --> 00:36:14,319
And I remember reading about this uh new technique.

708
00:36:14,320 --> 00:36:15,999
It came out of some folks at Google.

709
00:36:16,000 --> 00:36:18,879
I believe it came out of their Swiss office.

710
00:36:18,880 --> 00:36:20,430
It was called Swiss tables.

711
00:36:20,480 --> 00:36:22,479
>> Um I believe that's where the naming Yeah.

712
00:36:22,480 --> 00:36:24,719
Yeah. I believe where that's where the naming came from.

713
00:36:24,720 --> 00:36:26,159
um not 100% sure about that,

714
00:36:26,160 --> 00:36:29,598
but I remember reading about it and then I was working on Go for a long period of time

715
00:36:29,599 --> 00:36:33,919
and Go has this built-in map structure and it's a hasht.

716
00:36:33,920 --> 00:36:38,799
a very highly optimized um hasht because the go team is very competent um the go runtime

717
00:36:38,800 --> 00:36:42,799
tag team and various folks had taken an attempt at like you know um putting together

718
00:36:42,800 --> 00:36:47,519
a Swiss table implication for go and I tested some of them and I was like this is kind

719
00:36:47,520 --> 00:36:50,799
of fascinating what Swiss tables do and I'll explain how it works in just a second but

720
00:36:50,800 --> 00:36:54,719
I looked at it's like well it's really hard to beat the performance of the runtime the

721
00:36:54,720 --> 00:37:00,078
runtime was really good and I kept on I noodled on this for a little while and uh eventually

722
00:37:00,079 --> 00:37:05,439
I ended up having to take this business trip to uh to India to Bangalore and so I was

723
00:37:05,440 --> 00:37:06,639
on a long flight.

724
00:37:06,640 --> 00:37:10,959
No. >> Yeah. >> It always starts like this >> and I just like I'm just going to try to

725
00:37:10,960 --> 00:37:14,559
uh pull on this and I pulled on it sufficiently that I can get some of the benchmarks

726
00:37:14,560 --> 00:37:15,679
to be faster. >> Wow.

727
00:37:15,680 --> 00:37:18,959
>> And then like you know that like is kind of like catnip for an engineer like can I

728
00:37:18,960 --> 00:37:23,118
make it all faster figure all the rest of it you know got some help from the runtime

729
00:37:23,119 --> 00:37:27,118
folks. there's a an issue on the go issue tracker that you know where other people have

730
00:37:27,119 --> 00:37:30,719
been attempting this because people proposed like hey let's use Swiss tables and like

731
00:37:30,720 --> 00:37:33,759
the go like well you know we don't quite know all the details you're going to have to

732
00:37:33,760 --> 00:37:37,598
navigate this and that and there were some ideas there that combined them together and

733
00:37:37,599 --> 00:37:41,199
um got to the point where we had kind of a complete implementation that was faster on

734
00:37:41,200 --> 00:37:45,919
on most benchmarks not quite all of them but most of them and then the go folks eventually

735
00:37:45,920 --> 00:37:49,919
picked this up and pushed over to the finish line and >> and then so you you know like

736
00:37:49,920 --> 00:37:54,479
you came with the idea you you got to the point where you were able to show an implementation

737
00:37:54,480 --> 00:37:56,799
that showed how some of the benchmarks were faster.

738
00:37:56,800 --> 00:37:59,679
And then you started to work with some folks on the Go team, too.

739
00:37:59,680 --> 00:38:03,279
>> Well, I didn't I it wasn't quite that I I came up with an implementation that we ended

740
00:38:03,280 --> 00:38:04,879
up using at my company.

741
00:38:04,880 --> 00:38:06,399
It was good for our use case,

742
00:38:06,400 --> 00:38:09,759
but actually putting it into the runtime is a whole other, you know, kind of level.

743
00:38:09,760 --> 00:38:13,838
>> But then you you just showed like here's this implementation and then they they they

744
00:38:13,839 --> 00:38:17,358
kind of took the inspiration and the ideas and >> yeah, they're like this is great.

745
00:38:17,359 --> 00:38:20,719
We want to make all they they always are looking for ways to make runtime faster and

746
00:38:20,720 --> 00:38:24,879
there's like you know >> oh wait and then so so you did this in in this contribution

747
00:38:24,880 --> 00:38:28,399
this was a lot after you left Google right so this was from from the outside >> this

748
00:38:28,400 --> 00:38:31,039
is from the outside >> that's awesome yeah I know that people contribute stuff to the

749
00:38:31,040 --> 00:38:36,159
outside as well you know we had another colleague um he contributed one of the CRC implementations

750
00:38:36,160 --> 00:38:41,118
you know adapting some stuff that >> CRC >> uh cyclic redundancy checks sum you know

751
00:38:41,119 --> 00:38:45,279
Intel published some papers about here's how to do the CRC in assembly very fast and

752
00:38:45,280 --> 00:38:48,319
contributed one of the implementations you see a number of those things where you know

753
00:38:48,320 --> 00:38:50,479
people just like are contributing externally.

754
00:38:50,480 --> 00:38:51,759
It's not a lot honestly.

755
00:38:51,760 --> 00:38:55,358
I mean I actually don't know the full details but you know people are regularly contributing

756
00:38:55,359 --> 00:38:56,239
to these things.

757
00:38:56,240 --> 00:38:59,999
Peter just described how the Swiss table work came together using an issue tracker in

758
00:39:00,000 --> 00:39:04,239
the go tracker with different people contributing to the work and then the go team pushing

759
00:39:04,240 --> 00:39:06,159
all of this over the finish line.

760
00:39:06,160 --> 00:39:10,078
This is where I need to mention our season sponsor linear which is a place to coordinate

761
00:39:10,079 --> 00:39:12,830
work between humans as well as agents.

762
00:39:12,880 --> 00:39:17,039
One thing I've noticed about how most of us work with agents is how it's a pretty single

763
00:39:17,040 --> 00:39:20,879
player thing. You open a terminal UI, go back and forth with an agent,

764
00:39:20,880 --> 00:39:22,719
and it usually produces a PR,

765
00:39:22,720 --> 00:39:26,559
but the rest of your team has no idea what happened in that chat unless you tell them

766
00:39:26,560 --> 00:39:28,078
or copy the whole history.

767
00:39:28,079 --> 00:39:29,838
And when everyone in the team works like this,

768
00:39:29,839 --> 00:39:32,639
a lot of work happens that's invisible to the rest of the team.

769
00:39:32,640 --> 00:39:35,358
Linear's take is that agent work should be teamwork.

770
00:39:35,359 --> 00:39:38,319
Even today, teams already use linear to define the work to be done.

771
00:39:38,320 --> 00:39:41,519
Now, linear can already delegate an issue to a coding agent.

772
00:39:41,520 --> 00:39:46,078
This agent could be an AI agent that Linear integrates with like Codex or Cursor or Linear's

773
00:39:46,079 --> 00:39:48,029
own agent or a custom agent.

774
00:39:48,079 --> 00:39:51,519
Either way, the engine you're delegating stays responsible for the outcome.

775
00:39:51,520 --> 00:39:54,990
What I really like about how Linear works is how the work stays visible.

776
00:39:55,040 --> 00:39:58,319
Your teammates can follow the session of the agent, check out the PR producers,

777
00:39:58,320 --> 00:39:59,519
and join the review.

778
00:39:59,520 --> 00:40:03,630
We've gone from single player work to multiplayer engineering work with agents.

779
00:40:03,680 --> 00:40:06,719
Oh, and one more thing I like about linear, a focus on costs.

780
00:40:06,720 --> 00:40:10,990
Linear agents auto routing chooses a model that is the best suited for the task.

781
00:40:11,040 --> 00:40:13,759
Teams can also inspect usage and set limits, track usage,

782
00:40:13,760 --> 00:40:17,679
so you can use capable agents without having cost balloon out of control.

783
00:40:17,680 --> 00:40:19,760
Hop on board at linear.app/pragmatic.

784
00:40:20,640 --> 00:40:25,630
Peter also previously mentioned Gaia, Google's internal system that map login to users.

785
00:40:25,680 --> 00:40:28,959
It's not surprising that Google customuilt all systems, including this one,

786
00:40:28,960 --> 00:40:31,519
but most of us won't build our internal Gaia.

787
00:40:31,520 --> 00:40:34,430
This brings us to our season sponsor, Work OS.

788
00:40:34,480 --> 00:40:37,598
You can think of work as something like Gaia for the rest of us.

789
00:40:37,599 --> 00:40:41,309
Identity infrastructure you've otherwise spent quarters building yourself.

790
00:40:41,359 --> 00:40:45,759
Workers includes single sign on, skim directory sync, audit logs, rolebased access control.

791
00:40:45,760 --> 00:40:49,838
Basically everything a big customer security team asks for delivered as a handful of

792
00:40:49,839 --> 00:40:55,118
clean APIs. It's how companies go from we have a login to we can sell to a fortune 500

793
00:40:55,119 --> 00:40:57,919
without standing up their own internal identity platform.

794
00:40:57,920 --> 00:41:01,759
And work is already building for the next version of the agentic authorization problem.

795
00:41:01,760 --> 00:41:03,439
Their newest product is airlock.

796
00:41:03,440 --> 00:41:05,870
the authorization layer for AI agents.

797
00:41:05,920 --> 00:41:09,999
Think about what happens when you hand an agent a task like clean up the sale opportunities

798
00:41:10,000 --> 00:41:11,230
in our pipeline.

799
00:41:11,280 --> 00:41:15,358
The last thing you want is for this thing to have standing permissions to delete whatever

800
00:41:15,359 --> 00:41:20,239
it likes. Airlock sits between your agent and the tools they call.

801
00:41:20,240 --> 00:41:25,439
It evaluates every request against the agents intent and your rules and then it allows

802
00:41:25,440 --> 00:41:28,559
it, denies it, or routes it to a human for approval.

803
00:41:28,560 --> 00:41:30,959
You write down the policies in plain language.

804
00:41:30,960 --> 00:41:35,279
The agent never sees your credentials and every call and verdict gets logged.

805
00:41:35,280 --> 00:41:39,519
It works with coding agents like Cloud Code and Codeex and with MCP gateways.

806
00:41:39,520 --> 00:41:44,190
So if you're working out how to let agents do real work in production without overpermissioning

807
00:41:44,240 --> 00:41:47,519
them, take a look at work airlock at work.com/airlock.

808
00:41:48,720 --> 00:41:53,600
And with this, let's get back to Peter and why he left Google after building Colossus.

809
00:41:53,839 --> 00:41:56,719
You were at Google, you're building Colossus distributed file systems.

810
00:41:56,720 --> 00:42:01,598
you were at this point probably working on probably the largest system on the planet

811
00:42:01,599 --> 00:42:04,639
honestly. Why did you even consider leaving?

812
00:42:04,640 --> 00:42:05,759
>> Yeah. Yeah. Well,

813
00:42:05,760 --> 00:42:09,838
uh after Colossus I kind of dabbled in this other project called Google Goggles for a

814
00:42:09,839 --> 00:42:11,870
little while. Remember the glass holes?

815
00:42:11,920 --> 00:42:14,318
Yeah. >> He keeps coming back the idea by the way.

816
00:42:14,319 --> 00:42:14,879
So >> yeah. Yeah.

817
00:42:14,880 --> 00:42:16,318
Now it's still here and present.

818
00:42:16,319 --> 00:42:17,598
>> It seems like Google's early.

819
00:42:17,599 --> 00:42:21,039
>> Yeah. Yeah. And you know I I think that was a technology before its time.

820
00:42:21,040 --> 00:42:22,959
I don't think it was ready to do at that point.

821
00:42:22,960 --> 00:42:26,399
And looks like the uh actually doing the glasses is quite a bit harder.

822
00:42:26,400 --> 00:42:30,479
You know the Android phones we were trying to power it on were you know not powerful

823
00:42:30,480 --> 00:42:34,239
enough. And then you know I just kind of got wanderlust you know like h you know am I

824
00:42:34,240 --> 00:42:35,358
just kind of stagnating here.

825
00:42:35,359 --> 00:42:39,679
Google which is a strange thing to say but you know some other people feel as well and

826
00:42:39,680 --> 00:42:43,118
decided to go off and try my hand at another startup that didn't work out.

827
00:42:43,119 --> 00:42:45,039
We got aqua hired by Square.

828
00:42:45,040 --> 00:42:47,519
>> So yeah I want to pause for a second.

829
00:42:47,520 --> 00:42:49,439
So this company what was the company name?

830
00:42:49,440 --> 00:42:51,838
>> Uh the company that we founded is called Viewfinder.

831
00:42:51,839 --> 00:42:52,799
viewfound viewfinder.

832
00:42:52,800 --> 00:42:56,719
Yeah. >> Yeah. It was in the mobile photo sharing space which should sound familiar.

833
00:42:56,720 --> 00:42:57,679
This is like Instagram.

834
00:42:57,680 --> 00:42:59,598
This is like this was in 2012.

835
00:42:59,599 --> 00:43:02,078
Yeah. Right. As as Instagram and all of them were were taking off.

836
00:43:02,079 --> 00:43:02,559
>> Yeah. Yeah. No,

837
00:43:02,560 --> 00:43:06,239
we were right there in the play and uh we just didn't have the the right go to market

838
00:43:06,240 --> 00:43:07,919
kind of like how to attract the users,

839
00:43:07,920 --> 00:43:12,078
how to get viral growth kind of thing >> cuz from the outside like what I read when I

840
00:43:12,079 --> 00:43:16,399
when I checked you know the story just like oh you know like you you you co-ounded a

841
00:43:16,400 --> 00:43:18,318
startup it got acquired by Square.

842
00:43:18,319 --> 00:43:23,919
Hooray. Like sounds like you had bigger ambitions and this was a a decent outcome but

843
00:43:23,920 --> 00:43:25,519
not not the the dream, right?

844
00:43:25,520 --> 00:43:27,199
>> Yeah. Yeah. No, it wasn't the dream at all.

845
00:43:27,200 --> 00:43:29,759
So, uh the term I used was aqua hired.

846
00:43:29,760 --> 00:43:34,479
So, sometimes a company will get acquired get bought for you know their IP for their

847
00:43:34,480 --> 00:43:38,799
product for their business and other times they get bought just for the talent the people

848
00:43:38,800 --> 00:43:40,879
the people and we got bought just for the talent.

849
00:43:40,880 --> 00:43:43,679
They acquired the IP but I don't think Square ever did anything with it.

850
00:43:43,680 --> 00:43:45,519
It wasn't kind of like where they were working,

851
00:43:45,520 --> 00:43:49,118
but we had built up a kind of strong technical team and that's what we were hired for.

852
00:43:49,119 --> 00:43:51,118
And you know, like I can't remember the details.

853
00:43:51,119 --> 00:43:52,639
We'd raised a small amount of money.

854
00:43:52,640 --> 00:43:55,118
We were able to pay our investors back, make them whole.

855
00:43:55,119 --> 00:43:56,639
Maybe they got a little bit of a haircut,

856
00:43:56,640 --> 00:44:00,078
but maybe they actually got a little bit of a but it was it was okay.

857
00:44:00,079 --> 00:44:03,439
It's essentially they got their money back, which is like >> like you know,

858
00:44:03,440 --> 00:44:07,519
as a founder, you uh you know, investors are big boys.

859
00:44:07,520 --> 00:44:08,559
They're used to losing their money,

860
00:44:08,560 --> 00:44:10,799
but you kind of feel bad if you lose a lot of money for them.

861
00:44:10,800 --> 00:44:13,838
So, you know, getting them paid back kind of makes you feel a little bit better.

862
00:44:13,839 --> 00:44:17,358
>> Yeah. Yeah. >> And and then you you spent a little time at the company that acquired

863
00:44:17,359 --> 00:44:18,239
you was just Square.

864
00:44:18,240 --> 00:44:20,639
Yeah. And then you started issuing that a little bit again.

865
00:44:20,640 --> 00:44:21,519
>> Yeah. Yeah. Yeah.

866
00:44:21,520 --> 00:44:23,199
Because, you know, we've been working on these, you know,

867
00:44:23,200 --> 00:44:25,199
distributed file systems and storage systems.

868
00:44:25,200 --> 00:44:30,269
You know, Colossus, uh, one of the kind of sister projects to Colossus is Spanner.

869
00:44:30,319 --> 00:44:33,759
Um, and how how is Spanner different to Colossus?

870
00:44:33,760 --> 00:44:37,279
>> Well, Spanner is a essentially a distributed database.

871
00:44:37,280 --> 00:44:41,919
Colossus is a distributed storage system and kind of the the way I think about the difference

872
00:44:41,920 --> 00:44:44,239
between a distributed storage system and distributed database.

873
00:44:44,240 --> 00:44:45,999
You might think, oh, they're both storing data.

874
00:44:46,000 --> 00:44:51,598
I >> I was about to ask because a a distributed database will at some point be a storage

875
00:44:51,599 --> 00:44:52,318
system, >> right?

876
00:44:52,319 --> 00:44:53,598
Right. So >> So what's the difference?

877
00:44:53,599 --> 00:44:57,999
>> Yeah. So for Colossus, you know, was targeting large files, large appendon files.

878
00:44:58,000 --> 00:44:58,799
You can't update them.

879
00:44:58,800 --> 00:44:59,679
>> Append only. Yeah.

880
00:44:59,680 --> 00:45:01,279
>> Yeah. Large appendon files.

881
00:45:01,280 --> 00:45:05,759
So you know, 64 megabytes, maybe up to gigabytes in size.

882
00:45:05,760 --> 00:45:06,879
But if you're a database,

883
00:45:06,880 --> 00:45:10,479
you want to be storing like kind of small like you know kind of you know if you're using

884
00:45:10,480 --> 00:45:15,598
SQL or like the relational data you you might have table with you know billions or billions

885
00:45:15,599 --> 00:45:20,719
of you know rows those rows are broken up into columns the columns are typed that it

886
00:45:20,720 --> 00:45:24,318
just has a very different nature to the engineering challenge for a database than it

887
00:45:24,319 --> 00:45:29,039
does for a distributed storage system and usually distributed uh databases are implemented

888
00:45:29,040 --> 00:45:32,639
on top of some kind of distributed storage system and that was the the relationship.

889
00:45:32,640 --> 00:45:37,199
So Spanner was implemented on top of Colossus and some of the design decisions in Spanner

890
00:45:37,200 --> 00:45:42,959
kind of directly fell out of the appendon nature of the files in Colossus.

891
00:45:42,960 --> 00:45:44,719
You can't update a file in place.

892
00:45:44,720 --> 00:45:47,598
So you have to you know make the files immutable in your database.

893
00:45:47,599 --> 00:45:51,598
And this is where like you know log structured merge trees you know come into play.

894
00:45:51,599 --> 00:45:55,679
And they weren't invented at Google um but Google really popularized them with level

895
00:45:55,680 --> 00:45:59,039
DB which emerged out of the work on big table and spanner.

896
00:45:59,040 --> 00:46:01,279
Um that got popularized into Rox DB.

897
00:46:01,280 --> 00:46:05,358
I subsequently reimplement one of these things and this is what we use at Cockroach Labs.

898
00:46:05,359 --> 00:46:06,399
It's called Pebble.

899
00:46:06,400 --> 00:46:10,159
So I'm very familiar with the internals of that but it's kind of all based on this idea

900
00:46:10,160 --> 00:46:12,959
that the the data is kind of stored in immutable files.

901
00:46:12,960 --> 00:46:17,519
So how did you decide to found Cockroach Labs while we were working at Square?

902
00:46:17,520 --> 00:46:20,959
Um my co-founder and I was actually three of us.

903
00:46:20,960 --> 00:46:24,719
Uh we were all at Square and one of them is Spencer I mentioned earlier.

904
00:46:24,720 --> 00:46:28,639
He was working on the He's my college roommate and uh he was also at Google.

905
00:46:28,640 --> 00:46:29,838
>> He was also at Google.

906
00:46:29,839 --> 00:46:33,759
um this other guy Ben Darnell who's also at Google, he joined us um at Viewfinder,

907
00:46:33,760 --> 00:46:34,799
ended up at Square.

908
00:46:34,800 --> 00:46:38,559
We you know like we're kind of just noodling on a project to do and we we'd actually

909
00:46:38,560 --> 00:46:40,879
had this design back in um Viewfinder.

910
00:46:40,880 --> 00:46:46,639
We're like ah we didn't we looked around for a data base to be using in order to uh build

911
00:46:46,640 --> 00:46:49,199
Viewfinder on top of we didn't really like the things that were out there.

912
00:46:49,200 --> 00:46:51,118
The technologies inside Google looked better.

913
00:46:51,119 --> 00:46:51,999
We had the big table.

914
00:46:52,000 --> 00:46:53,039
We had Spanner and whatnot.

915
00:46:53,040 --> 00:46:57,118
We were looking around and you know like HBS existed but I wasn't quite happy and there's

916
00:46:57,119 --> 00:47:01,519
some other systems like React and others and you know at one point we just kind of like

917
00:47:01,520 --> 00:47:06,078
you know came up with a design for um cockroach to be an initial design then I was like

918
00:47:06,079 --> 00:47:09,598
no no guys we're doing a mobile photo sharing site we shouldn't build a distributed you

919
00:47:09,599 --> 00:47:13,999
know database so we we >> put it on the back burner which I think was absolutely the

920
00:47:14,000 --> 00:47:16,879
right thing maybe or maybe we should just pivoted away from doing the mobile photo sharing

921
00:47:16,880 --> 00:47:19,919
site given the way things worked out and then we got to square and we saw some of the

922
00:47:19,920 --> 00:47:23,358
same problems that they were experiencing with data storage systems And the Spencer is

923
00:47:23,359 --> 00:47:25,999
very convincing, managers convinced some of the management that like, hey,

924
00:47:26,000 --> 00:47:27,679
can you just work on this part-time, you know,

925
00:47:27,680 --> 00:47:31,999
to see if it had life behind the the design and then kind of conscripted uh Ben and I

926
00:47:32,000 --> 00:47:34,239
into it and eventually started getting, you know,

927
00:47:34,240 --> 00:47:37,999
attention externally and we're like, hey, can we go and spin this off into a company?

928
00:47:38,000 --> 00:47:39,759
And that's what ended up happening.

929
00:47:39,760 --> 00:47:42,559
>> And so you started a company,

930
00:47:42,560 --> 00:47:47,598
but I understand you didn't raise VC funding initially, right?

931
00:47:47,599 --> 00:47:51,999
>> That was the case at Viewfinder at uh we did it differently at Viewfinder.

932
00:47:52,000 --> 00:47:57,519
we kind of assued the VC money and you know in hindsight I wouldn't recommend that um

933
00:47:57,520 --> 00:48:01,039
but >> you wouldn't recommend >> yeah I wouldn't recommend taking the VC money because

934
00:48:01,040 --> 00:48:04,479
my experience the VCs are very very intelligent and they can help you navigate a lot

935
00:48:04,480 --> 00:48:08,399
of challenges you know I think sometimes there's this perception you know the VCs will

936
00:48:08,400 --> 00:48:12,318
push you into various areas and maybe there's some bad ones out there that do the VCs

937
00:48:12,319 --> 00:48:16,559
I've had experience with just like some of the sharpest people you know I've ever met

938
00:48:16,560 --> 00:48:20,559
and >> so so you kind of have like an extra like like person helping you on the team

939
00:48:20,560 --> 00:48:23,759
pretty much >> mentoring you, giving you guidance, telling you what they're seeing.

940
00:48:23,760 --> 00:48:26,639
They give you advice, seeing what they're seeing in the market, where things are going.

941
00:48:26,640 --> 00:48:29,358
That is very hard for sometimes for a founder,

942
00:48:29,359 --> 00:48:30,558
>> especially as a technical founder, right,

943
00:48:30,559 --> 00:48:32,159
that you're you're focused on the engineering part.

944
00:48:32,160 --> 00:48:32,799
>> Yeah. Yeah. Yeah.

945
00:48:32,800 --> 00:48:35,519
No, we actually took money right away for Cockroach Labs.

946
00:48:35,520 --> 00:48:38,799
Um, it was almost like as soon as we left, we went out and did a little road show,

947
00:48:38,800 --> 00:48:42,399
you know, kind of in the the Bay Area and got some interest and, you know,

948
00:48:42,400 --> 00:48:43,999
got a investor right away.

949
00:48:44,000 --> 00:48:45,999
>> I have to ask about the name though.

950
00:48:46,000 --> 00:48:48,509
>> Yeah. >> How how did the name of cockroach?

951
00:48:48,559 --> 00:48:54,399
>> Yeah. I uh so you know we named the Yes, that was uh that was mine.

952
00:48:54,400 --> 00:48:57,279
Pul fiction come out in college and like oh what should we name this thing?

953
00:48:57,280 --> 00:48:58,959
Oh the new image manipulation program.

954
00:48:58,960 --> 00:49:03,199
I think we're thinking image manipulation program initially and I'm like oh it's obvious.

955
00:49:03,200 --> 00:49:07,039
It just stuck at some point you know we're newly on the this new database and you kind

956
00:49:07,040 --> 00:49:08,318
of want to give things a name.

957
00:49:08,319 --> 00:49:11,039
Um you can't just say oh we're working on this distributed database.

958
00:49:11,040 --> 00:49:14,879
You kind of need something and this master is like oh cockroach DB like cockroaches are

959
00:49:14,880 --> 00:49:17,679
unkillable. I want these things this database to be unkillable.

960
00:49:17,680 --> 00:49:21,230
you know, cockroach is going to survive the nuclear apoc apocalypse.

961
00:49:21,280 --> 00:49:24,159
So that that was where the genesis was and just stuck.

962
00:49:24,160 --> 00:49:26,959
>> Yeah. We're we're whenever a bunch of our nodes goes down,

963
00:49:26,960 --> 00:49:27,919
this thing will still be up.

964
00:49:27,920 --> 00:49:30,959
>> Yeah. Yeah. And you know, this is where we're at today with cockroach DB.

965
00:49:30,960 --> 00:49:34,078
It's like one of the things that I point out, I'm like, "Holy crap, this is awesome.

966
00:49:34,079 --> 00:49:34,799
You kill a node.

967
00:49:34,800 --> 00:49:36,399
We we did this whole campaign last year,

968
00:49:36,400 --> 00:49:39,118
which was really just to prove out something already been present.

969
00:49:39,119 --> 00:49:40,959
The campaign was performance under adversity.

970
00:49:40,960 --> 00:49:43,598
But just like you can run a workload against it, you can kill a node,

971
00:49:43,599 --> 00:49:47,039
you can sometimes kill a whole region and the system keeps on going.

972
00:49:47,040 --> 00:49:48,558
And it's like stories like that, you know,

973
00:49:48,559 --> 00:49:49,999
like what we did on the marketing side there,

974
00:49:50,000 --> 00:49:52,078
but also we hear this from our customers, too.

975
00:49:52,079 --> 00:49:55,598
They've had fires in data centers and all the other data systems go down and cockroach

976
00:49:55,599 --> 00:49:56,318
DB keeps on going.

977
00:49:56,319 --> 00:49:57,919
I'm like, that's awesome.

978
00:49:57,920 --> 00:50:01,598
When you started out, who who were companies,

979
00:50:01,599 --> 00:50:06,959
startups that who wanted to use cockroach DB and and how has it changed since?

980
00:50:06,960 --> 00:50:10,959
Because you know like just making the case like okay I'm I'm starting a startup it's

981
00:50:10,960 --> 00:50:14,799
a small startup like I will need a database and I'll I don't know I'll split up I'll

982
00:50:14,800 --> 00:50:18,719
typically choose a postgress right it's free everyone's using it I I'm running it on

983
00:50:18,720 --> 00:50:21,200
node at what point

984
00:50:21,520 --> 00:50:25,759
did you see that typical typically tech companies are like okay like this is not enough

985
00:50:25,760 --> 00:50:30,159
for me that it's running on a node either because it can go down or because I'm out growing

986
00:50:30,160 --> 00:50:35,470
it what was it they outgrowing like I'm I'm trying to get a sense of like at what point

987
00:50:35,520 --> 00:50:40,799
did companies say like tell themselves like we need something distributed >> in a database.

988
00:50:40,800 --> 00:50:45,080
>> Yeah. I mean often times we have companies calling us up after they've had a disaster.

989
00:50:45,130 --> 00:50:50,239
[laughter] >> So so so like a node went up or a hard drive failed that kind of stuff.

990
00:50:50,240 --> 00:50:54,239
>> Uh you know we it's not quite like we're ambulance chasers but if you see an outage

991
00:50:54,240 --> 00:50:57,759
like a big outage from some you know company you know like we'll sometimes be trying

992
00:50:57,760 --> 00:51:02,318
to knock them up but also they they uh will call us you know and mean like there is a

993
00:51:02,319 --> 00:51:04,479
very big bank who's now a customer.

994
00:51:04,480 --> 00:51:08,719
um don't think I can name them but you can go read they had a very serious outage due

995
00:51:08,720 --> 00:51:13,838
to a weather event and after that >> which probably knocked down I'm assuming a region

996
00:51:13,839 --> 00:51:17,439
or or database or a networking cable or tree fell on something >> I think it knocked

997
00:51:17,440 --> 00:51:20,799
down a whole region you know was a regionwide power outage you knocked down the region

998
00:51:20,800 --> 00:51:24,558
and there's a mandate from the CEO it's like no we just have to you know be able to drive

999
00:51:24,559 --> 00:51:28,399
these things and that gets pushed down all the way and you see this in other places where

1000
00:51:28,400 --> 00:51:32,159
you know one of our early customers they were running on AWS and they just got to the

1001
00:51:32,160 --> 00:51:36,959
maximum size you can run in A rur instance on and then what would typically happen at

1002
00:51:36,960 --> 00:51:39,118
that point is then you have to shard your database.

1003
00:51:39,119 --> 00:51:40,719
You know this is very standard practice.

1004
00:51:40,720 --> 00:51:45,919
You you take your single node database you create 10 or 20 or 100 shards and this is

1005
00:51:45,920 --> 00:51:50,078
what Google done for some period of time and that's a a heavy burden on the application

1006
00:51:50,079 --> 00:51:53,919
developer and the way we always phrase this is like I mean the application developer

1007
00:51:53,920 --> 00:51:56,959
is becoming a database developer at that point and they're doing it poorly.

1008
00:51:56,960 --> 00:52:00,078
You know they're trying to implement distributed transactions or indexes and whatnot.

1009
00:52:00,079 --> 00:52:02,879
We felt the burden for that belongs on the database developer.

1010
00:52:02,880 --> 00:52:04,719
Can we talk about automatic sharding?

1011
00:52:04,720 --> 00:52:08,879
I think it's safe to assume most of us will know what what sharding is when when you're

1012
00:52:08,880 --> 00:52:14,318
but actually let's start from like manual sharding and then how you can implement automatic

1013
00:52:14,319 --> 00:52:18,639
sharding and if if you can tell us like you know tactics that a database like Hogwarts

1014
00:52:18,640 --> 00:52:21,358
DB can can do to actually just take that load off of you.

1015
00:52:21,359 --> 00:52:25,199
Yeah. Yeah. So I I I think the the very basic form of sharding is a little bit like the

1016
00:52:25,200 --> 00:52:28,959
hash table. Let's say you know you have a fixed number of shards like let's say it's

1017
00:52:28,960 --> 00:52:29,919
just a 100 shards.

1018
00:52:29,920 --> 00:52:33,519
your data model is a a user with a lot of data associated with the user,

1019
00:52:33,520 --> 00:52:37,358
you just take the user and you say like oh they map them to one of the shards and you

1020
00:52:37,359 --> 00:52:41,549
know you kind of just rely on the hash function to get like fairly even distribution.

1021
00:52:41,599 --> 00:52:43,439
The problem with this is at some point, you know,

1022
00:52:43,440 --> 00:52:47,118
one of your shards will get full and you have to kind of reshard and that's a very very

1023
00:52:47,119 --> 00:52:51,679
ownorous pro um >> resharding I guess simple way to do is like if if it's just a hard

1024
00:52:51,680 --> 00:52:52,399
drive, I don't know,

1025
00:52:52,400 --> 00:52:56,959
per per node where you write the user data gets it gets full and you're like okay well

1026
00:52:56,960 --> 00:52:58,399
I now need to split it somehow.

1027
00:52:58,400 --> 00:52:59,199
I need to move it.

1028
00:52:59,200 --> 00:53:00,078
I need to remap it.

1029
00:53:00,079 --> 00:53:03,199
I need to reject my metadata which knows where this data lives.

1030
00:53:03,200 --> 00:53:03,838
That kind of stuff.

1031
00:53:03,839 --> 00:53:07,279
>> Yeah. Yeah. And depends on exactly how you're doing that mapping from like you know

1032
00:53:07,280 --> 00:53:10,159
the user ID or whatever your shard key is to the shard.

1033
00:53:10,160 --> 00:53:12,159
you might have to remap them all, right?

1034
00:53:12,160 --> 00:53:13,039
This is very typical.

1035
00:53:13,040 --> 00:53:14,078
So, exactly. So, I mean,

1036
00:53:14,079 --> 00:53:17,679
and this happens in hashts where if you know oftentimes in order to grow the hash table,

1037
00:53:17,680 --> 00:53:19,759
you just have to essentially create a new hasht,

1038
00:53:19,760 --> 00:53:21,519
double the size and copy all the data over.

1039
00:53:21,520 --> 00:53:24,159
Now, that's kind of like the very basic straightforward way.

1040
00:53:24,160 --> 00:53:27,919
And there's uh various levels of complexity you add on it.

1041
00:53:27,920 --> 00:53:29,838
One of them is called consistent hashing.

1042
00:53:29,839 --> 00:53:31,919
Um, and there's various techniques to do this.

1043
00:53:31,920 --> 00:53:36,399
It's kind of fascinating like uh how they all work but in consistent hashing you can

1044
00:53:36,400 --> 00:53:39,838
add an additional node and then it only moves a fraction of the data from each shard

1045
00:53:39,839 --> 00:53:42,318
over there. Um there's various systems to do that.

1046
00:53:42,319 --> 00:53:47,118
I believe this is like when Cassandra the way cockroach DB does it is more akin to big

1047
00:53:47,119 --> 00:53:51,838
table more to spanner more to h base where instead of actually hashing we actually take

1048
00:53:51,839 --> 00:53:56,159
you know you can imagine all your keys in a system and this is always true in a system

1049
00:53:56,160 --> 00:54:00,399
that you can imagine just one big contiguous key space and then you kind of partition

1050
00:54:00,400 --> 00:54:04,639
contiguous spans of that and then you have to build up an index on top of those contiguous

1051
00:54:04,640 --> 00:54:09,199
spans and you what I just described there actually sounds a lot like a B tree.

1052
00:54:09,200 --> 00:54:13,358
So that there's this index on top that is like that that maps you from, you know,

1053
00:54:13,359 --> 00:54:17,759
like I need to have this range which node is it on and this is a little bit like a B

1054
00:54:17,760 --> 00:54:19,118
tree, you know, you kind of squint, you know,

1055
00:54:19,119 --> 00:54:22,239
it's like I think you squint and everything's either B tree or it's a hash table and

1056
00:54:22,240 --> 00:54:23,838
you know that index structure,

1057
00:54:23,839 --> 00:54:27,519
but it's it's now starting to make sense cuz when I I remember when I read it might have

1058
00:54:27,520 --> 00:54:29,279
been the Wikipedia article on B trees,

1059
00:54:29,280 --> 00:54:33,470
it it said this is data structure that is frequently used in databases.

1060
00:54:33,520 --> 00:54:36,239
It's now coming back to me because I didn't think too much of it.

1061
00:54:36,240 --> 00:54:38,239
I'm not a I'm not not someone who builds databases.

1062
00:54:38,240 --> 00:54:39,039
Now that we're talking,

1063
00:54:39,040 --> 00:54:42,479
we just like organically keep touching all the trees again and again.

1064
00:54:42,480 --> 00:54:44,639
>> And the the other place that it comes up in databases.

1065
00:54:44,640 --> 00:54:48,159
So this is where it kind of comes up in distributed databases and no one ever really

1066
00:54:48,160 --> 00:54:49,039
calls it a B tree.

1067
00:54:49,040 --> 00:54:52,318
I just kind of squint sometimes and I see like it's actually kind of a B tree.

1068
00:54:52,319 --> 00:54:56,239
But the other place it comes up in databases is for your indexes.

1069
00:54:56,240 --> 00:55:01,118
So if you have a table and you have like an index on your email address um and you want

1070
00:55:01,119 --> 00:55:03,279
to be able to scan over those email addresses in order,

1071
00:55:03,280 --> 00:55:05,439
that's a B tree under the hood in a database.

1072
00:55:05,440 --> 00:55:08,959
And you know like any kind of index you have it usually provides sorted order.

1073
00:55:08,960 --> 00:55:13,679
There are hash indexes but oftent times it's the B tree index and they are ubiquitous

1074
00:55:13,680 --> 00:55:16,799
in databases. There's actually a paper called the ubiquitous B tree.

1075
00:55:16,800 --> 00:55:20,479
And you know basically I just identified that I think that paper was written back in

1076
00:55:20,480 --> 00:55:22,639
the 80s and they're still ubiquitous today.

1077
00:55:22,640 --> 00:55:27,039
They are the foundations of single node databases and pretty much every data system I

1078
00:55:27,040 --> 00:55:29,838
worked on has had bries at some point place in them.

1079
00:55:29,839 --> 00:55:31,999
>> I want to ask about strong consistency.

1080
00:55:32,000 --> 00:55:35,199
Uh so Cockroach DB offers strong consistency.

1081
00:55:35,200 --> 00:55:39,598
Now for people who are a bit more newcomers to distributed systems,

1082
00:55:39,599 --> 00:55:43,999
can we talk about the consistency models and then why strong consistency is important

1083
00:55:44,000 --> 00:55:45,439
and why it's hard to implement it?

1084
00:55:45,440 --> 00:55:48,959
>> So I mean there's multiple ways to kind of approach this but I mean if you've used

1085
00:55:48,960 --> 00:55:52,959
a database you probably heard that uh of transactions and transactions are a way to perform

1086
00:55:52,960 --> 00:55:55,630
a whole bunch of mutations uh atomically.

1087
00:55:55,680 --> 00:55:59,838
So databases talk about atomisticity, consistency, isolation, durability.

1088
00:55:59,839 --> 00:56:01,439
The durability is really easy.

1089
00:56:01,440 --> 00:56:03,838
It's like when I write data to the database, it has to be durably written.

1090
00:56:03,839 --> 00:56:06,318
So if anything crashes, it comes back.

1091
00:56:06,319 --> 00:56:10,159
Um the atomicity is just referring to the fact I I want to do a whole bunch of changes.

1092
00:56:10,160 --> 00:56:13,039
I want them all committed or all aborted at the same time.

1093
00:56:13,040 --> 00:56:15,679
I don't want to have like some kind of partial operation.

1094
00:56:15,680 --> 00:56:16,879
And why is this important?

1095
00:56:16,880 --> 00:56:18,239
Why is the admissity important?

1096
00:56:18,240 --> 00:56:21,519
Well, the atomistity is what gets you to the point where it's like as an application,

1097
00:56:21,520 --> 00:56:25,118
I can do a bunch of operations and if there's an error hand an error occurs,

1098
00:56:25,119 --> 00:56:29,439
it all kind of gets rolled back and it's a much simpler um development model to work

1099
00:56:29,440 --> 00:56:33,118
within. And then there's the consistency and isolation which kind of get you know kind

1100
00:56:33,119 --> 00:56:36,719
of muddled. The isolation is referring to isolation between transactions.

1101
00:56:36,720 --> 00:56:39,358
I don't just want to one run one transaction at a time.

1102
00:56:39,359 --> 00:56:40,558
That's easy to do right.

1103
00:56:40,559 --> 00:56:41,919
I want to run a lot in parallel.

1104
00:56:41,920 --> 00:56:45,999
Oh yeah. And make it so that when they're running in parallel they are running as concurrently

1105
00:56:46,000 --> 00:56:49,919
as possible. But you want to have the appearance when they're running um as concurrently

1106
00:56:49,920 --> 00:56:53,519
as possible that there is kind of a serial order to them.

1107
00:56:53,520 --> 00:56:55,039
So this is like kind of the the whole trick.

1108
00:56:55,040 --> 00:56:59,390
The the kind of the gold standard for um isolation is called linearizability.

1109
00:56:59,440 --> 00:57:00,318
Don't worry about that.

1110
00:57:00,319 --> 00:57:02,749
There's a the step down from that is called serializability.

1111
00:57:02,799 --> 00:57:06,190
And that literally refers to having a serial order of your transactions.

1112
00:57:06,240 --> 00:57:10,399
But you're having to construct that in a way that you're doing everything as as concurrently

1113
00:57:10,400 --> 00:57:15,279
as possible. And the benefit of this the serializability is again it's a very simple

1114
00:57:15,280 --> 00:57:16,558
model for the application program.

1115
00:57:16,559 --> 00:57:20,798
They don't have to worry about weird kind of defects occurring in their program.

1116
00:57:20,799 --> 00:57:24,798
And some of the ones that can occur is like the classic description is of a bank, right?

1117
00:57:24,799 --> 00:57:27,519
where I might want to uh read, you know,

1118
00:57:27,520 --> 00:57:29,519
like have an operation that reads and says like,

1119
00:57:29,520 --> 00:57:32,639
do I have $100 in my bank account to transfer somewhere else?

1120
00:57:32,640 --> 00:57:37,519
And you could like arrange for lesser isolation levels that you might be able to subtract

1121
00:57:37,520 --> 00:57:39,039
that $100 twice.

1122
00:57:39,040 --> 00:57:40,239
And that's bad, right?

1123
00:57:40,240 --> 00:57:44,399
You know, we want to keep accurate, you know, track of your bank account or,

1124
00:57:44,400 --> 00:57:46,959
you know, what's in your shopping cart or, you know,

1125
00:57:46,960 --> 00:57:50,959
it's kind of like the the use cases are endless there and you do this wrong and you have

1126
00:57:50,960 --> 00:57:52,318
very egregious bugs.

1127
00:57:52,319 --> 00:57:56,399
But now going back to to to weak consistency and strong consistency.

1128
00:57:56,400 --> 00:58:00,239
>> Anything less than linearizability or serializability might be considered kind of

1129
00:58:00,240 --> 00:58:01,039
weak consistency.

1130
00:58:01,040 --> 00:58:04,879
But there's also like you know kind of strong consistency and eventual consistency.

1131
00:58:04,880 --> 00:58:08,959
So the eventual consistency is like sometimes I can do an operation and it might not

1132
00:58:08,960 --> 00:58:12,159
be immediately I might not be able to see all the the updates.

1133
00:58:12,160 --> 00:58:15,519
>> The read results will not necessarily be the current update >> but they'll come back

1134
00:58:15,520 --> 00:58:18,879
you know at some point you know I've written part of it and the rest of it will show

1135
00:58:18,880 --> 00:58:19,919
up at some point.

1136
00:58:19,920 --> 00:58:24,720
And often times when you're doing >> the easiest thing is your credit card balance, right?

1137
00:58:25,040 --> 00:58:27,118
>> Yeah. And uh it's faster to do it that way.

1138
00:58:27,119 --> 00:58:29,679
Faster in terms of just what the performance you can get out of the system.

1139
00:58:29,680 --> 00:58:31,999
But again, it's a little bit harder for the application to deal with.

1140
00:58:32,000 --> 00:58:35,919
One of the places this often comes up in in distributed databases or databases that have

1141
00:58:35,920 --> 00:58:40,078
any sort of replication is that I could write to the primary replica and then I read

1142
00:58:40,079 --> 00:58:41,838
from the secondary and it's not there yet.

1143
00:58:41,839 --> 00:58:43,199
This is that's eventual consistency.

1144
00:58:43,200 --> 00:58:44,749
>> Yeah, that's eventual consistency.

1145
00:58:44,799 --> 00:58:49,279
And you can often work around this but you just like puts bigger burden on the application

1146
00:58:49,280 --> 00:58:51,598
developer because you have to pay attention to that.

1147
00:58:51,599 --> 00:58:56,798
>> And then with cockroach DB you have strong consistency meaning as soon as you're writing

1148
00:58:56,799 --> 00:59:00,798
it when you're reading from the database you already get the written value back.

1149
00:59:00,799 --> 00:59:01,919
>> You get the written value back.

1150
00:59:01,920 --> 00:59:04,159
You know it's like you you read whatever you wrote.

1151
00:59:04,160 --> 00:59:07,199
It doesn't matter if you're reading from the same you know node you wrote it to.

1152
00:59:07,200 --> 00:59:10,239
If you read it from another node you still actually get the data you just wrote.

1153
00:59:10,240 --> 00:59:15,999
Is the trade-off logically not that you would have higher latency because clearly to

1154
00:59:16,000 --> 00:59:20,078
implement strong consistency you would somehow need to in the naive approach you would

1155
00:59:20,079 --> 00:59:25,039
need to write to all replicas right >> what are you doing inside cockroach >> well we

1156
00:59:25,040 --> 00:59:29,039
are writing to all the replicas so >> well well yeah >> yeah yeah you're doing it fast

1157
00:59:29,040 --> 00:59:31,679
>> you were just doing it fast you're making that efficient I think this is one of the

1158
00:59:31,680 --> 00:59:34,719
things that kind of also fascinates me about the software industry is we keep on finding

1159
00:59:34,720 --> 00:59:39,039
ways to be more and more sophisticated in order to provide you know like do do things

1160
00:59:39,040 --> 00:59:42,479
that make it easier to write the applications but do it in a very high performance way

1161
00:59:42,480 --> 00:59:46,399
and we've gotten you know very very good at this over over the years and this is the

1162
00:59:46,400 --> 00:59:49,999
area I know about the databases this is happening everywhere like I'm just fascinated

1163
00:59:50,000 --> 00:59:54,558
by how fast graphics have gotten where when I enter entered the industry like you were

1164
00:59:54,559 --> 00:59:58,239
literally like writing out each individual pixel and now you have these GPUs that are

1165
00:59:58,240 --> 01:00:01,519
doing like billions of triangles per second whatever the current numbers are and I'm

1166
01:00:01,520 --> 01:00:05,279
just like kind of astounded like how much sophistication got has gone into every area

1167
01:00:05,280 --> 01:00:08,798
of computer science wherever you look at it >> one more thing on cockroach DB I want

1168
01:00:08,799 --> 01:00:10,509
to ask about the raft consensus.

1169
01:00:10,559 --> 01:00:12,159
What is the raph consensus?

1170
01:00:12,160 --> 01:00:14,239
>> Yeah, I mean consensus protocols,

1171
01:00:14,240 --> 01:00:18,239
the original consensus protocol is called Paxos and it was famously hard to implement.

1172
01:00:18,240 --> 01:00:21,199
Raft was uh you might think of it as a variant of Paxos.

1173
01:00:21,200 --> 01:00:23,598
It was kind of like an alternative to Paxos,

1174
01:00:23,599 --> 01:00:27,598
but in my mind today >> and then the consistence algorithm being that you have like a

1175
01:00:27,599 --> 01:00:33,519
number of nodes like three to a lot more and then how do you get them to agree on what

1176
01:00:33,520 --> 01:00:34,479
do you typically agree on?

1177
01:00:34,480 --> 01:00:36,318
>> Yeah. Yeah. So you agree that the right occurred.

1178
01:00:36,319 --> 01:00:38,719
So like the way to think consensus.

1179
01:00:38,720 --> 01:00:43,279
So you might think I want to replicate data and I I I write it to a primary and I write

1180
01:00:43,280 --> 01:00:44,399
it to a secondary.

1181
01:00:44,400 --> 01:00:47,598
And you you can't actually have consensus when you only have two replicas.

1182
01:00:47,599 --> 01:00:51,999
And the reason you can't have consensus is if there's a crash and I come up like if I'm

1183
01:00:52,000 --> 01:00:54,719
on the secondary, how do I know if something was written to the primary?

1184
01:00:54,720 --> 01:00:57,439
If I'm on the primary, how do I know it was written on the secondary?

1185
01:00:57,440 --> 01:01:00,479
Right? And you're always going to be in this kind of confusing place where it's like

1186
01:01:00,480 --> 01:01:04,719
you either have to roll back a little bit or like you know you lose some data.

1187
01:01:04,720 --> 01:01:06,879
And consensus requires at least three,

1188
01:01:06,880 --> 01:01:09,679
but you can have consensus across more than three replicas.

1189
01:01:09,680 --> 01:01:12,558
And the idea with consensus is I'm going to write to three places.

1190
01:01:12,559 --> 01:01:14,959
And normally you don't actually when you're doing a read,

1191
01:01:14,960 --> 01:01:17,199
you don't read from multiple of them.

1192
01:01:17,200 --> 01:01:19,118
But if there's a crash, I have to do recovery.

1193
01:01:19,119 --> 01:01:20,399
Then I'm reading oh,

1194
01:01:20,400 --> 01:01:25,039
I can read from uh two of the uh any two of the three and I know I can like kind of determine

1195
01:01:25,040 --> 01:01:26,639
what had happened previously.

1196
01:01:26,640 --> 01:01:30,159
Um and it's usually just on the the recovery time that you're actually doing that consensus

1197
01:01:30,160 --> 01:01:33,039
read. So reads are typically just happening from one replica.

1198
01:01:33,040 --> 01:01:37,679
You have to write to all three and on a crash during that kind of failure is when the

1199
01:01:37,680 --> 01:01:42,399
consensus read occurs >> and inside cockroach DB how many replicas do you choose for

1200
01:01:42,400 --> 01:01:44,159
the consensus? >> It's typically three.

1201
01:01:44,160 --> 01:01:48,239
Um for some system tables it can be five and then customers also have control of this.

1202
01:01:48,240 --> 01:01:51,039
At the database level you can write to five, seven.

1203
01:01:51,040 --> 01:01:54,159
Five is like, you know, if you're really concerned, uh, you know,

1204
01:01:54,160 --> 01:01:56,399
about kind of the data durability, you might use five.

1205
01:01:56,400 --> 01:01:58,558
But there's a slowdown, you know, the more you're writing to.

1206
01:01:58,559 --> 01:02:02,318
It's like the more storage space that >> and and obviously like we're talking like of

1207
01:02:02,319 --> 01:02:05,999
slow down with nodes, but of of course if they're like between regions,

1208
01:02:06,000 --> 01:02:10,479
there's now you have a lot more resilience for let's say an earthquake or a power or

1209
01:02:10,480 --> 01:02:12,479
whatever. But now you will have additional latency.

1210
01:02:12,480 --> 01:02:13,919
Just speed of light basics, right?

1211
01:02:13,920 --> 01:02:15,199
Speed of light latency, right?

1212
01:02:15,200 --> 01:02:20,558
and it's you know tens you know or up to hundreds of milliseconds or even higher if you're

1213
01:02:20,559 --> 01:02:21,439
going across the globe.

1214
01:02:21,440 --> 01:02:24,879
So you know you have to be very careful with that um in terms of how you architect your

1215
01:02:24,880 --> 01:02:28,959
queries. I mean one of the things that is just a general truism of distributed databases

1216
01:02:28,960 --> 01:02:31,039
is you don't want to have a lot of back and forth.

1217
01:02:31,040 --> 01:02:35,199
You want to kind of like do all your reads in one kind of parallel read set get them

1218
01:02:35,200 --> 01:02:39,039
back then do your writes right but if you're doing like kind of serial operations where

1219
01:02:39,040 --> 01:02:42,959
I I read a row I write a row I read a row I write a row I mean the the latencies just

1220
01:02:42,960 --> 01:02:45,199
add up. We talked about founding Cockroach DB,

1221
01:02:45,200 --> 01:02:47,919
but how how is the company growing and where are you today?

1222
01:02:47,920 --> 01:02:51,519
>> Yeah. Yeah. I mean, we're being used in like we're powering mission critical applications.

1223
01:02:51,520 --> 01:02:52,558
That's our bread and butter.

1224
01:02:52,559 --> 01:02:53,118
>> But by the way,

1225
01:02:53,119 --> 01:02:56,239
can can you elaborate on mission critical because it's like uh >> Yeah.

1226
01:02:56,240 --> 01:02:58,959
>> If you're not you are in the industry where you know what this means,

1227
01:02:58,960 --> 01:03:03,519
but from the outside it can feel hard to put a thumb on what is mission critical is is

1228
01:03:03,520 --> 01:03:07,358
is is my SAS that is like showing as mission critical probably not.

1229
01:03:07,359 --> 01:03:11,679
>> Yeah. Uh so mission critical to my mind is like these uh kind of the other term of

1230
01:03:11,680 --> 01:03:15,999
art is tier zero applications the ones that are like just the the core crown jewels of

1231
01:03:16,000 --> 01:03:20,159
what's running a company you know like a trading system you know your trading system

1232
01:03:20,160 --> 01:03:24,159
can't go down if the trading system goes down this is kind of a critical problem for

1233
01:03:24,160 --> 01:03:28,318
the firm that is running the trading system you know banking systems um but also you

1234
01:03:28,319 --> 01:03:32,959
know like we work with Door Dash you know like some people might think that delivering

1235
01:03:32,960 --> 01:03:37,439
burrito is a kind of a mission critical system and it certainly is for Door Dash right?

1236
01:03:37,440 --> 01:03:39,439
You know, if that goes down, it's problematic.

1237
01:03:39,440 --> 01:03:41,118
We power shopping carts.

1238
01:03:41,119 --> 01:03:43,439
Um, you know, other stuff like this where it's like, well,

1239
01:03:43,440 --> 01:03:46,719
if the shopping cart goes down, you know, that company is losing, you know,

1240
01:03:46,720 --> 01:03:49,279
hundreds of thousands, millions of dollars per hour.

1241
01:03:49,280 --> 01:03:51,118
So, that that's kind of the critical you think about.

1242
01:03:51,119 --> 01:03:54,639
>> Yeah. I guess of course they're losing, but but this is like when Yeah.

1243
01:03:54,640 --> 01:03:58,879
their customers are also like they're used to this just working like running water and

1244
01:03:58,880 --> 01:04:01,598
then when it's not the same thing as when your utility breaks, right?

1245
01:04:01,599 --> 01:04:05,519
Your water, electricity is out, you'll survive, but it's not what you expected.

1246
01:04:05,520 --> 01:04:06,399
>> Yeah. Yeah. Yeah.

1247
01:04:06,400 --> 01:04:07,679
And everybody's like what?

1248
01:04:07,680 --> 01:04:10,639
How how what age are we living in that the electricity goes out?

1249
01:04:10,640 --> 01:04:13,358
We kind of had this ingrained into our heads at Google.

1250
01:04:13,359 --> 01:04:15,038
It's like Gmail cannot go down.

1251
01:04:15,039 --> 01:04:16,798
People are, you know, relying on it.

1252
01:04:16,799 --> 01:04:17,838
Search cannot go down.

1253
01:04:17,839 --> 01:04:21,358
You know, if it's go goes down too long, people are going to move to other systems.

1254
01:04:21,359 --> 01:04:22,639
Um and it's not like, you know,

1255
01:04:22,640 --> 01:04:25,519
like in some ways search isn't as mission critical except oh wait,

1256
01:04:25,520 --> 01:04:27,598
every single search is ad dollars behind it.

1257
01:04:27,599 --> 01:04:29,759
And you actually notice the blip in the revenue.

1258
01:04:29,760 --> 01:04:33,038
And it's not just the blip in the revenue, it's the blip in reputation as well.

1259
01:04:33,039 --> 01:04:36,719
I mean, I think that's the one that really poisons companies is like if your bank is

1260
01:04:36,720 --> 01:04:38,318
down for a serious amount of time,

1261
01:04:38,319 --> 01:04:41,358
the reputational damage there will be horrifically bad.

1262
01:04:41,359 --> 01:04:43,279
And you know, like we often talk about like, oh,

1263
01:04:43,280 --> 01:04:45,439
and then grandma won't be able to pay her rent and she'll get evicted.

1264
01:04:45,440 --> 01:04:48,078
You got to take this like responsibility really, really seriously.

1265
01:04:48,079 --> 01:04:51,679
>> No, but also like just Gmail being mission critical just on the way here.

1266
01:04:51,680 --> 01:04:55,598
We only exchanged numbers later, but we were communicating over email like, "Oh,

1267
01:04:55,599 --> 01:04:58,639
I'm I was telling you that that you were telling me that you're here.

1268
01:04:58,640 --> 01:05:02,078
I emailed." And I never for a second thought that it it could go down.

1269
01:05:02,079 --> 01:05:04,879
And I think we were like responding within 30 seconds, right?

1270
01:05:04,880 --> 01:05:06,959
And it's just I just know it's it's there.

1271
01:05:06,960 --> 01:05:11,118
Like I I didn't like bother setting up a secondary communication line.

1272
01:05:11,119 --> 01:05:12,159
So >> yeah. Yeah.

1273
01:05:12,160 --> 01:05:15,999
No, it's like when people just like when you have that kind of level of of trust with

1274
01:05:16,000 --> 01:05:18,479
your users, you got to maintain it and invest in it.

1275
01:05:18,480 --> 01:05:21,838
But then it leads to this kind of freedom for the user as well where you just don't think

1276
01:05:21,839 --> 01:05:23,519
about like I don't have to worry about it.

1277
01:05:23,520 --> 01:05:24,479
It's just going to work.

1278
01:05:24,480 --> 01:05:28,399
>> And then in terms of the the company like how many engineers do you have roughly?

1279
01:05:28,400 --> 01:05:30,350
>> We have some hundreds.

1280
01:05:30,400 --> 01:05:35,439
I don't actually know the precise engineering number 110 but there might be 150 in R&D

1281
01:05:35,440 --> 01:05:38,318
overall. Um there's other folks besides just engineers.

1282
01:05:38,319 --> 01:05:39,759
I'm clear engineering managers.

1283
01:05:39,760 --> 01:05:41,038
Those are engineers as well.

1284
01:05:41,039 --> 01:05:43,279
And then you you know like we've been growing steadily.

1285
01:05:43,280 --> 01:05:45,519
It takes quite a while to build a distributed database.

1286
01:05:45,520 --> 01:05:47,040
>> Not for the faint of heart.

1287
01:05:47,075 --> 01:05:49,759
[laughter] >> So it took a couple years a couple times.

1288
01:05:49,760 --> 01:05:50,879
>> Yeah. Yeah. Uh yeah.

1289
01:05:50,880 --> 01:05:52,239
Well, I did a distributed storage system.

1290
01:05:52,240 --> 01:05:53,598
Now I did a distributed database.

1291
01:05:53,599 --> 01:05:54,798
It's not for the faint of heart, right?

1292
01:05:54,799 --> 01:05:59,199
So there's a lot of work getting it to a a level of stability then a level of kind of

1293
01:05:59,200 --> 01:06:01,038
quality beyond that level of stability.

1294
01:06:01,039 --> 01:06:05,038
Um getting all the bugs out um and then continue to innovate and put more performance

1295
01:06:05,039 --> 01:06:09,390
into the system adding functionality to integrate better within within enterprises.

1296
01:06:09,440 --> 01:06:13,838
Our revenue has been you know kind of steadily growing over the years and you know it's

1297
01:06:13,839 --> 01:06:18,078
at this place now where we we see a path to future success as well.

1298
01:06:18,079 --> 01:06:20,879
And I I want to ask about your coding habits.

1299
01:06:20,880 --> 01:06:26,798
So when you co-founded the company, how much code did you write for the first few years?

1300
01:06:26,799 --> 01:06:27,838
>> I wrote a lot.

1301
01:06:27,839 --> 01:06:30,078
So I've always been a very prolific coder,

1302
01:06:30,079 --> 01:06:33,038
I wrote a lot of code early days and early days,

1303
01:06:33,039 --> 01:06:38,479
I mean I was uh kind of we were all technical co-founders, Ben Spencer and I.

1304
01:06:38,480 --> 01:06:41,759
Um and we were all writing a lot of code and I was no exception.

1305
01:06:41,760 --> 01:06:45,919
But you know like I look back at my GitHub output and you know it's like kind of peak

1306
01:06:45,920 --> 01:06:50,798
years maybe 100 thousand lines of code in a year which is a year which is a lot.

1307
01:06:50,799 --> 01:06:51,439
>> Yeah. >> Yeah.

1308
01:06:51,440 --> 01:06:56,639
No. So I mean >> like we're talking preai >> prei right this is when you're back doing

1309
01:06:56,640 --> 01:06:57,759
this manually right?

1310
01:06:57,760 --> 01:07:01,439
You know at some point you know we started out using a system called Rox DB which is

1311
01:07:01,440 --> 01:07:07,118
an LSM. At some point I think it's back in 2019 you ran into limitations with it.

1312
01:07:07,119 --> 01:07:11,919
I decided um we wanted to rewrite it and did a big push to you write and that might have

1313
01:07:11,920 --> 01:07:16,318
been like 40 50,000 lines of code and then a bunch of other people come up and helped

1314
01:07:16,319 --> 01:07:19,999
and like you kind of look at that output I'm just like oh my goodness that was a lot

1315
01:07:20,000 --> 01:07:24,798
to keep in your head a it's a lot just to type you know 100,000 lines of code the average

1316
01:07:24,799 --> 01:07:28,719
kind of like uh that the industry talks about is 3,000 lines of code from an engineer

1317
01:07:28,720 --> 01:07:34,798
in a month and so if you multiply that out maybe 36,000 in a year that's good right so

1318
01:07:34,799 --> 01:07:37,038
I was doing a lot I kind of look at had.

1319
01:07:37,039 --> 01:07:40,719
It's like there's kind of a max that you can hold in your head at a time.

1320
01:07:40,720 --> 01:07:43,519
Um the tools have gotten a lot better since I first entered the industry.

1321
01:07:43,520 --> 01:07:46,639
We gotten better debugging techniques, better testing techniques,

1322
01:07:46,640 --> 01:07:48,798
but still um quite significant.

1323
01:07:48,799 --> 01:07:52,318
>> And you were CTO from from from the beginning, co-founder of CTO,

1324
01:07:52,319 --> 01:07:57,919
but there was a time sometime around like 2022 when you decided to kind of be a bit more

1325
01:07:57,920 --> 01:07:58,798
hands-off, right?

1326
01:07:58,799 --> 01:08:00,239
>> Yeah. Yeah. >> Can you tell me about that?

1327
01:08:00,240 --> 01:08:04,479
>> I mean, the general rule of thumb for uh engineering leaders is well,

1328
01:08:04,480 --> 01:08:05,598
you got to have your team.

1329
01:08:05,599 --> 01:08:08,479
at a manager team and we had a VP of engineering.

1330
01:08:08,480 --> 01:08:13,118
Um, but I was kind of getting to the point of like okay is my are my coding days done

1331
01:08:13,119 --> 01:08:17,758
you know like kind just direct from a higher level and you know I got this advice for

1332
01:08:17,759 --> 01:08:21,119
a long period of time and I pushed back on it but you know I kind of acquiesced at some

1333
01:08:21,120 --> 01:08:24,238
point and I think it was the right advice I'm not saying that the advice was wrong at

1334
01:08:24,239 --> 01:08:29,919
the time um but there was a time period from about 2022 to 2024 where I was like my my

1335
01:08:29,920 --> 01:08:31,119
output declined.

1336
01:08:31,120 --> 01:08:33,519
I think I actually did the Swiss tables thing in that time period,

1337
01:08:33,520 --> 01:08:36,718
but I wasn't doing much on the the core >> the the business.

1338
01:08:36,719 --> 01:08:38,238
>> Yeah, the the core business.

1339
01:08:38,239 --> 01:08:39,999
You know, I would get in there and do some work,

1340
01:08:40,000 --> 01:08:44,559
but at like it's really hard that if you're in meetings all day to also do coding.

1341
01:08:44,560 --> 01:08:46,718
I think this is the fundamental tension if you like.

1342
01:08:46,719 --> 01:08:50,318
>> So you kind of took on the kind of the meeting burden, the coordination burden,

1343
01:08:50,319 --> 01:08:54,959
the yeah the the stuff that was if I'm reading correctly before you spent a lot of your

1344
01:08:54,960 --> 01:08:57,919
head in the code and now you're spending a lot of your head like above the code,

1345
01:08:57,920 --> 01:08:59,599
the business, the engineering,

1346
01:08:59,600 --> 01:09:03,439
the or the whatever customers that kind of stuff >> the the customers and just being

1347
01:09:03,440 --> 01:09:04,559
an executive as well.

1348
01:09:04,560 --> 01:09:07,309
Yeah. So very hard to wear all those hats simultaneously.

1349
01:09:07,359 --> 01:09:10,559
And then I got back into it because AI started to emerge.

1350
01:09:10,560 --> 01:09:13,758
And >> so how when did you get when did you start using AI?

1351
01:09:13,759 --> 01:09:16,959
But when do you start to find it useful in terms of coding?

1352
01:09:16,960 --> 01:09:20,559
>> Yeah. Well, it's interesting because you know those initial versions of like kind

1353
01:09:20,560 --> 01:09:24,718
of glorified autocomplete came out and >> we're talking about the the GitHub copilot,

1354
01:09:24,719 --> 01:09:26,559
the cursor >> co-pilot.

1355
01:09:26,560 --> 01:09:28,959
I that was the one I I had the first exposure to.

1356
01:09:28,960 --> 01:09:30,318
We dabbled with cursor at the time,

1357
01:09:30,319 --> 01:09:33,678
but they're all like kind of glorified autocomplete and it was kind of crazy that you

1358
01:09:33,679 --> 01:09:36,158
could just start typing something you like the rest of the function.

1359
01:09:36,159 --> 01:09:39,119
You look at I'm like wait kind of got that right.

1360
01:09:39,120 --> 01:09:40,879
This is crazy, right?

1361
01:09:40,880 --> 01:09:44,798
And um you know we're trying to encourage our engineers to use this and at some point

1362
01:09:44,799 --> 01:09:49,119
you know I can't remember if this is my idea or my co-founders or someone and basically

1363
01:09:49,120 --> 01:09:54,158
like you know like in order to like really guide people about how to use it you have

1364
01:09:54,159 --> 01:09:57,919
to be a user yourself you know I think this is true of like engineering management in

1365
01:09:57,920 --> 01:10:01,279
general like you want to like manage engineers you have to know how to be an engineer

1366
01:10:01,280 --> 01:10:04,479
like if you if you don't know how to be a good engineer it's like really hard to manage

1367
01:10:04,480 --> 01:10:08,879
other engineers >> I feel like you'll have a hard time like just relating to them at

1368
01:10:08,880 --> 01:10:10,399
the very least. Exactly.

1369
01:10:10,400 --> 01:10:12,639
Exactly. So kind of took it on being like no.

1370
01:10:12,640 --> 01:10:17,119
I mean this is like it was clear very early on like this is probably going to go somewhere

1371
01:10:17,120 --> 01:10:22,158
but it wasn't quite clear how far how fast it would go and you start dabbling this and

1372
01:10:22,159 --> 01:10:25,119
I was like oh okay well it's not quite good enough.

1373
01:10:25,120 --> 01:10:28,238
It's not quite good enough but you know let me start getting back into the coding very

1374
01:10:28,239 --> 01:10:32,639
rapidly. You know you start seeing the signs of life like you know kind of the opus models

1375
01:10:32,640 --> 01:10:35,759
coming out. It was Sonnet first, then the Opus, and you're like,

1376
01:10:35,760 --> 01:10:37,599
you're looking at these and you're like, "Oh, wow."

1377
01:10:37,600 --> 01:10:41,599
Okay. Well, they seem to be able to do quite a lot, but you know,

1378
01:10:41,600 --> 01:10:43,359
the code is still not great.

1379
01:10:43,360 --> 01:10:47,710
But then it was just like the just this cadence of continuing improvements.

1380
01:10:47,760 --> 01:10:52,879
And, you know, I was starting to do a lot with Sonnet and then starting to use Opus.

1381
01:10:52,880 --> 01:10:56,158
And, you know, I had that same moment everybody else did.

1382
01:10:56,159 --> 01:10:58,399
And this was last year, last uh November.

1383
01:10:58,400 --> 01:11:01,359
Uh, >> no. November, December, winter break, right?

1384
01:11:01,360 --> 01:11:02,639
>> Yeah. It was Thanksgiving.

1385
01:11:02,640 --> 01:11:06,158
I distinctly remember it because you know >> you were not chilling,

1386
01:11:06,159 --> 01:11:07,759
you were coding, weren't you?

1387
01:11:07,760 --> 01:11:08,799
>> You were agenting.

1388
01:11:08,800 --> 01:11:11,279
>> I was uh agenting just like everyone else.

1389
01:11:11,280 --> 01:11:16,319
I had this thing I'd wanted to do for a long period of time um on cockroach DB which

1390
01:11:16,320 --> 01:11:20,959
is like I wanted to like test like all these configurations of cockroach DB across like

1391
01:11:20,960 --> 01:11:23,919
you know different vertical scaling like how many CPUs you have on a node,

1392
01:11:23,920 --> 01:11:26,079
how many stores you uh discs you have on a node,

1393
01:11:26,080 --> 01:11:29,759
how how many nodes you have in the system and test across all this huge matrix of it.

1394
01:11:29,760 --> 01:11:33,519
This is one of these things I couldn't ever quite get prioritized appropriately because

1395
01:11:33,520 --> 01:11:34,799
it never seemed like kind of critical,

1396
01:11:34,800 --> 01:11:37,599
but I've like I always had this intuition there was something there.

1397
01:11:37,600 --> 01:11:39,759
And then over this like four day span,

1398
01:11:39,760 --> 01:11:45,119
the code just like materialized as I was using um I think that was Opus 47 or is it 45?

1399
01:11:45,120 --> 01:11:48,399
Whatever the number was like the recollection I had is, you know,

1400
01:11:48,400 --> 01:11:53,359
like I'm pretty fast typer and then I just remember having this feeling of like well

1401
01:11:53,360 --> 01:11:56,158
the code is just like materializing before my eyes.

1402
01:11:56,159 --> 01:11:59,519
you know, you would ask for these things and I got out of the habit of actually typing

1403
01:11:59,520 --> 01:12:02,910
and you just kind of like ask for something materializes.

1404
01:12:02,960 --> 01:12:05,198
If you've ever seen like some of those um you know,

1405
01:12:05,199 --> 01:12:09,839
a movie where they they have this kind of archetype of a a software engineer gets in

1406
01:12:09,840 --> 01:12:11,759
front of the keyboard and they start typing, you show the screen,

1407
01:12:11,760 --> 01:12:16,079
it's just like it's going way beyond human speed and then it was it was beyond that,

1408
01:12:16,080 --> 01:12:19,919
right? like it was I think as software engineers like we used to laugh at these like

1409
01:12:19,920 --> 01:12:25,119
I I still remember swartfish the the thing when they're visualizing the things are going

1410
01:12:25,120 --> 01:12:29,919
or the code appearing you know you're seeing that the person's typing and then it's a

1411
01:12:29,920 --> 01:12:34,079
big line and a software engineers like we're laughing cuz it's not how it is but but

1412
01:12:34,080 --> 01:12:38,959
it's crazy that like that effect right >> yeah yeah but now now it's even better than

1413
01:12:38,960 --> 01:12:42,879
that effect right >> because it actually works this time >> and it's that's slow in comparison

1414
01:12:42,880 --> 01:12:46,319
it it materializes faster than that like you can literally No,

1415
01:12:46,320 --> 01:12:48,559
like we've been talking about B trees a whole bunch.

1416
01:12:48,560 --> 01:12:51,039
I implemented another B tree in in the past month.

1417
01:12:51,040 --> 01:12:51,759
Of course you did.

1418
01:12:51,760 --> 01:12:57,198
>> Yeah. And it took about 30 minutes to implement probably 10,000 lines of highly optimized

1419
01:12:57,199 --> 01:12:59,999
Rust. I mean it's just like it boggles the mind.

1420
01:13:00,000 --> 01:13:04,319
Um I mean I we you should look up later 10,000 lines is like a crazy amount.

1421
01:13:04,320 --> 01:13:06,319
You physically cannot type that fast.

1422
01:13:06,320 --> 01:13:09,919
you you started get back to coding like are we talking about kind of vibe coding prototyping

1423
01:13:09,920 --> 01:13:14,079
or are we actually talking you started to contribute like proper production ready code

1424
01:13:14,080 --> 01:13:20,799
that is up the level of what you're doing at DB >> well it started with this tool um

1425
01:13:20,800 --> 01:13:25,439
that was like this kind of benchmarking tool that tested this matrix I I'm the CTO we

1426
01:13:25,440 --> 01:13:31,119
have an office of the CTO um the office of the CTO's mandate is to innovate and I was

1427
01:13:31,120 --> 01:13:35,678
looking for places like we can have innovation and one of the things that we want to

1428
01:13:35,679 --> 01:13:41,599
innovate in um was you know better uh autoscaling of a cockroach DB cluster and kind

1429
01:13:41,600 --> 01:13:45,439
of in the January time frame I came out with like what I would think is like kind of

1430
01:13:45,440 --> 01:13:49,678
a a research breakthrough you might say and it came about because I was dabbling in this

1431
01:13:49,679 --> 01:13:52,879
area and just working with models trying to understand it and they're not just good at

1432
01:13:52,880 --> 01:13:56,238
coding they're also good at like helping you explore design ideas and I think this is

1433
01:13:56,239 --> 01:13:59,519
kind of the fascinating thing where it's you have to get out of the mindset of like I

1434
01:13:59,520 --> 01:14:03,839
know exactly what I'm going to build um but more like hey we have this problem talk through

1435
01:14:03,840 --> 01:14:05,919
it be a partner with mem Mhm.

1436
01:14:05,920 --> 01:14:07,279
And >> be a sparring partner.

1437
01:14:07,280 --> 01:14:08,479
>> Be a sparring partner.

1438
01:14:08,480 --> 01:14:10,959
You know, like there's this advice you might have heard that like, you know,

1439
01:14:10,960 --> 01:14:13,599
if you're stuck on a problem, you just go yellow duck it.

1440
01:14:13,600 --> 01:14:14,799
Go >> rubber duck.

1441
01:14:14,800 --> 01:14:15,599
>> Rubber duck it.

1442
01:14:15,600 --> 01:14:17,279
Uh I think I've heard is yellow duck.

1443
01:14:17,280 --> 01:14:19,039
You know, go just go talk to something.

1444
01:14:19,040 --> 01:14:20,639
You don't even need it to respond.

1445
01:14:20,640 --> 01:14:22,479
Now you can talk to this system.

1446
01:14:22,480 --> 01:14:26,959
Uh this intelligence and it will give you back stuff and it's not always right.

1447
01:14:26,960 --> 01:14:27,919
I mean, this is the thing.

1448
01:14:27,920 --> 01:14:30,238
Even today, these models are fantastic.

1449
01:14:30,239 --> 01:14:31,119
Fable's fantastic.

1450
01:14:31,120 --> 01:14:32,238
Astro is fantastic.

1451
01:14:32,239 --> 01:14:35,119
And they're not always right, but they they kind of like they know.

1452
01:14:35,120 --> 01:14:41,759
so much. It's just encyclopedic and you can explore ideas super super fast and then be

1453
01:14:41,760 --> 01:14:45,039
pointing out well that doesn't sound right to me and it'll be like oh yeah you're absolutely

1454
01:14:45,040 --> 01:14:49,678
right you know like I I hate that sick fancy it's like it kills me >> or you're you're

1455
01:14:49,679 --> 01:14:53,119
right to push on back on you were right to push it back on that that's absolutely right

1456
01:14:53,120 --> 01:14:59,198
>> yeah and yet like just able to make such fast progress um and I I I find it absolutely

1457
01:14:59,199 --> 01:15:03,999
incredible so it quickly moved from just doing kind of side things building up some tools

1458
01:15:04,000 --> 01:15:09,470
and whatnot to building you know essentially over the last eight months it's January

1459
01:15:09,520 --> 01:15:14,799
um we've been building towards you know a new launch and you know like a lot of that

1460
01:15:14,800 --> 01:15:19,519
has been produced gently powered engineers and like the entire company's got on board

1461
01:15:19,520 --> 01:15:23,678
with this now where I think it's like probably 100% of engineers are using it to a greater

1462
01:15:23,679 --> 01:15:28,479
or lesser degree and a lot of code is being produced I think it's very high quality code

1463
01:15:28,480 --> 01:15:34,109
these models not test things adequately they don't look quite intently enough about performance

1464
01:15:34,159 --> 01:15:36,799
But if you're like uh you can guide them in the right way.

1465
01:15:36,800 --> 01:15:40,718
And I think there's actually a huge advantage to anybody who's done management before.

1466
01:15:40,719 --> 01:15:45,599
It's a little bit like being a manager of people where you're a manager of a large group.

1467
01:15:45,600 --> 01:15:47,279
You're not looking to every line of code,

1468
01:15:47,280 --> 01:15:49,519
but you're definitely kind of helping architect the system.

1469
01:15:49,520 --> 01:15:51,759
And I think there's a very strong analogy there.

1470
01:15:51,760 --> 01:15:53,439
>> I sometimes push back on this analogy.

1471
01:15:53,440 --> 01:15:55,678
The reason being I I was an engineering manager.

1472
01:15:55,679 --> 01:15:58,158
My take is that working with these agents,

1473
01:15:58,159 --> 01:16:01,599
it's not like management because management has so much of the human stuff.

1474
01:16:01,600 --> 01:16:05,279
the like as a manager when I think of all this a bunch of stuff I dealt with which was

1475
01:16:05,280 --> 01:16:10,319
the people side of things the conflicts between people the performance reviews the meeting

1476
01:16:10,320 --> 01:16:14,479
etc and you have none of that you do have the orchestration like like and this this is

1477
01:16:14,480 --> 01:16:18,559
like almost like I guess a really like naive way of management where like you know they

1478
01:16:18,560 --> 01:16:22,718
don't push back they they start to do it sometimes they're like unreliable but you know

1479
01:16:22,719 --> 01:16:26,799
I almost like to use like orchestration a bit more because I feel management is is so

1480
01:16:26,800 --> 01:16:30,559
much more involved like I think this is the thing where like a tech lead who has no management

1481
01:16:30,560 --> 01:16:32,238
responsibilities but they have a group of interns,

1482
01:16:32,239 --> 01:16:34,799
but they don't need to do with their performance, with their anything.

1483
01:16:34,800 --> 01:16:37,678
Like, it's a lot closer to that, if you know what I mean.

1484
01:16:37,679 --> 01:16:38,959
>> Yeah, I I I do agree.

1485
01:16:38,960 --> 01:16:40,959
I I use the engine manager shorthand,

1486
01:16:40,960 --> 01:16:44,830
but it's really about being a tech lead for like a 30 or 40 person organization,

1487
01:16:44,880 --> 01:16:46,639
you know, or being an architect for one.

1488
01:16:46,640 --> 01:16:50,079
I think the the term architect gives me a little bit of a distaste,

1489
01:16:50,080 --> 01:16:53,439
but it's being an architect, he's also on the ground, >> a hands-on architect.

1490
01:16:53,440 --> 01:16:54,750
>> Hands-on architect.

1491
01:16:54,800 --> 01:16:58,718
and you don't have any of the management stuff, which is a blessing and curse,

1492
01:16:58,719 --> 01:17:01,439
but it's kind of remarkable that you can spin these things up.

1493
01:17:01,440 --> 01:17:04,959
If they make a mistake, you can keep on correcting them until they get it right.

1494
01:17:04,960 --> 01:17:08,559
We've always been able to do that on the human side, and I can do it faster.

1495
01:17:08,560 --> 01:17:12,639
I think the ultimate result for me is you just have to be more ambitious about everything

1496
01:17:12,640 --> 01:17:18,399
you do. You can produce more higher performance, higher quality, more secure.

1497
01:17:18,640 --> 01:17:20,399
So, our ambitions have to raise up.

1498
01:17:20,400 --> 01:17:24,639
You also mentioned that with with AI like you're everyone's using and you're building

1499
01:17:24,640 --> 01:17:29,999
stuff but also with Cockor DB you're now building something that is also related to AI.

1500
01:17:30,000 --> 01:17:31,119
Can you talk about that?

1501
01:17:31,120 --> 01:17:32,879
>> Yeah. Yeah. We're building multiple things.

1502
01:17:32,880 --> 01:17:35,070
I mean AI is the future.

1503
01:17:35,120 --> 01:17:37,439
>> Well, it's here to say I think that's >> here to say.

1504
01:17:37,440 --> 01:17:42,590
Yeah. I mean like uh you know one of our thesis which is uh not crazy.

1505
01:17:42,640 --> 01:17:44,959
Every application in the future is going to be written by AI.

1506
01:17:44,960 --> 01:17:48,959
You know I think there will be some kind of bespoke software you know handcrafted software.

1507
01:17:48,960 --> 01:17:53,919
I think we will see that um continue to exist um just as people write assembly still

1508
01:17:53,920 --> 01:17:58,238
you know um but it's going to be diminishing in size so you know asmtoically approaching

1509
01:17:58,239 --> 01:18:02,559
100% of software will be written by uh AI >> generated right >> yeah it's hard to say

1510
01:18:02,560 --> 01:18:06,158
if like when it's going to end where the humans are the agent you know kind of like supplying

1511
01:18:06,159 --> 01:18:10,799
the agency and the vision behind it I think that might exist for many many years but

1512
01:18:10,800 --> 01:18:15,039
like I think the code will like fundamentally be written by the AI and I think we're

1513
01:18:15,040 --> 01:18:20,319
going to just see this explosion of applications and We're we're seeing that inside Cockroach

1514
01:18:20,320 --> 01:18:21,599
Labs. We're seeing it elsewhere.

1515
01:18:21,600 --> 01:18:22,319
Earlier this year,

1516
01:18:22,320 --> 01:18:27,439
we kind of rolled out this um internal platform where non-engineers could write kind

1517
01:18:27,440 --> 01:18:28,399
of many applications.

1518
01:18:28,400 --> 01:18:30,238
I mean, this is like you're hearing this at other companies.

1519
01:18:30,239 --> 01:18:31,439
We did the same thing.

1520
01:18:31,440 --> 01:18:35,759
And over the course of just a couple months, you know, 500 applications,

1521
01:18:35,760 --> 01:18:40,669
a thousand applications >> by by non-engineers, >> by non-engineers, primarily by non-engineers.

1522
01:18:40,719 --> 01:18:41,919
And I thought it was awesome.

1523
01:18:41,920 --> 01:18:44,479
Like our HR team is building like these little applications.

1524
01:18:44,480 --> 01:18:48,158
Like this is the thing I've always dreamed about doing and like they were never serviced.

1525
01:18:48,159 --> 01:18:51,198
Like like your CFO is all over the place and ours is no exception.

1526
01:18:51,199 --> 01:18:54,079
Producing dashboards that they can never created before.

1527
01:18:54,080 --> 01:18:55,759
And I think it's very empowering.

1528
01:18:55,760 --> 01:18:56,959
I'm married. I have a wife.

1529
01:18:56,960 --> 01:18:57,919
She needs software.

1530
01:18:57,920 --> 01:18:59,919
She cannot produce that software on her own.

1531
01:18:59,920 --> 01:19:03,119
And I have never actually helped her produce the software which is my own failing.

1532
01:19:03,120 --> 01:19:07,039
But um I think there's a world in the future where she gets like everyone's getting custom

1533
01:19:07,040 --> 01:19:08,399
software built for them.

1534
01:19:08,400 --> 01:19:12,238
And you're just going to also see greater and greater applications and higher quality

1535
01:19:12,239 --> 01:19:15,439
um uh systems being produced by every company as well.

1536
01:19:15,440 --> 01:19:17,359
>> Today, what is your stack?

1537
01:19:17,360 --> 01:19:20,750
What what do you work with in terms of harness model?

1538
01:19:20,800 --> 01:19:21,839
How you run agents?

1539
01:19:21,840 --> 01:19:22,718
Is it one agent?

1540
01:19:22,719 --> 01:19:25,599
Is it multiple? What kind of terminal do you use?

1541
01:19:25,600 --> 01:19:27,439
>> Yeah. Yeah. It's evolved over time.

1542
01:19:27,440 --> 01:19:30,959
So, you know, like when I first started dabbling AI stuff again, it was, you know,

1543
01:19:30,960 --> 01:19:34,639
GitHub co-pilot, was an Emacs user for like 20some years.

1544
01:19:34,640 --> 01:19:37,839
I got convinced to move to VS Code, but that's all gone now.

1545
01:19:37,840 --> 01:19:40,879
Um, at some point I moved to using cloud code.

1546
01:19:40,880 --> 01:19:43,279
That was a thing that whatever reason I just got started using it.

1547
01:19:43,280 --> 01:19:45,359
It was cloud code in the terminal of late.

1548
01:19:45,360 --> 01:19:46,718
Uh, I use a mixture.

1549
01:19:46,719 --> 01:19:48,799
So, um, I'll just describe my current setup.

1550
01:19:48,800 --> 01:19:52,559
I use the cloud desktop app, cloud code via the cloud desktop app.

1551
01:19:52,560 --> 01:19:54,750
It's fantastic. Good job, Anthropic.

1552
01:19:54,800 --> 01:19:59,759
I also sometimes use the codec desktop app just so I I have an alternative model to turn

1553
01:19:59,760 --> 01:20:02,559
to for some very critical things we're working on.

1554
01:20:02,560 --> 01:20:06,079
I will get, you know, one of those models sometimes like the best model.

1555
01:20:06,080 --> 01:20:08,399
oftentimes I'm using the best model like Fable, you know,

1556
01:20:08,400 --> 01:20:09,999
I've recently started using that.

1557
01:20:10,000 --> 01:20:12,238
Sometimes it's Astra, sometimes it's the other ones,

1558
01:20:12,239 --> 01:20:15,119
but you you like you have one produced design, you have the other one being like,

1559
01:20:15,120 --> 01:20:16,639
"Hey, my colleague produced this.

1560
01:20:16,640 --> 01:20:19,999
Can you uh kind of tear it up, you know, adversarial review it?"

1561
01:20:20,000 --> 01:20:21,599
And it's not always perfect, right?

1562
01:20:21,600 --> 01:20:25,359
You know, but I think there is utility especially for something that's super super critical.

1563
01:20:25,360 --> 01:20:28,799
We're pushing towards the launch really soon and kind of rushing towards the finish line.

1564
01:20:28,800 --> 01:20:30,479
I'm telling folks on the team, it's like, you know,

1565
01:20:30,480 --> 01:20:33,599
especially our very senior folks like you should use the best model right now.

1566
01:20:33,600 --> 01:20:35,279
Um it's worthwhile to do that.

1567
01:20:35,280 --> 01:20:37,039
I'm generally just using the best model.

1568
01:20:37,040 --> 01:20:40,879
And part of the reason is I don't actually know that I'm getting a lot more intelligence

1569
01:20:40,880 --> 01:20:45,359
from it, but I don't want to have the cognitive overhead of deciding on a case- by case

1570
01:20:45,360 --> 01:20:47,198
basis. Yeah. >> Should I use Sonnet?

1571
01:20:47,199 --> 01:20:48,238
Should I use Opus?

1572
01:20:48,239 --> 01:20:49,359
Should I use Fable?

1573
01:20:49,360 --> 01:20:51,119
Should I use Soul or Astra?

1574
01:20:51,120 --> 01:20:55,759
And I think, you know, maybe if you like I really need speed, >> I would make that decision.

1575
01:20:55,760 --> 01:20:58,238
But often times I'm like I'm doing things in parallel.

1576
01:20:58,239 --> 01:21:01,119
So you're asking how many agents I'm spinning up.

1577
01:21:01,120 --> 01:21:02,479
>> Well, it depends.

1578
01:21:02,480 --> 01:21:05,919
Like often times there there's like a certain number of sessions you might be using.

1579
01:21:05,920 --> 01:21:11,759
I find like kind of cognitive overhead is about five to 10 sessions concurrently but

1580
01:21:11,760 --> 01:21:15,039
sometimes those sessions will have many sub agents doing things.

1581
01:21:15,040 --> 01:21:16,639
>> Yeah. >> And they'll be long running.

1582
01:21:16,640 --> 01:21:19,039
>> Yeah. Yeah. It'll depend on what I'm doing.

1583
01:21:19,040 --> 01:21:23,119
So as I was coming here I was on the train I spun up something to do a little bit of

1584
01:21:23,120 --> 01:21:25,439
research and that was just probably using three sub agents.

1585
01:21:25,440 --> 01:21:30,718
Other times it might be 20 sub aents you know with uh Anthropic has dynamic workflows.

1586
01:21:30,719 --> 01:21:32,639
I can't remember what the codeex thing is called,

1587
01:21:32,640 --> 01:21:36,879
but they have various ways of spinning up kind of graphs of agents and, you know,

1588
01:21:36,880 --> 01:21:37,839
prosecuting them.

1589
01:21:37,840 --> 01:21:40,718
I mean, I I think it's sometimes I'm probably using 100 sub agents,

1590
01:21:40,719 --> 01:21:41,919
others it's only like five.

1591
01:21:41,920 --> 01:21:45,198
It depends on what what's happening and sometimes I'm not doing any of that agentic stuff

1592
01:21:45,199 --> 01:21:46,959
because I'm off thinking about something else.

1593
01:21:46,960 --> 01:21:51,119
>> Since you were an Emacs user for 20 years, you know, using the terminal, right?

1594
01:21:51,120 --> 01:21:55,839
Or well, Emacs, but how how come you move to graphical interface with agents?

1595
01:21:55,840 --> 01:21:58,559
A lot a lot of people still use to uh terminal UIs.

1596
01:21:58,560 --> 01:22:02,638
>> Yeah. Yeah. Well, I so I moved from Emacs to VS Code.

1597
01:22:02,639 --> 01:22:05,919
I was just like a a colleague was like, "Hey, these IDs are really good.

1598
01:22:05,920 --> 01:22:06,638
You need to try them out."

1599
01:22:06,639 --> 01:22:07,999
I was like, "Emac is an ID."

1600
01:22:08,000 --> 01:22:11,198
And then I I kind of moved and like, you know, some of the stuff was just more seamless.

1601
01:22:11,199 --> 01:22:12,638
And like Emacs keeps on catching up.

1602
01:22:12,639 --> 01:22:14,559
It always feels a little bit, you know,

1603
01:22:14,560 --> 01:22:17,678
6 months to a year behind and like every time they have a new version,

1604
01:22:17,679 --> 01:22:20,238
my setup would break and I just got tired of that.

1605
01:22:20,239 --> 01:22:21,359
So, I moved to VS Code.

1606
01:22:21,360 --> 01:22:24,158
But then I was like, I'd have many terminals open in VS Code.

1607
01:22:24,159 --> 01:22:27,999
Um, my setup would always be like uh terminals in VS Code tabs.

1608
01:22:28,000 --> 01:22:29,599
the terminals and tabs, you know,

1609
01:22:29,600 --> 01:22:32,638
have like eight of them open and then they started taking over.

1610
01:22:32,639 --> 01:22:34,959
It used to be like just cloud code in the terminals.

1611
01:22:34,960 --> 01:22:38,399
Yeah. >> Yeah. And then, you know, just some point recently is like six weeks ago,

1612
01:22:38,400 --> 01:22:41,599
a colleague was like, well, have you tried out the cloud desktop app recently?

1613
01:22:41,600 --> 01:22:42,079
It's really good.

1614
01:22:42,080 --> 01:22:44,399
I was like, yeah, no, no, I'm very very happy.

1615
01:22:44,400 --> 01:22:47,439
And I made the switch and it was a little bit awkward at first and then and suddenly

1616
01:22:47,440 --> 01:22:49,198
I'm like, holy crap, this is awesome.

1617
01:22:49,199 --> 01:22:51,678
>> You you can like manage the agents a bit better, easier.

1618
01:22:51,679 --> 01:22:55,198
I mean it's like you have the sessions there the sessions are down the side you know

1619
01:22:55,199 --> 01:22:59,519
and it's like they're kind of like tabs in some regard and it's just but the whole integration

1620
01:22:59,520 --> 01:23:03,759
that anthropic's been doing and openi is doing the same thing now by the way like I'm

1621
01:23:03,760 --> 01:23:08,270
not dissing cursor and factory and cognition they're all pushing the same thing it's

1622
01:23:08,320 --> 01:23:14,638
incredible how fast these systems are innovating right now and evolving what do you think

1623
01:23:14,639 --> 01:23:19,039
good software engineering looks today compared to like you know four years ago before

1624
01:23:19,040 --> 01:23:22,959
we had a has it has it changed has change or or that's not really changed.

1625
01:23:22,960 --> 01:23:26,399
>> Yeah, I I I think the ambition has to increase, you know.

1626
01:23:26,400 --> 01:23:29,039
Um I think the quality has to be higher, security has to be higher.

1627
01:23:29,040 --> 01:23:30,319
I >> I'm glad you mentioned quality.

1628
01:23:30,320 --> 01:23:35,119
Can we talk about that cuz I I'm seeing across the industry just quality declines,

1629
01:23:35,120 --> 01:23:37,359
which you cannot fully put your finger on AI,

1630
01:23:37,360 --> 01:23:42,319
but often times it is people pushing out more and more and just not paying attention

1631
01:23:42,320 --> 01:23:46,430
to just small regressions here and there.

1632
01:23:46,480 --> 01:23:51,599
again like you're you're building a database like yeah >> have you noticed any or have

1633
01:23:51,600 --> 01:23:55,759
you gotten feedback of any quality regressions or or if not like how come because like

1634
01:23:55,760 --> 01:24:01,599
what when like this is just basic you know like we talk about the law of of physics this

1635
01:24:01,600 --> 01:24:07,039
is an observed thing that when you start to have more output you increase your deployment

1636
01:24:07,040 --> 01:24:12,960
frequency you often not always but you often have um like more regressions

1637
01:24:13,360 --> 01:24:16,479
and more bugs if you produce a certain number of lines of you're probably going to have

1638
01:24:16,480 --> 01:24:18,158
a certain number of defects per line of code.

1639
01:24:18,159 --> 01:24:19,599
And now you can produce more lines of code,

1640
01:24:19,600 --> 01:24:21,999
so you probably would have more defects, right?

1641
01:24:22,000 --> 01:24:26,079
But the thing that pushes against that is that you can be telling these agents,

1642
01:24:26,080 --> 01:24:28,638
you have to give them quite a firm hand at this.

1643
01:24:28,639 --> 01:24:31,999
This is a thing that I I hope that the model providers are listening to,

1644
01:24:32,000 --> 01:24:34,079
but you need to give them quite a hand.

1645
01:24:34,080 --> 01:24:37,439
They kind of get a little bit lazy on the testing side,

1646
01:24:37,440 --> 01:24:39,519
and you have to make sure the tests are comprehensive,

1647
01:24:39,520 --> 01:24:42,158
but also they're using all the tech testing techniques.

1648
01:24:42,159 --> 01:24:44,158
And there's a lot of testing techniques out there.

1649
01:24:44,159 --> 01:24:48,399
we have a lot of knowledge about how to do testing well and you know what the agents

1650
01:24:48,400 --> 01:24:53,119
are lazy humans are a little bit lazy so getting the humans to actually be very disciplined

1651
01:24:53,120 --> 01:24:57,039
about their testing is also challenging and I think it's actually easier with the agents

1652
01:24:57,040 --> 01:25:00,319
you know you can kind of instill that you can get them set up you have to give them the

1653
01:25:00,320 --> 01:25:05,198
guidance use property based testing use metamorphic testing use like these advanced testing

1654
01:25:05,199 --> 01:25:09,279
techniques deterministic simulation testing you know there there's like technique after

1655
01:25:09,280 --> 01:25:14,238
technique you can use and humans have always like I I'm lazy myself Like I'm pretty good

1656
01:25:14,239 --> 01:25:15,599
about being disciplined about testing.

1657
01:25:15,600 --> 01:25:18,079
At some point you're just like okay that was enough.

1658
01:25:18,080 --> 01:25:22,158
We gota ship and like now you can get a little bit like you know a little bit stricter

1659
01:25:22,159 --> 01:25:25,999
and firmer. I think the same thing applies over on the security side the performance

1660
01:25:26,000 --> 01:25:30,959
side. I mean we've always had at cockroach and in the industry we've known security coding

1661
01:25:30,960 --> 01:25:32,238
practices. >> Yep.

1662
01:25:32,239 --> 01:25:36,479
>> You can literally have every single line of code on every commit reviewed by a security

1663
01:25:36,480 --> 01:25:41,839
expert >> doing like an advers adversarial uh security review by an agent or multiple

1664
01:25:41,840 --> 01:25:43,198
agents >> or multiple agents.

1665
01:25:43,199 --> 01:25:43,999
Right. And you know,

1666
01:25:44,000 --> 01:25:47,678
we're going through a tough time in the industry right now with like kind of um hacks

1667
01:25:47,679 --> 01:25:49,198
and security leaks and whatnot.

1668
01:25:49,199 --> 01:25:51,678
There's only a limited number of bugs that can be in software.

1669
01:25:51,679 --> 01:25:55,519
I think we'll get it, you know, added on the security side and on the quality side.

1670
01:25:55,520 --> 01:25:58,479
And then also just on the p when I I say quality, it's not just bugs,

1671
01:25:58,480 --> 01:26:02,319
but it's also like, you know, just little things like, oh, that UX element is wrong.

1672
01:26:02,320 --> 01:26:04,238
And there's no excuse for that right now.

1673
01:26:04,239 --> 01:26:06,638
Like fixing it is so so easy.

1674
01:26:06,639 --> 01:26:11,119
And what we're also starting to see evolve is like, you know, designers using Figma.

1675
01:26:11,120 --> 01:26:12,718
Well, I think that should be a thing of the past.

1676
01:26:12,719 --> 01:26:18,238
Right now, our designer just deals with HTML and CSS and JavaScript directly and sometimes

1677
01:26:18,239 --> 01:26:22,879
is even just producing pull requests, you know, PRs, which she loves.

1678
01:26:22,880 --> 01:26:24,879
We all love everybody's happy with this.

1679
01:26:24,880 --> 01:26:26,589
There's not any this waterfall.

1680
01:26:26,639 --> 01:26:30,479
>> So, she she's producing pull requests for the production codebase.

1681
01:26:30,480 --> 01:26:33,999
>> Yeah. Yeah. And you know, this is on the UX side, not on the core database,

1682
01:26:34,000 --> 01:26:36,959
>> of course. But this is the area that that like she owns, right?

1683
01:26:36,960 --> 01:26:39,678
>> Yeah. Yeah. It's just I mean, she loves it.

1684
01:26:39,679 --> 01:26:40,238
Everybody loves it.

1685
01:26:40,239 --> 01:26:41,599
I mean there's no downside.

1686
01:26:41,600 --> 01:26:42,638
>> This is not new.

1687
01:26:42,639 --> 01:26:43,759
>> Yeah, >> for sure.

1688
01:26:43,760 --> 01:26:46,079
>> Yeah. Yeah. >> What about code review?

1689
01:26:46,080 --> 01:26:47,439
What's your take on code review?

1690
01:26:47,440 --> 01:26:50,879
I think it's pretty it's starting to get a bit controversial like is it going to stay

1691
01:26:50,880 --> 01:26:54,959
or not? Because it's been a practice that's been around like think about I mean you you

1692
01:26:54,960 --> 01:26:55,599
worked at Google.

1693
01:26:55,600 --> 01:26:59,919
Google has been so big on code review as I understand there you there are two code reviews,

1694
01:26:59,920 --> 01:27:03,999
right? The the domain expert reviews it and then there's a language expert reviews it.

1695
01:27:04,000 --> 01:27:05,839
So you went through that.

1696
01:27:05,840 --> 01:27:08,718
How did you think of it and how are you thinking of it now?

1697
01:27:08,719 --> 01:27:13,678
I used to be part of the I was one of the early members of the C++ coding >> review as

1698
01:27:13,679 --> 01:27:16,479
I'm asking right okay you will have strong opinions tell me.

1699
01:27:16,480 --> 01:27:21,759
>> Yeah. Yeah. I mean I I I see that code review I think was fairly essential before

1700
01:27:21,760 --> 01:27:26,238
the age of AI but now we're getting to the point where I'm reviewing and looking at code.

1701
01:27:26,239 --> 01:27:30,718
I I I look at most of the code that the the agents producing still and yet it's like

1702
01:27:30,719 --> 01:27:33,759
you're not giving it the same level of scrutiny, right?

1703
01:27:33,760 --> 01:27:35,198
And this has always been the case.

1704
01:27:35,199 --> 01:27:37,519
If you get a a poll request from a junior engineer,

1705
01:27:37,520 --> 01:27:40,879
you have to give it more scrutiny than if you get it from your most senior engineer and

1706
01:27:40,880 --> 01:27:44,238
you ask any, you know, tech lead, any engineering manager, any software,

1707
01:27:44,239 --> 01:27:45,439
they'll they'll say, "Yeah, yeah,

1708
01:27:45,440 --> 01:27:48,479
the junior engineer or someone new to the codebase, you know,

1709
01:27:48,480 --> 01:27:50,079
you just have to give more scrutiny."

1710
01:27:50,080 --> 01:27:52,079
And what I'm finding as the agents are getting better,

1711
01:27:52,080 --> 01:27:53,919
you're having to give less and less scrutiny.

1712
01:27:53,920 --> 01:27:55,759
You do have to give scrutiny to some things.

1713
01:27:55,760 --> 01:27:58,750
It's like I said, it's like the testing will be incomplete.

1714
01:27:58,800 --> 01:28:00,479
Maybe they didn't follow the security stuff.

1715
01:28:00,480 --> 01:28:02,799
Maybe, you know, like there was a performance regression.

1716
01:28:02,800 --> 01:28:07,039
And I think it's like you know assuring there's kind of constraints on the system so

1717
01:28:07,040 --> 01:28:09,230
it's hard for them to do the wrong thing.

1718
01:28:09,280 --> 01:28:13,999
I suspect that you know I don't know if it's going to be this year or next that we're

1719
01:28:14,000 --> 01:28:17,839
materially going to stop looking at the code in the same way that we don't look at assembly

1720
01:28:17,840 --> 01:28:22,399
anymore like you will >> we we trust the compiler to produce pretty good assembly.

1721
01:28:22,400 --> 01:28:26,238
We trust the compiler to produce good assembly and most >> unless you're one of those

1722
01:28:26,239 --> 01:28:30,638
people where it's really important you're and you're maybe a game developer or or someone

1723
01:28:30,639 --> 01:28:33,519
and you look at it and but there's fewer and fewer of those folks.

1724
01:28:33,520 --> 01:28:38,959
>> Yeah. And even then I think what we'll be migrating too is why why does the human

1725
01:28:38,960 --> 01:28:40,319
have to look at the assembly?

1726
01:28:40,320 --> 01:28:42,079
The AI should look at the assembly.

1727
01:28:42,080 --> 01:28:47,790
like I'm doing something right now where I I want to have a zero overhead abstraction,

1728
01:28:47,840 --> 01:28:51,119
you know, for doing something that is done at test time and when it's in production is

1729
01:28:51,120 --> 01:28:55,439
compiled out. Babel set up for me a little system where it actually looks at the decompiled

1730
01:28:55,440 --> 01:28:59,279
code to verify like that there's only a few extra instructions put in place.

1731
01:28:59,280 --> 01:29:00,638
I never would have put that in place,

1732
01:29:00,639 --> 01:29:03,359
but now it has this guardrail that every time it changes this,

1733
01:29:03,360 --> 01:29:05,359
it can verify that there's no regression.

1734
01:29:05,360 --> 01:29:08,718
And I I think you're going to see more and more of this where you kind of put guard rails

1735
01:29:08,719 --> 01:29:13,279
in place. I think the model kind of likes it because it can work within that guard rail.

1736
01:29:13,280 --> 01:29:16,638
>> Now, for what, like 20 plus years,

1737
01:29:16,639 --> 01:29:20,079
you were writing so much code like I'm sure you were in the zone.

1738
01:29:20,080 --> 01:29:22,479
Do you remember being in the zone and just turning it out?

1739
01:29:22,480 --> 01:29:27,839
You wrote wrote a lot of like production ready code now that you're coding with AI like

1740
01:29:27,840 --> 01:29:29,519
do you get into the zone?

1741
01:29:29,520 --> 01:29:30,718
>> Yeah, absolutely.

1742
01:29:30,719 --> 01:29:32,718
>> And h how is that zone?

1743
01:29:32,719 --> 01:29:34,238
Is it the same? Is it different?

1744
01:29:34,239 --> 01:29:36,319
>> Feels It definitely feels a little bit different.

1745
01:29:36,320 --> 01:29:41,710
It's maybe a little bit less intense, but you're managing more things cognitively.

1746
01:29:41,760 --> 01:29:45,439
>> Like, can you describe me like what is it right now when you're in the zone?

1747
01:29:45,440 --> 01:29:50,799
>> Yeah. Well, I'm thinking of ideas that normally would have taken me a a week to experiment

1748
01:29:50,800 --> 01:29:55,839
with and I think of multiple of these experiments and then I fire them off all simultaneously

1749
01:29:55,840 --> 01:29:59,678
and then I'm kind of like reviewing like what else should I do while that's that's being

1750
01:29:59,679 --> 01:30:03,119
complete. Sometimes I'm kind of reviewing like they'll be giving me progress updates

1751
01:30:03,120 --> 01:30:04,559
of like, oh, hey, this is coming in.

1752
01:30:04,560 --> 01:30:05,359
We're seeing this stuff.

1753
01:30:05,360 --> 01:30:07,198
And I'm being like, well, that doesn't sound right.

1754
01:30:07,199 --> 01:30:07,919
Hey, what about this?

1755
01:30:07,920 --> 01:30:09,919
Or do we do this, you know, correctly?

1756
01:30:09,920 --> 01:30:13,519
You know, maybe I had design and it's not implementing the design quite perfectly.

1757
01:30:13,520 --> 01:30:16,399
But I have this little feeling that, you know, I haven't been a college professor,

1758
01:30:16,400 --> 01:30:18,079
but maybe I'm I was a, you know,

1759
01:30:18,080 --> 01:30:21,839
if I was a college professor and I had a whole swarm of research assistants and they're

1760
01:30:21,840 --> 01:30:25,439
all got off doing things and it's coming back, but it's coming back just really rapidly.

1761
01:30:25,440 --> 01:30:26,238
>> Really rapidly.

1762
01:30:26,239 --> 01:30:27,519
You're not waiting months or weeks.

1763
01:30:27,520 --> 01:30:28,799
>> I'm not waiting months or weeks.

1764
01:30:28,800 --> 01:30:29,519
And then I'm iterating.

1765
01:30:29,520 --> 01:30:30,718
I'm like, "Oh, that one failed.

1766
01:30:30,719 --> 01:30:33,279
That's fine." You know, you just got to let go.

1767
01:30:33,280 --> 01:30:35,759
And this this is the nature of the the software I build.

1768
01:30:35,760 --> 01:30:38,319
I think there's other pieces where it's like you just whip out a website.

1769
01:30:38,320 --> 01:30:41,519
You can whip out something that doesn't have this level of kind of scrutiny,

1770
01:30:41,520 --> 01:30:42,879
you know, very quickly.

1771
01:30:42,880 --> 01:30:46,238
>> But I I talked about this in I have a whole bunch of analogies about like the the

1772
01:30:46,239 --> 01:30:48,158
>> the let's talk about analogies.

1773
01:30:48,159 --> 01:30:51,599
What analogies do you have about about using AI or AI?

1774
01:30:51,600 --> 01:30:55,599
the the one I was um you know advocating for uh I had kind of two that I was advocating

1775
01:30:55,600 --> 01:31:01,039
for like late last year and this year which is you know AI is coming for us it's here

1776
01:31:01,040 --> 01:31:05,359
right and it's like uh you know you're producing software you're like walking down the

1777
01:31:05,360 --> 01:31:08,559
road and sometimes you know someone will pass you they're running they've got an efficient

1778
01:31:08,560 --> 01:31:13,759
gate and whatnot but everyone's under their own locomotion and the these agent coding

1779
01:31:13,760 --> 01:31:18,399
agents came and it was like a car pulled up next to you get into the car you don't how

1780
01:31:18,400 --> 01:31:21,839
to drive you don't understand the controls but you got to get in start you might forgot

1781
01:31:21,840 --> 01:31:24,990
the gas pedal and it takes off and crashes into a tree.

1782
01:31:25,040 --> 01:31:26,799
But you have to learn how to drive, you know.

1783
01:31:26,800 --> 01:31:30,959
I think using all these coding tools, it just isn't doesn't just happen naturally.

1784
01:31:30,960 --> 01:31:32,479
It's learning how to drive.

1785
01:31:32,480 --> 01:31:36,479
I think we might be in the era of the F1 driver right now,

1786
01:31:36,480 --> 01:31:39,919
which is like the really good people can drive these systems a lot harder,

1787
01:31:39,920 --> 01:31:42,479
a lot faster than the people who just picking them up.

1788
01:31:42,480 --> 01:31:45,039
If you've never used an agent coding tool,

1789
01:31:45,040 --> 01:31:48,319
there's a vast difference between someone who's like really expert in them,

1790
01:31:48,320 --> 01:31:49,919
knows where they break,

1791
01:31:49,920 --> 01:31:53,519
can pay attention to that versus someone who's just picking up for the first time.

1792
01:31:53,520 --> 01:31:57,519
>> One analogy I've heard is we used to talk about 10x engineer.

1793
01:31:57,520 --> 01:32:00,799
You remember like this this used to be a debate for a very long time.

1794
01:32:00,800 --> 01:32:02,238
Is it or is is it not?

1795
01:32:02,239 --> 01:32:04,479
But now what I'm hearing is the 100x engineer.

1796
01:32:04,480 --> 01:32:09,839
>> Yeah. And so you're saying that you are seeing some folks who maybe let's not use

1797
01:32:09,840 --> 01:32:13,678
the term 100x engineering but like this like F1 driver who is just really really good

1798
01:32:13,679 --> 01:32:17,999
at it like how would you describe a person who who you've seen this is is it just like

1799
01:32:18,000 --> 01:32:22,638
rock solid engineering basics and they picked up uh they lean into using these tools

1800
01:32:22,639 --> 01:32:23,839
or what are they like?

1801
01:32:23,840 --> 01:32:28,638
Yeah. Yeah. I mean there there is uh quite a bit of a it feels like you know it's directional

1802
01:32:28,639 --> 01:32:32,479
that if they're good at software engineering before I mean I sometimes think it's like

1803
01:32:32,480 --> 01:32:35,599
you know everyone's in this kind of spectrum of capability of software engineering and

1804
01:32:35,600 --> 01:32:40,238
this is just like you know taking that line and spread it out and it's not quite true

1805
01:32:40,239 --> 01:32:43,359
you know I think it's helped some people more than others but you know feels like it's

1806
01:32:43,360 --> 01:32:47,519
just stretched it out so you know your ability before is now amplified >> I'm always

1807
01:32:47,520 --> 01:32:54,559
interested to learn that for example Boris Churnney Tibo at uh uh open AI I

1808
01:32:54,560 --> 01:32:57,279
they both have been really really good software engineers.

1809
01:32:57,280 --> 01:33:01,119
Boris wrote one of the first Typescript books, the first TypeScript book for O'Reilly.

1810
01:33:01,120 --> 01:33:02,638
He built some massive systems.

1811
01:33:02,639 --> 01:33:06,559
Same same with Tibo who built it and now you're kind of seeing oh these people are building

1812
01:33:06,560 --> 01:33:09,999
all these tools and innovating like yeah they >> they were really good before as well.

1813
01:33:10,000 --> 01:33:10,879
>> Yeah. Yeah. Well,

1814
01:33:10,880 --> 01:33:14,638
I mean this is I mean I have two other analogies to give you about like the what feels

1815
01:33:14,639 --> 01:33:19,119
like an AI. You you've undoubtedly heard about the term pair programming and pair programming.

1816
01:33:19,120 --> 01:33:20,319
Yeah, pair programming, you know,

1817
01:33:20,320 --> 01:33:23,839
and the the idea behind pair programming is it's good to just like, you know,

1818
01:33:23,840 --> 01:33:26,479
have one keyboard, one module, and two engineers at it,

1819
01:33:26,480 --> 01:33:29,759
one at the keyboard and the other one sitting beside them kind of like looking over the

1820
01:33:29,760 --> 01:33:30,718
shoulder and giving guidance.

1821
01:33:30,719 --> 01:33:35,039
And I think there's an aspect of that feeling where I actually got chat to do a little

1822
01:33:35,040 --> 01:33:36,238
image of this where, you know,

1823
01:33:36,239 --> 01:33:40,510
it's like the Android is at the computer typing and you're just there giving instructions.

1824
01:33:40,560 --> 01:33:42,559
But that was maybe the way it felt like a year ago.

1825
01:33:42,560 --> 01:33:44,399
I think it feels a little bit different now.

1826
01:33:44,400 --> 01:33:49,519
Um, the one I I I've just started recently saying is that I feel like the domain experts,

1827
01:33:49,520 --> 01:33:52,718
the people who were really strong before are now massively amplified.

1828
01:33:52,719 --> 01:33:56,238
Have you seen all this mathematical stuff coming out like the crazy proofs?

1829
01:33:56,239 --> 01:33:58,959
>> Well, I don't understand it, but I >> I don't understand them either.

1830
01:33:58,960 --> 01:34:01,519
Yeah. >> Yeah. So, I told you I wasn't very good at math, you know,

1831
01:34:01,520 --> 01:34:05,919
it's not complete like I just was like I I stopped in the freshman year of college, right?

1832
01:34:05,920 --> 01:34:09,678
Yeah. But you know I kind of like wa watch along with these uh advancements and you know

1833
01:34:09,679 --> 01:34:13,999
Terrence Tao he's like probably the most famous living mathematician you know super genius

1834
01:34:14,000 --> 01:34:17,599
he actually posted this session there was this recent breakthrough I think it's called

1835
01:34:17,600 --> 01:34:22,079
the Jacobian conjecture I don't even know what it meant but he posted this chat GBT session

1836
01:34:22,080 --> 01:34:26,718
where he's interacting I think it was chat GBT and you could see him interacting with

1837
01:34:26,719 --> 01:34:31,919
this intelligence and it was crazy because he's talking to it as a peer colleague and

1838
01:34:31,920 --> 01:34:36,399
it's responding and l it honestly looks like I'd encourage everyone to go this up.

1839
01:34:36,400 --> 01:34:38,718
It looks like, you know, almost a foreign language.

1840
01:34:38,719 --> 01:34:43,279
It's like his domain expertise is getting amplified by the system he's interacting with.

1841
01:34:43,280 --> 01:34:47,759
And you can see how he's like kind of learning and exploring ideas just really rapidly.

1842
01:34:47,760 --> 01:34:50,079
Now, there's all this controversy about AI mathematics,

1843
01:34:50,080 --> 01:34:55,919
but I feel like the the analogy that comes to mind to me though is uh the domain experts.

1844
01:34:55,920 --> 01:34:58,718
They're a little bit like sorcerers in their particular domain.

1845
01:34:58,719 --> 01:35:02,158
You know, you have the earth sorcerers, the earth wizards, the water ones, whatnot.

1846
01:35:02,159 --> 01:35:06,559
And if you know the the magic incantations, the right words to say the right order,

1847
01:35:06,560 --> 01:35:08,399
you actually get something kind of magical.

1848
01:35:08,400 --> 01:35:09,198
And if you don't,

1849
01:35:09,199 --> 01:35:12,718
you just get sparkle stuff that doesn't have anything there behind it, you know,

1850
01:35:12,719 --> 01:35:15,279
you you would probably you don't know much about distributed databases.

1851
01:35:15,280 --> 01:35:17,919
If you're going to ask Fable or Astra,

1852
01:35:17,920 --> 01:35:20,959
build me a distributed database like Cocker DB, you will get something out,

1853
01:35:20,960 --> 01:35:23,198
but it'll kind be ultimately hollow inside.

1854
01:35:23,199 --> 01:35:26,399
If you're an expert in databases and you ask to build a distributed database and you

1855
01:35:26,400 --> 01:35:29,599
can point out all the various things you have to know about distributed database,

1856
01:35:29,600 --> 01:35:30,479
here's what you have to worry about.

1857
01:35:30,480 --> 01:35:33,279
storage layer, the networking layer, here's the various data structures,

1858
01:35:33,280 --> 01:35:37,519
runtime inside, you can actually get something quite magical very very rapidly out of

1859
01:35:37,520 --> 01:35:43,678
it. >> So what would your advice be for advice be for engineers with like mid-level to

1860
01:35:43,679 --> 01:35:50,959
to senior level who you know who have been figuring out how to do coding to become strong

1861
01:35:50,960 --> 01:35:53,439
engineers in this I guess age of AI?

1862
01:35:53,440 --> 01:35:58,879
Well, the first off is you have the most amazing tutor kind of readily at hand.

1863
01:35:58,880 --> 01:36:02,799
And I mean, one of the things that I would always do, you know,

1864
01:36:02,800 --> 01:36:05,359
throughout my career, and now I've kind of stopped doing it,

1865
01:36:05,360 --> 01:36:07,599
but it's like the reason why is going to become obvious,

1866
01:36:07,600 --> 01:36:10,479
which is like I would always look at other people's code.

1867
01:36:10,480 --> 01:36:13,550
So, I was at, you know, Google early on.

1868
01:36:13,600 --> 01:36:15,198
You probably heard of Jeff Dean.

1869
01:36:15,199 --> 01:36:16,959
Jeff Dean was amazing coder.

1870
01:36:16,960 --> 01:36:20,559
His colleague Sanjay Gowat also just an incredible coder.

1871
01:36:20,560 --> 01:36:23,439
And you know, I would be looking at their poll requests.

1872
01:36:23,440 --> 01:36:24,559
I'd be looking at their changes.

1873
01:36:24,560 --> 01:36:25,999
It wasn't called poll requests at Google.

1874
01:36:26,000 --> 01:36:29,039
There's a different name for it, but I'd be looking at my CL, right?

1875
01:36:29,040 --> 01:36:32,238
>> Yeah. >> CL. Um, this is P4.

1876
01:36:32,239 --> 01:36:33,759
It's a different version control system.

1877
01:36:33,760 --> 01:36:36,718
But I'd be looking at their changes, be like, how'd they do what they did, right?

1878
01:36:36,719 --> 01:36:37,759
Be looking at their code.

1879
01:36:37,760 --> 01:36:40,479
Oh my goodness, Sanj's code is really always very elegant.

1880
01:36:40,480 --> 01:36:41,919
You know, Jeff is high performance.

1881
01:36:41,920 --> 01:36:42,718
How's he doing that?

1882
01:36:42,719 --> 01:36:44,079
You know, like how's he going about it?

1883
01:36:44,080 --> 01:36:47,069
It's almost just like you acquire via osmosis.

1884
01:36:47,119 --> 01:36:49,839
But now what you can do is not just acquire vossmosis,

1885
01:36:49,840 --> 01:36:53,519
but you can literally like I mean I would be encouraging if you're a junior engineer

1886
01:36:53,520 --> 01:36:57,359
and you know there's a senior engineer nearby like you could ask them how they're doing

1887
01:36:57,360 --> 01:37:01,678
what they're doing but you could just ask the AI to dissect what they've done and explain

1888
01:37:01,679 --> 01:37:05,919
it to you and explain to you at various different levels like how does this code work?

1889
01:37:05,920 --> 01:37:07,118
What is it doing?

1890
01:37:07,119 --> 01:37:07,999
Give me a diagram.

1891
01:37:08,000 --> 01:37:09,118
Explain it to me like I'm five.

1892
01:37:09,119 --> 01:37:10,238
Explain it to me like I'm 10.

1893
01:37:10,239 --> 01:37:11,198
Explain it to me in French.

1894
01:37:11,199 --> 01:37:16,319
whatever like you want like I mean fundamentally the the to some degree AI is a translation

1895
01:37:16,320 --> 01:37:20,319
tool they translate it from whatever you know kind of language you're understanding it's

1896
01:37:20,320 --> 01:37:24,559
in and keep on interrogating it till it gets it you know increases your understanding

1897
01:37:24,560 --> 01:37:29,359
>> you were saying how domain experts are very much amplified I I guess one strategy

1898
01:37:29,360 --> 01:37:32,718
as a software engineer is like obviously become a great software engineer and use it

1899
01:37:32,719 --> 01:37:37,919
as a tool you can get a lot faster you can get good at distributed systems like I'm not

1900
01:37:37,920 --> 01:37:43,439
a distributed databases expert at all but I I use AI to explain a few things for me to

1901
01:37:43,440 --> 01:37:47,439
understand up front which was very helpful and it would have taken me a lot longer time

1902
01:37:47,440 --> 01:37:51,118
beforehand. So I can use this >> but I wonder if there's another part of like as a software

1903
01:37:51,119 --> 01:37:54,399
engineer you can use it to become more of a domain expert wherever you're working if

1904
01:37:54,400 --> 01:37:55,359
it's a payments company.

1905
01:37:55,360 --> 01:37:57,839
I mean use it to learn about payments as well.

1906
01:37:57,840 --> 01:38:02,559
So you can help the business you can help your team and honestly you'll just learn more

1907
01:38:02,560 --> 01:38:03,359
right. >> Yeah. Yeah.

1908
01:38:03,360 --> 01:38:07,678
I I think you know I would encourage everyone you have to have a little curiosity right

1909
01:38:07,679 --> 01:38:11,839
don't don't be bound in by you know the area you're working on explore outside of it

1910
01:38:11,840 --> 01:38:16,319
you know I was working on Gmail but I was fascinated about how like the the main Google

1911
01:38:16,320 --> 01:38:19,839
search engine worked I was fascinated by like how the internals of big table worked even

1912
01:38:19,840 --> 01:38:24,479
though I wasn't directly working on big table just explore in look at those things and

1913
01:38:24,480 --> 01:38:29,919
now it's so much easier because you have this super advanced patient intelligence there

1914
01:38:29,920 --> 01:38:33,919
to explain to it like why do you think it was done this way and then Like I mean once

1915
01:38:33,920 --> 01:38:36,319
you become an expert you can be integrity and I see it's done that way.

1916
01:38:36,320 --> 01:38:37,359
What happens if we change this?

1917
01:38:37,360 --> 01:38:40,799
Would this be helpful and you know that's where you go from just kind of learning to

1918
01:38:40,800 --> 01:38:42,158
actually contributing back.

1919
01:38:42,159 --> 01:38:44,879
I think uh everyone has to have the personal agency to do this.

1920
01:38:44,880 --> 01:38:48,079
You know if you're just sitting there waiting for someone to educate you on how to do

1921
01:38:48,080 --> 01:38:51,999
this, it's going to be really hard right now because anyone who's coming in explaining

1922
01:38:52,000 --> 01:38:54,879
how to use AI or explain how to be a better software engineer,

1923
01:38:54,880 --> 01:38:56,559
they're going to be out of date, right?

1924
01:38:56,560 --> 01:39:01,279
You just got to get in there be using the tools all the time yourself and using it like

1925
01:39:01,280 --> 01:39:01,999
you have to learn.

1926
01:39:02,000 --> 01:39:07,039
I feel like I've learned more in the past probably even year than the previous five years

1927
01:39:07,040 --> 01:39:12,238
combined which is weird given >> given your trajectory and given the environment you

1928
01:39:12,239 --> 01:39:13,599
were working in.

1929
01:39:13,600 --> 01:39:16,638
Right. >> Yeah. I mean like everybody's been in the industry for a while.

1930
01:39:16,639 --> 01:39:18,079
Like I'm a definitely a better coder.

1931
01:39:18,080 --> 01:39:20,638
I was a better coder 10 years ago than when I first got in the industry.

1932
01:39:20,639 --> 01:39:24,718
It's like I could look back every decade and realize like I got a lot better and I feel

1933
01:39:24,719 --> 01:39:26,718
like I just got a lot better over this past year.

1934
01:39:26,719 --> 01:39:29,599
And and this was also one of the reasons I was really excited to talk to you because

1935
01:39:29,600 --> 01:39:32,399
when we started to just exchange messages,

1936
01:39:32,400 --> 01:39:35,039
the first thing you wrote to me when I asked you like, hey, you know,

1937
01:39:35,040 --> 01:39:36,319
how how are things going?

1938
01:39:36,320 --> 01:39:37,599
You said like you wouldn't believe,

1939
01:39:37,600 --> 01:39:42,238
but my coding output is insane and it's high quality and it's database quality level.

1940
01:39:42,239 --> 01:39:44,399
And those were things I don't really usually see it.

1941
01:39:44,400 --> 01:39:48,350
I I usually see, okay, I'm now producing more code, but it's slo.

1942
01:39:48,400 --> 01:39:49,519
But again, like to me,

1943
01:39:49,520 --> 01:39:53,839
this is a bit of an inspiration like look like you can use these tools to just like amplify

1944
01:39:53,840 --> 01:39:55,118
yourself as a software engineer.

1945
01:39:55,119 --> 01:39:56,799
like you are one example, right?

1946
01:39:56,800 --> 01:39:57,999
>> Yeah. Hopefully one of many.

1947
01:39:58,000 --> 01:40:00,879
>> Yeah. Yeah. No, and we I'm not the only one in Cockroach Labs.

1948
01:40:00,880 --> 01:40:02,238
We have other people doing this as well.

1949
01:40:02,239 --> 01:40:03,519
I find it very exciting.

1950
01:40:03,520 --> 01:40:06,319
You know, it's a little bit exhausting right now, but it's very exciting.

1951
01:40:06,320 --> 01:40:10,319
Like I got into software engineering um because I like building stuff.

1952
01:40:10,320 --> 01:40:11,999
You can build stuff faster.

1953
01:40:12,000 --> 01:40:15,599
You know, the stuff you might have had to compromise on the in the past and you can take

1954
01:40:15,600 --> 01:40:16,799
away some of those compromises.

1955
01:40:16,800 --> 01:40:19,118
I mean, you see this in the UX of software coming out.

1956
01:40:19,119 --> 01:40:20,879
I think the UX is a lot higher.

1957
01:40:20,880 --> 01:40:23,678
You see all the fancy like web animations and whatnot.

1958
01:40:23,679 --> 01:40:25,118
That's only just like the surface level.

1959
01:40:25,119 --> 01:40:27,439
It just extends way way below that.

1960
01:40:27,440 --> 01:40:28,799
Peter, this was awesome.

1961
01:40:28,800 --> 01:40:30,079
Thanks for coming on the podcast.

1962
01:40:30,080 --> 01:40:31,039
>> Yeah, this is wonderful.

1963
01:40:31,040 --> 01:40:31,919
Thanks for having me.

1964
01:40:31,920 --> 01:40:35,759
One reason I was excited to talk to Peter is because he's been a very high-profile and

1965
01:40:35,760 --> 01:40:40,399
productive engineer preai building some of the most resilient distributed systems in

1966
01:40:40,400 --> 01:40:44,479
production. Cockroach DB is known for its resilience and how even if several nodes are

1967
01:40:44,480 --> 01:40:47,279
destroyed, the [music] database still operates without data loss.

1968
01:40:47,280 --> 01:40:50,158
Basically, it's as hard to get rid of as cockroaches are.

1969
01:40:50,159 --> 01:40:54,079
Hence the name. One interesting part of our conversation was how Peter built more efficient

1970
01:40:54,080 --> 01:40:56,718
data structures than the standard coding libraries had.

1971
01:40:56,719 --> 01:41:00,879
Thanks to him and colleagues paying attention to parts of the library that seemed slow.

1972
01:41:00,880 --> 01:41:05,359
He did it for the C++ STL map and in Go for the Swiss table implementation.

1973
01:41:05,360 --> 01:41:09,439
I found both stories a good reminder that you can improve the existing library or even

1974
01:41:09,440 --> 01:41:12,718
the language especially if you measure which parts feel slow.

1975
01:41:12,719 --> 01:41:17,198
Another part of his conversation that I liked was how Peter came a bit of a full circle.

1976
01:41:17,199 --> 01:41:21,839
He used to write 100,000 lines of code per year being a very productive engineer and

1977
01:41:21,840 --> 01:41:27,439
CTO. He didn't stop writing code aiming to coach engineers between 2022 and 2024.

1978
01:41:27,600 --> 01:41:31,839
And then he started to code again because with AI tools he wanted to coach his engineers

1979
01:41:31,840 --> 01:41:34,844
better. But it's hard to do if you don't use the tools yourself.

1980
01:41:34,845 --> 01:41:38,030
[music] And now he finds himself being extremely productive.

1981
01:41:38,080 --> 01:41:40,879
And this time the team around him is productive as well.

1982
01:41:40,880 --> 01:41:43,999
And we're not talking about vioded software, but database worthy,

1983
01:41:44,000 --> 01:41:46,634
highquality code generated and committed to production.

1984
01:41:46,635 --> 01:41:50,109
[music] Peter is convinced that AI amplifies existing expertise.

1985
01:41:50,159 --> 01:41:54,479
And this is one reason why he probably learned more this last year building with AI than

1986
01:41:54,480 --> 01:41:56,158
the previous 5 years combined.

1987
01:41:56,159 --> 01:42:00,319
And I find it a valuable reminder that learning and building deep expertise in software

1988
01:42:00,320 --> 01:42:01,999
engineering, this is very valuable.

1989
01:42:02,000 --> 01:42:05,839
And as closing, I appreciated that Peter said that not only is he excited,

1990
01:42:05,840 --> 01:42:07,519
but he's also exhausted.

1991
01:42:07,520 --> 01:42:08,879
There's [music] a lot to learn,

1992
01:42:08,880 --> 01:42:12,879
but it's tiring and neither him nor anyone I know is immune to this.

1993
01:42:12,880 --> 01:42:17,439
So, if you're also exhausted with all of the things going on with AI,

1994
01:42:17,440 --> 01:42:18,718
know that you're not alone.

1995
01:42:18,719 --> 01:42:22,079
Check the show list for more the pragmatic engineering deep dives on Google's engineering

1996
01:42:22,080 --> 01:42:23,839
culture and on distributed systems.

1997
01:42:23,840 --> 01:42:25,279
If you like this episode,

1998
01:42:25,280 --> 01:42:28,238
please make sure you're subscribed in your [music] podcast player and a special thank

1999
01:42:28,239 --> 01:42:29,678
you if you leave a rating.

2000
01:42:29,679 --> 01:42:32,160
Thanks. And I'll see you in the next