Distributed databases with Peter Mattis

Summary

In this extensive conversation, Peter Mattis reflects on a three-decade engineering career spanning open-source creation, hyperscale distributed infrastructure at Google, and the founding of Cockroach Labs. The discourse begins with Mattis’s early work co-founding GIMP and GTK, emphasizing how productive naivety and relentless execution often matter more than competitor announcements. This foundations-first mindset carried into Google, where Mattis helped architect the backend indexing and storage engine for Gmail’s groundbreaking launch, authored the Python-based declarative build files that evolved into Bazel, and co-founded Colossus—Google’s successor to GFS that pioneered Reed-Solomon erasure coding and Bigtable metadata bootstrapping. Throughout these deep infrastructure projects, Mattis consistently highlights the imperative of understanding low-level hardware constraints, CPU cache hierarchies, and foundational data structures such as B-trees and Swiss Tables.

Transitioning to the inception of CockroachDB, the discussion examines the fundamental differences between append-only distributed storage systems and strongly consistent, relational databases. Mattis details CockroachDB’s mission to provide unkillable, zero-downtime operation through contiguous range-based automatic sharding, strict serializable ACID transactions, and Raft consensus quorum replication. By taking the burden of sharding and transaction isolation off application developers, CockroachDB prevents subtle and catastrophic data inconsistencies across mission-critical enterprise workloads.

The conversation culminates in an analysis of how generative AI and agentic coding workflows are transforming modern software engineering. Mattis recounts how coding agents brought him back to writing immense volumes of high-performance production code after years in executive leadership. He argues that modern AI tools do not equalize skill levels but rather massively amplify deep domain expertise—acting as a force multiplier for experienced engineers who know the precise technical incantations. While cautioning against agent laziness in testing and emphasizing the necessity of rigorous techniques like property-based and metamorphic testing, Mattis envisions a future where traditional manual code review recedes in favor of automated guardrails, empowering proactive engineers to accelerate their learning and build with higher ambition.

Topics

Early Foundations: GIMP, GTK, and Engineering Persistence

• Mattis transitioned from mechanical engineering to computer science in college after discovering a natural affinity for programming.
• GIMP and GTK originated as ambitious college projects built without formal computer graphics training, proving that initial naivety can prevent premature discouragement.
• A competitor’s pre-announcement on Usenet nearly halted GIMP’s release, reinforcing the lesson that execution matters more than overlapping ideas.

Hyperscale Google Systems: Gmail, Build Tools, and Colossus

• Gmail launched in 2004 with 1 GB of storage—drastically surpassing standard 4 MB quotas—powered by a custom B-tree and inverted index backend for message threading.
• Mattis authored the original declarative build files for Google 3 to replace monolithic Makefiles, creating the architecture that evolved into Blaze and Bazel.
• Colossus succeeded GFS to scale past 1,000-node cluster limits, implementing Reed-Solomon erasure coding for 2x storage overhead and bootstrapping metadata inside Bigtable.

Hardware Constraints, Caches, and Data Structure Optimization

• Modern systems engineering must navigate dramatic hardware speed shifts, from 5–10ms HDD latencies to 30–50μs NVMe SSD access and speed-of-light fiber limits.
• Achieving peak performance requires optimizing for CPU cache lines (L1/L2/L3) and utilizing lock-free programming over heavy synchronization.
• Mattis engineered cache-aware B-trees to replace pointer-heavy STL std::map structures and authored a Swiss Table implementation that accelerated Go’s map runtime.

CockroachDB Architecture: Automatic Sharding, Consistency, and Raft

• Distributed databases differ from distributed storage by managing structured, typed, relational data on immutable, append-only file systems via Log-Structured Merge-trees like Pebble.
• CockroachDB eliminates the developer burden of manual database sharding by indexing contiguous key ranges like a distributed B-tree.
• The system enforces strict serializable ACID transactions and uses Raft consensus across 3–5 replicas to survive node, rack, and regional data center outages without downtime.

AI Agentic Workflows, Code Verification, and Domain Amplification

• Advanced coding models enabled Mattis to return from executive management to shipping high-volume, database-grade production code, operating akin to a hands-on technical architect.
• Multi-agent workflows running 5–10 concurrent sessions require strict testing guidance (property-based, metamorphic, and deterministic simulation) to prevent agent laziness.
• Traditional human code review will decline as AI automated guardrails take over, mirroring how developers ceased reviewing compiler-generated assembly.
• AI functions as a force multiplier for domain experts (‘sorcerers’) who know precise architectural requirements, while providing an on-demand tutor for engineers eager to learn.

[00:00:00] Introduction and Overview

The host introduces Peter Mattis, co-founder and CTO of Cockroach Labs and original co-creator of GIMP, framing the discussion around distributed databases, large-scale systems engineering, and how modern AI tools have dramatically amplified production coding output.

• Peter Mattis co-founded Cockroach Labs, created GIMP and GTK, and worked on Gmail and Colossus at Google.
• The conversation covers distributed storage architectures, B-trees, consensus mechanisms, and leveraging AI coding models for high-quality systems engineering.

““For the last 30 years, I’ve always been a prolific coder, but my current output is a bit insane. And this isn’t vioded junk, but database worthy, highquality, high performance code”.”

[00:02:42] Peter’s Path into Technology

Mattis narrates his early exposure to programming through gaming and home computers, his initial enrollment in mechanical engineering in college, and his subsequent switch to computer science.

• Mattis began programming on early Apple computers after getting interested through video games.
• He initially enrolled in mechanical engineering following his father, but switched to computer science after finding CS coursework intuitive and engaging.

[00:04:00] Creating GIMP, GTK, and Entrepreneurial Lessons

Mattis recounts co-developing GIMP and the GTK graphics library as a college side project with Spencer Kimball, discussing naive persistence in the face of competitor announcements and early open-source adoption.

• GIMP began as a college project to build a Photoshop-like tool without formal computer graphics training.
• Mattis notes that a degree of naivety helps when starting massive projects that would otherwise seem intimidating.
• A pre-announcement from a competitor on Usenet almost discouraged the release, illustrating that execution matters far more than overlapping ideas.

““There’s always going to be someone else working on your idea. You can’t get dissuaded if they pre-announce it. Nothing ever comes of it.””

[00:07:16] Joining Google and Engineering Gmail’s Storage Backend

Mattis describes declining an early Google offer before joining in 2002 to work on the backend threading, indexing, and storage systems for Project Caribou (Gmail).

• Google founders Larry and Sergey reached out to Mattis because the original Google logo was designed in GIMP.
• Gmail launched on April 1, 2004, offering 1 GB of storage when competitors offered roughly 4 MB, intentionally designed as an unprecedented product leap.
• The storage and retrieval engine was rewritten to support message threading and fast indexed search using B-trees and inverted indexes.

““It was like kind of a shock and awe campaign for the industry. I think it was actually four megabytes for like Hotmail or Yahi Mail and then you had this and not only that we had so much more storage but it was indexed really fast.””

[00:14:51] Google’s Monorepo and Build Infrastructure Evolution

Mattis details the transition from Google 2’s monolithic Makefiles to Google 3’s declarative build file system, which eventually evolved into Blaze and the open-source Bazel.

• Google 2 relied on large, unwieldy Makefiles that made dependency declaration error-prone and hard to maintain.
• Mattis helped design the initial Google 3 build system using higher-level, Python-based BUILD files.
• This declarative dependency model laid the foundation for Blaze, Bazel, and Buck, dramatically optimizing large-scale monorepo build caching and execution.

[00:17:28] Colossus and Distributed Storage Architecture

Mattis describes founding the Colossus project as the next-generation successor to the Google File System (GFS), highlighting Reed-Solomon erasure coding and distributed metadata management.

• GFS faced scalability bottlenecks around single-node master architectures and cluster sizes capped at ~1,000 machines.
• Colossus introduced Reed-Solomon erasure coding to provide higher redundancy with roughly 2x storage overhead compared to 3x replication in GFS.
• Colossus solved metadata scaling by storing file chunks inside Bigtable, creating a circular bootstrapping dependency managed through a foundational base Bigtable.

““You have kind of nine chunks of data, but any five of those chunks can be used to reconstruct it. And what this means is you can lose any four copies and you can still reconstruct your data”.”

[00:23:59] Hardware Evolution and Modern Latency Constraints

The dialogue explores shifts in storage hardware from rotational HDDs to NVMe SSDs and how software systems must adapt to speed-of-light networking and hardware cache hierarchies.

• Storage media latencies transitioned from 5–10 milliseconds on HDDs to 30–50 microseconds on NVMe SSDs.
• At data center and global scales, speed of light in fiber becomes the dominant bottleneck for cross-zone and cross-region operations.
• High-performance systems engineering requires managing CPU cache levels (L1/L2/L3) and minimizing synchronization locks.

[00:30:04] Optimizing Core Data Structures: B-Trees and Swiss Tables

Mattis explains his work replacing standard data structures with cache-aware alternatives, specifically implementing B-trees for C++ STL maps and contributing Swiss Tables to Go.

• Standard balanced binary trees (like red-black trees in std::map) incur high cache-miss penalties and pointer overhead compared to B-trees.
• B-trees optimize spatial locality by packing sorted arrays into nodes and splitting them dynamically.
• Mattis implemented an open-addressing Swiss Table hash map for Go during a long flight, which helped inspire and accelerate the Go runtime’s upgraded map implementation.

[00:41:52] Leaving Google, Viewfinder, and the Roots of CockroachDB

Mattis discusses leaving Google, founding mobile photo startup Viewfinder, getting acqui-hired by Square, and conceiving an open-source distributed database modeled after Spanner.

• Distributed databases differ from distributed storage systems by handling typed, relational data with small record mutations on top of immutable file systems.
• Log-Structured Merge-trees (LSMs) like LevelDB, RocksDB, and Pebble enable relational databases to sit on append-only storage systems.
• Frustrated by the lack of robust Spanner-like open-source databases while building Viewfinder and working at Square, the founders spun out CockroachDB.

[00:46:10] CockroachDB’s Mission-Critical Resilience and Sharding

Mattis describes the founding philosophy of CockroachDB, its automatic range-based sharding mechanism, and its focus on zero-downtime survival of hardware and regional outages.

• CockroachDB was designed to be resilient against node, rack, and whole-region data center failures.
• Manual sharding places an immense operational burden on application engineers, turning them into poor database implementers.
• CockroachDB partitions a single contiguous key space into ordered, contiguous spans indexed like a distributed B-tree.

““The application developer is becoming a database developer at that point and they’re doing it poorly… We felt the burden for that belongs on the database developer.””

[00:55:28] Strong Consistency, Serializability, and Raft Consensus

Mattis details ACID transaction semantics, the distinction between eventual and strong consistency (serializability), and the mechanics of Raft consensus across replicas.

• Serializability ensures concurrent transactions appear to execute sequentially, preventing critical bugs like double-spending.
• Eventual consistency forces application developers to handle race conditions and stale reads manually.
• CockroachDB uses the Raft consensus protocol across typically 3 to 5 replicas, requiring quorum writes while serving reads efficiently from single replicas.

[01:06:15] AI Re-igniting Coding Output and Engineering Leadership

Mattis explains how AI coding models brought him back to writing heavy production code after stepping into management, comparing the interaction to technical leadership and hands-on architecture.

• Pre-AI, Mattis wrote up to 100,000 lines of code annually before management responsibilities reduced his hands-on output between 2022 and 2024.
• To effectively guide engineering teams on AI adoption, leaders must be active daily practitioners.
• Working with modern models feels like acting as a hands-on tech lead or architect orchestrating a large team of fast assistants.

[01:19:12] Tooling, Multi-Agent Workflows, and Quality Control

Mattis describes his current setup with Claude Code desktop, Codex, and multi-agent workflows, emphasizing the need for rigorous, automated testing to overcome agent laziness.

• Mattis transitioned from Emacs and VS Code terminals to dedicated desktop agent interfaces, running 5–10 concurrent sessions with multiple sub-agents.
• AI agents are prone to laziness in testing and require firm guidance to employ property-based, metamorphic, and deterministic simulation testing.
• AI enables non-engineers across departments (e.g., HR, Finance) and UX designers to build functional tools and commit direct pull requests.

““They kind of get a little bit lazy on the testing side, and you have to make sure the tests are comprehensive, but also they’re using all the tech testing techniques.””

[01:26:39] The Evolution of Code Review and Verification

The discussion examines how traditional manual code reviews will decline as AI automated verification, compiler guardrails, and adversarial review agents take over routine inspection.

• Human code review will decline similarly to how developers stopped inspecting compiler-generated assembly language.
• Verification will shift toward automated guardrails, decompilation checks for zero-overhead abstractions, and automated adversarial review agents.

[01:29:17] Domain Experts as Amplified ‘Sorcerers’

Mattis argues that AI acts as a massive force multiplier for deep domain expertise, comparing experienced engineers to sorcerers who know the precise incantations needed to generate complex systems.

• AI amplifies existing engineering skill rather than leveling everyone to the same plateau.
• Without domain expertise, prompts yield superficial software; with deep knowledge, models generate highly optimized, production-grade architectures rapidly.

““The domain experts, the people who were really strong before are now massively amplified… They’re a little bit like sorcerers in their particular domain… If you know the the magic incantations, the right words to say the right order, you actually get something kind of magical.””

[01:35:33] Advice for Leveling Up Engineering Skills

Mattis offers actionable advice for engineers to accelerate their career growth by using AI as an interactive, multi-level tutor and maintaining relentless technical curiosity.

• Engineers should use AI to dissect and explain elite engineers’ pull requests and complex architectures at various conceptual depths.
• Technical professionals must exercise personal agency and proactive curiosity to master adjacent domains and stay current with fast-evolving tools.

““You have the most amazing tutor kind of readily at hand… you could just ask the AI to dissect what they’ve done and explain it to you and explain to you at various different levels””