Protobuf

#

1. Origins & History

#

Protobuf was born inside Google around 2001 back when Jeff Dean and Sanjay Ghemawat were building the core infrastructure. In the beginning, it was yet another attempt to fix a recurring scale problem: thousands of servers needed to exchange structured data over the network and store it on disk, and the existing options were bad. Most nonstandard binary formats broke every time someone added a field. XML was oversized and slow to parse at the scale they needed. So they built an in-house format that was eventually named proto1. It was compact, fast, and reliably adaptable. It allowed old code to ingest new message formats and new code could read the old without anything breaking. Between 2005 and 2010 it had become the connective tissue in Google’s infrastructure — pretty much every server-to-server call and most stored data at the company was (and still is) in the protobuf format. In July 2008, Google open-sourced the cleaned-up second generation as proto2, in a release led by Kenton Varda. Proto3 followed with its stable 3.0 release in 2016, simplifying the language and allowing it to support many languages consistently — the “serialization” layer of gRPC was made available to the open-source community in 2015 as the public descendant of its internal Stubby RPC system. Currently, the “editions” models (released in 2023 & 2024) comprise the current generation. A fun genealogical footnote: Varda, the engineer who open-sourced protobuf and maintained proto2, later left and built Cap’n Proto, a competing format designed around eliminating the parse step entirely. Facebook’s Thrift was built by ex-Googlers reproducing the idea. Protobuf isn’t just a format; it’s the common ancestor of a whole family.

Adoption

#

Inside Google, it’s total — the company has stated there are hundreds of thousands of distinct message types in its codebase. Outside, adoption rode two vehicles. The first is gRPC, which made protobuf the default serialization of the cloud-native world: Kubernetes components, etcd, Envoy, and most service-mesh internals speak it. The second is network telemetry: the OpenConfig ecosystem (gNMI, gNOI) standardized on protobuf over gRPC, which is precisely why Juniper’s JTI, Cisco’s model-driven telemetry, and Radware’s stream API all reach for it. It’s also embedded in TensorFlow model formats, game networking stacks, and countless mobile apps where payload size and battery matter.

Who owns the protobuf Intellectual Property?

#

Google, more or less, but the answer is layered.

The Implementation: Google owns the copyright on the reference implementation (the protoc compiler and runtime libraries) and licenses it under the BSD 3-clause license — a permissive license that lets anyone use, modify, and redistribute it, commercially or otherwise, with attribution. So Google “owns” it in the copyright sense but has granted the world nearly unrestricted rights.

The Specification: Google owns this outright in a governance sense. As we saw (see?) in the IETF media-type draft, even the IETF’s own document lists Google’s protobuf team as the change controller and cites protobuf.dev as the normative reference. There is no standards body — no ISO, no IETF, no ECMA — with authority over the format. So, Google decides what protobuf is.

The Format: here’s the subtle part — a wire format as such is very hard to “own.” Copyright doesn’t protect ideas or methods of operation, and the encoding rules are published openly. Anyone can (and many do) write independent, clean-room protobuf implementations without touching Google’s code.

One caveat worth knowing when looking to hitch your horse:

BSD-3 contains no express patent grant (unlike Apache 2.0, which is what gRPC uses). In practice Google has never asserted patents over protobuf in seventeen years of open availability, and the ecosystem treats it as unencumbered — but the formal patent posture rests on implied license and Google’s behavior rather than an explicit grant. The “BSD 3-clause” and “Apache 2.0” are two standardized, off-the-shelf open-source license texts that any copyright holder can apply to their own work.

So, my interpretation is that Google owns the trademark-level identity, the reference code copyright, and complete change control over the spec — while the format itself functions as a public good because independent implementation is trivial and legally uncontested.

That said, I’m not a lawyer and that’s the practical picture, not legal advice.

License Comparison: Protobuf vs. gRPC

Both are permissive licenses from the same owner; the load-bearing difference is the patent column — Apache 2.0 puts Google’s patent promise in writing, BSD-3 leaves it to implied license and Google’s non-assertion track record. Practical engineering picture, not legal advice.

Protobuf Innovation

#

Protobuf needed to solve an interoperability problem in a largely heterogeneous and rapidly evolving environment: thousands of services, written in different languages, deployed independently, all exchanging structured data that changed shape constantly. To do this, it needed to perform three functions effectively within a single format — functions that had previously been handled by separate standards, tools, and conventions that rarely worked well together.

First, it needed to describe the data — a formal, language-neutral definition of message structure that both sides of an exchange could treat as a standard. Previously, this was done by interface definition languages and schema notations: CORBA’s OMG IDL, ASN.1’s abstract syntax, XML Schema, or — as I’m told — a header file and a design doc that drifted out of sync with reality. Protobuf’s .proto file collapsed this into a small, readable schema where the field numbers are the contract.

Next, it needed to deliver the data efficiently on the wire — a compact, fast-to-parse binary encoding. Previously, this was a separate standards layer entirely: ASN.1’s competing encoding rule sets (BER, DER, PER), Sun’s XDR, CORBA’s CDR, or the homespun TLV and die-cast formats every C shop maintained. Protobuf replaced the menu with exactly one encoding — varint-based, four wire types, no options to debate — it exchanged theoretical flexibility for a format simple enough to implement quickly and audit for security.

Finally, it needed to age well. To connect any two endpoints across languages and across time — turning the schema into idiomatic, type-safe code in every language in use, and guaranteeing that yesterday’s compiled code could safely exchange messages with tomorrow’s. Previously, code generation belonged to expensive vendor compilers (ASN.1) or framework-locked stub generators (rpcgen, CORBA ORBs), and schema evolution belonged to nothing at all — it was managed by version fields, flag days, and luck. Protobuf shipped a free, polished compiler as the product itself, and made compatibility structural: unknown fields are skipped, names never travel, and retired field numbers are reserved forever.

Serialization- What it actually does

#

Serialization converts a structured, in-memory object — with its types, its named fields, its nested sub-objects — into a flat sequence of bytes that can cross a wire or rest on disk, in a way that lets a receiver who shares the schema reconstruct the identical structure. It does so by creating an somewhat agnostic and minimalistic format, while parsing out expensive artifacts (usually verbosity) in the native data. “Flattening” the data (i.e. dumping the stream at the binary level) is not really the hard part. The challenge is in the interpretation of the structure. PCAP, for example, can be streamed, but PCAP needs to be understood by sender and consumer alike in order to be effective. This associated overhead is the baggage that limits PCAP to represent traffic flows, and not any other data structures. Does PCAP work well? Yes. But it will not scale as effectively as protobuf in heavy applications.

Protobuf’s simple approach is a single recursive convention: every unit of data is preceded by a compact instruction for reading it. Feild types are assigned tags, and the tag declares which field is coming and what shape it takes — a self-terminating varint, a fixed 4- or 8-byte value, or a run of bytes whose exact length is stated up front. The parser never searches for boundaries and never relies on terminators; it advances recursively through addition. Read the instruction, consume exactly what it prescribes, and consistantly land on the next instruction in the bitstream.

This is the same engineering discipline that framing at the lower layers has always used. An IP header’s IHL and Total Length fields declare the header’s extent and the packet’s end before the parser reaches either; a TCP header’s Data Offset does the same for its options. Nothing in a well-designed protocol scans for an end marker, because content can always impersonate a marker. Protobuf takes that principle — declare extent in a prefix, never delimit with a sentinel — and applies it recursively to arbitrary application data. This single property is what makes the format extensible: the scaffolding that frames today’s fields frames tomorrow’s identically, so old parsers glide over new data without breaking. A nested message is simply a length-delimited field whose bytes happen to be another protobuf message; the parser can descend into it, or skip it wholesale, using the same length prefix either way.

Because the convention composes, structures of any depth — messages inside messages, lists of messages, maps of messages — reduce on the wire to the same primitive: instruction, then exactly-measured content. And because every unit is self-measuring, a receiver can traverse data it only partially understands, skipping unknown fields by their declared extent. That single property is what makes the format extensible: the scaffolding that frames today’s fields frames tomorrow’s identically, so old parsers glide over new data without breaking.

What the format deliberately does not do is frame itself at the top. A serialized message has no header, no footer, no overall length — it is only the concatenation of its fields.

The outermost boundary is delegated to whatever carries the message: gRPC frames it on the HTTP/2 stream with a five-byte prefix, a Kafka record frames it with the record’s length, a file frames it with the file’s size. Each layer measures its own payload and lets the layer below carry it.

Protobuf Field Mapping: Ethernet / IPv4 / TLS Record

#

*Field numbers are per-message (each protocol is its own protobuf message); they are schema choices, not standards. Wire types: varint = self-terminating variable-length integer; fixed32 = always 4 bytes; len-delim = length prefix

• raw bytes.*

A Happy Ending

#

Focusing on serialization is my doing(for better or worse), but the story and conditions that led to the rise of protobuf and its continuing impact on traffic telemetry is a good one. The innovation was not any one function per se, but the refusal to treat them as separate concerns. Protobuf is one solution, a single artifact checked into version control, delivering the interface contract, the wire format, and the compatibility guarantee.

This makes Protobuf one of those rare elements that simply show up and provide universal value without garnering revenue intrinsically. Google considers it to be the core “glue” for its internal services and infrastructure. As it reaches across systems and applications, I think it’s like a network of highways. It’s value is really measured in proportion to what it makes possible, not what it actually costs to use.

Protobuf succeeded so completely that it became boring, which is the reason gRPC/protobuf packet streaming is quietly appearing in vendor products without fanfare. The roads were already built; vendors and operators need only to add an on-ramp.