Avro and Protocol Buffers: When a File Cannot Be Read Without Its Schema
Two schema-driven formats that made opposite choices about where the schema lives. One file explains itself, the other is unreadable on its own, and neither situation is a bug.
A CSV can be opened by anything. A .pb file cannot be opened by anything, including the tool that wrote it, unless you also have the schema that defines what the bytes mean. That is not a defect and it is not recoverable by trying harder. It is a design decision, and Avro and Protocol Buffers made it in opposite directions.
Knowing which kind of file you have in front of you decides whether the next step is a conversion or a search through a repository for a .proto file.
Avro puts the schema in the file#
An .avro file is an Object Container File. The header holds a metadata map, and in that map is avro.schema: the full writer schema, as JSON, inside the file. A reader never needs the definition out of band, because the data and the definition travel together. That self-description is the format's entire argument for itself, and it is why Avro to JSON needs nothing from you except the file.
The rest of the design follows from streaming. Data sits in blocks of record count, byte length, payload and a 16-byte sync marker, so a reader can resynchronise at a block boundary, which is what makes an Avro file splittable across a cluster. Integers are zig-zag encoded varints, so small values cost one byte and negative numbers do not cost ten. Field names live once in the header rather than on every record. Schema resolution matches reader and writer fields by name with defaults, which is how a consumer written this year reads a file written two years ago.
Protocol Buffers leaves the schema behind#
A .pb file is the opposite bargain. Each field is a numeric identifier and a wire type followed by a value. There are no field names in the payload, no comments, no type descriptions, nothing that says what field 3 was called or what it meant. You get compactness, fast decoding and forward-compatible evolution, and you pay by requiring the matching .proto schema to exist somewhere else. The Protocol Buffers format page sets out the wire format in more detail.
So Protocol Buffers to JSON works differently from every other data route here: it needs the .pb file and exactly one self-contained .proto file, supplied together in a folder or ZIP. Schema imports are blocked, so conversion cannot reach out to a relative or remote resource, and exactly one message type per schema is accepted so that nothing silently picks the wrong root message. A .pb extension does not identify which message the bytes are, and guessing produces plausible nonsense rather than an error.
Why anyone chose the harder option#
Because the schema is usually not missing. Protocol Buffers is designed for services that generate code from a shared .proto at build time: both ends already have the definition, and the wire never needs to carry it. Repeating field names on every message between two systems that already know them is pure overhead. Avro made the other call because it was built for data at rest in a lake, where a file might outlive the job that wrote it and be read by a team that has never seen the pipeline.
- Avro: row-oriented, self-describing, splittable, designed for ingestion and long-lived event logs where the schema will change and old files must stay readable.
- Protocol Buffers: schema-external, designed for messages in flight between services that compile the same definition.
- Parquet: columnar and self-describing, which is where Avro landing data usually gets compacted to for analytics.
- A file that is self-describing can be handed to a stranger. A file that is not can only be handed to somebody who also has the schema.
If your file turns out to be columnar rather than row-oriented, the guide to opening a Parquet file covers that route and the type loss involved in flattening it.
What JSON can and cannot carry out#
Converting either format to JSON gives you the values and the declared field names. It does not give you the schema's documentation strings, its evolution rules or its defaults, because JSON has nowhere to put them. For Avro the schema is still in the original file, so nothing is lost permanently; for protobuf the .proto remains the only place that information exists. When the records are regular, Avro to CSV flattens the schema to a header row, which is convenient for a look and discards the nesting.
The short version: if the file starts with Obj and a version byte, it can explain itself and you can just convert it. If it is a bare protobuf message, find the .proto first. No amount of tooling substitutes for the definition, and any tool that claims otherwise is guessing on your behalf.
Troubleshooting
Most often a compression codec other than null or deflate, usually Snappy. The file is not damaged; it is outside what the local decoder covers, and a partial decode would be worse than the refusal.
The .pb file and exactly one self-contained .proto must be supplied together in a folder or ZIP. A .pb file alone cannot be decoded by anything.
Imports are blocked deliberately so conversion never fetches a relative or remote file. Flatten the definitions you need into one self-contained schema with a single message type.
Likely the wrong root message type for these bytes. Protobuf will decode against a mismatched schema and produce structured nonsense, so confirm the message type rather than trusting the output.
Frequently asked questions
Why can I open an Avro file but not a .pb file?
Avro stores the writer schema in the file header, so the data explains itself. Protocol Buffers stores only field numbers and wire types, so the meaning lives in a .proto file that never travels with the payload.
Can a protobuf message be decoded without the schema?
Not meaningfully. The field numbers and coarse wire types can be walked, but a length-delimited field could be a string, a nested message or a packed number array, and choosing wrongly yields output that looks fine and is not. That is why the schema is required here.
Why does the protobuf route need a folder or ZIP?
Because it needs two files: the message and exactly one self-contained .proto. Imports are blocked so nothing is fetched from elsewhere, and a single message type is required so the root is never chosen by guesswork.
Which Avro files will not convert?
Those using a compression codec other than null or raw deflate, Snappy being the common case. They are refused rather than partially decoded, because a dataset that silently stops halfway is the worse failure.
Is the schema I upload sent anywhere?
No. Both the data file and the .proto stay in your browser. A schema often describes an internal system more completely than the payload does, which is a good reason not to hand it to a remote service.
