Why FlatBuffers

· FlatBuffers

This is a post in an ongoing series on FlatBuffers.

I’m going to try to make you excited about serialization formats. Bear with me here.

As a developer, when you send chunks of data over the internet, or save it to a hard drive, do you stop to consider how you should store that data?

How do you encode it to a file so that, in the future, another program can read your data? How do you choose a standard format to store your precious user data (and cat pictures)? Do you care about speedy writes and speedy reads, so your expensive servers can get back to doing real work?

You could choose JSON or XML, which are human readable and commonly supported, but they are slow to write and slow to parse. They also can’t protect you from writing bad data, and they don’t help you evolve your format over time.

You could choose Protocol Buffers, which helps you guarantee data integrity by using static typing. It’s also fast compared to interpreted formats like JSON or XML. Unfortunately, an entire Protocol Buffer must be parsed to read any data inside of it. This makes random access impossible.

(Protocol Buffers also require you to allocate memory to read them. This means there will be inevitable memory churn as you allocate and deallocate objects.)

These formats, and their cousins, feel like using an old VCR: put in the video, hit play, then fast-forward (remember this?) to the scene you’re looking for. It’s fundamentally a scan operation. There’s no seeking to the data.

This seems kind of… unexciting.

What if serialization could be better, so that you could have your cake and eat it too? Incredibly fast at random access — like native structs — while being schema-versioned for data integrity? And, to boot, no extraneous memory allocations?

FlatBuffers is what you get when you optimize for speed above all… and then, after the dust settles, realize you didn’t give up any of the features you love. FlatBuffers is feature-competitive with Protocol Buffers, and also has true random access. It can be orders of magnitude faster than JSON and XML, and is often faster than Protocol Buffers.

About the author

I'm Robert Winslow, a consulting engineer. I help teams launch great products: Learn more.