SofaBuffers
Streaming-first binary serialization · MIT licensed

SofaBuffers

Structured Objects For Anyone

… so optimized, feels amazing.

A compact, streamable binary format for structured data — tightly packed, schema-driven, and tiny enough for the smallest microcontroller yet fast enough for the cloud.

Explore on GitHub Get the Code Generator Read the Documentation →
11 supported languages 1 schema → all languages 0 heavyweight dependencies
Core libraries

One format, eleven languages

Each core library implements the same streamable wire format and shares a common suite of conformance test vectors — and some languages ship more than one build, each tuned for a different target.

CC / C++Heap-free C99 / C++20 for embedded MCUs C++C++Pure C++20, no C backend, speed-tuned RsRustHigh-speed std build for big machines RsRust#![no_std], heap-free, for microcontrollers GoGoStreaming, dependency-free runtime core PyPythonPure Python 3.9+, zero dependencies TsTypeScriptRuns on Node, browsers, Deno & Bun JvJavaAllocation-light, runs anywhere a JVM does KtKotlinOne multiplatform codec: JVM, JS & native C#C#Allocation-light .NET, cloud-ready DtDartPure Dart, streaming, zero dependencies ZigZigAllocator-free, comptime-dispatched, max throughput
Get started

Installed in one command

Pick the generator or your core library, hit copy. Every core library is dependency-free and installs with a single command through the package manager you already use — only the generator itself differs per operating system.

Instructions for

SofaBuffers Code Generator

sofabgen is a single static binary with no runtime dependencies. It turns a YAML/JSON object definition into typed code for every supported language.

Requires Nothing at all — the binary is self-contained

Full README on GitHub →

1Install the CLI

Detects your OS and architecture, verifies the SHA-256 and drops the binary on your PATH. Set SOFABGEN_VERSION to pin a release, SOFABGEN_INSTALL_DIR to choose where it lands.

bash 
curl -fsSL https://raw.githubusercontent.com/sofa-buffers/generator/main/install.sh | sh

1Install the CLI

Detects your OS and architecture, verifies the SHA-256 and drops the binary on your PATH. Set SOFABGEN_VERSION to pin a release, SOFABGEN_INSTALL_DIR to choose where it lands.

zsh 
curl -fsSL https://raw.githubusercontent.com/sofa-buffers/generator/main/install.sh | sh

1Install the CLI

There is no install script for Windows — this pulls the released binary straight into a folder that is already on your PATH. On an ARM machine swap in sofabgen-windows-arm64.exe.

PowerShell 
$dst = "$env:LOCALAPPDATA\Microsoft\WindowsApps\sofabgen.exe"
Invoke-WebRequest "https://github.com/sofa-buffers/generator/releases/latest/download/sofabgen-windows-amd64.exe" -OutFile $dst

2Or use the channel you already manage tools with

Every channel ships the same binary from the same release — pick one.

npm — pinned per project

any shell 
npm install --save-dev @sofa-buffers/generator

PyPI — standalone CLI

any shell 
uv tool install sofabgen   # or: pipx install sofabgen

Go toolchain

any shell 
go install github.com/sofa-buffers/generator/cmd/sofabgen@latest

3Generate typed sources

One flag per language — c, cpp, go, python, typescript, rust, csharp, java, kotlin, zig, dart — or docs for a self-contained HTML reference page.

any shell 
sofabgen --version
sofabgen --lang rust --in messages/ --out src/generated/
What makes it cool

Small on the wire, simple in your code

Every byte and every design decision is tuned for moving structured data fast — across slow links and onto the tiniest devices — without complicating the code you write.

Streaming-first

Serialize and deserialize in arbitrarily small chunks — you never need the whole message in memory. No length-prefixed envelope, so encoders emit data before the final size is even known.

Tightly packed

Varint (LEB128) encoding shrinks small integers to a single byte, and zig-zag mapping keeps negative numbers compact. Low overhead is reserved for the field types you actually use most.

Self-describing TLV

Type information lives in the stream itself: each field header packs (id << 3) | type. Decoders can read, descend, or skip any field — even ones they don't recognize.

Nested sequences

Delimiter-based sequences create fresh ID namespaces for deeply nested, hierarchical structures — without ever pre-computing a size or buffering a sub-tree.

Schema → code

Write a definition once; the generator emits dead-simple, language-native objects with .serialize() / .deserialize() plus streaming variants.

Runs everywhere

A tiny footprint with no heavy runtime makes it equally at home on resource-constrained MCUs and high-throughput cloud services. Little-endian on the wire, identical on every architecture.

Why SofaBuffers?

It fills the gap the others left open

Born out of real IoT and embedded communication needs, where every existing format made an awkward trade-off.

JSONHuman-readable, but carries significant byte overhead.
MsgPack / BSON / CBORMore efficient, but lack first-class message definitions.
Protocol BuffersPowerful, but generate unnecessarily heavy code.
FlatBuffersRandom-access, but not tightly packed for slow interfaces.

"Because JSON is too fat,
and Protobuf is too Google."

Under the hood

A wire format you can reason about

Simple, predictable building blocks — varints, a 3-bit type tag, and delimiter-based scopes.

One header, eight wire types

Each field starts with a single varint header. The low three bits select the wire type; everything above is the field ID (0 … 2,147,483,647).

  • Unsigned & signed integers (varint / zig-zag)
  • Fixed-length values: float32, float64, UTF-8 strings, binary blobs
  • Arrays of integers and of fixed-length values
  • Sequence start / end markers for nested structures
  • No top-level envelope — a message is just a flat stream of fields
wire-format.txt 
# Field header = (id << 3) | type
0x0  unsigned int      (varint)
0x1  signed int        (zig-zag varint)
0x2  fixlen value      (f32/f64/str/blob)
0x3  unsigned int array
0x4  signed int array
0x5  fixlen value array
0x6  sequence start    (nested scope)
0x7  sequence end

# Varints are little-endian, 7 bits each:
300 -> 0xAC 0x02
# Zig-zag keeps small negatives tiny:
-1  -> 1     -2 -> 3
example.py 
# Generated objects are dead simple
person = Person()
person.name = "Ada"
person.id   = 42

# One-shot convenience
data    = person.serialize()
person2 = Person.deserialize(data)

# ...or stream it in arbitrarily small chunks,
# never holding the whole message in memory.

Two ways to use it

Power users can drive the raw encoder and decoder by hand for maximum control. Everyone else writes a schema and lets the generator produce language-native classes.

  • Generated objects with intuitive field access
  • Convenience serialize() / deserialize() helpers
  • Streaming variants for constrained memory
  • Shared conformance test vectors across every language
Quality & performance

Battle-tested, and benchmarked

Every core library is held to the same bar — proven interoperable and kept fast — by two dedicated harnesses that run across all of them.

Proven correct, everywhere

Our Crucible differential-fuzzing harness feeds identical bytes to every language at once and treats any disagreement as a bug. The result: implementations that stay byte-for-byte interoperable, round-trip clean, and hardened against malformed input.

Explore Crucible →

Benchmarked to stay fast

The Arena suite measures every library head-to-head against Protocol Buffers — from raw encode/decode throughput on server CPUs to firmware footprint on bare-metal ARM and RISC-V. Fair, reproducible, and always watching for regressions.

Explore the Arena →

Ready to slim down your wire format?

Browse the core libraries, read the format specification, and start serializing structured data the lightweight way.

Browse all repositories Read the spec →