sofab package¶

Submodules¶

Module contents¶

SofaBuffers — runtime for the SofaBuffers binary wire format.

Byte-for-byte compatible with the C/C++/Rust/Go/Java/C# core libraries. Import the Encoder and Decoder and the wire-format types from here.

Encoder / Decoder resolve to the compiled native accelerator (sofab._speedups, built by Cython) when it is available, and to the pure-Python implementations otherwise — the two are byte-for-byte interchangeable (see IMPL).

class sofab.Encoder¶

Bases: object

Native encoder — see sofab.encoder.Encoder for the full contract.

One buffer-ownership model, byte-identical to the pure-Python encoder: the encoder writes into a fixed buffer and drains it through a flush sink, and never grows a buffer (CORELIB_PLAN S5.1).

Encoder.over_buffer(buffer, offset, flush) — the primitive: writes into a caller-owned bytearray, draining through flush when it fills.
Encoder(writer=None, sticky=False) — the same over a scratch buffer of _SCRATCH_SIZE bytes installed with a sink, which forwards to writer.write or, with no writer, appends into the result getvalue() hands back.

buffer_set()¶

bytes_used()¶

error¶

flush()¶

getvalue()¶

classmethod over_buffer()¶

write_bool()¶

write_bool_array()¶

write_bytes()¶

write_float32()¶

write_float32_array()¶

write_float32_array_bits()¶

write_float32_bits()¶

write_float64()¶

write_float64_array()¶

write_sequence_begin_lazy()¶

write_sequence_end()¶

write_sequence_end_keep()¶

write_signed()¶

write_signed_array()¶

write_string()¶

write_unsigned()¶

write_unsigned_array()¶

class sofab.Decoder¶

Bases: object

Native push decoder — see sofab.decoder.Decoder for the contract.

Bytes go in through feed and fields come out at a visitor or a binding. Incoming bytes are held in one contiguous buffer and parsed by advancing a C cursor with direct pointer indexing; a construct that runs off the end suspends and resumes from its first byte on the next feed, so the same path serves a whole message and a reader that dribbles one byte at a time.

error¶

feed()¶

Consume data and report the outcome for the bytes so far (§5.2).

See sofab.decoder.Decoder.feed() for the contract; this is the same call with the loop in C.

reset()¶: Forget the stream and start a new message, keeping the compiled binding and its destinations. See the pure engine for the contract.

class sofab.Visitor[source]¶

Bases: object

Base visitor: override the hooks for the fields you handle.

Every hook is keyed by the wire type the decoder recovered. field_id is the decoded field id. Unhandled hooks default to a no-op, which still consumes the value (so unknown fields are skipped safely).

destinations()[source]¶

The slots this handler wants its fields written into, or None.

Asked once, when the sofab.Decoder is built, and never again — so nothing the wire says can change the answer, which is what §6.6 asks of a decode’s storage. Return (binding, words, objects): a sofab.Binding mapping field ids to slots, a writable 8-byte-aligned buffer for the scalar and array slots, and a list for string/blob slots (or None when the table names none).

A field the table names is written straight into its slot and no typed hook fires for it — the same bargain on_array_begin(), on_string_begin() and on_blob_begin() strike per field, made once for the whole message. A field the table does not name reaches this visitor’s hooks exactly as it would have without a table, and the count/maxlen a table entry declares is answered from the table rather than from on_schema_bound().

This is not a second decode surface (§5.3.1). The decoder still drives, the walk is the same walk, and every rule — the receiver cap, the schema bound, the §7.3 tag test, the UTF-8 check, the declared element width, the resume transaction — has one implementation that runs for a mapped field and an unmapped one alike. The table says where a value goes; it never says how it is decoded.

Decoder(binding=…, words=…, objects=…) is the constructor shorthand for a handler that declares exactly this and nothing else.

Return type:: tuple[Binding, Any, list[Any] | None] | None

on_field(field)[source]¶

Called for every non-sequence field before its value is decoded. Return False to skip the value entirely; any other return proceeds to decode it and dispatch to the typed hook below.

Parameters:: field (Field)
Return type:: bool | None

on_schema_bound(field_id, n, wtype, subtype)[source]¶

The count or length the schema declares for this field, or -1.

Asked once, at the count/length header — after on_field(), before a payload byte is read or any storage is written — for a string, a blob or an array this handler has accepted, and for nothing else. A scalar carries neither a count nor a length, so none is asked for, and a field the handler skipped is never asked (§6.7.2).

n is what the wire announced: the byte length for a string or blob, the element count for an array.

wtype/subtype are the tag the wire carried, and they are here so that a handler can apply MESSAGE_SPEC §7.3 to its own declaration before answering. This is the only hook that spans more than one kind — on_string_begin() fires for a string and nothing else, on_array_begin() for an integer array and nothing else, so the decoder has already matched the tag for them. Here it has not, and an id the schema bounds can arrive under a tag the schema never declared for it. §7.3 says such a field is skipped like an unknown id, so a handler must answer -1 for a tag it did not declare — a bound answered for someone else’s field is a bound applied to a length that was never the handler’s, and the INVALID that follows contradicts §7.3. A table entry (destinations()) gets the same test run for it by the decoder, which is why the two routes agree.

subtype is the fixlen subtype for a string, a blob and a fixlen array, and None for an integer array, which carries none. Both are enum members recovered by index, not built, so overriding this hook still costs no allocation per field.

Returning n >= 0:

a wire count/length above it is INVALID — sofab.SofaDecodeError (MESSAGE_SPEC §7.1): the message contradicts the schema;
the receiver-side max_dyn_* cap stops applying to the field (§6.2.1) — a schema bound is a statement about validity, a receiver limit about capacity.

Returning -1 (the default) leaves the field to the receiver caps, unchanged.

A handler that declares destinations (destinations()) answers this from its table for every field the table names; this hook is what the rest go through, and both reach the same rule in the same place.

def on_schema_bound(self, field_id, n, wtype, subtype):
    if (field_id == 0 and wtype is WireType.FIXLEN
            and subtype is FixlenSubtype.STRING):
        return 32          # the schema's maxlen for this field
    return -1              # not the field the schema declared

Parameters:

field_id (int)
n (int)
wtype (WireType)
subtype (FixlenSubtype | None)

Return type:

int

on_sequence_begin(field_id)[source]¶

A nested sequence is opening; nothing inside it has been decoded.

Three answers:

False: skip the whole sub-tree — its end marker is consumed and on_sequence_end() is not called.
another Visitor: descend into it: every field of that sub-tree goes to the visitor returned, its on_sequence_end() fires when the scope closes, and this visitor resumes afterwards. That is how a hand-written object hands a nested message to the object that models it. Generated code is flat and does not descend; it grows a wrapper array’s list with the helpers in sofab.collectors.
anything else: decode the sub-tree into this same visitor, as a flat event stream.

A sub-tree opens a fresh id scope (§4.9), so the ids inside it mean what the nested schema says, not what the enclosing one does.

Parameters:: field_id (int)
Return type:: bool | TypeAliasForwardRef(‘sofab.Visitor’) | None

on_sequence_end()[source]¶

The current nested sequence closed.

Return type:: None

on_array_begin(field_id, wtype, count)[source]¶

An integer array’s header has been read; no element has been decoded.

This is the only place a handler can say anything about the array’s elements, because the typed hook below receives them already decoded. Return None to take the default — a list, handed to on_unsigned_array() / on_signed_array() — or a (dst, elem_min, elem_max) triple:

dst: Somewhere to put the elements, or None to keep the list. A writable buffer of at least count slots: an array of the right typecode, a memoryview over one, or any object supporting the buffer protocol. The decoder writes into it and does not call the typed hook — the handler already has the values where it wanted them, and none of them was ever a Python object. A buffer too short is sofab.SofaArgumentError; the decoder never grows one (CORELIB_PLAN §6.6).
elem_min / elem_max: The element width the schema declares, or None for an open side. The decoder applies it at each element, so a value outside it is INVALID whether the array completes or is truncated behind it (§7.1), which is also §5.2’s INVALID-over-INCOMPLETE for free. A handler cannot do this itself: by the time it holds the list, an array that never arrived is indistinguishable from one that did.

Called again for the same array if a chunk boundary suspends the read, so return the same answer each time; the decoder restarts the array from its first element and fills dst from the beginning.

Not called for float arrays, which carry no declared width to state. Their destination hook is on_float_array_begin().

A configured max_dyn_array_count does not gate this hook, on the same reasoning as on_blob_begin(): it is asked first and told count, and a dst it hands back is storage it sized itself. The cap governs the list the decoder would otherwise build — the None answer — not a destination of the handler’s own.

Parameters:

field_id (int)
wtype (WireType)
count (int)

Return type:

tuple[Any, int | None, int | None] | None

on_blob_begin(field_id, size)[source]¶

A blob’s length has been read; no payload byte has been copied yet.

Return None to take the default — a bytes, handed to on_bytes(). Return a writable, contiguous buffer of at least size bytes and the decoder copies the payload straight into it and does not call on_bytes(). One too short is sofab.SofaArgumentError; the decoder never grows one (CORELIB_PLAN §6.6), and the refusal comes at the length word, before a byte is written.

This is §6.6.3’s second shape for an aggregate: a callback carrying a whole blob obliges the codec to build one, and the only size available to build it from is the wire’s. A megabyte blob costs a megabyte allocation per message that way; into a destination it costs none.

Called again for the same blob if a chunk boundary suspends the copy, so return the same answer each time; the decoder restarts the payload from its first byte.

The string twin is on_string_begin().

A configured max_dyn_blob_len does not gate this hook. It is asked first, and asked whatever the announced size is. The limit is there to stop the sender dictating the receiver’s allocation (§6.2.1), and a handler that hands back a buffer has sized that buffer itself — there is no allocation of the decoder’s left to prevent. It is told size before a byte is copied precisely so that a receiver unwilling to take that many can refuse it here, which is its call to make. Return None and the cap applies again, because then the bytes is the decoder’s to build and the wire is its only size.

Parameters:

field_id (int)
size (int)

Return type:

Any

on_string_begin(field_id, size)[source]¶

A string’s byte length has been read; no payload byte has been copied yet, and none has been validated.

Return None to take the default — a str, handed to on_string(). Return a writable, contiguous buffer of at least size bytes and the decoder validates the payload as UTF-8, copies the wire bytes straight into it, and does not call on_string(). One too short is sofab.SofaArgumentError; the decoder never grows one (CORELIB_PLAN §6.6), and the refusal comes at the length word, before a byte is written.

size is the wire byte length, which is what a schema maxlen bounds (MESSAGE_SPEC §1) — not a character count. What lands in the buffer is UTF-8, so a target that wants Python text decodes it itself; what this saves is the str the decoder would otherwise have had to build, sized by the wire.

This is §6.6.3’s second shape for the third aggregate, and it is the sharpest of the three: with a caller reassembly= buffer and a Binding, a 1 MiB string still cost a 1 MiB allocation inside the codec, because there was no third opt-out to take.

The payload is still validated (§6.7.2: a field the handler reads is materialized and validated). Validation walks the bytes — §6.4.3’s utf8_valid primitive — so nothing the wire sizes is built to check them. Invalid UTF-8 is INVALID and the destination is left untouched.

Called again for the same string if a chunk boundary suspends the copy, so return the same answer each time; the decoder restarts the payload from its first byte.

A configured max_dyn_string_len does not gate this hook, on the same reasoning as on_blob_begin().

Parameters:

field_id (int)
size (int)

Return type:

Any

on_float_array_begin(field_id, subtype, count)[source]¶

A fixlen (fp32/fp64) array’s count has been read; no element has been decoded.

Return None to take the default — a list, handed to on_float32_array() / on_float64_array() — or a writable buffer of at least count 8-byte slots (an array("d"), a memoryview over one, a NumPy float64 array). The decoder widens each element into it and does not call the typed hook. A buffer too short is sofab.SofaArgumentError; the decoder never grows one (CORELIB_PLAN §6.6).

subtype is sofab.FixlenSubtype.FP32 or FP64, so one hook serves both and a handler that only wants one returns None for the other.

Slots are 8 bytes for both subtypes because a Python float is a double and that is what the values become. A consumer that needs an fp32’s wire bits intact takes on_float32_array_bits() instead (§6.5).

Called again for the same array if a chunk boundary suspends the read, so return the same answer each time.

A configured max_dyn_array_count does not gate this hook, on the same reasoning as on_blob_begin().

Parameters:

field_id (int)
subtype (FixlenSubtype)
count (int)

Return type:

Any

on_unsigned(field_id, value)[source]¶

Handle a decoded unsigned-integer field.

Parameters:

field_id (int)
value (int)

Return type:

None

on_signed(field_id, value)[source]¶

Handle a decoded signed-integer field.

Parameters:

field_id (int)
value (int)

Return type:

None

on_float32(field_id, value)[source]¶

Handle a decoded 32-bit float field.

The value is a Python float — a C double — because Python has no other float. CORELIB_PLAN §6.5 permits that for a value consumer; a consumer that has to reproduce the wire bytes takes on_float32_bits() instead.

Parameters:

field_id (int)
value (float)

Return type:

None

on_float32_bits(field_id, bits)[source]¶

The raw wire bits of a 32-bit float field, as an int.

Override this and the decoder calls it instead of on_float32() for every scalar fp32. bits is the little-endian payload read as an unsigned 32-bit integer, exactly as it lay on the wire, and sofab.Encoder.write_float32_bits() puts it back verbatim.

This is CORELIB_PLAN §6.5’s required channel for a double-only target, which Python is. IEEE widening fp32 to a double sets the quiet bit, so a signaling NaN’s payload is destroyed the instant the value passes through the wider float — and no later code can recover it. A port on such a target therefore “MUST provide a raw-wire-bytes path for bit-exact consumers (transcode, round-trip, any re-encode) that re-emits those bytes verbatim” and “MUST NOT re-encode an fp32 from the widened value”.

(This port also preserves an sNaN through the widened float, by doing the conversion on the bit pattern by hand rather than letting the hardware quiet it — so on_float32() is bit-exact here today. That is a property of this implementation on this platform; the raw channel is the guarantee.)

The array twin is on_float32_array_bits().

Parameters:

field_id (int)
bits (int)

Return type:

None

on_float64(field_id, value)[source]¶

Handle a decoded 64-bit float field.

Parameters:

field_id (int)
value (float)

Return type:

None

on_string(field_id, value)[source]¶

Handle a decoded UTF-8 string field.

Parameters:

field_id (int)
value (str)

Return type:

None

on_bytes(field_id, value)[source]¶

Handle a decoded raw byte-blob field.

Parameters:

field_id (int)
value (bytes)

Return type:

None

on_unsigned_array(field_id, values)[source]¶

Handle a decoded unsigned-integer array field.

Parameters:

field_id (int)
values (list[int])

Return type:

None

on_signed_array(field_id, values)[source]¶

Handle a decoded signed-integer array field.

Parameters:

field_id (int)
values (list[int])

Return type:

None

on_float32_array(field_id, values)[source]¶

Handle a decoded 32-bit float array field.

Parameters:

field_id (int)
values (list[float])

Return type:

None

on_float32_array_bits(field_id, count, payload)[source]¶

The raw wire bytes of a 32-bit float array, undecoded.

Override this and the decoder calls it instead of on_float32_array(). payload is a read-only memoryview of exactly 4 * count little-endian bytes — the array’s payload as it lay on the wire — and sofab.Encoder.write_float32_array_bits() puts it back verbatim.

§6.5’s requirement is stated over “every fp32 position — a scalar fp32 (§4.6) and each element of an fp32 array (§4.8)”, so the scalar channel alone would not meet it.

The bytes do not outlive the call. They are the caller’s own input, borrowed for the duration of this callback exactly as a fed chunk is (§6, chunk lifetime), and a handler that still needs them afterwards copies them. That is §6.7’s second route — “the codec passes the value through the callback … and the caller copies it. The second route is not a view” — and it is why nothing is allocated to deliver an array of any length.

Parameters:

field_id (int)
count (int)
payload (Any)

Return type:

None

on_float64_array(field_id, values)[source]¶

Handle a decoded 64-bit float array field.

Parameters:

field_id (int)
values (list[float])

Return type:

None

class sofab.Binding(closed=False)[source]¶

Bases: object

Where each field id’s value belongs. Build once, decode many times.

Every binder method returns self, so a table reads as one statement. at is a slot index — into words for the numeric kinds, into objects for string(), bytes(), string_into() and blob_into(). count_at is an optional words slot the decoder writes the field’s arrival into: 1 for a scalar that turned up, the element count for an array, the byte length for a string_into() / blob_into() payload, the number of occurrences for a sequence. Slots the decoder never writes are left exactly as the caller prepared them, which is how a decode reports absence without inventing a sentinel.

An array’s count_at is written when the array’s count header is read, not when its last element lands — that is where the destination is settled (§6.6.3). On a decode that completes the value is the same either way; a decode that ends INCOMPLETE or INVALID inside an array may already have written it.

closed decides what happens to an id this table does not name. Open (the default), it goes to the decoder’s visitor, as it would without a table. Closed, it is skipped exactly as a decoder with no visitor skips it — no hook, nothing materialized, no cap spent, decode stays COMPLETE — and a nested sequence the table does not name is skipped whole.

That is what a child table wants. The decoder descends into a bound sequence without telling the visitor, so the visitor still believes the walk is in the parent’s scope, and an id the child does not name would reach it under the parent’s identity: an unknown field a newer sender added inside a struct would land in whichever parent field shares its id. A closed child cannot hand one over. The flag belongs to the table, so a child bound from two places behaves the same in both.

Parameters:: closed (bool)

property entries: tuple[Entry, ...]¶: The rows, in the order they were bound. The engines compile this.

property closed: bool¶: Whether an id this table does not name is skipped rather than handed to the visitor; see Binding.

property words_required: int¶: Slots the words buffer must hold — i.e. it must be at least words_required * 8 bytes. Counts this table only; a child sequence() binding shares the same buffer, so take the maximum over the whole tree (or give every table disjoint slots, which is what generated code does).

property objects_required: int¶: Entries the objects list must hold.

property tree_words_required: int¶: words_required over this table and every table reachable through sequence(). A child shares the parent’s storage, so this is the size the one buffer has to have.

property tree_objects_required: int¶: objects_required over the whole tree; see tree_words_required.

freeze()[source]¶

Close the table — this one and every child — and return the whole reachable set.

A binding is a build-once artifact: a sofab.Decoder derives its storage requirements and one handler per table from it and holds them, so a table that changed afterwards would decode against a stale copy. Freezing at first use makes that a clear error instead. Called for you — building the decoder is what freezes the table — and idempotent, so calling it yourself is harmless.

It is deliberately the whole tree: a child bound into a parent is reachable only downwards, so freezing the root is the only moment at which every table in it can be reached at once.

Return type:: list[Binding]

unsigned(field_id, at, count_at=None, max_value=None)[source]¶

Bind an unsigned-integer field to words slot at (uint64).

max_value is the schema’s declared width (0xFF for a u8, or for a bitfield whose highest pos is 7). The slot is 64 bits wide whatever the field declares, so nothing about the storage enforces a narrower width, and MESSAGE_SPEC §1 then requires an explicit check: given, a value above it is INVALID at the value, before it is stored — so a message truncated behind it is INVALID, not INCOMPLETE (§5.2).

Parameters:

field_id (int)
at (int)
count_at (int | None)
max_value (int | None)

Return type: