Skip to content

serialization.md

Latest commit

 

History

History
104 lines (72 loc) · 2.99 KB

serialization.md

File metadata and controls

104 lines (72 loc) · 2.99 KB
Raw

Serialization

Arrow serialization layer for automatic schema generation and IPC validation.

ArrowSerializableDataclass

Use ArrowSerializableDataclass as a mixin for dataclasses that need Arrow serialization. The Arrow schema is generated automatically from field annotations:

from dataclasses import dataclass
from typing import Annotated

import pyarrow as pa

from vgi_rpc import ArrowSerializableDataclass, ArrowType


@dataclass(frozen=True)
class Measurement(ArrowSerializableDataclass):
    timestamp: str
    value: float
    count: Annotated[int, ArrowType(pa.int32())]  # explicit Arrow type override

# Auto-generated schema
print(Measurement.ARROW_SCHEMA)
# timestamp: string
# value: double
# count: int32

# Serialize / deserialize
m = Measurement(timestamp="2024-01-01T00:00:00Z", value=42.0, count=7)
data = m.serialize_to_bytes()
m2 = Measurement.deserialize_from_bytes(data)
assert m == m2

These dataclasses work directly as RPC parameters and return types. They're also the base class for StreamState — any field you add to a stream state is automatically serialized between calls.

Type mappings

Python type Arrow type
str utf8
bytes binary
int int64
float float64
bool bool_
list[T] list_<T>
dict[K, V] map_<K, V>
frozenset[T] list_<T>
Enum dictionary(int32, utf8)
Optional[T] nullable T
nested ArrowSerializableDataclass struct
Annotated[T, ArrowType(...)] explicit type override
Annotated[T | None, ArrowType(...)] explicit type override, still nullable
Annotated[T, ArrowType(..., nullable=...)] explicit type AND nullability

Nullability

Optional[T] is a nullable column whether or not it is wrapped in Annotated. Before 0.43.0 the wrapper hid it — get_origin(Annotated[T | None, ...]) is Annotated, never a union — so every annotated optional field was described as non-null while its values were free to be null. Nothing in one SDK notices that, because the same schema describes both the write and the read; a peer comparing schemas field by field rejects the batch.

Set nullable only where the wire genuinely disagrees with the Python type:

# Optional in Python because it has a default, never null on the wire.
cursor: Annotated[bytes | None, ArrowType(pa.binary(), nullable=False)] = None

# Not Optional in Python, but a peer may omit the value.
tag: Annotated[str, ArrowType(pa.string(), nullable=True)]

An explicit value is a claim about the bytes, so make it only where the serializer backs it up. Deriving from the annotation is right almost always.

API Reference

ArrowSerializableDataclass

::: vgi_rpc.utils.ArrowSerializableDataclass

ArrowType

::: vgi_rpc.utils.ArrowType

IpcValidation

::: vgi_rpc.utils.IpcValidation

ValidatedReader

::: vgi_rpc.utils.ValidatedReader

Validation

::: vgi_rpc.utils.validate_batch

Errors

::: vgi_rpc.utils.IPCError