Skip to content

Heterogeneous batched writes (option C): N arbitrary (pkg, method, args) in one transaction #255

Description

@ohohoreilly

Gated enhancement / follow-up to #253. Design RFC: docs/design/batched-writes.md §2.C.

Summary

Batched writes v1 (#253, merged in PR #254) shipped option B — a batch of N calls of one (pkg, method) against N argument records, folded into a single transaction. Option C is the fully-general form: a batch of N arbitrary, distinct calls — different packages/methods — committed atomically together:

[ graph add_mention <{...}>,
  views increment_view <{...}>,
  ledger post <{...}> ]      // N different methods, one transaction

This is the most powerful shape (atomic cross-method multi-write), but it was deferred from v1 as the most semantics to pin down and the least-requested. File now to track; implement only on demand.

Why it's a clean follow-up, not a redesign

B's server path already generalizes to C. TSession::TryBatch (orly/server/session.cc) opens the pov / builds the TContext / loops func->Call accumulating effects into one TUpdate committed once, with the per-call results aggregated into one list reply. To get C, the loop just resolves a different func per element instead of reusing one — the deferred-entry fold, the single commit, the #234 backpressure, and the array reply are all already in place.

To settle in review before coding

  • Grammar / wire shape. B reuses the try keyword with a bracketed list of arg records (one method named once). C needs each element to carry its own (pkg, method, args) — a distinct statement form (e.g. a bracketed list of pkg method <{...}> triples). Decide the surface and whether it's a new keyword.
  • Per-call resolution. Resolve each element's package + function independently within the one context; confirm a single TIndyContext over one pov is correct for calls into different packages.
  • Mixed commutative / assign ordering. Within one op_by_key, non-commutative assigns/deletes to the same key across different methods collapse in statement order (as in B). Commutative ops still fold. State this explicitly for the cross-method case.
  • Partial-failure policy. v1 (B) is all-or-nothing. Decide whether C wants the same, or a best_effort variant (also deferred from Batched / pipelined writes: fold N method calls into one transaction (~3-5x write throughput) #253) — they're naturally settled together.
  • Meta record shape. B records one method + N arg sets. C records N (method, args) pairs — define the TMetaRecord encoding and confirm one UpdateProgress notification per batch is still acceptable to every consumer.
  • Clients. A heterogeneous call_batch overload taking a list of (pkg, method, args) in python/go/ts.

Out of scope

Any engine/storage change — same constraint as #253: the fold already exists; C only feeds it entries from per-element-varying functions.

Refs: #253 (option B, the v1 this builds on), PR #254 (B implementation), PR #252 (RFC).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions