feat: produce and transport resolved symbol tables from Python

[leongdl]

Two commits, both in service of one use case: letting a scheduler produce a
resolved symbol table and hand it to a worker on another host — the flow
SerializedSymbolTable's own docstring describes as "a scheduler serializes the
symbol table to JSON and sends it to a worker that may be on a different OS".

Today neither half of that is reachable from Python: there is no supported way to
get the transport text out, and create_job discards the tables it builds.

Commit	Adds
feat: add SerializedSymbolTable JSON accessors	to_json_str() / from_json_str()
feat: return resolved symbol tables from create_job	create_job_with_symbol_tables() → JobWithSymbolTables

Happy to split these into two PRs if you'd prefer to review them separately.

---

1. JSON accessors

SerializedSymbolTable exposes from_symtab, to_symtab and __reduce__. To
move the transport text across a boundary today, a caller has to reach into the
pickle protocol:

_helper, (json_text,) = serialized.__reduce__()   # not a contract

or call the private module-level pickle helper to get back in:

from openjd._openjd_rs import _reconstruct_serialized_symtab   # underscore

The Rust side already has the capability (from_json_str, and serde_json via the
type's Serialize impl). This exposes it.

Design notes

• from_json_str validates syntax, not contents. Malformed JSON raises
  ValueError; a well-formed document that is not a valid table is accepted and
  rejected later by to_symtab. That matches the existing split — to_symtab is
  where entry validation lives — and keeps from_json_str cheap. Tests pin both
  halves so the behaviour is deliberate.
• __reduce__ is unchanged, still routing through
  _reconstruct_serialized_symtab, so existing pickles keep working. Happy to
  re-point it and drop the private helper if you prefer, but that changes what old
  pickled bytes reference.
• Output is stable. SymbolTable's Serialize impl emits entries in canonical
  path order, which the type already relies on for Eq/Hash, so the text is safe
  to store and compare.

2. Returning the resolved symbol tables

create_job builds a job-scope symbol table, and instantiate_model extends it
per step with Step.Name and the step's template-scope let bindings
(StepTemplate._extend_step_symtab). Both are then discarded. openjd-rs keeps the
equivalent table on each Step as resolved_symtab; the Python model has no
equivalent, so a Python scheduler cannot produce what a Rust session consumes.

result = create_job_with_symbol_tables(
    job_template=job_template, job_parameter_values=values
)
result.job                                  # same Job create_job returns
result.job_symbol_table                     # Param.*, RawParam.*, Job.Name
result.step_symbol_tables["render"]         # + Step.Name, + template-scope let

Design notes

• Returned, not attached to the model. Adding a field to the pydantic Step
  would change the model's serialized form, and Step.dict() output is persisted
  as a worker contract by at least one consumer. A separate return value avoids
  that entirely.
• create_job is untouched behaviourally. Its body moved to
  _create_job_and_symbol_table, which both entry points call. A test asserts the
  two produce equal jobs.
• The per-step tables come from the model's own hook. Rather than
  reimplementing the step scope, this re-invokes
  step_template._job_creation_metadata.extends_symtab — the same callable
  instantiate_model uses. It depends only on the step's name, its let bindings
  and the job-scope table, so re-invoking it is deterministic and cannot drift from
  what the job was instantiated with.
• Serialization goes through the engine. symtab_to_expr_values for the typed
  coercion, then SerializedSymbolTable.from_symtab — the same serializer
  openjd-rs uses.
• Imports stay lazy. openjd.expr is imported inside the helper and under
  TYPE_CHECKING, so importing openjd.model still does not load the Rust
  bindings. Your test_importing_parser_does_not_load_rust_expr_surface caught my
  first attempt at this — thanks, it's a good test.
• Not filtered. openjd-rs narrows each step's table to the symbols that step
  references. This returns the full scope, which is a valid superset: a session
  layers its own scopes on top either way. Filtering would be a payload
  optimization and I did not want to reimplement filter_symtab_for_step here.

Parity with openjd-rs

The point of this is that a v0 producer can feed a Rust consumer, so I measured
that rather than assuming it. 18 templates, 21 step comparisons, diffed entry by
entry on name, type and value against openjd-rs' Step.resolved_symtab:

Zero missing symbols, zero semantic mismatches. Cases covered: scalar types,
BOOL, LIST[INT], LIST[STRING], LIST[PATH], RANGE_EXPR, PATH
(RawParam-only convention), template-scope let (frozen), script-scope let
(inputs only, results absent), parameterSpace ranges from a let,
hostRequirements from a let, CHUNK[INT], step environments, multi-step
scoping, explicit parameter values, negative/zero/empty values, and the EXPR
function library.

RANGE_EXPR was the case I expected to break, since symtab_to_expr_values
omits it from types so the engine infers it from the value. Both sides produce
('range_expr', '1-9:2'), including the same 1-10:2 → 1-9:2 normalization.

test_matches_the_rust_implementation encodes this as a regression test.

One cosmetic difference worth flagging. For an integral-valued float written
1.0 or 1.00, openjd-rs emits "1" and this Python path emits "1.0". Both
decode to the same ExprValue, so it is a transport-text difference rather than a
value one — confirmed by round-tripping each through to_symtab and comparing.

It still has a consequence. SerializedSymbolTable derives Eq and Hash
structurally over the JSON, so anything comparing two serialized tables byte-wise
sees a false mismatch on integral floats. Callers should compare decoded
SymbolTables instead. Normalizing float display between the two implementations
would close it at the source, but that changes transport output, which felt out of
scope here.

Testing

hatch run test — 5478 passed, 24 skipped, 3 xfailed. hatch run lint clean
(ruff, black, mypy). cargo fmt --check and cargo check clean.

Total coverage moves from 93.91% to 93.98%; every line added is covered. The 94%
gate was already short before this change.

17 new tests: 9 for the JSON accessors in
test/openjd/expr/test_symbol_table.py, 8 for the symbol tables in
test/openjd/model_v0/test_create_job.py.

Two things to flag

scripts/generate_stubs.sh does not run on current mainline for me, so I
hand-wrote the two stub entries in _openjd_rs.pyi to match the generator's style
and verified with mypy. The script clones pyo3-stub-gen at --depth 1 with no
pinned revision, and today's tip no longer compiles against this repo's pyo3:

error[E0599]: no method named `downcast` found ... help: there is a method `cast`
error: could not compile `pyo3-stub-gen` (lib) due to 8 previous errors

Pinning that clone to a known-good tag would make stub regeneration reproducible.
Happy to send it separately.

hatch run test fails if CONDA_PREFIX is set alongside VIRTUAL_ENV
(maturin failed: Both VIRTUAL_ENV and CONDA_PREFIX are set). Not caused by this
change, and easy to work around with env -u CONDA_PREFIX, but it bites anyone
developing from a conda base environment.

[github-actions]

The two guarantees documented here — an array of name/type/value objects "in canonical (lexicographic) path order", and "stable for a given table, so it is safe to store or compare" — only hold for instances built via from_symtab.

from_json_str is explicitly lazy (per its own docstring, and confirmed by test_from_json_str_defers_content_validation_to_to_symtab, which stores a non-array document successfully). So inner can hold arbitrary well-formed JSON, and to_json_str re-emits it as-is. Consequences:

• to_json_str() on a from_json_str-derived instance is not necessarily an array, and not necessarily in canonical path order.
• Byte-comparing two to_json_str() outputs is not equivalent to comparing table contents: a hand-built transport string with entries in non-lexicographic order, or with duplicate names, survives the round trip and compares unequal to the from_symtab form of the same table.

Since the docstring actively invites callers to store and compare the text, consider scoping the claim (e.g. canonical order and byte-stability hold for tables built via from_symtab; from_json_str preserves the input text as given), or normalizing on ingest in from_json_str so the invariant is unconditional.

[github-actions]

This test does not exercise PATH-typed values, so it cannot catch a regression in path serialization.

SymbolTable({"RawParam.Scene": "/proj/scene.blend"}) goes through py_to_expr_value with no target type, which infers the native type — i.e. a plain STRING, not PATH. path_format is then irrelevant to the round trip. The existing test at line 43 shows the way to actually build a PATH value:

ExprValue(str(input_file), type="path", path_format=HOST_PATH_FORMAT)

Separately, the only assertion is membership ("RawParam.Scene" in restored), which holds for any surviving key. Asserting the restored value and its type code (e.g. restored["RawParam.Scene"].type == TypeCode.PATH plus the expected host-format string) would make the test meaningful — and would be the one place in this suite verifying that type: "path" survives the JSON transport with the requested path_format applied on the way back.

[github-actions]

This asserts determinism (same input, same output twice in one process) but not the canonical-ordering property the docstring actually promises. Both from_symtab calls receive the identical dict, so the test would still pass if the transport order tracked insertion order rather than being sorted.

To pin down "canonical (lexicographic) path order" — and catch nondeterminism from the underlying HashMap iteration order, which is the real risk here — compare two tables built with opposite insertion orders:

first = SerializedSymbolTable.from_symtab(SymbolTable({"Param.B": 2, "Param.A": 1})).to_json_str()
second = SerializedSymbolTable.from_symtab(SymbolTable({"Param.A": 1, "Param.B": 2})).to_json_str()
assert first == second

(Python dict literals preserve insertion order, so these two do reach the Rust side in different orders.)

[github-actions]

This test is circular and cannot fail. __reduce__ (symbol_table.rs:342) serializes with the same serde_json::to_string(&self.inner) that to_json_str uses, and _reconstruct_serialized_symtab calls the same SerializedSymbolTable::from_json_str. So both sides of the comparison are literally the same code path — as the docstring itself notes ("round-trips through the same transport text"). It adds test surface without adding coverage.

If the intent is to guard against pickle and JSON diverging in the future, comparing the two text forms directly would be a tighter check:

assert pickle.loads(pickle.dumps(serialized)).to_json_str() == serialized.to_json_str()

Otherwise dropping it leaves nothing untested — test_round_trip_preserves_values already covers JSON content fidelity.

[github-actions]

Design note on the lazy-validation choice: for the use case the docstring names — a scheduler persisting the transport text and later handing it to a worker — deferring content validation to to_symtab pushes the error to the worst possible place. The scheduler happily accepts and stores a bad payload, and the failure surfaces later on the worker, far from the input that caused it.

Validating structure at ingest (i.e. parsing into the real entry type rather than a loose JSON document) would make from_json_str a real trust boundary and give the error at the point where the caller still has the offending text in hand. If the laziness is deliberate — e.g. openjd_expr::SerializedSymbolTable::from_json_str intentionally keeps the raw document so unknown future entry kinds pass through untouched — it would help to say so in the docstring, since as written it reads as an accident rather than forward-compatibility.

[seant-aws]

PR #332 Review: create_job_with_symbol_tables + JSON accessors

Verdict: Approve ✅ — good to merge.

---

Executive Summary

Clean, well-designed PR that adds two features we directly consume:

1. SerializedSymbolTable.from_json_str() / .to_json_str() — the public JSON transport API (replaces the private _reconstruct_serialized_symtab we're currently using in #1063)
2. create_job_with_symbol_tables() — DataPlane-side producer returning resolved tables per-step

Architecture is sound, 17 tests cover the critical paths, and the PR was verified against openjd-rs with zero semantic mismatches across 21 step comparisons.

---

No Critical Issues

The reviewer bot flagged the to_json_str Rust method as potentially missing a return — this is a false positive. The actual code uses a tail expression (serde_json::to_string(&self.inner).map_err(...)) with no semicolons, which is valid Rust implicit return of PyResult<String>.

---

Important Issues (should discuss, not blocking)

#	Issue	Severity	Assessment
1	Deferred validation in from_json_str — validates JSON syntax only, pushes content errors to to_symtab() call time	Important	This is fine for our consumption path because we call to_symtab() immediately after deserializing. But worth a docstring clarifying that callers should validate early.
2	Bot comment #2 — stability claim scope	Minor	The "canonical order" guarantee holds for from_symtab-built instances but not from_json_str-derived ones. A docstring scoping the claim would help.

[seant-aws]

Also have this as a followup: aws-deadline/deadline-cloud-worker-agent#1063