The contract for native mode: DOOM's tic simulation and renderer, written as
ClickHouse SQL, run against the shareware doom1.wad. SPEC.md is the
contract for emulation mode and does not apply here; PURITY.md says which of
its rules apply to which mode.
Native mode is correct when it agrees with the real engine. The reference emulator runs the real DOOM binary and reports the engine's game state at every tic and its framebuffer at every frame. Native mode has to produce the same state row for every tic and the same 64,000-byte frame for every frame.
The driver inserts the WAD as raw lumps: one row per lump with its index, name,
enclosing map marker and bytes. Everything derived from a lump is derived in
SQL: map geometry decoded from the fixed-width records, texture composition
from PNAMES and TEXTURE1, sprite frames, flats, colormaps, the blockmap
cell lists, the BSP ancestor paths, and the level's initial state.
The engine's own constant tables (states, mobjinfo, sprnames,
weaponinfo, finesine, finetangent, tantoangle, rndtable,
gammatable, fuzzoffset, checkcoord, the sound ids, the animation and
switch name lists) are program data. They are generated from the vendored
engine source under rom/vendor/doomgeneric/ by
native/src/bin/gen_tables.rs into native/tables/*.tsv, and a test fails
if regeneration differs from what is committed. A table that cannot be
traced to the vendored source this way does not belong in the tree.
One row per tic in native_state, keyed by the tic number. The field list and
its order are spec/src/native_state.rs; the reference emulator's probe writes
the same fields. Every fixed_t is Int32, angles are UInt32, enums are
their C values. Mobjs and sector thinkers are parallel array columns indexed by
slot in thinker-list order. A thinker's identity is the value of a global
counter taken when it was added; pointers between thinkers hold that identity,
0 for none. The probe cannot read identities out of RAM and writes 0 for them,
so parity compares by slot and ignores the identity columns. setup_things
says how many mobj slots the thinker list holds ahead of the sector thinkers,
and the probe reads that off its own walk, so parity compares it. A message the
player or the heads-up display holds is stored as the xxh64 of its text, which
is what the probe can compute from a C string.
One input row (tic, source, keys, mouse_dx, mouse_dy) produces the state row
for tic from the state row for tic - 1. source 0 takes the tic command
from the demo lump; source 1 builds it from the key bits in
spec::native_state::key and the mouse deltas, as G_BuildTiccmd does. The
tic runs P_PlayerThink, the thinker list in creation order with the same
random-number draws as the engine, P_UpdateSpecials, then the status bar,
heads-up display and menu tickers. P_SpawnSpecials adds the sector thinkers
after P_LoadThings and P_AddThinker appends, so the sector thinkers run
after the first setup_things mobj slots and before anything spawned during
play. The compaction takes that count down for each thing it drops at or below
it. Nothing takes it up.
One input row (frame, tic, melt passes) produces the frame for frame from
the state row for tic and the previous frame. The melt's pass count per frame
is what the engine's clock gave it, read off the reference run and loaded as
data with its provenance; SQL turns the per-frame passes into the melt's
running step. The frame is 320×200 8bpp bytes,
row-major, plus the palette index chosen by the status bar, an RGB rendering of
the two, and the frame hash xxHash64(framebuffer || palette) defined by
spec::fb_hash. The framebuffer persists between frames, as it does in the
engine: pixels the renderer does not draw keep their previous value.
The simulation opens as two residents chained through native_stage, and the
renderer opens as a third. The first carries the player and the thinkers: it
reads the state row for tic - 1 and writes one row per tic into
native_stage, keyed by tic the same way native_state is. The second
carries the specials and G_Ticker: it reads that same tic's own row back out
of native_stage and writes native_state. native_stage holds every column
native_state holds, in the same order, plus px_crossed_line,
tx_crossed_line and mt_light_index: the player's own crossed line, the
moved things' own, and where the sector thinkers start reading the random
table. The first resident computes them, the second reads them, and neither
table's own contract otherwise carries them. A Join engine table refuses
ALTER TABLE ... ADD COLUMN, so the two column lists are
declared separately in schema.sql, and a change to one is a change to both.
Each resident is one long-lived INSERT INTO ... SELECT ... FROM input(...)
over a chunked HTTP body, analysed once per session. The statement text leads
the body, terminated by a newline, because a URL parameter is limited to about
64 KB and these statements are larger; any WITH clause sits after INSERT INTO ... and before SELECT. Settings travel as URL parameters:
max_insert_block_size = 1, min_insert_block_size_rows = 1,
min_insert_block_size_bytes = 1, input_format_parallel_parsing = 0,
max_block_size = 1, max_threads = 1, max_insert_threads = 1,
async_insert = 0, optimize_and_compare_chain = 0, and max_query_size
set to the statement's byte length plus 64. The server reads that many bytes
before it parses, so the first row
after the statement is padding, tic = 0, at least 128 bytes, and is filtered
out. A statement error surfaces on the response only after the body closes,
so the driver reads the response concurrently and treats an early response as
failure. A statement that has already failed keeps accepting rows and commits
none, so the driver detects death by rows that stop landing.
The driver feeds the row for tic t to the first simulation resident, waits for
native_stage to hold a row for t, feeds a row carrying t alone to the
second, and waits for native_state to hold t before feeding t+1 to the
first. All three residents open at once, so a session's first tic pays for the
largest of their analyses rather than the sum. A resident that ends is
reopened, and the session resumes both simulation statements from the same
tic: a native_stage row a resumed run cannot show was ever read by the
second statement is not one to trust. Restarting the simulation on a database
it has already run against empties native_stage alongside native_state; a
staged row left behind would let the driver's own presence check for that tic
find it before the new run's own first statement has written it.
Static data enters a statement as scalar constants evaluated once. A constant
array is held as one value per element, and in a statement dozens of
subqueries deep each element costs kilobytes, so the pixel pools and every
other large constant are Strings indexed with substring; a string constant
is one value whatever its length. A per-frame array captured inside a lambda is
copied once per element of the array being mapped, so per-pixel lookups go to
constants only and per-frame data is consumed element-wise.
A statement costs what it holds rather than what a tic asks of it. Every node
is evaluated for every row, both arms of an if included, and a lambda's body
is evaluated even where the array it maps over is empty. arrayFold is the
one exception: it runs its body once per element and not at all for none, so
a stage a tic has no work for is written as the body of a fold over what it
has to do. A lambda body that reads neither of its parameters is evaluated
outside the lambda whatever the fold does, so such a body has to lead back to
one of them.
Analysis is paid once per session. On ClickHouse 26.8.2.7 the analyser's
logical expression pass hashes the other side of every comparison with a
constant that is a direct operand of and, and of every equals with a
constant that is a direct operand of or. It hashes each one afresh, and the
hash of a column walks the whole subquery the column comes from. The stages
are nested subqueries, so each such hash walks most of the statement. The
generator wraps every comparison that is an operand of AND or OR in
identity(), which returns its argument and which the pass does not look
inside (native/src/sql/sim/chains.rs). On a development machine the first
statement analyses in about 1.4 s wrapped and 44 s unwrapped.
One shape is measured on ClickHouse 26.7.5.10: writing a thirty-column
compaction as arrayFilter((v, a) -> a = 1, X, mt_kept) costs 5.2 s more than
writing it
as the surviving places worked out once and each column read through them,
and the same 5.2 s more than one arrayZip of every column filtered once.
arrayZip(a, b) costs what arrayFilter((v, k) -> …, a, b) costs, so
whatever this is, it is not the lambda.
The same rewrite applied to the rest of mobj.rs saves nothing. Sixty-six
calls over two or more arrays were taken out of the statement two ways, by
indexing and by zipping a row per stage, and neither moved the analysis;
indexing added 9 s by repeating the array text. So the 5.2 s belongs to that
compaction and not to a count of call sites, and a rewrite of this kind is
worth doing only against a measurement of the piece in hand.
arrayConcat over several arrays costs about 20 ms a site, which is small
enough that list growth is written the plain way. Every piece added to a
statement is measured with QueryAnalysisMicroseconds from
system.query_log.
A body that has to be one binding, such as a fold whose steps each read what
the step before wrote, is kept to what the dependency forces inside it. The
driver's budget for the first tic (FIRST_TIC_TIMEOUT in
driver/src/native/session.rs) is sized for this analysis on a CI runner,
which is about four times slower than a development machine.
Rows reach the statement one block each. Rows written into a statement from a
VALUES list share one block whatever max_insert_block_size says, so every
row of that block reads the state from before the block; tests that drive a
statement without the driver use numbers() or one insert per row.
The reference emulator's probe writes one row per frame in the shape of
section 3, with the frame index and the engine's gametic. A parity run loads
those rows and reports the first tic at which any field differs, with the
field, both values and the thinker involved, then the first frame whose hash
differs. Two hashes pin demo3: frame 220 is aa27f0470c7c5f3a and the final
frame is d303721d8116e877.
One field is partly outside the comparison. T_VerticalDoor never reads a
door's topcountdown while the door goes up, and the engine leaves it as the
zone allocator returned it until the door reaches the top and writes it. So
s_count for a door thinker whose direction is up is left out of the
comparison, and every other element of every other field is compared.
A tic the statement could not produce exactly is not compared at all.
native_state.unresolved says so for one tic, decided fresh each tic, with
its bits named in sim::unresolved; native_state.unimplemented says the
level itself carries a path native mode does not model, decided once when
the level loads, with its bits named in sim::unimplemented. native diff
reads both columns for every tic it ran and stops at the first tic either
sets, before any field is compared, with exit 3 and a message naming the tic
and the bits. native demo and native play stop at the same tic rather
than drawing past it.
No SQL path in native mode reads a clock, a random function or the host environment. Pacing to 35 Hz happens in the driver and never changes a computed value.