Add passive movie checkpoints and controlled GCC/LLVM investigation

.github/workflows/build.yml

@@ -53,6 +53,7 @@ jobs:
           python tools/experience/test_boss_catalog.py
           python tools/experience/test_repository.py
           python tools/experience/test_diagnostics.py
+          python tools/experience/test_runtime_checkpoint.py
           python tools/experience/test_normal_lifecycle.py
 
       # The window checks need a display. xvfb is what gives them one; with

docs/diagnostics/MOVIE-REGRESSION.md

@@ -0,0 +1,82 @@
+# Controlled intro-movie investigation
+
+The B01 intro CTD remains unresolved. Boss-disabled installed executables and
+all 512 previous recipes match the published Experience release. The native
+Display build difference is an isolation lead, not a confirmed cause.
+
+`tools/experience/prepare_movie_ab.py` assembles independent local JPRE or Retail
+installations from verified pristine files and the B01 package. Supply the B01
+GCC artifact and LLVM payload as native inputs. Output contains original game
+assets: keep it local and never upload it to GitHub or release artifacts.
+
+The two installations differ only in `ddraw.dll`, `vodisplay.dll`, and the
+integrity/recovery metadata containing their hashes. The assembler rejects
+other differences. No shared mutable directories, hardlinks or DLL swapping
+are used. Launch one installation at a time.
+
+## Development evidence
+
+Diagnostics remain Off by default. F11 → Diagnostics → Development uses the
+existing preference. Local UAT also provides explicit enable/disable commands
+for setting that same preference before the first movie; restart after enabling.
+
+The passive collector writes atomic, flushed checkpoints outside the game
+process. Changes are persisted immediately after observation; unchanged state
+is refreshed at most every five seconds. Windows sampling remains at 500 ms.
+Four bounded files retain current, previous, last completed and last incomplete
+sessions. Completion means the collector observed process exit, not successful
+gameplay. A missing shutdown remains `INCOMPLETE_SESSION`, not proof of a crash.
+
+| Evidence | Classification |
+|---|---|
+| Recognized movie window exists / visible | OBSERVED `MOVIE_WINDOW_CREATED` / `MOVIE_VISIBLE` |
+| Movie window disappears | OBSERVED `WINDOW_CLOSED_UNKNOWN` |
+| Any owned top-level window is visible | OBSERVED `PROCESS_WINDOW_VISIBLE`; not proof of a game frame |
+| Process handle signals exit | OBSERVED `PROCESS_EXITED_UNKNOWN`, with exit code |
+| Matching Windows Application Error event | CONFIRMED `SOFTWARE_EXCEPTION_CONFIRMED` |
+| Natural completion, skip request, skip confirmation | UNKNOWN: no safe internal observation point added |
+| Surface release/restoration, first game frame, attract state | UNKNOWN: window visibility cannot establish these operations |
+
+Manual bypass is never inferred from duration or disappearance. Normal and
+nonzero exits are not automatically classified as normal shutdown or exception.
+No raw input, global hooks, renderer callbacks or game-code hooks are added.
+
+Windows event collection reads at most 32 recent Application Error events after
+exit with a five-second timeout. It accepts only `v_on.exe`, matching process ID
+and session time. Export retains exception code, fault offset and a numeric
+module category: 0 unknown, 1 game, 2 ddraw, 3 vodisplay, 4 avifil32, 5 ntdll.
+An event published later may be missed; missing evidence remains UNKNOWN.
+There is no stack or loaded-module inventory without a dump.
+
+WER LocalDumps setup is not installed. Its usual machine-wide registry location
+requires elevated access and the executable-name scope cannot distinguish the
+two installations. Existing Windows crash policy is left untouched. No dump
+collection, registry writes or replacement exception handlers are introduced.
+Checkpoints are the mandatory fallback.
+
+Manual export adds sanitized `runtime-state.json` to the existing archive.
+It never includes dumps, game binaries/assets, raw input, arbitrary strings,
+private paths or process memory. Off stops new collection and retains prior
+records for manual export. Storage is four checkpoints of less than 8 KiB each;
+the existing event-log retention policy remains unchanged.
+
+## Windows acceptance
+
+Use a Windows desktop already set to 2560×1440. The test does not change Windows
+resolution. Initial settings: bosses Off, intro On, Single Display, 16:9. The
+validated internal rendering profile remains independent of physical output.
+
+1. Enable Development explicitly in each installation, then launch `v_on.exe`.
+2. Allow natural intro completion and attract mode. Do not skip during the
+   primary comparison. Run each installation at least three times.
+3. Export diagnostics after each run, retaining the run's archive.
+4. Repeat paired tests in Dual Mirror using the same monitor assignments.
+5. Separately test manual bypass and record the intentional action yourself;
+   the passive collector will correctly leave the skip classification UNKNOWN.
+6. Repeat representative paired tests with bosses enabled through the patcher.
+   Rebuild both installations with identical options; do not hand-edit EXEs.
+
+Record build, mode, run number, duration, complete cycles, natural/skip intent,
+CTD/continuation, diagnostic completion and exception metadata. Three successes
+are useful evidence, not proof against intermittent failure. Synthetic tests do
+not validate Windows video decoding or establish the cause of this CTD.

tools/experience/crash_evidence.py

@@ -0,0 +1,36 @@
+"""Read bounded Windows Application Error metadata; never enable dumps or upload."""
+import datetime
+import subprocess
+import xml.etree.ElementTree as ET
+
+
+def parse(text, pid, started_ms):
+    try: root = ET.fromstring('<Events>'+text+'</Events>')
+    except ET.ParseError: return None
+    for event in root:
+        def elements(name): return [e for e in event.iter() if e.tag.split('}')[-1] == name]
+        data = {e.attrib.get('Name'): e.text or '' for e in elements('Data')}
+        try:
+            provider = elements('Provider')[0].attrib.get('Name')
+            stamp = elements('TimeCreated')[0].attrib['SystemTime']
+            when = datetime.datetime.fromisoformat(stamp.replace('Z', '+00:00')).timestamp()*1000
+            process = int(data.get('ProcessId', ''), 0)
+            code = int(data.get('ExceptionCode', ''), 16)
+            offset = int(data.get('FaultingOffset', ''), 16)
+        except (ValueError, KeyError, IndexError): continue
+        if provider != 'Application Error' or process != pid or when < started_ms: continue
+        if data.get('AppName', '').lower() != 'v_on.exe': continue
+        module = {'v_on.exe': 1, 'ddraw.dll': 2, 'vodisplay.dll': 3,
+                  'avifil32.dll': 4, 'ntdll.dll': 5}.get(data.get('ModuleName', '').lower(), 0)
+        return {'exception_code': code, 'fault_offset': offset, 'fault_module': module}
+    return None
+
+
+def collect(pid, started_ms):
+    try:
+        result = subprocess.run(['wevtutil.exe', 'qe', 'Application',
+            '/q:*[System[(EventID=1000)]]', '/c:32', '/rd:true', '/f:xml'],
+            capture_output=True, timeout=5, creationflags=0x08000000, check=False)
+        if result.returncode or len(result.stdout)>262144: return None
+        return parse(result.stdout.decode('utf-8-sig', errors='replace'), pid, started_ms)
+    except (OSError, subprocess.TimeoutExpired): return None

tools/experience/diagnostics.py

@@ -174,7 +174,8 @@ def export(root, *, game_directory=None, return_name="RETURN-THIS-VO-PATCH-EX-G0
         edition = json.loads(prepared.read_text()).get('edition')
         if edition in ('retail', 'jpre'):
             versions['game'] = edition
-    payload = {'summary.json': {'versions': versions, 'physical_acceptance': 'PENDING',
+    import runtime_checkpoint
+    payload = {'runtime-state.json': runtime_checkpoint.exported(root), 'summary.json': {'versions': versions, 'physical_acceptance': 'PENDING',
                                'missing_module_logs': [k for k, v in reports.items() if not v],
                                'warnings': [e for e in events if e['event'] in ('warning', 'error', 'recovery', 'previous_session_incomplete')],
                                'assignment_evidence': 'saved configuration only; runtime ownership not inferred'},

tools/experience/movie_uat.py

@@ -0,0 +1,22 @@
+"""Explicit local UAT consent using the existing Diagnostics preference."""
+import configparser
+import sys
+from pathlib import Path
+
+
+def main():
+    action=sys.argv[1] if len(sys.argv)>1 else ''
+    if action not in ('enable','disable'):raise SystemExit('Use enable or disable')
+    path=Path(__file__).resolve().parent.parent/'player-displays.ini'
+    cfg=configparser.ConfigParser(interpolation=None)
+    data=path.read_bytes() if path.exists() else b''
+    cfg.read_string(data.decode('utf-16' if data.startswith((b'\xff\xfe',b'\xfe\xff')) else 'utf-8-sig'))
+    section=next((s for s in cfg.sections() if s.lower()=='diagnostics'),'Diagnostics')
+    if not cfg.has_section(section):cfg.add_section(section)
+    cfg.set(section,'Level','DEVELOPMENT' if action=='enable' else 'OFF')
+    temp=path.with_suffix('.new')
+    with temp.open('w',encoding='utf-16') as stream:cfg.write(stream)
+    temp.replace(path)
+    print('Development diagnostics enabled for next launch.' if action=='enable' else 'Diagnostics Off. Existing local evidence retained.')
+
+if __name__=='__main__':main()

tools/experience/normal_lifecycle.py

@@ -7,6 +7,8 @@
 import diagnostics as d
 import discovery
 import movie_geometry
+import runtime_checkpoint
+import crash_evidence
 
 RETURN='RETURN-THIS-VO-PATCH-EX-F01-P04-F02.zip'
 STATE={'VO_EX_REQUESTED_MODE':'selected_mode','VO_EX_EFFECTIVE_MODE':'effective_mode',
@@ -84,7 +86,7 @@ def movie_state(pid):
     user.EnumWindows.argtypes=[callback,w.LPARAM]
     user.EnumChildWindows.argtypes=[w.HWND,callback,w.LPARAM]
     user.GetWindowRect.argtypes=[w.HWND,ctypes.POINTER(w.RECT)];user.GetWindowRect.restype=w.BOOL
-    found=set();visible=set();seen=set();bounds=[]
+    found=set();visible=set();seen=set();bounds=[];process_window_visible=[False]
     def visit(hwnd,unused):
         if hwnd in seen:return True
         if len(seen)>=256:return False
@@ -104,10 +106,12 @@ def visit(hwnd,unused):
     def top(hwnd,unused):
         visit(hwnd,unused)
         owner=w.DWORD();user.GetWindowThreadProcessId(hwnd,ctypes.byref(owner))
-        if owner.value==pid:user.EnumChildWindows(hwnd,child,0)
+        if owner.value==pid:
+            if user.IsWindowVisible(hwnd):process_window_visible[0]=True
+            user.EnumChildWindows(hwnd,child,0)
         return True
     if not user.EnumWindows(callback(top),0):raise ctypes.WinError(ctypes.get_last_error())
-    return len(found),len(visible),tuple(sorted(bounds))
+    return len(found),len(visible),tuple(sorted(bounds)),process_window_visible[0]
 
 
 def attach(root,pid,event_name):
@@ -131,6 +135,12 @@ def ready():
         def wait():
             previous=None;probe_failed=False;prior=None;ticks=0
             geometry=None
+            checkpoints=None
+            session_ms=int(time.time()*1000)
+            if d.level(root)==3:
+                try:mode=int(os.environ.get("VO_EX_EFFECTIVE_MODE", "-1"))
+                except ValueError:mode=-1
+                checkpoints=runtime_checkpoint.Checkpoints(root,pid,mode)
             while True:
                 result=k.WaitForSingleObject(process,500)
                 if result==0:break
@@ -142,22 +152,33 @@ def wait():
                     try:prior=sample(root,prior)
                     except OSError:d.event(root,'error',{'module':5,'code':3})
                 if selected<3:continue
+                if checkpoints is None:
+                    try:mode=int(os.environ.get("VO_EX_EFFECTIVE_MODE", "-1"))
+                    except ValueError:mode=-1
+                    checkpoints=runtime_checkpoint.Checkpoints(root,pid,mode)
                 if geometry is None:geometry=movie_geometry.evidence(root,os.environ)
                 try:
                     current=movie_state(pid)
+                    checkpoints.windows(current[0],current[1],current[3])
                     if current!=previous:
                         d.event(root,'movie',{'count':current[0],'visible':current[1],
                                              'event':1 if current[1] and (not previous or not previous[1]) else 3 if previous and previous[1] and not current[1] else 2})
-                        for i,(x,y,w,h) in enumerate(current[2]):
-                            d.event(root,'movie',{'window_index':i,'window_x':x,'window_y':y,'window_width':w,'window_height':h,
-                                                 'geometry_match':not geometry['geometry_fallback'] and (x,y,w,h)==tuple(geometry[k] for k in ('expected_x','expected_y','expected_width','expected_height'))})
+                        for i,(x,y,width,height) in enumerate(current[2]):
+                            d.event(root,'movie',{'window_index':i,'window_x':x,'window_y':y,'window_width':width,'window_height':height,
+                                                 'geometry_match':not geometry['geometry_fallback'] and (x,y,width,height)==tuple(geometry[k] for k in ('expected_x','expected_y','expected_width','expected_height'))})
                         previous=current
-                except OSError:
+                except OSError as error:
+                    try:checkpoints.update(checkpoints.row['state'],error_code=getattr(error,'winerror',0) or 0)
+                    except OSError:pass  # a failed checkpoint must not affect the game
                     if not probe_failed:d.event(root,'error',{'module':4,'code':1})
                     probe_failed=True
             if previous and previous[0]:d.event(root,'movie',{'count':0,'visible':0,'event':3})
             code=w.DWORD()
             if not k.GetExitCodeProcess(process,ctypes.byref(code)):raise ctypes.WinError(ctypes.get_last_error())
+            if checkpoints and d.level(root)==3:
+                checkpoints.exited(code.value)
+                evidence=crash_evidence.collect(pid,session_ms)
+                if evidence:checkpoints.exited(code.value,evidence)
             return code.value
         return run(root,wait,ready)
     finally:

tools/experience/package_normal.py

@@ -33,7 +33,7 @@ def main():
   shutil.copy2(internal/name,runtime/name)
  shutil.copytree(internal/'DLLs',runtime/'DLLs')
  (runtime/'python312._pth').write_text('python312.zip\n.\nDLLs\n')
- for name in ('normal_lifecycle.py','diagnostics.py','discovery.py','movie_geometry.py'):shutil.copy2(HERE/name,runtime/name)
+ for name in ('normal_lifecycle.py','diagnostics.py','discovery.py','movie_geometry.py','runtime_checkpoint.py','crash_evidence.py'):shutil.copy2(HERE/name,runtime/name)
  shutil.copy2(HERE/'EXPORT-DIAGNOSTICS.cmd',payload/'EXPORT-DIAGNOSTICS.cmd')
  # Package the notices for the exact dependencies selected by the build.
  assert '__version__ = "2026.07.22"' in (internal/'Lib/site-packages/certifi/__init__.py').read_text()

tools/experience/prepare_movie_ab.py

@@ -0,0 +1,67 @@
+"""Assemble local-only controlled movie installations from verified player files.
+
+Never upload the output: it contains original game assets. Both builds use the
+same B01 patcher and recipes. Native provenance and recovery hashes follow the
+selected DLLs; all other installed runtime bytes must match.
+"""
+import argparse
+import hashlib
+import importlib.util
+import json
+import shutil
+import sys
+from pathlib import Path
+
+
+def sha(path): return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def main():
+    p=argparse.ArgumentParser(description=__doc__)
+    for name in ('package','game','gcc','llvm','output'): p.add_argument(name,type=Path)
+    p.add_argument('--bosses',action='store_true',help='Enable bosses identically in both installations')
+    a=p.parse_args();a.output.mkdir(parents=True,exist_ok=False)
+    package=a.package.resolve();here=Path(__file__).resolve().parent
+    sys.dont_write_bytecode=True
+    sys.path.insert(0,str(package/'_internal'))
+    spec=importlib.util.spec_from_file_location('movie_ab_patcher',package/'_internal/v-on-patcher.py')
+    v=importlib.util.module_from_spec(spec);spec.loader.exec_module(v)
+    raw=(a.game/'v_on.exe').read_bytes()
+    assert hashlib.md5(raw).hexdigest() in v.BUILDS,'Verified pristine game required'
+    assert (a.game/'von.avi').is_file(),'Original intro movie required'
+    # Apply once, then copy a complete independent installation, never hardlinks.
+    first=a.output/'A-GCC-MSVCRT';shutil.copytree(a.game,first)
+    patcher=v.Patcher();assert patcher.load(str(first/'v_on.exe'))[1]
+    wanted=v.default_state();wanted.update(bosses=a.bosses,bossunlock=False,movie=True)
+    ok,logs=patcher.apply(wanted);assert ok,logs
+    for name in ('normal_lifecycle.py','diagnostics.py','runtime_checkpoint.py','crash_evidence.py'):
+        shutil.copy2(here/name,first/'experience-runtime'/name)
+    (first/'player-displays.ini').write_text('[Presentation]\nMode=0\nGameAspect=1\n[Diagnostics]\nLevel=OFF\n')
+    # Explicit UAT opt-in; default installation stays Off.
+    (first/'ENABLE-DEVELOPMENT.cmd').write_text('@echo off\r\ncd /d "%~dp0"\r\n"experience-runtime\\python.exe" "experience-runtime\\movie_uat.py" enable\r\npause\r\n')
+    (first/'DISABLE-DEVELOPMENT.cmd').write_text('@echo off\r\ncd /d "%~dp0"\r\n"experience-runtime\\python.exe" "experience-runtime\\movie_uat.py" disable\r\npause\r\n')
+    shutil.copy2(here/'movie_uat.py',first/'experience-runtime/movie_uat.py')
+    second=a.output/'B-LLVM-UCRT';shutil.copytree(first,second)
+    inventories={}
+    for root,native in ((first,a.gcc),(second,a.llvm)):
+        for name in ('ddraw.dll','vodisplay.dll'):shutil.copy2(native/name,root/name)
+        meta=json.loads((root/'experience-version.json').read_text())
+        meta.update(wrapper_sha256=sha(root/'ddraw.dll'),helper_sha256=sha(root/'vodisplay.dll'))
+        (root/'experience-version.json').write_text(json.dumps(meta,indent=2)+'\n')
+        # Keep Restore correct after diagnostic additions and native selection.
+        journal=root/'.vo-ex-backup/ownership.json';ownership=json.loads(journal.read_text())
+        for row in ownership['files']:
+            file=root/row['name']
+            if file.is_file():row['installed']=sha(file)
+        journal.write_text(json.dumps(ownership,indent=2)+'\n')
+        inventories[root.name]={f.relative_to(root).as_posix():sha(f) for f in root.rglob('*') if f.is_file()}
+        (a.output/(root.name+'-SHA256.json')).write_text(json.dumps(inventories[root.name],indent=2)+'\n')
+    left,right=inventories.values();assert left.keys()==right.keys()
+    changed={n:{'A':left[n],'B':right[n]} for n in left if left[n]!=right[n]}
+    assert set(changed)=={'ddraw.dll','vodisplay.dll','experience-version.json','.vo-ex-backup/ownership.json'},changed
+    (a.output/'comparison.json').write_text(json.dumps({'expected_native':['ddraw.dll','vodisplay.dll'],
+        'derived_native_hash_metadata':['experience-version.json','.vo-ex-backup/ownership.json'],
+        'unexpected':[], 'files':changed,'options':wanted},indent=2)+'\n')
+    print('PASS two complete independent installations; only native DLLs and their required integrity/recovery hash metadata differ')
+
+if __name__=='__main__':main()