Add e2e rendered image verification tests#221
Merged
Conversation
✅ CI Results🧪 Tests
📦 Artifactsscreenshots-ubuntu-latest · test-results · extension-vsix · ts-unit-test-results · python-unit-test-results |
…image-verification-utils.ts (canvas screenshot + pixel assertions)\n- New: tests/ui-test/image-rendering.test.ts (rendering verification tests)\n- Modified: tests/ui-test/DebugTestHelper.ts (captureCanvasImage)\n- Modified: tests/ui-test/display-options.test.ts (pixel assertions for display options)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pe declaration errors The jimp package has upstream type declaration errors (missing pngjs types and @jimp/plugin-print/load-font). These cause yarn test:compile to exit with code 2, blocking CI. skipLibCheck suppresses these third-party .d.ts issues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nel assertion
Two bugs causing all new tests to fail:
1. image-rendering.test.ts: viewVariable() did not call getWebviewEditor() after
clicking 'View Image', so the Image View panel was not focused. element.takeScreenshot()
on #gl-canvas captured the VS Code background (dark gray ~44/255) instead of actual
rendered pixels. Fix: call getWebviewEditor() + extra 1.5s wait in viewVariable().
2. display-options.test.ts: assertChannelSwapped for bgr_test right half had reversed
channel args. bgr_test right half is stored as BGR-red ([0,0,255]) which the extension
displays as blue (b>r) before the Swap RGB/BGR filter and red (r>b) after.
Fix: change ('r','b') to ('b','r').
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Dispatch window resize event inside webview frame before taking element screenshot, forcing WebGL to re-render the canvas. Without this, the backbuffer (preserveDrawingBuffer=false) may be cleared after the initial render, yielding a blank/dark capture. - Add 400ms sleep after resize dispatch to allow the RAF to complete. - Log center pixel sample in captureCanvasImage to aid CI diagnosis. - Retry clickDisplayOption up to 5 times to handle webview render delays. - Increase viewVariable post-getWebviewEditor wait from 500ms to 1500ms. - Add wait(1000) before getWebviewEditor() to match the working pattern in display-options.test.ts (viewVariableAndScreenshot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…able requestAnimationFrame is throttled in a freshly-opened webview with no prior user interaction. Clicking any display-option button causes Yew to schedule a full-speed RAF cycle. Using 'Reset' (no visual side-effect at default settings) before captureCanvasImage ensures the canvas has rendered real pixels rather than the cleared backbuffer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The VS Code webview outer frame (.webview class) contains a nested vscode-webview:// iframe with the actual extension content. On fresh sessions, this nested iframe may not yet exist when switchToWebviewFrame is first called, causing ChromeDriver to find elements via cross-frame traversal while clicks are intercepted by the nested iframe. Fix: after switching to the outer webview frame, poll for the nested content iframe up to 8 seconds before proceeding. Applied to both image-verification-utils.ts and display-options.test.ts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VS Code webviews use a 3-level iframe structure:
Level 1: main window → <iframe class="webview [ready]"> (outer container)
Level 2: <iframe src="vscode-webview://..."> (VS Code content manager)
Level 3: <iframe id="active-frame"> (actual user HTML)
Previous code only navigated 2 levels deep. Clicking buttons found from
level 2 would fail with ElementClickInterceptedError because the #active-frame
iframe was intercepting. This is also why the canvas was dark — the warmup
Reset click was never actually executed.
Fix: after switching to level-2, wait up to 15s for #active-frame and switch
into it. This matches the pattern used by vscode-extension-tester's own
WebviewMixin.switchToFrame() (which uses activeFrame = By.id('active-frame')).
Applied to both image-verification-utils.ts and display-options.test.ts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the 15-second #active-frame wait that caused ElementClickInterceptedError in display-options (the wait caused #active-frame to load and overlay buttons). Both display-options and image-verification-utils now: - Log all iframe attributes at each level for diagnostics - Switch outer→level-2 immediately without long waits - Check for #active-frame immediately (no polling) and switch if present - Fall back to level-2 if no #active-frame found yet Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Reset/channel buttons in the webview are covered by a Monaco panel sash (resize handle) in CI, causing WebDriver's coordinate-based click to fail with ElementClickInterceptedError. Using JavaScript .click() dispatches the event directly on the element, bypassing hit-detection. Also adds scrollIntoView before clicking for additional robustness. CI diagnostics confirmed: - Iframe structure: main -> .webview.ready -> #active-frame (user HTML) - Canvas capture works: r=217 correctly shown for Red Channel - Only button clicks were failing due to the sash overlay Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All sampleRegion y-ranges updated to y=0.22–0.52 (h=0.30) to stay within the rendered image extent (empirically y≈0.13–0.56). The previous y=0.40–0.65 extended ~10% past the image bottom, mixing in dark background pixels and reducing the measured luminance enough to fail the brightness-difference assertions. x-coordinates also tightened: - Red band: x=0.52–0.62 (clear of RED/GREEN transition at x≈0.68) - Green band: x=0.75–0.85 (solidly GREEN, away from transition) - Blue band: x=0.93–0.98 (the only visible slice of the blue column) - BGR-swap right region and red-channel-filter middle region updated to match the same calibrated x values Remove horizontal color-profile debug logging from captureCanvasImage now that all 7 tests pass locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
elazarcoh
force-pushed
the
feat/e2e-image-verification
branch
from
43f6ac2 to
1d40b96
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Export switchToWebviewFrame from image-verification-utils. Remove local switchToWebviewFrame, switchToMainContent, clickDisplayOptionButton, testDisplayOption, and viewVariableAndScreenshot from display-options.test.ts. Replace with imported switchToWebviewFrame and clickDisplayOption from utils. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Exclude docs/ from ESLint to avoid lint noise from TypeScript code blocks in Markdown plan files - Rename viewVariable parameter name -> name to match image-rendering.test.ts - Add explanatory comment on getWebviewEditor() call Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New DebugAnnotator class in image-verification-utils.ts:
- record(relX, relY, relW, relH, color, assertFn, label): executes
assertion, records pass/fail, re-throws on failure
- addRegion(...): records region without assertion (info-only)
- saveHtml(): writes <testName>.html with canvas screenshot +
SVG rectangle overlays (green=pass, red=fail, grey=info)
Only runs when SVIFPD_DEBUG_IMAGES=1 is set.
New captureAnnotatedCanvas(driver, testName) helper returns
{ img, annotator } pair for use in tests.
Integrated into image-rendering.test.ts (7 tests) and
display-options.test.ts (2 assertion blocks).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ports - display-options.test.ts: nest try/finally so annotatorBefore.saveHtml() always runs when capturedBefore succeeds (matches image-rendering pattern) - image-verification-utils.ts: move SVG text labels above box when they would overflow the viewBox bottom edge - image-verification-utils.ts: skip HTML write when no annotations recorded Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Switch captureCanvasImage from #gl-canvas (full viewport including webview sidebar) to .view-container (main rendering cell only). Apply img.autocrop() to trim VS Code background from all 4 edges. Coordinates in image-rendering.test.ts and display-options.test.ts updated to image-space fractions based on actual measured band positions (left≈0.0–0.43, mid≈0.43–0.86, right≈0.86–1.0 for rgb_gradient; bgr_test right half at x≈0.64–0.86) — no more 0.47 sidebar x-offset needed. waitForCanvasToRender sampling updated to cover the full image width instead of the old x>0.50 render-area heuristic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Gate 'used .view-container' debug log on screenshot success - Log warn on screenshot failure (was silently swallowed) - Remove #gl-canvas fallback: all test coordinates are calibrated for .view-container geometry; fallback would silently produce wrong geometry - Update stale JSDoc on captureCanvasImage and waitForCanvasToRender Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- sampleRegion: clamp x0/y0 to >= 0 to prevent negative-index reads
into previous bitmap rows via getPixelColor
- image-rendering.test.ts: treat clickDisplayOption('Reset') failure as
test error (was silently ignored, leaving stale state for next test)
- display-options.test.ts: throw when capturedAfter is null instead of
silently skipping pixel assertions; same Reset hardening applied
- DebugTestHelper.ts: update stale JSDoc (was #gl-canvas, now .view-container)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rovements as they are no longer relevant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add image-rendering verification to the e2e suite.\n\nThis PR: \n- Adds tests/ui-test/image-verification-utils.ts (canvas screenshot + pixel assertions).\n- Adds tests/ui-test/image-rendering.test.ts (rgb, grayscale, heatmap, bgr swap, channel filter tests).\n- Adds captureCanvasImage() to DebugTestHelper.\n- Adds pixel assertions to display-options.test.ts to ensure UI actions change visuals.\n\nUses WebDriver element screenshots and Jimp for decoding; assertions are region-based (dominant channel, luminance) to be robust across CI/GPU differences.\n\nCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com