Skip to content

feat(camera): desk view for Full Camera sections, with a covered tilt - #989

Open
christian-wr wants to merge 1 commit into
getopenscreen:mainfrom
christian-wr:feat/desk-view
Open

christian-wr wants to merge 1 commit into
getopenscreen:mainfrom
christian-wr:feat/desk-view

Conversation

@christian-wr

@christian-wr christian-wr commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Record the papers on your desk with the same webcam that films your face: tilt it down, and a Full Camera section shows the desk the right way up.

  • Desk view (inspector, Full Camera section): turns the camera 180°, switches the mirror off for that section — a selfie-mirrored page reads back to front — and shows the whole camera frame instead of the face crop. Rotation (0° / 180°) and mirror (Auto / On / Off) are also available separately. The timeline marks such a section with a rotate icon.
  • The tilt is covered. At both ends of a desk section the camera picture is blurred and dimmed while the Full Camera grow/shrink runs (the moment the camera is physically moving), fading in and out around it, with a translated "Desk mode" label over it. The label can be switched off per section; the blur stays.
  • Preview and export render it identically: everything goes through the compositor's frame plan. Old projects are unchanged — the new region fields are optional and defaults are never written.

How it is built

  • Rust compositor: per-frame WebcamOrientation (u/v bound swaps, crop bypass) and cover strength from the Full Camera region in effect; a new LayerCB.cover lane (appended last — Rust, HLSL, WGSL and MSL agree at offset 176 / 192 bytes, pinned by the existing size/offset test); the blur reuses the background effect's Vogel kernel and is clamped to the camera's valid area, so aligned decoder padding (1080 lines in a 1088-line texture) never bleeds in; two label animations in text_anim.
  • App: rotation / mirror / deskLabel on Full Camera regions, one undo step per change, persistence normalises unknown values; the label is generated like captions, one start/end pair per projected piece of the section, so it always matches the blur — also for sections split across clips.
  • Translations for every locale.

Known limitation: inside a speed region the label fades on source time while the blur runs on screen time (as every annotation does), so at 2× the label fades half as fast as the blur.

Related issue

None — new feature.

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Screenshots / video

Can follow on request (a short before/after of the tilt).

Testing

  • npm run test (303 files, 4179 passed), both tsc configs, npm run lint (0 errors), npm run i18n:check.
  • cargo test -p openscreen-compositor --lib on Windows ARM64: all new tests pass, including a naga validation of layer.wgsl (with the LAYER_MODELS prefix) that runs on every host. The pipeline_windows::tests that need hardware video decode fail on this host with or without this change.
  • Measured on real headless exports (electron . export, synthetic asymmetric 1080p and 720p cameras):
    • steady part of a desk section equals the source turned 180° at encode-floor difference (5.1), and is not mirrored with the project mirror on;
    • covered ends: camera luma 0.64× and fine detail 0.008× of the steady part; no dark/green band at the turned edge on a 1080p camera (130.4 vs 130.0);
    • label only while covered, absent with the label switched off;
    • a 2.6 s section changes smoothly (largest step 15.6 per frame, no one-frame jump).
  • Real device: a webcam tilted onto a printed page and back on Windows 11 ARM64 (Snapdragon X Elite) — Full Camera section with Desk view, tilt hidden at both ends, page upright and sharp in between, label switch, MP4 export. Logged in technical-documentation/testing/manual-e2e-checklist.md together with the measurements.
  • Not covered here: macOS and Linux rendering — the Metal shader is reviewed but compiled only by the macOS CI job.

Summary by CodeRabbit

  • New Features
    • Full Camera sections now support 180° rotation, mirror options, and full-frame display.
    • Desk view adds a blurred, dimmed camera cover and optional labels that fade at section transitions.
    • Camera orientation and desk-label settings are saved with undo support and are available in all supported languages.
  • Bug Fixes
    • Camera previews and exports now reflect the selected orientation and avoid unintended cropping in full-frame mode.
    • Changing the app language now refreshes translated text in the native compositor.

A Full Camera section can now show a webcam tilted down onto the desk: "Desk view"
in the inspector turns the camera 180°, switches the mirror off for that section
(otherwise text on the page reads back to front) and shows the whole camera frame
instead of the face crop. The timeline marks such sections with a rotate icon.

The moment the camera is tilted is covered at both ends of the section: the whole
camera picture is blurred and dimmed for the Full Camera grow (and the shrink),
fading in and out around it, with a translated "Desk mode" label over it that can be
switched off per section. Preview and export render it identically through the
compositor's frame plan.

- Rust: per-frame orientation (u/v bound swaps, crop bypass) and cover strength,
  a `LayerCB.cover` lane (HLSL, WGSL, MSL), the blur clamped to the camera's valid
  area so aligned decoder padding never bleeds in, two label text animations.
- App: `rotation` / `mirror` / `deskLabel` on Full Camera regions (defaults are not
  stored), store and persistence, inspector controls, generated label annotations
  per projected piece, translations in every locale.
- Tests for each layer; WGSL is validated with naga on every host.
@coderabbitai

coderabbitai Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (3)
technical-documentation/testing/manual-e2e-checklist.md — configured
technical-documentation/engineering/release-and-secrets.md — configured
AGENTS.md — auto-discovered

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 12f95830-4a1e-4bf2-98b3-470b06ad4fed
📥 Commits

Reviewing files that changed from the base of the PR and between b2e8635 and 92cb6d5.

⛔ Files ignored due to path filters (1)
  • crates/compositor/src/shaders.hlsl is excluded by !**/*.hlsl
📒 Files selected for processing (44)
  • crates/compositor/src/compositor_linux.rs
  • crates/compositor/src/compositor_macos.rs
  • crates/compositor/src/compositor_windows.rs
  • crates/compositor/src/frame_geometry.rs
  • crates/compositor/src/regions.rs
  • crates/compositor/src/scene.rs
  • crates/compositor/src/shaders.metal
  • crates/compositor/src/text_anim.rs
  • crates/compositor/src/vk_shaders/layer.wgsl
  • src/components/ai-edition/NativeCompositorOverlay.test.tsx
  • src/components/ai-edition/NativeCompositorOverlay.tsx
  • src/components/ai-edition/v4/EditorShellV4.module.css
  • src/components/ai-edition/v4/FloatingInspector.test.tsx
  • src/components/ai-edition/v4/FloatingInspector.tsx
  • src/components/ai-edition/v4/V4Timeline.tsx
  • src/components/video-editor/projectPersistence.test.ts
  • src/components/video-editor/projectPersistence.ts
  • src/components/video-editor/types.ts
  • src/i18n/locales/ar/settings.json
  • src/i18n/locales/cs/settings.json
  • src/i18n/locales/de/settings.json
  • src/i18n/locales/en/settings.json
  • src/i18n/locales/es/settings.json
  • src/i18n/locales/fr/settings.json
  • src/i18n/locales/it/settings.json
  • src/i18n/locales/ja-JP/settings.json
  • src/i18n/locales/ko-KR/settings.json
  • src/i18n/locales/pt-BR/settings.json
  • src/i18n/locales/ru/settings.json
  • src/i18n/locales/tr/settings.json
  • src/i18n/locales/vi/settings.json
  • src/i18n/locales/zh-CN/settings.json
  • src/i18n/locales/zh-TW/settings.json
  • src/lib/ai-edition/store/documentWriteAudit.test.ts
  • src/lib/ai-edition/store/useTimeline.test.ts
  • src/lib/ai-edition/store/useTimeline.ts
  • src/lib/ai-edition/timeline/timelineMap.test.ts
  • src/lib/cameraOrientation.test.ts
  • src/lib/cameraOrientation.ts
  • src/lib/deskCover.test.ts
  • src/lib/deskCover.ts
  • src/native/sceneDescription.test.ts
  • src/native/sceneDescription.ts
  • technical-documentation/testing/manual-e2e-checklist.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

Full Camera regions now support rotation, mirroring, full-frame display, and desk-label settings. Scene generation and compositor rendering apply those settings, including transition cover effects. The editor exposes the controls, and locale changes refresh the native scene.

Changes

Full Camera desk view

Layer / File(s) Summary
Orientation settings and timeline updates
src/lib/cameraOrientation.ts, src/components/video-editor/types.ts, src/components/video-editor/projectPersistence.ts, src/lib/ai-edition/store/useTimeline.ts, src/lib/ai-edition/timeline/timelineMap.test.ts, related tests
Full Camera regions support normalized rotation, mirroring, and label visibility. Persistence retains non-default values, and timeline actions update regions with undo history.
Editor controls and translations
src/components/ai-edition/v4/FloatingInspector.tsx, src/components/ai-edition/v4/V4Timeline.tsx, src/components/ai-edition/v4/EditorShellV4.module.css, src/i18n/locales/*/settings.json, related tests
The inspector adds desk-view, rotation, mirror, and label controls. Rotated timeline pills use a rotation icon. The settings labels are translated across the listed locales.
Scene orientation and desk labels
src/native/sceneDescription.ts, src/lib/deskCover.ts, related tests
Scene descriptions serialize resolved orientation and add localized labels at the start and end of turned camera regions. Label timing uses bounded transition and fade windows.
Compositor orientation and cover geometry
crates/compositor/src/scene.rs, crates/compositor/src/regions.rs, crates/compositor/src/frame_geometry.rs, crates/compositor/src/compositor_*.rs, manual E2E checklist
Frame planning resolves active camera orientation and cover strength. The Linux, macOS, and Windows compositors use transformed source bounds, bypass the crop for full-frame regions, and pass cover parameters to rendering.
Blurred cover and label rendering
crates/compositor/src/shaders.metal, crates/compositor/src/vk_shaders/layer.wgsl, crates/compositor/src/text_anim.rs
The shaders add radius-based blur and blend and dim the image during the cover. Desk-label animations fade opacity at the region edges.

Locale-triggered scene refresh

Layer / File(s) Summary
Refresh native scene on locale change
src/components/ai-edition/NativeCompositorOverlay.tsx, src/components/ai-edition/NativeCompositorOverlay.test.tsx
The native-scene effect reruns when the locale changes. The test checks that switching locales calls setNativeScene.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~50 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Editor as Full Camera editor
  participant Timeline as useTimeline
  participant Scene as buildSceneDescription
  participant Compositor as Native compositor
  Editor->>Timeline: Save orientation and desk-label settings
  Timeline->>Scene: Provide camera fullscreen regions
  Scene->>Compositor: Push serialized camera regions and projected labels
Loading

Merge Risk: ⚪ Minimal · up to 92cb6

The Full Camera desk-view change looks consistent across the editor, scene description and compositor. The author reports the tests pass. Rendering on macOS and Linux was not tested manually, and the Metal shader is compiled only by CI.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 92cb6

The camera settings remain within the existing project workflow, with compatible defaults and bounded rendering behavior. The main remaining concern is whether overlapping edits preserve earlier changes and a coherent undo history. No new security vulnerability was established.

Retained concerns

  • Medium · reliability · inferred: The new orientation and desk-label mutations write render-captured whole-document snapshots without serialization. Overlapping edits may overwrite an earlier change and record the same pre-edit document twice, weakening project-level undo/redo integrity. Successful-save and epoch guards are counterevidence for isolated failures and interruptions, but do not locally prevent ordinary concurrent writes. Native stale-write rejection remains unverified. This extends an existing mutation pattern rather than introducing a new privilege boundary.
Security review details

Security Blast Radius

  • inferred — The examined changes affect the active project's camera composition, clip-owned annotations, and whole-document save/undo state. The orientation and label fields themselves carry scalar rendering intent rather than new resource paths, credentials, or execution capabilities.

Trust Boundaries and Controls

  • observed — Project orientation values are normalized before scene serialization, and Rust gates full-frame behavior on an exact 180-degree rotation. Generated desk text comes from a fixed localization key rather than arbitrary project-provided label content.
  • observed — The HLSL, WGSL, and Metal cover kernels clamp every blur tap below the valid camera extent with a half-chroma-texel margin, providing counterevidence to decoder-padding sampling through the new blur path.

Resilience and Maintainability Implications

  • observed — Camera-setting patches preserve region IDs and timing while updating every fragment belonging to the selected pill. Failed saves do not install the requested frontend document or create its undo entry, containing isolated persistence failures.

Hardening Proposals

  • proposed — Apply the new camera mutations through a shared serialized document-mutation path that reads current state after preceding saves settle. Preserve project identity and interruption guards, and validate rapid edits, failed saves, undo/redo, and recovery against the native persistence contract.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.93% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 57 functions across 25 files. (19 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: desk view for Full Camera sections with a covered tilt.
Description check ✅ Passed The description includes a detailed summary, issue status, change type, release and platform impact, visual-change note, and comprehensive testing details. It is mostly complete against the template.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 71.93% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 57 functions across 25 files. (19 skipped: 19 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

🔧 Clippy (1.98.1)

Clippy execution failed


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant