fix: Make surface definitions transactional

[kroese]

No description provided.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Screen-transition preservation needs human validation, and the legacy memory-budget regression remains unresolved.

Review effort: Balanced
Findings: 1

Open (1)

• Legacy surfaces can exceed the advertised 512 MiB allocation limit · New

What changed in this PR

Refactors VMware SVGA surface definitions to prepare replacements before publishing them, preserving existing surfaces on failure.

Changes:

• Separates surface preparation, object-table publication, and commit.
• Uses the guest-backed memory budget on capable devices.
• Adds detailed D3D11 resource-creation failure traces.

File	Description
hw/​display/​vmware_vga_dxvk.c	Adds resource-creation failure diagnostics.
hw/​display/​vmware_vga_3d.c	Introduces transactional surface preparation and commit, and changes budget selection.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Failure handling across guest metadata and live scanout needs correction and human validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Legacy surfaces can exceed the advertised 512 MiB allocation limit

Previously missed (1)

In code that hasn't changed since last review

Restore direct scanout binding when surface redefinition fails

hw/​display/​vmware_vga_3d.c:3776

When redefining the active direct-scanout surface, vmsvga3d_surface_prepare() detaches its direct binding before this OTable write. If the write fails, this path frees the candidate but does not restore that binding. Detach preserves the current pixels in a copy, but the rejected redefine still changes the live scanout source while leaving the old surface installed. Stage the fallible materialization before publishing and defer clearing the binding until commit, or restore the prior binding on failure.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Budget inconsistencies remain unresolved, and rollback and scanout storage lifetimes require human validation.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Separate GB and legacy surface memory accounting

hw/​display/​vmware_vga_3d.c:3228

GB and legacy commits both update state->surface_bytes, but GB allocations now check that total against 1 GiB while legacy definitions still check it against 512 MiB. With 576 MiB of GB shadows installed, a 64 MiB legacy DEFINE is rejected; creating that legacy surface first and then the same GB surfaces succeeds. Allocation order therefore determines whether the same surface set fits. Track GB and legacy usage separately, including replacements between the two kinds and destruction, so each advertised budget applies to the corresponding surfaces.

Use GB memory sizing during guest-backed surface recovery

hw/​display/​vmware_vga_3d.c:3481

vmsvga3d_d3d10_recover_gb_surface_live() still calls this installer, so guest-backed surface recovery uses 512 MiB while GB DEFINE now allows 1 GiB. When a valid OTable entry has no host surface, recovering a 64 MiB depth surface with 512 MiB already installed fails the budget check, although DEFINE would accept it; vmsvga3d_d3d10_dsv_realize_live() then returns false. Route GB recovery through preparation/commit with vmsvga3d_gb_surface_memory_size(s), preserving the legacy limit for legacy DEFINE calls.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Surface lifetime, scanout transitions, and publication ordering require human review and runtime validation of failure paths.

Review effort: Balanced
Findings: None