Skip to content

Resolve open Engine issues and Prose Guardian regression#4122

Merged
SpicyMarinara merged 4 commits into
stagingfrom
agent/issues-sweep-4114-4120
Jul 26, 2026
Merged

Resolve open Engine issues and Prose Guardian regression#4122
SpicyMarinara merged 4 commits into
stagingfrom
agent/issues-sweep-4114-4120

Conversation

@SpicyMarinara

@SpicyMarinara SpicyMarinara commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Why

Resolve the current open issue set with focused fixes, include the maintainer credits update, restore Prose Guardian rewrites reported against latest staging, and add the requested explicit sidecar VRAM control.

Scope

Validation

  • pnpm check
  • pnpm localization:check
  • pnpm regression:issues
  • pnpm smoke:ui
  • Manually verify the affected UI paths in a browser

Automated validation run while preparing this PR: pnpm check, pnpm regression:issues, and pnpm smoke:ui (130 passed, 68 skipped). The only lint notice is the pre-existing NoodleHome.tsx exhaustive-deps warning.

All validation boxes intentionally remain unchecked for human verification.

Summary by CodeRabbit

  • New Features

    • Configure up to five LoRAs for ComfyUI image and video generation.
    • Generate game music and sound effects with ElevenLabs.
    • Load and unload local models directly from Connections settings.
    • Improve generated game audio playback and caching.
    • Added accessibility semantics to the Memory Recall toggle.
  • Bug Fixes

    • Improved persistence and display of background-generated messages.
    • Refined text-rewrite handling and scene audio generation controls.
    • Added clearer local model loading, unloading, and error states.
    • Updated credits with additional acknowledgements.

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a3c08e2d-05dc-403b-96f9-3693572670a2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds ComfyUI LoRA discovery and workflow support, ElevenLabs-generated game audio with caching, explicit sidecar model load/unload controls, scene-analysis audio modes, message-cache corrections, accessibility semantics, and regression coverage.

Changes

Client and generation correctness

Layer / File(s) Summary
Accessibility, cache, and rewrite handling
packages/client/src/components/chat/ChatSettingsDrawer.tsx, packages/client/src/hooks/*, packages/server/src/routes/generate*, packages/server/src/services/generation/prose-guardian-settings.ts
Memory Recall exposes checkbox state, persisted background messages update the cache from SSE events, stream buffers clear before streaming begins, and strict text rewrites accept explicit boolean-like requests.
Regression coverage and credits
scripts/regressions/open-issues.regression.ts, packages/client/src/components/chat/HomeCreditsModal.tsx
Regression assertions cover the updated UI, cache, rewrite, LoRA, audio, and sidecar behavior; two special-thanks entries are added.

ComfyUI LoRA configuration and propagation

Layer / File(s) Summary
LoRA data contracts and normalization
packages/shared/src/types/*generation-defaults.ts, packages/shared/src/constants/*generation-defaults.ts
ComfyUI defaults now contain up to five normalized LoRA settings and generate %LORA_n% and strength placeholder replacements.
Connection editor and model discovery
packages/client/src/components/connections/ConnectionEditor.tsx, packages/client/src/hooks/use-connections.ts, packages/server/src/routes/connections.routes.ts, packages/client/src/localization/locales/en.json
Fetched ComfyUI LoRAs populate image and video editors with five-slot selectors, strength controls, and workflow guidance.
Generation propagation
packages/server/src/services/video/*, packages/server/src/services/image/image-generation.ts, packages/server/src/routes/{gallery,game,sprites}.routes.ts, packages/server/src/services/generation/media-connection-fallback.ts
Connection, runtime, fallback, image, video, storyboard, scene, and animated-expression generation paths carry LoRA settings into workflow placeholder replacement.

ElevenLabs game-audio generation

Layer / File(s) Summary
TTS configuration
packages/shared/src/types/tts.ts, packages/client/src/components/panels/settings/TTSConfigCard.tsx, packages/client/src/localization/locales/en.json
ElevenLabs profiles gain opt-in game music and sound-effect generation settings with persisted UI toggles.
Scene-analysis audio modes
packages/shared/src/schemas/scene-analysis.schema.ts, packages/server/src/services/sidecar/scene-analyzer.ts, packages/server/src/services/sidecar/scene-postprocess.ts, packages/server/src/routes/sidecar.routes.ts, packages/server/src/routes/game.routes.ts
Scene analysis and post-processing support generated audio prompts separately from Spotify selection and existing asset matching.
Generation, caching, and playback
packages/client/src/components/game/GameSurface.tsx, packages/server/src/routes/tts.routes.ts
Game scenes request generated audio, cache deterministic assets, materialize returned tags, rebuild the asset manifest, and resolve generated assets during playback.

Sidecar model load and unload controls

Layer / File(s) Summary
Store and process lifecycle
packages/client/src/stores/sidecar.store.ts, packages/server/src/services/sidecar/sidecar-inference.service.ts, packages/server/src/services/sidecar/sidecar-process.service.ts
Explicit load and unload operations update sidecar state, preserve downloaded status after unloading, and prevent automatic startup while manually unloaded.
Connection-panel controls
packages/client/src/components/panels/ConnectionsPanel.tsx, packages/client/src/localization/locales/en.json
The local-model card provides guarded load/unload actions, progress indicators, power icons, toasts, and localized lifecycle labels.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Suggested labels: bugfix, feature

Suggested reviewers: thetopham

Poem

LoRAs line up in slots of five,
New game sounds wake worlds alive.
Sidecars sleep, then power on,
Stale streams fade before the dawn.
Checkbox states now speak with care—
A tidy patch, if one dares.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.45% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the PR's main theme: resolving open engine issues plus a Prose Guardian regression.
Description check ✅ Passed The description mostly follows the template, covering why, scope, and validation with concrete test results.
Linked Issues check ✅ Passed Issues [#4114-#4123] are covered by the UI, cache, LoRA, audio, and sidecar changes, with regression coverage reinforcing the fixes.
Out of Scope Changes check ✅ Passed No clear out-of-scope code changes stand out; the Prose Guardian and credits updates are explicitly described in the PR scope.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/issues-sweep-4114-4120

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SpicyMarinara
SpicyMarinara marked this pull request as ready for review July 26, 2026 07:42
@SpicyMarinara

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot added bugfix Bug fix feature New feature or enhancement labels Jul 26, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (1)
packages/client/src/components/game/GameSurface.tsx (1)

2600-2619: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Two "scoped asset map" flavors, one silently stale.

Ah, a fascinating specimen of divergent evolution! scopedAssetMap (the useMemo) spreads generatedAudioAssetsRef.current at compute time, but its dependency array (assetManifest?.assets, gameAssetExcludedFolders) can never include a ref — so it never re-includes newly generated audio tags once generateGameAudioAsset mutates the ref. getScopedAssetMap() recomputes fresh every call and is correctly used everywhere generated-audio freshness matters today (handleSegmentEnter, applySceneResult, processSceneRef). No active bug in this diff, but the stale twin is left in place as bait for the next unfortunate experimenter who reaches for scopedAssetMap expecting a freshly generated tag.

A brief comment near the scopedAssetMap definition noting it excludes fresh generated-audio tags (or consolidating the two into one helper) would spare a future colleague an... unpleasant surprise.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/client/src/components/game/GameSurface.tsx` around lines 2600 -
2619, Add a brief comment beside the scopedAssetMap useMemo documenting that its
generatedAudioAssetsRef contents are captured only at memoization time and may
exclude newly generated audio tags; direct callers requiring fresh generated
assets should use getScopedAssetMap().
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/client/src/components/connections/ConnectionEditor.tsx`:
- Line 345: Update the localProvider-change cleanup effect to also reset
remoteLoras, alongside remoteModels and fetchError, so LoRA results from the
previous provider are cleared whenever the provider or base URL changes.

In `@packages/client/src/components/game/GameSurface.tsx`:
- Around line 2620-2675: Wrap the api.post call in generateGameAudioAsset with
the existing withTimeout helper, using the established timeout and
error-handling pattern from runGameAssetGeneration, handleGenerateSceneVideo, or
previewTurnStoryboardPrompts. Preserve the current successful asset registration
and null-return behavior when generation times out or fails.

In `@packages/client/src/hooks/use-background-autonomous.ts`:
- Around line 198-223: Update the text_rewrite handling in the background stream
event loop to apply any string editedText regardless of rewriteApplied,
including the no-rewrite fallback. When updating the latest assistant message,
mirror use-generate.ts by removing extra.postProcessingPending while preserving
other extra fields, and continue ignoring events without valid editedText or
without a latest assistant message.

In `@packages/client/src/localization/locales/en.json`:
- Line 6370: Inspect usages of the localization key
ui.panels.sidecarcard.download and remove it if it is an unused composition
fragment; otherwise replace its value with the intended complete, properly
capitalized download label consistent with neighboring sidecarcard translations,
without retaining the trailing comma.

In `@packages/server/src/routes/connections.routes.ts`:
- Around line 792-801: Ensure the shared ComfyUI LoRA lookup using
fetchComfyLoaderModelNames is executed for both image_generation and
video_generation connections. Update the models handler and its provider split
so the video branch also returns loras, while preserving the existing model and
LoRA mapping behavior.

In `@packages/server/src/routes/tts.routes.ts`:
- Around line 191-201: Change the asset-generation flow around
buildAssetManifest() so cache-miss writes do not synchronously rescan and
rewrite the full manifest for every generated asset. Debounce or batch rebuild
requests, preserving eventual manifest updates while allowing multiple
generations in one turn to share a single rebuild and avoiding blocking the
event loop per asset.

In `@packages/server/src/services/sidecar/scene-postprocess.ts`:
- Line 267: Update the per-beat music assignment in postProcessSceneResult to
sanitize and retain out.music only when ctx.generateMusic &&
!ctx.useSpotifyMusic; otherwise leave it undefined. Also update
buildSceneAnalyzerSystemPrompt so generatedAudio uses ctx.generateSoundEffects
|| (ctx.generateMusic && !ctx.useSpotifyMusic), matching
buildSceneAnalyzerUserPrompt.

In `@packages/server/src/services/sidecar/sidecar-process.service.ts`:
- Around line 205-208: Update unload() so manuallyUnloaded is assigned inside
the same lifecycle-lock-protected operation as stop(), preventing a concurrent
restart from overwriting the unload intent before shutdown completes.

---

Nitpick comments:
In `@packages/client/src/components/game/GameSurface.tsx`:
- Around line 2600-2619: Add a brief comment beside the scopedAssetMap useMemo
documenting that its generatedAudioAssetsRef contents are captured only at
memoization time and may exclude newly generated audio tags; direct callers
requiring fresh generated assets should use getScopedAssetMap().
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: becb3729-5203-439d-bc6a-94e7b548fbca

📥 Commits

Reviewing files that changed from the base of the PR and between 66fd75e and fe3a679.

📒 Files selected for processing (35)
  • packages/client/src/components/chat/ChatSettingsDrawer.tsx
  • packages/client/src/components/chat/HomeCreditsModal.tsx
  • packages/client/src/components/connections/ConnectionEditor.tsx
  • packages/client/src/components/game/GameSurface.tsx
  • packages/client/src/components/panels/ConnectionsPanel.tsx
  • packages/client/src/components/panels/settings/TTSConfigCard.tsx
  • packages/client/src/hooks/use-background-autonomous.ts
  • packages/client/src/hooks/use-connections.ts
  • packages/client/src/hooks/use-generate.ts
  • packages/client/src/localization/locales/en.json
  • packages/client/src/stores/sidecar.store.ts
  • packages/server/src/routes/connections.routes.ts
  • packages/server/src/routes/gallery.routes.ts
  • packages/server/src/routes/game.routes.ts
  • packages/server/src/routes/generate.routes.ts
  • packages/server/src/routes/generate/retry-agents-route.ts
  • packages/server/src/routes/sidecar.routes.ts
  • packages/server/src/routes/sprites.routes.ts
  • packages/server/src/routes/tts.routes.ts
  • packages/server/src/services/generation/media-connection-fallback.ts
  • packages/server/src/services/generation/prose-guardian-settings.ts
  • packages/server/src/services/image/image-generation.ts
  • packages/server/src/services/sidecar/scene-analyzer.ts
  • packages/server/src/services/sidecar/scene-postprocess.ts
  • packages/server/src/services/sidecar/sidecar-inference.service.ts
  • packages/server/src/services/sidecar/sidecar-process.service.ts
  • packages/server/src/services/video/game-video-runtime.ts
  • packages/server/src/services/video/video-generation.ts
  • packages/shared/src/constants/image-generation-defaults.ts
  • packages/shared/src/constants/video-generation-defaults.ts
  • packages/shared/src/schemas/scene-analysis.schema.ts
  • packages/shared/src/types/image-generation-defaults.ts
  • packages/shared/src/types/tts.ts
  • packages/shared/src/types/video-generation-defaults.ts
  • scripts/regressions/open-issues.regression.ts

Comment thread packages/client/src/components/connections/ConnectionEditor.tsx
Comment thread packages/client/src/components/game/GameSurface.tsx
Comment thread packages/client/src/hooks/use-background-autonomous.ts
Comment thread packages/client/src/localization/locales/en.json
Comment thread packages/server/src/routes/connections.routes.ts
Comment thread packages/server/src/routes/tts.routes.ts
Comment thread packages/server/src/services/sidecar/scene-postprocess.ts Outdated
Comment thread packages/server/src/services/sidecar/sidecar-process.service.ts
@SpicyMarinara
SpicyMarinara merged commit e0de3c5 into staging Jul 26, 2026
6 checks passed
@SpicyMarinara
SpicyMarinara deleted the agent/issues-sweep-4114-4120 branch July 26, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Bug fix client feature New feature or enhancement server shared

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant