Skip to content

feat: pipeline multiple per progetto con PipelineBar (#162)#179

Merged
nikazzio merged 30 commits into
mainfrom
feat/multi-pipeline
May 26, 2026
Merged

feat: pipeline multiple per progetto con PipelineBar (#162)#179
nikazzio merged 30 commits into
mainfrom
feat/multi-pipeline

Conversation

@nikazzio
Copy link
Copy Markdown
Owner

Summary

  • Nuova tabella pipelines come source of truth per configurazione, stage, stato di esecuzione e chunk tradotti per ogni pipeline; il testo sorgente è condiviso nel record projects
  • pipelineService.ts nuovo servizio che gestisce CRUD pipeline, saveFullState, loadTranslations, restoreTranslations, saveChunkCheckpoint
  • PipelineBar UI con cerchi numerati (1, 2…), nome inline editabile con font font-display italic (stesso stile di "Testo Originale"/"Traduzione Candidata"), pulsante "+" per aggiungere pipeline — si inserisce tra header e contenuto principale
  • Test completamente aggiornati: nuovo pipelineService.test.ts, projectService.test.ts e projectStore.test.ts riscritti per la nuova architettura

Architettura

Il testo sorgente vive in projects (condiviso tra pipeline). Ogni pipeline ha: configurazione completa, stage, stato di esecuzione e chunk tradotti isolati. La colonna source_display_text nelle pipeline è nullable e predispone a un futuro override di sorgente per-pipeline senza schema changes.

Le migrazioni DB sono idempotenti (INSERT OR IGNORE, ensureColumn) e non rompono istanze esistenti — i dati vengono migrati da pipeline_configspipelines al primo avvio.

Test plan

  • Aprire un progetto esistente → si apre correttamente con una sola pipeline
  • Aggiungere una nuova pipeline con "+" → compare nella PipelineBar
  • Rinominare una pipeline (doppio clic sul nome) → il nome si aggiorna
  • Passare da una pipeline all'altra → i chunk tradotti cambiano correttamente
  • Avviare la traduzione → i chunk vengono salvati nella pipeline attiva
  • Riaprire il progetto → si ripristinano i chunk della pipeline attiva
  • npm test → 341/341 test passano
  • npx tsc --noEmit → nessun errore

nikazzio added 8 commits May 20, 2026 17:19
Ogni progetto supporta ora più pipeline di traduzione indipendenti.
Il testo sorgente è condiviso a livello progetto; i chunk tradotti,
la configurazione e lo stato di esecuzione sono isolati per pipeline.

Architettura:
- Nuova tabella `pipelines` (sostituisce pipeline_configs come source
  of truth); testo sorgente migrato in `projects`
- Nuovo `pipelineService.ts`: CRUD pipeline, checkpoint chunk,
  loadTranslations, restoreTranslations, saveFullState
- `projectService.ts` semplificato: solo CRUD progetto + sorgente
- `projectStore.ts`: gestione multi-pipeline (switch, create,
  rename, duplicate, delete), activePipelineId spostato qui
- `pipelineStore.ts`: rimosso activePipelineId, solo config + runState
- `usePipeline.ts`, `RunResumeBanner.tsx`: aggiornati ai nuovi servizi

UI:
- `PipelineBar`: cerchi numerati + nome inline editabile (font-display
  italic, stesso stile di "Testo Originale"), pulsante "+" per aggiungere
  pipeline; appare tra header e contenuto principale

Test:
- `pipelineService.test.ts`: nuovo, copre getPipelineConfig, saveFullState,
  loadTranslations, restoreTranslations (adattati da projectService.test)
- `projectService.test.ts`: riscritto per getProjectSource/saveProjectSource
- `projectService.sql.test.ts`: aggiornato per leggere pipelineService.ts
- `projectStore.test.ts`: mock aggiornati per nuova architettura biservizio
release-please aveva aggiornato Cargo.toml a 0.8.0 ma non
aveva rigenerato Cargo.lock, lasciando un'incongruenza che
faceva apparire il file come modificato ad ogni esecuzione
di cargo. Aggiornata solo la riga della versione del crate.
- rounded-[28px], shadow editoriale, backdrop 35% (era 70%)
- struttura header/body/footer con separatori border
- titolo font-display text-2xl italic (era text-lg senza stile)
- bottone chiudi pill con bordo (era icona nuda)
- pulsanti azione rounded-full pill (erano squadrati)
…se pipeline modificata

- Spostata da flex-shrink-0 globale a colonna flex interna al main
  document: ora appare sopra solo a DocumentView, non a InsightsDrawer
- Rimosso check viewMode dalla PipelineBar (non serve: è renderizzata
  solo nel branch document di App.tsx)
- Warning conferma eliminazione solo se pipeline è stata modificata:
  lastRunConfig != null oppure updatedAt > createdAt + 5s
…t, empty state redesign

- PipelineBar: pill bianca elevata per pipeline attiva (rounded-full), cerchio accent rosso pieno, inattive su beige senza sfondo
- InsightsDrawer: parte chiuso al caricamento (showDocumentDrawer: false), rimosso auto-open in setViewMode
- Empty state DocumentView: ridisegnato con sfondo beige, headline display, tre card placeholder in stile dashboard
- i18n: aggiunte stringhe emptyCardPipeline/Translations/Quality in it e en
- DocumentView: padding interno ridotto a py-3/py-4 per avvicinare toolbar alla PipelineBar
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces multi-pipeline support per project by moving pipeline configuration/state/translations into a new pipelines table (and a new pipelineService), adds a PipelineBar UI to switch/manage pipelines, and updates persistence logic + tests to align with the new architecture.

Changes:

  • Add pipelines as the source of truth for per-pipeline config/run-state/translations; keep source document text on projects.
  • Introduce pipelineService.ts and refactor stores/hooks to load/save pipeline state by pipeline_id.
  • Add PipelineBar to the document UI and update/replace service/store tests accordingly.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/types.ts Add Pipeline entity type used across store/service/UI.
src/stores/uiStore.ts Adjust default showDocumentDrawer behavior when switching view modes.
src/stores/projectStore.ts Track pipelines + active pipeline; refactor open/save to use pipelineService.
src/stores/projectStore.test.ts Rewrite tests to mock pipeline/project/db services with new persistence flow.
src/stores/pipelineStore.ts Remove activePipelineId from pipeline store (now owned by project store).
src/services/projectService.ts Split out project “source text” persistence and create default pipeline on project creation.
src/services/projectService.test.ts Update tests to validate getProjectSource/saveProjectSource.
src/services/projectService.sql.test.ts Point SQL structural validation at pipelineService.ts.
src/services/pipelineService.ts New service for pipeline CRUD + run state + translations persistence/restore.
src/services/pipelineService.test.ts Add unit tests for the new pipeline service.
src/services/dbService.ts Add pipelines table + migrations for multi-pipeline + translations.pipeline_id.
src/i18n/it.json / src/i18n/en.json Add strings used by the new document empty-state UI cards.
src/hooks/usePipeline.ts Switch checkpointing/run-state updates to pipelineService + active pipeline id from project store.
src/components/layout/PipelineBar.tsx New UI to select/rename/delete/add pipelines.
src/components/layout/index.ts Export PipelineBar.
src/components/document/DocumentView.tsx Refresh empty state layout and add placeholder cards.
src/components/common/RunResumeBanner.tsx Update run-state persistence to use pipelineService and project’s active pipeline id.
src/components/common/ConfirmDialog.tsx Visual/layout adjustments for confirm modal.
src/App.tsx Insert PipelineBar above DocumentView in document mode.
src-tauri/Cargo.lock Bump app crate version.
CLAUDE.md Add Git workflow guidance.
Comments suppressed due to low confidence (1)

src/services/pipelineService.ts:355

  • Same issue as above: this bulk translation INSERT/UPSERT also omits project_id, which can leave rows orphaned on project deletion because only project_id has a cascading FK. Ensure new translation rows keep project_id populated (or add a proper FK/cascade on pipeline_id).
      `INSERT INTO translations (
         id, pipeline_id, original_text, final_translation, position, chunk_status, stage_results,
         judge_status, judge_rating, translation_locked, judge_issues, coherence_result, footnotes,
         source_display_text, source_processing_text, translation_display_text, translation_processing_text,
         blob_id, blob_order, blob_reference_chunk_ids
       ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/services/pipelineService.ts Outdated
Comment thread src/services/pipelineService.ts Outdated
Comment thread src/stores/projectStore.ts Outdated
Comment thread src/components/layout/PipelineBar.tsx
Comment thread src/components/layout/PipelineBar.tsx Outdated
Comment thread src/services/projectService.sql.test.ts
…, language pair in createPipeline, i18n dialog, CI

- switchPipeline: salva lo stato corrente prima di caricare la pipeline target (evita perdita di edit unsaved)
- saveFullState/saveTranslations/saveChunkCheckpoint: aggiunge project_id alle INSERT di translations per preservare il cascade ON DELETE
- createPipeline: propaga source_language/target_language dalla pipeline corrente invece di usare i default della tabella
- createAndOpen/saveCurrentProject: passano currentProjectId a saveFullState
- setViewMode: chiude showDocumentDrawer solo su transizione verso non-document (fix CI)
- PipelineBar: stringhe del dialog di cancellazione via i18n, role="tablist" → role="group"
- Empty state: flex-1 per occupare tutto lo spazio verticale disponibile
- SQL test: rinomina test "pipelines INSERT" → "translations upsert" per riflettere cosa verifica
- Test: aggiorna firme saveFullState in projectStore.test e pipelineService.test
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

src/services/pipelineService.ts:484

  • In restoreTranslations, parseJson<CoherenceResult>(row.coherence_result, undefined as unknown as CoherenceResult) again relies on an unsafe cast to allow undefined. This makes it easy to accidentally treat coherenceResult as always-present in future refactors. Prefer a parseJson<T>(value): T | undefined overload (or a dedicated parseJsonOptional) so optional JSON columns remain correctly typed without casts.
    const judgeResult = restoreJudgeResult(row);
    const stageResults = parseJson<Record<string, PipelineResult>>(row.stage_results, {});
    const coherenceResult = parseJson<CoherenceResult>(row.coherence_result, undefined as unknown as CoherenceResult);
    const footnotes = row.footnotes ? parseJson<Footnote[]>(row.footnotes, []) : undefined;
    const blobReferenceChunkIds = row.blob_reference_chunk_ids
      ? (parseJson<unknown>(row.blob_reference_chunk_ids, []) as string[]).filter((v): v is string => typeof v === 'string')
      : undefined;

Comment thread src/stores/projectStore.ts Outdated
Comment thread src/services/pipelineService.ts
Comment thread src/components/layout/PipelineBar.tsx Outdated
Comment thread src/components/document/DocumentView.tsx Outdated
Comment thread src/services/pipelineService.ts Outdated
nikazzio added 4 commits May 24, 2026 18:39
…eys, unused import

- pipelineService: add parseJson overloads to remove undefined-as-unknown casts,
  drop unused runInTransaction import
- PipelineBar: replace hardcoded aria-label/title strings with t() calls
- DocumentView: stable React keys for empty-state cards (was i18n strings)
…store leak

- Replace targetChunkCount/splitIntoTargetChunks with wordsPerChunk/splitByWordTarget
  throughout algorithm, stores, services, DB schema, and tests. The parameter is now
  a direct word-count size instead of an indirect chunk count.

- Remove obsolete footnote warning from buildImportWarnings (dead code).

- ImportPreviewDialog now uses local state for sourceLanguage, targetLanguage,
  provider and model — no store mutations during interaction. On Confirm, all
  config changes are applied to usePipelineStore and persisted via savePipelineConfig
  to the active pipeline. Cancel leaves the store untouched.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/components/pipeline/ProductionStream.tsx:274

  • recommendedChunks is computed with recommendChunkCount(inputText) (default 700 words/chunk), but the UI’s “Use recommendation” button applies chunkPresetMedium to wordsPerChunk. This can easily make the displayed recommendation inconsistent with what the button applies (especially if presets are user-tuned). Consider passing chunkPresetMedium into recommendChunkCount so the recommendation matches the preset baseline, or deriving a recommended wordsPerChunk from the computed chunk count.
  const { glossaryHighlightEnabled, setGlossaryHighlightEnabled, chunkPresetMedium } = useUiStore();
  const { t } = useTranslation();

  const stats = useMemo(() => estimateTextStats(inputText), [inputText]);
  const recommendedChunks = useMemo(() => recommendChunkCount(inputText), [inputText]);
  const enabledStages = useMemo(() => config.stages.filter((s) => s.enabled), [config.stages]);
  const hasGlossary = config.glossary.length > 0;
  const showHighlight = glossaryHighlightEnabled && hasGlossary;

Comment thread src/services/dbService.ts
Comment on lines +374 to +376
// Add pipeline_id to translations (translations belong to a pipeline, not a project).
await ensureColumn('translations', 'pipeline_id', 'TEXT DEFAULT NULL');

Comment thread src/services/pipelineService.ts Outdated
source_display_text, source_processing_text, translation_display_text, translation_processing_text,
blob_id, blob_order, blob_reference_chunk_ids
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21)
ON CONFLICT(id) DO UPDATE SET
Comment thread src/services/pipelineService.ts Outdated
nikazzio added 3 commits May 25, 2026 10:29
…odels

- ProductionStream: recommendChunkCount now uses chunkPresetMedium as target
  so the "Chunks consigliati" display and the "Usa suggerimento" button are
  consistent even when the user has customized the medium preset.

- Header: savePipelineConfig failure is now logged instead of silently swallowed.
  Removed the spurious `as const` on updatedConfig; added explicit DocumentFormat
  and DocumentRenderProfile casts to preserve the correct narrow types.

- HelpGuide i18n: update recommended models (GPT-5, llama3.2, qwen2.5, gemma3,
  llama3.3:70b) to reflect the current model catalog.
… stage id comparison

- chunksStore: sourceProcessingText is now set directly from the chunk text,
  which is already clean (definitions stripped upstream by deriveSourceDocumentState).
  Removed the redundant extractFootnotes call and the now-unused import.

- ImportPreviewDialog: context overflow warning now identifies stage0 by id
  instead of object reference, making the intent explicit and the guard
  against an undefined stage0 unambiguous.
@nikazzio
Copy link
Copy Markdown
Owner Author

Copilot review — all items addressed

Fixed in earlier commits:

  • ✅ `translations` INSERT missing `project_id` — included in both `saveChunkCheckpoint` and `saveTranslationsInternal`
  • ✅ `createPipeline` not passing `sourceLanguage`/`targetLanguage` — both params now passed
  • ✅ `saveProjectSource` inside `runInTransaction` (deadlock) — moved outside the transaction
  • ✅ PipelineBar confirm/delete dialog strings — all use `t()` keys
  • ✅ `projectService.sql.test.ts` test name — corrected to match tested table
  • ✅ `parseJson` unsafe cast — replaced with typed overloads
  • ✅ PipelineBar aria-labels — all use `t()` keys
  • ✅ `key={label}` translated string — replaced with stable keys (`'pipeline'`, `'translations'`, `'quality'`)
  • ✅ `runInTransaction` unused import — removed

Fixed in commit 4578444:

  • ✅ `PipelineBar` `aria-label="Pipeline"` hard-coded — now uses `t('pipeline.ariaGroup')`
  • ✅ `restoreTranslations` — `blob_reference_chunk_ids` guarded with `Array.isArray` before `.filter()`

Design decision — `ON CONFLICT(id)` in translations:

Chunk IDs are generated as `chunk-${Date.now()}-${index}` and are globally unique by construction — cross-pipeline collision is not possible. `ON CONFLICT(id)` on the primary key is therefore semantically correct. Adding `pipeline_id` to the conflict target would require dropping and recreating the primary key (not supported by SQLite `ALTER TABLE`), introducing a risky migration with no practical benefit given the current ID generation strategy.

nikazzio added 4 commits May 25, 2026 12:12
Three interconnected bugs fixed:

1. DB migration gap (root cause of bugs 1 & 3): the `pipelines` table lacked
   `ensureColumn` coverage, so saves failed on older DBs missing columns like
   `blob_budget_tokens`. Added whitelist entries and migration calls for all
   15 columns that may be absent on pre-existing databases.

2. State inconsistency on first save (bug 3): `set({ pipelines })` was called
   mid-save before the transaction completed. On failure, pipelines appeared in
   the UI but `currentProjectId` remained null. Moved the pipelines update into
   the success `set()` so state is only committed when the full save succeeds.

3. "+" button disabled when no project is saved (bug 2): `createNewPipeline`
   silently returns without a project ID. The button is now visually disabled
   with a tooltip explaining the prerequisite.

Bug 1 (tab switch does nothing) was a symptom of bug 3: `switchPipeline`
exits early when `currentProjectId` is null, which only happened because the
premature `set({ pipelines })` made the bar visible in an inconsistent state.
…mpt badges

- scrollbar-gutter:stable su LibraryPanel e DictionaryEntryEditor — X non coperto dalla scrollbar
- PromptTemplatesTab: filtri icon-only + label corsivo (pattern LibraryPanel); pulsante Nuovo solo icona
- DictionariesTab: pulsanti azione icon-only; rimosso bg rosa dalla card assegnata
- DictionaryEntryEditor: layout tabella sticky header, scroll interno, aggiunta in cima, niente rosa
- StageCard: badge CUSTOM e pulsante reset al prompt di default
- PipelineConfig: AuditPromptEditor e PersonaEditor con edit/close/reset; textarea rows=12
- ProductionStream: rimosso pulsante "ri-valuta blocco" (duplicato dell'AuditPanel)
- handleRerunAll: resetta tutto (chunks + log) prima del riesegui
- SettingsModal: pill-selector per init nuova pipeline
- usePipeline: stripFootnoteMarkers su tutti i percorsi LLM; judge usa config base
- CLAUDE.md: documentato pattern barra navigazione icon-only + label corsivo
- createNewPipeline ora chiama switchPipeline dopo la creazione: la nuova
  pipeline eredita testo sorgente e chunk dal progetto (comportamento atteso
  — le pipeline sono varianti di traduzione dello stesso testo)
- resetAllChunks: svuota il buffer RAF pending prima del reset, azzera
  cancelRequested e activeStreamId — elimina token stale e stato sporco
  che causavano interruzioni inattese al riesegui
nikazzio added 5 commits May 25, 2026 23:43
…unreachable cancel check

- projectStore: openProject/switchPipeline now assign judgePrompt/Model/Provider directly
  from DB instead of falling back to in-memory state; DB is authoritative on project load
- pipelineService: drop run_in_progress column from setPipelineRunState UPDATE (field is
  never read back; run_status alone tracks pipeline state)
- pipelineService: remove export from saveTranslations (only used internally via saveFullState)
- usePipeline: remove unreachable cancelRequested check after runJudgeForChunk returns
  (the function already handles cancellation and returns 'cancelled')
…ation log

New pipeline now starts with the same source document as the active one:
- snapshot chunks before switchPipeline saves+clears them
- after the switch, restore and reset chunks to ready state (no translations)
- clear the operation log so the new pipeline starts with no run history
…line fixes

- originalText è ora immutabile (impostato in chunksFromTexts, mai sovrascritto in ON CONFLICT)
- restoreChunkSourceText ripristina sourceDisplayText a originalText; pulsante RotateCcw visibile quando divergono
- Pulsante matita spostato nell'header del pannello sorgente; sorgente non più bloccata dopo completed
- AuditPanel: pulsanti Check/X per marcare ogni problema come risolto o rifiutato (stato locale, mutuamente esclusivi)
- Test mode: esegue sempre i primi N chunk (rerun-unlocked), mai resume da mid-point
- createNewPipeline: generateChunks() anziché setChunks+reset per evitare ON CONFLICT(id) sulla pipeline vecchia
- pipelineService: persiste originalText nella colonna original_text; restoreTranslations legge original_text con fallback
- DictionaryEntryEditor: scrollbar non copre più la X (pr-2 invece di scrollbar-gutter:stable)
- Lingua default cambiata a italiano
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 55 out of 57 changed files in this pull request and generated 6 comments.

Comment thread src/services/pipelineService.ts
Comment thread src/services/pipelineService.ts
Comment thread src/stores/projectStore.ts
Comment thread src/stores/projectStore.ts
Comment thread src/components/layout/PipelineBar.tsx Outdated
Comment thread src/stores/chunksStore.ts
Comment on lines +329 to +338
const sourceFootnotes = usePipelineStore.getState().sourceFootnotes;
const nextChunks = updateSingleChunk(state.chunks, chunkId, (chunk) => {
if (chunk.sourceDisplayText === chunk.originalText) return chunk;
const hasTranslation = !!(chunk.currentDraft || Object.keys(chunk.stageResults).length > 0);
const updated = updateChunkSourceFields(
chunk,
chunk.originalText,
chunk.originalText,
buildChunkFootnotes(chunk.originalText, sourceFootnotes),
);
nikazzio added 5 commits May 26, 2026 16:28
…xt in restore

- withSyncedChunkFields now syncs originalText = sourceDisplayText (legacy field kept in step with canonical)
- restoreTranslations prefers source_display_text over original_text for originalText mapping
- usePipeline test beforeEach resets pipelineMode to 'document' to avoid test-mode forcing rerun-unlocked on batch skip tests
…rsing

- createPipeline/duplicatePipeline use generateId('pipeline') instead of Date.now()
- createPipeline omits judge columns — DB defaults (gemini/gemini-3-flash-preview) apply
- openProject/switchPipeline guard judgeModel and judgeProvider against empty DB values
- isModified normalizes SQLite timestamp to ISO 8601 before Date() parsing (WebKit fix)
- usePipeline test beforeEach sets pipelineMode: 'production' (was: 'test' default → forced rerun-unlocked on batch skip tests)
- Aggiunto UNIQUE INDEX su (pipeline_id, id) in translations: due pipeline
  non possono più sovrascriversi anche in caso di chunk ID identici.
- ON CONFLICT(id) → ON CONFLICT(pipeline_id, id) in entrambi gli upsert di
  pipelineService: il conflict target ora corrisponde al nuovo constraint.
- chunksFromTexts: sourceProcessingText ora usa stripFootnoteMarkers — i
  marker [^id] non vengono più passati al modello.
- restoreChunkSourceText: sourceProcessingText derivato con
  stripSuperscriptMarkers, le footnote preesistenti vengono preservate
  invece di tentare una riassegnazione su testo in forma superscript.
@nikazzio nikazzio merged commit 9a86267 into main May 26, 2026
6 checks passed
@nikazzio nikazzio deleted the feat/multi-pipeline branch May 26, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants