Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
eccab36
docs(mssql-export): record approved wizard design and implementation …
shadowjohn Oct 12, 2026
87a3498
feat(mssql-export): own export session and immutable requests
shadowjohn Oct 12, 2026
dd675dd
docs(mssql-export): record task 1 verification
shadowjohn Oct 12, 2026
af30227
feat(mssql-export): collect native metadata and preflight snapshot ex…
shadowjohn Oct 12, 2026
b91e2aa
fix(mssql-export): preserve cancellation during command startup
shadowjohn Oct 12, 2026
fce1f29
docs(mssql-export): record metadata verification and identity contract
shadowjohn Oct 12, 2026
4e1ec67
feat(mssql-export): verify and publish exports and mirrors safely
shadowjohn Oct 12, 2026
0ca3f63
fix(mssql-export): bind publication to original consent
shadowjohn Oct 12, 2026
a2b0480
feat(mssql-export): stream typed row values within batch budgets
shadowjohn Oct 12, 2026
4e5d3e1
feat(mssql-export): script native schema and ordered dependencies
shadowjohn Oct 12, 2026
6703cb2
docs(mssql-export): record native schema validation
shadowjohn Oct 12, 2026
dee79d8
merge: integrate reviewed MSSQL values and artifact pipelines
shadowjohn Oct 12, 2026
f409c5a
docs(mssql-export): record reviewed values and artifact integration
shadowjohn Oct 12, 2026
6aed738
feat(mssql-export): reconstruct large values with guarded SQL chunks
shadowjohn Oct 12, 2026
57e7349
docs(mssql-export): record reviewed large-value restoration
shadowjohn Oct 12, 2026
c1f553b
feat(mssql-export): orchestrate cancellable export with measured prog…
shadowjohn Oct 12, 2026
400b497
docs(mssql-export): record reviewed export orchestration
shadowjohn Oct 12, 2026
14ba2fc
fix(mssql-export): stage small legacy text rows with native collation
shadowjohn Oct 12, 2026
2634975
feat(mssql-export): add native export wizard and Punky progress
shadowjohn Oct 12, 2026
c67e9b4
test(mssql-export): checkpoint isolated live acceptance harness
shadowjohn Oct 12, 2026
6dcfea8
fix(mssql-export): preserve modal cancellation results
shadowjohn Oct 12, 2026
fcf790e
docs(mssql-export): record reviewed native wizard and implementation …
shadowjohn Oct 12, 2026
72801a4
feat(mssql-export): route MSSQL interactive SQL exports through wizard
shadowjohn Oct 12, 2026
5657d6d
docs(mssql-export): record reviewed interactive routing
shadowjohn Oct 12, 2026
0a5410a
docs(mssql-export): record verified large single-session restore
shadowjohn Oct 12, 2026
bb38aa3
merge: integrate verified MSSQL export acceptance and legacy text fix
shadowjohn Oct 12, 2026
fa906b5
docs(mssql-export): record combined verification and acceptance bound…
shadowjohn Oct 12, 2026
0c0009a
Fix MSSQL export review findings
shadowjohn Oct 12, 2026
1410b25
docs(mssql-export): finalize reviewed implementation and verified bou…
shadowjohn Oct 12, 2026
5edf69d
docs(mssql-export): record final native UI retry
shadowjohn Oct 12, 2026
6f7712b
docs(mssql-export): plan native geometry serialization support
shadowjohn Oct 12, 2026
92fda81
Support native geometry in streaming MSSQL SQL exports
shadowjohn Oct 12, 2026
e3d4998
docs(mssql-export): record geometry restore acceptance and boundaries
shadowjohn Oct 12, 2026
6bbc3ba
docs(mssql-export): complete geometry review and acceptance record
shadowjohn Oct 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/MSSQL_SQL_EXPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# MSSQL SQL logical export

This exporter creates a local UTF-8 SQL script for explicitly selected SQL Server tables, optionally including supported dependent views. It is a logical export, not a native `.bak`, and does not change scheduled backups or other providers.

## Source and restore requirements

Data export uses one owned connection and one SNAPSHOT transaction. The source database must already have `ALLOW_SNAPSHOT_ISOLATION ON` with state `ON`; the exporter does not enable it or fall back to weaker isolation. Schema-only export does not require a data snapshot. The account needs SELECT plus readable definitions for the selected objects and dependencies. Missing prerequisites and unsupported features block preflight.

Restore to a prepared, compatible database using **one sqlcmd process/session for the entire artifact**:

```powershell
sqlcmd -S '<target-server>' -d '<target-database>' -E -b -x -f 65001 -i '<export.sql>'
```

`-b` stops on SQL errors; `-x` disables sqlcmd variable substitution; `-f 65001` specifies UTF-8 input. Do not split/reconnect between GO batches: large rows use session-local staging. The existing GUI SQL import splitter is not the validated import path. Review the destination before restoring: schema-and-data creates objects and does not drop/replace existing objects; data-only assumes compatible existing objects. Re-running a whole artifact is not an idempotent restore.

For the tested official **Go sqlcmd v1.10.0**, input is natively UTF-8 and `-f` is not supported. Use its explicit alternative command:

```powershell
& '<verified-portable-go-sqlcmd.exe>' -S '<target-server-or-named-pipe>' -d '<target-database>' -E -b -x -i '<export.sql>'
```

The installed classic sqlcmd 15.0.1300.359 fails when this fixture's input crosses 2 GiB. Do not use that version for >2 GiB artifacts. The Go client's UTF-8 decoder and one retained database connection are documented in its [version-pinned source](https://github.com/microsoft/go-sqlcmd/blob/v1.10.0/pkg/sqlcmd/sqlcmd.go#L339-L373). Local compatibility tests check Unicode codepoints/catalog names, literal substitution syntax, GO parsing, same-session temporary state and fail-fast behavior. On this LocalDB environment Go requires the exact running instance pipe (`np:\\.\pipe\LOCALDB#<instance-id>\tsql\query`); the shorthand `(localdb)\name` did not resolve. Never substitute another instance. See verification for the large-file result and client provenance.
## Supported behavior and boundaries

Native scalar types retain precision and temporal offsets, raw ANSI/Unicode/binary values retain bytes, and NULL differs from empty or trailing spaces. Supported schema includes identity seed/increment, computed columns, rowversion definitions, defaults, keys, supported indexes, checks, foreign keys, comments and supported dependent views. Computed and rowversion values are regenerated, not copied. Identity source allocation history/gaps are not promised. Empty tables retain their declared identity seed. Named filegroups must already exist on the destination.

Large values are streamed in bounded chunks; a row may span multiple GO batches. The importer validates ordered chunks, totals and completion before its one target INSERT. Replaying a consumed row's terminal batch does not insert it twice. A failed row is not replaced with NULL, even if a client continues after errors. **Earlier successfully inserted rows may remain:** restoration of the whole file is not all-or-nothing. Stop and investigate any sqlcmd error.

Legacy `text` rows use typed staging even for small values because direct varbinary assignment to text is rejected by SQL Server. This preserves the source codepage but increases script size and restore work for those rows. `ntext`, `image`, untyped XML and ordinary WKT text are supported within the documented tests; native SQL Server `sys.geometry` now uses `Serialize()` -> bounded binary chunks -> `geometry::Deserialize()`, with small values optionally inline. Native geometry DDL, NULL/empty, serialized bytes, SRID and Z/M are retained without WKT conversion, simplification or MakeValid. Geography, user-defined/alias CLR types, spatial indexes, typed XML, sql_variant, unsupported dependencies/table/column/index features and unreadable definitions are rejected rather than silently omitted. Tables with spatial indexes remain blocked in all modes, including data-only; geometry value support does not imply spatial-index scripting. Refer to the current preflight details for the affected object/column.

The transaction remains open while source rows are exported, retaining snapshot versions. It is released before local hash verification and mirror work. Allow disk space for source version store, SQL hex expansion (often more than twice binary payload), partial output, verified local output and optional mirror copies. Progress reports actual source bytes/rows and separate file verification/mirror work; estimates can remain unknown. More bytes in an export do not mean rows have been successfully restored.

Cancellation waits for owned reader/command/transaction/file cleanup. The final consent stamp detects destination changes observed before publication. Windows File.Replace does not provide an atomic compare-and-swap against unrelated external writers: a change in the narrow final-check/replace window can still be displaced. Recovery paths are retained and reported; after a partial replacement error, the previous file may be at a recovery path instead of its original location. Use a new filename for externally managed targets. Before replacement starts, failure or cancellation preserves the existing output. If the local file was published but the mirror fails or is canceled, the result is **local completed, mirror incomplete**, not complete success. Any retained recovery paths are reported.

## Reproducing the isolated acceptance

The opt-in runner defaults to instructions only:

```powershell
.\tests\Run-MssqlExportIntegration.ps1
.\tests\Run-MssqlExportIntegration.ps1 -Run -Server '(localdb)\mySQLPunkExportTest_20261012' -Profile Core -OutputDirectory '<new-empty-directory-under-authorized-FixtureRoot>'
# Profiles Geometry, Large and Concurrency use the same explicit arguments.
# Official portable Go client is opt-in: -Client Go -SqlcmdPath <verified-exe>.
# Profile ClientProbe verifies client semantics before a large restore.
# Large -RestoreArtifact <prior-owned-export.sql> validates prior SHA/token, then restores it in a new fixture.
```

This task's runner deliberately accepts only its authorized dedicated instance and the local authorization manifest at `.superpowers/sdd/implementation-plan/fixture-environment.json`. It requires that instance to be running, installed sqlcmd, Integrated Security and sufficient free C: space. It does not install/start databases, read saved connections/credentials, or accept a user database name. Each invocation generates GUID `_src`/`_dst` databases with an ownership token, bounded MDF/LDF sizes and exact file paths, then validates token/names/paths before DROP. Outputs and evidence remain local under the fixture root; no raw large artifact is committed. Never work around an antivirus block.

Core uses actual service artifacts, catalog comparison and streaming SHA-256 row digests. Large uses independent 1×/10× processes, actual file length, single-session restore and 25 ms memory sampling by service phase. Concurrency uses actual source connections with controlled second-connection updates, DDL, blocking/cancellation, disconnect and filesystem faults. The verifier retains only per-row digests, never whole field values. SQL fixture setup batching is separate from actual artifact restoration.

See [verification evidence](design/mssql-export/verification.md) for exact PASS/FAIL/PENDING scope. Native wizard/device sound/long-job UI, SSH lifecycle and GUI import require their own evidence; local engine acceptance does not establish those, production, IIS or user-account acceptance.

## Geometry restore compatibility

The Geometry profile verifies native geometry on SQL Server 2025 (17.0.4025.3), including a 17,600,048-byte serialized value, with real service output restored through one classic sqlcmd session. It compares streamed Serialize() hashes plus catalog, SRID, Z/M and empty/validity observations; invalid but storable shapes remain invalid and unchanged. Malformed serialized bytes or invalid chunk ordering fail the guarded row without inserting a partial value.

This is SQL Server's internal spatial serialization format, not portable OGC WKB. Cross-version engine compatibility and theoretical maximum-size geometries require their own restore acceptance. The client remains bounded; source Serialize(), target Deserialize() and staging still require SQL Server memory/tempdb/log/disk resources. See [Geometry design](design/mssql-export/geometry-design.md) and [verification](design/mssql-export/verification.md) for exact tested boundaries.
44 changes: 44 additions & 0 deletions docs/design/mssql-export/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# MSSQL SQL 匯出精靈:互動畫面原型

這是 2026-10-12 設計討論用的獨立 HTML 原型;正式 WinForms 精靈及匯出引擎另外實作。所有資料、速度、ETA、檢查結果、成功與取消狀態均為模擬;不連線資料庫,也不產生 SQL 檔案。

使用者已於 2026-10-12 同意此畫面及修訂後的正式功能規格:[design.md](design.md)。[實作計畫](implementation-plan.md) 已同意並執行。正式功能與實際驗證請看 [操作說明](../../MSSQL_SQL_EXPORT.md)、[驗證紀錄](verification.md);下列紀錄只描述當時的 HTML 原型。

## 開啟

可直接開啟 `index.html`,或以 Python 3.8+ 啟動本機預覽:

```powershell
python docs/design/mssql-export/serve-preview.py --port 17643
```

網址:`http://127.0.0.1:17643/docs/design/mssql-export/index.html`

預覽伺服器只綁定 loopback,僅提供此頁與三個明確允許的 Punky 素材,不提供資料夾索引。按 Ctrl+C 停止。

## 可以操作

- 上方原型導覽直接切換選表、輸出設定、確認、進度與完成。
- 選表預設全選;可搜尋表名/註解,跨搜尋保留勾選,零選取停用下一步。
- 選擇結構與資料/僅結構/僅資料;編輯示意輸出路徑。
- 右下情境選單可模擬未啟用快照,查看確認頁的提示與停用開始按鈕。
- 從確認頁按「開始模擬匯出」,約 40 秒內走過資料匯出、檔案驗證與完成;取消則經過停止與清理的模擬狀態。
- 深淺色切換;進度頁 Punky 自動循環播放,只有靜音控制。實際開始模擬時嘗試播放一次原音;仍受瀏覽器播放政策與裝置音量影響。
- 開啟資料夾是示意動作,會提示沒有實際輸出。

## 素材與邊界

直接引用既有 `mySQLPunk/image/punky_report_loading_atlas.png`(384px、73 格、83ms/格)、靜態圖與 WAV,不複製大型素材。正式版仍需以真正的 MSSQL 工作狀態、吞吐量與檔案驗證結果驅動畫面;本原型的數字與進度算法不可當成實作。

原型導覽是設計檢視工具,正式精靈執行中不能藉此切換設定。本 HTML 原型未實作 Windows 儲存檔案對話框、真實連線/SSH、快照條件、取消 SQL、串流匯出與還原驗證;正式版本的證據另列於驗證紀錄。

## 本機驗證紀錄(2026-10-12)

- JavaScript 與 Python 語法檢查通過;本機 HTTP 回應 200。
- 在 Brave 實際操作:預設 8/8 全選、清空停用下一步、註解「訂單」搜尋出 2 表、跨搜尋保留 7/8 選取,空路徑停用下一步。
- 快照情境切為未啟用時,確認頁出現原因並停用開始;切回可用後可模擬匯出至完成。
- 取消依序呈現「正在取消」與「本次匯出已取消」;深淺色、Punky loop 及畫面截圖已檢查。
- 原型視窗限制高度、內容區捲動、固定底部操作列;目前瀏覽器尺寸下完整進度與三列工作狀態可見。其他 DPI/視窗尺寸的正式 WinForms 驗收尚未進行。
- 未實際驗證裝置聽到的音效,也未執行應用程式/測試 EXE 或連線 DB。

已保存 [淺色進度圖](preview-progress-light.png) 與 [深色進度圖](preview-progress-dark.png)。
Loading