TAPDW

Processing, quality-control, feature-engineering, figure-generation, and analysis code for TAPDW 1.0 (build revision 5), the Thoracic Anesthesia Perioperative Data Warehouse described in A perioperative anesthesia dataset for thoracic surgery.

TAPDW contains de-identified, controlled-access perioperative records from 1,775 thoracic surgical encounters in 1,651 patients treated between 2023 and 2025. The code preserves the distinction between observed, standby, not-observed, unknown, and conflicting evidence and supports field-level provenance.

Important data boundary

This repository contains no patient-level data. It does not contain the TAPDW DuckDB database, Parquet/CSV extracts, source-document images, OCR text, review workbooks, identifiers, unshifted dates, date-shift values, API audit payloads, credentials, or local identity mappings.

Access to TAPDW data is governed by hospital policy and must be requested at tapdw.periopdata.org. Do not open an issue or pull request containing clinical records, screenshots, identifiers, or access credentials.

Repository contents

scripts/periop_pipeline/       Core extraction, canonicalization, validation,
                               feature, publication, and release modules
scripts/structured_pipeline.py Command-line entry point
tests/                         Data-independent unit tests
examples/analysis_template/    Ready-to-use analysis scaffold for an approved
                               local TAPDW DuckDB file
requirements-*.txt             Core, publication, and optional GPU OCR stacks

The raw-source pipeline is included for methodological transparency. Commands that process source documents or restricted review records require an institutionally approved local environment and are not runnable from this code-only repository alone.

Quick start: analysis template

After approval, download TAPDW_v1.0.duckdb from the controlled portal and place it at:

examples/analysis_template/data/TAPDW_v1.0.duckdb

On Windows PowerShell:

Set-Location examples/analysis_template
powershell -ExecutionPolicy Bypass -File .\bootstrap.ps1
powershell -ExecutionPolicy Bypass -File .\run_all.ps1

The template opens the database read-only, verifies the release checksum and data contract, constructs an encounter-level analysis cohort, generates a small descriptive report, and runs its tests. The database remains excluded from Git by default.

Pipeline environment

Use 64-bit Python 3.12 or 3.13:

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-core.txt
.\.venv\Scripts\python.exe -m pip install -r requirements-publication.txt
.\.venv\Scripts\python.exe -m pytest -q --basetemp=.pytest_tmp

GPU OCR is optional and requires the CUDA-compatible PaddlePaddle wheel before installing requirements-ocr.txt. Review-workbook generation additionally requires Node.js and @oai/artifact-tool; configure their locations with TAPDW_NODE_EXE and TAPDW_NODE_MODULES.

The DeepSeek-compatible extraction components read DEEPSEEK_API_KEY from the environment or an untracked .env. Never commit a real key. .env.example contains only blank variable names.

Run the command inventory with:

.\.venv\Scripts\python.exe scripts\structured_pipeline.py --help

Reproducibility and governance

• REPRODUCIBILITY.md maps the code to the manuscript workflow and explains which steps need controlled inputs.
• DATA_GOVERNANCE.md states the public/controlled/local boundary.
• scripts/check_public_boundary.py fails if a database, clinical document, archive, secret, populated notebook output, or local absolute path is added.
• GitHub Actions runs the boundary scan, Python compilation, and unit tests on each push and pull request.

Citation

Please cite the TAPDW Data Descriptor and institutional dataset record. Citation metadata are provided in CITATION.cff.

License

Code is released under the MIT License. This license applies to the software only; it does not grant access to or rights in the controlled TAPDW dataset.