Isometric Drawing to Automated MTO Generator

Full-stack AI engineering assessment submission for extracting a piping Material Take-Off (MTO) from one uploaded piping isometric drawing.

Live Demo

Frontend: https://isometric-to-mto.vercel.app

Backend API Docs: https://isometric-to-mto.onrender.com/docs

The application accepts one PDF/PNG/JPG/JPEG piping isometric, validates and preprocesses it in a FastAPI backend, uses a Gemini Vision pipeline when GEMINI_API_KEY is present, falls back to a deterministic mock MTO when no key is configured, validates output with Pydantic, normalizes piping MTO conventions, derives gasket/bolt consumables, renders the MTO in a Next.js UI, and exports CSV.

1. Project overview

Browser / Next.js 15
  └─ upload one drawing
      └─ POST /api/upload
          └─ FastAPI validates type + size and stores file
              └─ POST /api/extract
                  └─ preprocessing: PDF/image -> normalized PNG page(s)
                      └─ Gemini Vision extraction OR mock fallback
                          └─ compact extraction contract parsed into Pydantic MTO schema
                              └─ post-processing: normalize rows, derive gasket/bolt quantities, compute summary
                                  └─ JSON response -> frontend MTO table + summary cards
                                      └─ GET /api/export/{file_id}.csv

Why this architecture?

• Next.js frontend keeps the evaluator-facing UI clean and responsive.
• FastAPI backend owns file validation, preprocessing, AI calls, schema validation, post-processing, and CSV generation.
• Pipeline abstraction keeps Gemini and mock extraction behind one run_pipeline entry point.
• Pydantic models define the canonical API contract and protect the UI from malformed model output.
• Mock fallback means the app works on a fresh machine without secrets or paid services.

2. Folder structure

project/
├── frontend/
│   ├── app/
│   │   ├── page.tsx                     # upload + immediate results page
│   │   ├── history/page.tsx             # upload history
│   │   └── results/[fileId]/page.tsx    # cached result page
│   ├── components/
│   │   ├── UploadForm.tsx               # upload + client-side validation
│   │   ├── ResultsTable.tsx             # drawing preview + metadata + MTO table + CSV link
│   │   ├── HistoryList.tsx              # uploaded files and extraction status
│   │   └── ui/                          # small UI primitives
│   ├── lib/api.ts                       # typed backend API client
│   └── types/index.ts                   # frontend mirror of backend schemas
├── backend/
│   ├── app/
│   │   ├── main.py                      # FastAPI app and CORS
│   │   ├── core/config.py               # typed environment settings
│   │   ├── models/schemas.py            # canonical MTO Pydantic models
│   │   ├── routers/                     # upload, extract, export, history
│   │   ├── services/
│   │   │   ├── pipeline.py              # orchestration
│   │   │   ├── preprocessing.py         # PDF/image normalization
│   │   │   ├── gemini_vision.py         # live Gemini prompt + compact MTO extraction contract
│   │   │   ├── mock_pipeline.py         # deterministic sample MTO fallback
│   │   │   ├── postprocessing.py        # MTO normalization + derived values
│   │   │   ├── csv_export.py            # CSV renderer
│   │   │   └── storage/                 # memory/sqlite storage adapters
│   │   └── utils/validation.py          # upload validation
│   ├── sample_assets/
│   │   └── sample_isometric_loop3.pdf
│   └── tests/
├── .env.example
├── .gitignore
├── PROJECT_PROGRESS.md
└── README.md

3. Requirements

Recommended versions:

• Python 3.12 recommended
• Python 3.11 or newer should also work
• Node.js 20 or newer
• npm 10 or newer

4. Backend setup

Open a terminal from the project root:

cd backend
python -m venv .venv

Activate the virtual environment.

Windows PowerShell:

.\.venv\Scripts\activate

macOS/Linux:

source .venv/bin/activate

Install backend dependencies:

pip install -r requirements.txt

Create the backend environment file:

cp ../.env.example .env

On Windows PowerShell:

copy ..\.env.example .env

Run the backend:

python -m uvicorn app.main:app --reload

Backend URLs:

• API base: http://localhost:8000
• Swagger docs: http://localhost:8000/docs
• Health check: http://localhost:8000/api/health

5. Frontend setup

Open a second terminal from the project root:

cd frontend
npm install

Create frontend environment file:

cp ../.env.example .env.local

On Windows PowerShell:

copy ..\.env.example .env.local

Run the frontend:

npm run dev

Open:

http://localhost:3000

6. Environment variables

The root .env.example documents both backend and frontend values.

Backend

GEMINI_API_KEY=
GEMINI_MODEL=gemini-2.5-flash-lite
ENVIRONMENT=development
MAX_UPLOAD_SIZE_MB=20
CORS_ALLOW_ORIGINS=http://localhost:3000
STORAGE_BACKEND=sqlite
DB_PATH=./data/app.db
UPLOAD_DIR=./data/uploads

Frontend

NEXT_PUBLIC_API_BASE_URL=http://localhost:8000

Important:

• Never commit a real GEMINI_API_KEY.
• If GEMINI_API_KEY is empty, the app automatically uses the mock pipeline.
• The submitted ZIP should include .env.example, not .env or .env.local.

7. API endpoints

This project uses a simple upload-then-extract flow. It is easier to test and debug than a hidden background job queue for a short assessment.

Method	Endpoint	Purpose
GET	/api/health	liveness and active pipeline mode
POST	/api/upload	upload one PDF/PNG/JPG/JPEG drawing
POST	/api/extract	run preprocessing + Gemini/mock + post-processing
GET	/api/export/{file_id}.csv	download cached MTO as CSV
GET	/api/history	list uploads and extraction status
GET	/api/results/{file_id}	fetch cached MTO JSON

The assessment suggested /api/mto/{job_id} endpoints. The shipped API is documented here and equivalent for this take-home because extraction is synchronous and cached by file_id.

8. AI pipeline explanation

Stage 1: Upload

POST /api/upload accepts exactly one drawing file. The backend stores it using the configured storage backend.

Stage 2: Validation

Validation is done twice:

• Frontend checks extension and 20 MB size for fast user feedback.
• Backend checks extension, actual byte size, and empty-file cases because the client cannot be trusted.

Accepted extensions:

.pdf, .png, .jpg, .jpeg

Stage 3: Preprocessing

backend/app/services/preprocessing.py converts input into model-friendly page images.

• PDFs are rasterized with PyMuPDF into PNG images.
• Images are normalized with Pillow into RGB PNG.
• Each page becomes a PreprocessedPage object.

The assessment scope is one drawing per upload. Multi-page PDFs are still handled as multiple page images and sent together, but the expected use case is one isometric sheet.

Stage 4: Gemini Vision extraction

backend/app/services/gemini_vision.py contains:

• the extraction prompt;
• a compact key-value response contract;
• parsing of Gemini output into a normalized internal MTO payload;
• validation of the final result using the canonical Pydantic schema in schemas.py.

The live Gemini prompt asks for one compact line in this shape:

MTO|PIPE_M=0|FITTINGS=0|FLANGES=0|VALVES=0|GASKETS=0|BOLT_SETS=0|SUPPORTS=0|INSTRUMENTATION=0|WELDS=0|LINE_NUMBER=NA|NPS=NA|SCHEDULE=NA|MATERIAL=NA|END_TYPE=NA|PROJECT=NA|CONFIDENCE=0.6|NOTE=NA

This compact contract was chosen to avoid long or truncated JSON responses on dense scanned isometrics. The backend then maps the compact values into the full Pydantic MTO schema used by the API, frontend, and CSV export.

The extraction attempts to read:

• drawing metadata: line number, NPS, project when visible;
• material rows: pipe, fittings, flanges, valves, gaskets, bolts, supports, instrumentation, welds;
• engineering details: size, schedule/rating, material, and end type when visible;
• confidence and notes when uncertain.

Domain rules embedded in the prompt:

• pipe length is measured in metres and stored in length_m;
• pipe quantity is a count, usually 1 when a pipe row exists;
• non-pipe materials are counted as discrete items;
• bolts are counted in sets;
• gaskets and bolt sets are tied to flanged joints;
• supports, instrumentation, and welds are included as bonus categories;
• unknown drawing values are marked as NA, Not visible, or similar instead of being silently hallucinated.

Stage 5: Mock fallback

If GEMINI_API_KEY is missing or the live call fails, pipeline.py uses mock_pipeline.py instead of crashing.

This is intentional because evaluators may run the project without secrets. The mock result is clearly labelled in the UI and includes assumptions explaining that values are illustrative.

The bundled sample outputs include one live Gemini extraction and two mock fallback runs. This is intentional: the live output demonstrates the AI extraction path, while the mock outputs demonstrate graceful degradation when the Gemini free-tier quota is exhausted or no API key is configured.

Stage 6: Post-processing

postprocessing.py cleans and completes the MTO:

• renumbers rows to 1..N;
• forces pipe rows to use unit M;
• keeps pipe quantity as a count and pipe length in length_m;
• removes length_m from non-pipe rows;
• derives or updates gasket and bolt-set quantities from estimated flanged joints;
• computes summary cards for pipe length, fittings, flanges, valves, gaskets, bolt sets, supports, instrumentation, and welds.

Stage 7: CSV export

csv_export.py writes one row per MTO item with metadata repeated for spreadsheet context.

Exported columns:

item_no, category, description, size_nps, schedule_rating, material_spec,
end_type, quantity, unit, length_m, confidence, remarks, drawing_no,
revision, line_number, nps, material_class, service

9. MTO schema

Each material row follows this model:

{
  "item_no": 1,
  "category": "PIPE",
  "description": "Pipe, Seamless, BE, ASME B36.10",
  "size_nps": "6\"",
  "schedule_rating": "SCH 40",
  "material_spec": "ASTM A106 Gr.B",
  "end_type": "BW",
  "quantity": 1,
  "unit": "M",
  "length_m": 12.45,
  "confidence": 0.91,
  "remarks": ""
}

Important convention:

PIPE quantity = count of pipe row/items
PIPE length_m = total pipe length in metres
Non-pipe quantity = count
BOLT unit = SET

Supported categories:

PIPE, FITTING, FLANGE, VALVE, GASKET, BOLT, SUPPORT, INSTRUMENTATION, WELD

Supported units:

M, EA, NO, SET

10. Tests

Run backend tests:

cd backend
PYTHONPATH=. pytest -q

On Windows PowerShell:

cd backend
$env:PYTHONPATH="."
pytest -q

Included tests cover:

• mock MTO schema;
• post-processing and gasket/bolt derivation;
• CSV columns;
• Gemini compact response parsing and error handling using monkeypatching;
• SQLite save/cache/list behavior;
• pipeline logging regression.

11. Assumptions made

• The upload represents one drawing / one isometric sheet.
• No production database is required; SQLite is enough for a demo and survives local restarts.
• The backend uses a synchronous extraction call because the assessment workload is small and easier to verify.
• If a BOM table is visible, the pipeline should prefer it. If no BOM table is visible, the model infers cautiously from symbols and dimensions.
• Some scanned drawings do not expose schedule, material class, or end type clearly. In such cases the app marks fields as Not visible instead of inventing authoritative values.
• Mock output is not meant to be an authoritative extraction from the sample drawing.

12. Known limitations

• Vision extraction accuracy depends on scan quality, handwriting clarity, symbol density, and whether a BOM table is visible.
• Dense drawings with many overlapping callouts may produce lower confidence values.
• Rotated/blurred text may be missed by the model.
• The compact Gemini response contract is intentionally short for reliability. A production version would use a more robust OCR plus structured-output hybrid.
• The gasket/bolt derivation uses an estimated flanged-joint count. A production system should explicitly detect each flanged joint.
• Schedule/rating, material grade, and exact end type are only reported as real values when visible or confidently inferable.
• No bounding-box overlay is included.
• No async job queue is included. Very large PDFs may block the request until extraction finishes.
• No authentication or rate limiting is included.

13. Trade-offs

• Synchronous extraction vs async jobs: synchronous is simpler and easier for evaluators to run. A production version would use a background queue.
• SQLite/on-disk storage vs cloud storage: SQLite keeps setup simple. Production would use object storage plus Postgres.
• Compact Gemini response vs full JSON output: compact output avoids truncated model responses on dense drawings. The backend still validates the final normalized result against the Pydantic MTO schema.
• Vision LLM first vs full OCR/CV system: Gemini Vision provides a strong 1 to 2 day baseline. A production system would combine OCR, title-block parsing, BOM-table extraction, symbol detection, and human review.
• Mock fallback: improves reproducibility but should never be confused with real extraction. The UI labels it clearly.

14. What I would improve with more time

• Add OCR table extraction for BOM blocks.
• Add provider structured-output mode with a shorter schema and retry/repair strategy.
• Add symbol bounding boxes and overlay them on the drawing preview.
• Add async job queue with progress states.
• Add Excel export.
• Add multi-sheet rollup.
• Add manual correction UI for MTO rows.
• Add confidence heatmap and review workflow.
• Add Docker Compose for one-command local startup.
• Add cloud object storage and a real database for hosted deployments.

15. Final ZIP checklist

Before submitting, ensure the ZIP does not contain generated/dependency folders or secrets:

node_modules/
.next/
.venv/
venv/
__pycache__/
.git/
.env
.env.local
backend/data/
frontend/tsconfig.tsbuildinfo

The ZIP should contain:

frontend/
backend/
README.md
.env.example
.gitignore
PROJECT_PROGRESS.md
sample_assets/ if present
tests/ if present

Recommended ZIP name:

Kumar_Rishav_isometric_mto.zip

16. Quick verification commands

Backend:

cd backend
.\.venv\Scripts\activate
python -m uvicorn app.main:app --reload

Frontend, in a second terminal:

cd frontend
npm run dev

Backend tests:

cd backend
.\.venv\Scripts\activate
$env:PYTHONPATH="."
pytest -q