IsoMTO - AI-Powered Piping Isometric Drawing to MTO Generator

IsoMTO is a full-stack web application designed to automate the generation of a Material Take-Off (MTO) from piping isometric drawings. By leveraging a Google Gemini Vision LLM pipeline, the application processes drawings, extracts engineering metadata, identifies piping materials (pipes, fittings, flanges, valves, gaskets, and bolt sets), and displays them side-by-side with the drawing for verification.

---

1. Project Overview & Architecture

Preparing piping MTOs is traditionally a manual, slow, and error-prone process. IsoMTO automates this workflow by taking an unstructured engineering drawing (PDF or image) and generating a structured, validated bill of materials.

Architecture Flow Diagram

               +----------------------------------------+
               |        Piping Isometric Drawing        |
               |             (PDF, PNG, JPG)            |
               +-------------------+--------------------+
                                   |
                                   v
               +-------------------+--------------------+
               |          Next.js Frontend              |
               |     (File validation, Upload UI)       |
               +-------------------+--------------------+
                                   |
                                   | Multipart Upload
                                   v
               +-------------------+--------------------+
               |           FastAPI Backend              |
               |      (CORS, Server validation)         |
               +-------------------+--------------------+
                                   |
             +---------------------+---------------------+
             |                                           |
             | If API Key Configured                     | If No Key (Mock Mode)
             v                                           v
+------------+-------------+               +-------------+-------------+
|    PDF Preprocessing     |               |    Mock MTO Generator     |
| (Render to 3x Zoom PNG)  |               | (Returns sample schemas)  |
+------------+-------------+               +-------------+-------------+
             |                                           |
             v                                           |
+------------+-------------+                             |
|    Gemini Vision API     |                             |
| (JSON Structured Output) |                             |
+------------+-------------+                             |
             |                                           |
             v                                           |
+------------+-------------+                             |
|   Pydantic Validation    |                             |
| (Validate schema models) |                             |
+------------+-------------+                             |
             |                                           |
             v                                           |
+------------+-------------+                             |
|  Heuristic Derivations   |                             |
| (Compute bolts/gaskets)  |                             |
+------------+-------------+                             |
             |                                           |
             +---------------------+---------------------+
                                   |
                                   v
               +-------------------+--------------------+
               |         Structured MTO Output          |
               |       (Interactive Table, CSV)         |
               +----------------------------------------+

---

2. Interactive Workspace Screenshot

---

3. Pipeline Design & Technical Approach

Frontend (Next.js 15 App Router)

• User Interface: Designed with custom responsive CSS styling (sleek dark mode, glassmorphism panel styles, and themed badge pills).
• Upload Mechanism: Implemented a drag-and-drop area + file-picker with client-side type and size validation (maximum 20MB).
• Verification & Export: Results are shown side-by-side with the uploaded drawing. The tabular items feature confidence metrics, category color-coding, and a "Export as CSV" button linked directly to the download stream endpoint.

Backend (FastAPI)

• PDF-to-Image Preprocessing: Raw PDFs are intercepted and processed using PyMuPDF (fitz). The first page is rendered to a high-resolution PNG at a 3.0x zoom factor (approx 216 DPI). This is crucial for structural text legibility, enabling the vision model to read tiny print annotations.
• SDK Version & KeyError Safeguard: In older versions of google-generativeai, passing Pydantic models directly to response_schema raises a KeyError: 'properties' because of nested enums or $defs in Pydantic v2. A custom resolve_schema method was implemented to programmatically resolve, flatten, and clean $ref/allOf/anyOf JSON schema properties before passing the config as a dictionary to Gemini.
• Heuristic Derivation: In accordance with piping conventions, the backend automatically computes derived consumables: it appends 1 gasket and 1 bolt set per flanged joint based on the total flange quantities.
• Graceful Mock Fallback: If API_KEY is not present in .env, the pipeline gracefully falls back to returning a schema-valid mock MTO response, keeping the app 100% testable out-of-the-box.

---

4. Step-by-Step Running Guide

System Requirements

• Node.js: v18.0.0 or higher
• Python: v3.10.0 or higher

---

Backend Setup

1. Open your terminal and navigate to the backend directory:

   cd backend

2. Create and activate a Python virtual environment:

   # Create virtual environment
   python -m venv .venv
   
   # Activate on Windows (PowerShell)
   .\.venv\Scripts\Activate.ps1
   
   # Activate on macOS/Linux
   source .venv/bin/activate

3. Install the dependencies:

   pip install -r requirements.txt

4. Create a .env file from the example:

   copy .env.example .env

   Edit .env and fill in your Gemini API key:

   API_KEY=your_gemini_api_key_here

5. Start the FastAPI server using the virtual environment interpreter:

   uvicorn app.main:app --reload

   The backend API will run at http://127.0.0.1:8000. You can inspect the Swagger documentation at http://127.0.0.1:8000/docs.

---

Frontend Setup

1. Open a new terminal and navigate to the frontend directory:

   cd frontend

2. Install the Node packages:

   npm install

3. Create a .env.local file from the example:

   copy .env.example .env.local

   Keep the default backend URL proxy setup:

   BACKEND_API_URL=http://127.0.0.1:8000

4. Run the Next.js development server:

   npm run dev

   Open your browser and navigate to http://localhost:3000 to access the workspace.

---

Running Tests

To run the backend test suite:

cd backend
.\.venv\Scripts\python -m pytest

---

5. Environment Variables Configuration

Backend .env

API_KEY=AIzaSy...   # Google Gemini API key (optional: activates mock fallback if missing)

Frontend .env.local

BACKEND_API_URL=http://127.0.0.1:8000   # FastAPI backend host address

---

6. Assumptions & Known Limitations

Assumptions

1. Single Sheet Scope: Each uploaded file represents one piping isometric drawing sheet.
2. Standard MTO Rules: Pipe items are quantified by total cumulative length in meters (M), while fittings, valves, and flanges are quantified by piece count (EA).
3. Consumable Heuristics: Derives 1 gasket and 1 bolt set per flanged joint directly based on the count of flanged connections.

Limitations

1. Dense Layouts: Hand-drawn isometrics or scans with dense, overlapping lines and very small annotations can reduce extraction accuracy.
2. Text Orientation: Blueprints containing heavily rotated text can occasionally be misread by generic vision models.
3. PDF Scope: Preprocessing renders the first page of multi-sheet PDF documents.

---

7. Future Enhancements

1. Symbol Bounding Boxes: Detect and draw bounding-box highlights over inline components (valves, flanges) on the preview canvas.
2. Job Queue (Async Processing): Shift the extraction pipeline to an asynchronous task queue (e.g., Celery/Redis) with WebSockets to query job status, which is ideal for production and batch uploads.
3. BOM-to-Drawing Verification: Add a validation step that automatically reconciles the drawing's built-in BOM text table against the model's visual symbol extractions, highlighting discrepancies.
4. Excel Export: Add custom styling formatting options during downloads (e.g. .xlsx export with sheets/grouping).