Engineering Component Extractor

A lightweight proof-of-concept for extracting structured engineering component information from schematics, diagrams, and equipment nameplates using Google's Gemini multimodal model.

Instead of allowing the LLM to generate free-form text, this project validates every response against a strict schema before presenting it to the user, providing a safer and more reliable extraction pipeline.

---

Features

• Upload engineering drawings, schematics, or equipment images
• Extract structured component information using Gemini 2.5 Flash
• Validate outputs with Pydantic models
• Interactive Streamlit interface
• FastAPI backend for API-based integration
• JSON output suitable for downstream automation

---

Project Architecture

                Engineering Image
                       │
                       ▼
            Streamlit Frontend (app.py)
                       │
          POST /extract (Multipart Image)
                       │
                       ▼
            FastAPI Backend (main.py)
                       │
                       ▼
      Gemini 2.5 Flash Vision Model
                       │
                JSON Response
                       │
                       ▼
        Pydantic Validation (schema.py)
                       │
             Validated Component Data
                       │
                       ▼
        Interactive Table in Streamlit

---

Project Structure

engineering-extractor/
│
├── app.py              # Streamlit frontend
├── main.py             # FastAPI server
├── extractor.py        # Gemini inference logic
├── schema.py           # Pydantic validation models
├── requirements.txt
└── README.md

---

Technology Stack

Component	Technology
Backend	FastAPI
Frontend	Streamlit
AI Model	Gemini 2.5 Flash
Validation	Pydantic v2
Image Processing	Pillow
API	Google GenAI SDK

---

Data Schema

Each detected engineering component is validated against the following structure:

Component(
    tag_id: str,
    component_type: str,
    spec_value: str,
    unit: str,
    connections: List[str]
)

Any response that does not conform to this schema is rejected before reaching the frontend.

---

Installation

Prerequisites

• Python 3.10+
• Google AI Studio API Key

---

1. Clone the Repository

git clone <repository-url>

cd engineering-extractor

---

2. Create a Virtual Environment

Windows

python -m venv venv

venv\Scripts\activate

macOS / Linux

python3 -m venv venv

source venv/bin/activate

---

3. Install Dependencies

pip install fastapi \
uvicorn \
pydantic \
google-genai \
pillow \
python-multipart \
streamlit \
requests

Running the Application

This project consists of two services.

Terminal 1 — FastAPI Backend

uvicorn main:app --reload

Runs on:

http://localhost:8000

---

Terminal 2 — Streamlit Frontend

streamlit run app.py

Runs on:

http://localhost:8501

---

Workflow

1. Upload an engineering drawing.
2. The image is sent to the FastAPI backend.
3. Gemini analyzes the image.
4. The model returns structured JSON.
5. Pydantic validates the response.
6. Valid data is displayed in Streamlit.

---

Current Limitations

1. No Ground Truth Verification

The system has no reference engineering database or asset registry.

As a result, it cannot automatically determine:

• whether every component was extracted,
• whether an object was missed,
• or whether the extracted values are factually correct.

The application validates structure, not semantic correctness.

---

2. LLM-Based Connectivity Detection

The connections field is inferred entirely from Gemini's visual reasoning.

No computer vision techniques (such as OpenCV line tracing or graph extraction) are used to verify physical wiring or pipe connectivity.

---

3. Image Quality Dependency

Extraction accuracy depends heavily on input quality.

Blurred, low-resolution, poorly scanned, or low-contrast engineering drawings may produce incorrect identifiers or specification values.

---

Future Improvements

• OpenCV-based line tracing
• OCR-assisted extraction
• Engineering symbol detection
• Asset registry integration
• Confidence scoring
• Batch image processing
• Export to CSV / Excel
• Human-in-the-loop verification
• Support for P&ID and electrical CAD drawings

---

Example Use Cases

• Engineering drawing digitization
• Equipment inventory generation
• P&ID preprocessing
• Legacy documentation modernization
• Industrial asset extraction
• Engineering data pipelines

---

Disclaimer

This project is intended as a proof-of-concept demonstrating multimodal information extraction with schema validation.

It is not intended for safety-critical engineering workflows without additional verification mechanisms.