ClauseIQ — Contract Risk Analyzer

Upload a contract PDF → get an automated risk report: clause-by-clause risk findings, contract-type detection, extracted parties/dates/values, an auto-generated executive summary, and automatic renewal-deadline reminders. Analysis runs through a rule engine (V1) refined by a locally-trained ML classifier (V2). No external AI/LLM API keys used anywhere, and admins can extend detection coverage from the panel without a code deploy.

Architecture (Use Python 3 not Python 2 at all in workers)

clauseiq/                       (npm workspaces monorepo — services still deploy independently)
├── frontend/                   Next.js 14 + TypeScript. Talks to both microservices
│                                through next.config.mjs rewrites() — no separate API gateway.
├── services/
│   ├── auth-service/           Node + Express. Signup/login/OTP/JWT/sessions, admin auth.
│   └── core-service/           Node + Express. Contracts, uploads, risk reports, custom rules,
│                                ML feedback loop, admin analytics, BullMQ producers.
├── workers/                    Python. BullMQ consumers, concurrency 1 each.
│   ├── pdf_parser_worker/          PDF -> per-page text (+ OCR fallback for scans)
│   ├── clause_extractor_worker/    boilerplate-stripped text -> individual clauses
│   ├── document_classifier_worker/ contract type detection (NDA/MSA/Lease/...)
│   ├── rule_engine_worker/         V1: deterministic + admin custom rules + compound risk
│   ├── ml_classifier_worker/       V2: calibrated scikit-learn classifier (local, no API key)
│   ├── entity_extraction_worker/   parties/dates/values (spaCy NER + regex) + reminders
│   ├── summary_worker/             local TextRank executive summary (final stage)
│   └── reminder_worker/            daily BullMQ repeatable job — dispatches due reminders
├── packages/
│   └── db/                     Shared Prisma schema/migrations/seed — single source of truth.
└── infra/
    ├── docker-compose.yml      Full stack: Postgres, Redis, both services, frontend, all 8 workers.
    └── kubernetes/             Ready-to-apply k3s + KEDA manifests (see infra/kubernetes/README.md).

Why one Prisma schema for everything: both auth-service and core-service import @clauseiq/db as a workspace dependency so there is exactly one migration history and one generated Prisma Client, even though the two services run as separate processes (and, in production, separate deployments).

The pipeline (8 stages, all Python workers, concurrency=1 each):

Upload → PDF_PARSING → CLAUSE_EXTRACTION → DOCUMENT_CLASSIFICATION → RULE_ANALYSIS
       → ML_CLASSIFICATION → ENTITY_EXTRACTION → SUMMARY_GENERATION → COMPLETED

Each worker enqueues the next stage itself on success; REMINDER_DISPATCH runs separately as a daily BullMQ repeatable job (no external cron needed) rather than being part of the per-contract pipeline. queue_runner.py distinguishes fatal errors (bad input — fails immediately via bullmq.UnrecoverableError, no wasted retries) from recoverable ones (network blips — normal backoff/retry applies).

What makes this more than a V1 demo

• Contract-type awareness — a fast, explainable keyword classifier tags each contract (NDA/MSA/Employment/Lease/SaaS/Sales/Partnership), which the rule engine uses to weight which risk categories matter most for that document type.
• Compound risk detection — some rule combinations are riskier together than either finding alone (e.g. auto-renewal + no termination clause = no exit path at all). A dedicated compound-rules module flags these explicitly.
• Admin-extensible rule engine — admins can add custom regex-based detection rules from the panel (/admin/custom-rules); they're merged into every subsequent analysis run with zero code changes or redeploys.
• Active learning loop — admins can correct a clause's ML classification directly from the contract detail page; the correction is written back into the training set (ADMIN_CURATED) and improves the next model retrain. Training runs are logged with accuracy/F1 metrics, visible on /admin/ml-health.
• Entity extraction & reminders — parties, effective date, term, governing law, and monetary values are extracted automatically (spaCy NER + regex/dateutil, fully local). When a renewal-notice deadline can be derived, a reminder is scheduled and emailed automatically ahead of time.
• Executive summaries — a local TextRank algorithm (no LLM) picks the most structurally important clauses to generate a short summary automatically.
• OCR fallback — scanned/image-only PDFs are handled via an optional Tesseract OCR path; without it installed, the system fails with a clear, actionable message instead of a crash.
• Duplicate detection — file checksums are computed at upload time, so re-uploading the same contract is flagged immediately.

Build phases — all complete

• Phase 1 — Foundation: monorepo scaffold, Prisma schema, docker-compose, seed script.
• Phase 2 — auth-service: signup/login/OTP (console fallback), JWT access+refresh, sessions, password reset, admin auth, Zod validation, Redis-backed rate limiting.
• Phase 3 — core-service: contract upload (Cloudinary), BullMQ producer, risk report + knowledge base endpoints, admin analytics + queue-health endpoints.
• Phase 4 — Python workers: pdf_parser, clause_extractor, rule_engine (V1), ml_classifier (V2, local scikit-learn — no external API keys anywhere).
• Phase 5 — Next.js frontend: 4-theme system, full auth flow, dashboard with upload + live status polling, risk report page with charts, public knowledge base, navbar/footer/FAB.
• Phase 6 — Admin panel: AdminSidebar + AdminNavbar (synced via SidebarContext, nested dropdowns, collapse/expand), analytics dashboard with date-range filtering, user management (block/unblock/roles), contracts overview, queue health, audit logs, knowledge base CRUD.
• Phase 7 — SEO & polish: metadata on every page, robots.ts, sitemap.ts (placeholder domain — update NEXT_PUBLIC_SITE_URL after your first deployment).
• Infrastructure: Dockerfiles for all 4 image types, a full Docker Compose stack (one command runs everything, auto-restart on crash), CI (typecheck/lint on every push) and a manual CD workflow (build + push images to GHCR) via GitHub Actions, and a complete, ready-to-apply Kubernetes + KEDA manifest set for when you outgrow a single server.

Running it

Recommended: Docker Compose (one command, everything supervised)

cd infra
cp .env.example .env    # fill in Cloudinary + generate real JWT secrets — see infra/README.md
docker compose up -d --build

That's Postgres, Redis, both Node services, the frontend, and all 8 Python workers — each auto-restarted by Docker if it crashes, no manual terminals. First-time migration

• seed steps and full details are in infra/README.md.

Outgrowing one server? infra/kubernetes/README.md is a complete, ready-to-apply Kubernetes + KEDA manifest set (queue-depth-based autoscaling) with a full step-by-step migration guide — same Docker images, no code changes needed.

Alternative: run natively (best for active development / hot-reload)

cd clauseiq
npm install                                   # installs frontend + both services (npm workspaces)
pip install -r workers/requirements.txt --break-system-packages   # or use a venv
python -m spacy download en_core_web_sm       # powers better sentence splitting + party extraction
                                                # (both workers fall back gracefully if this is skipped)

# Environment files — copy every .env.example and fill in real values
cp packages/db/.env.example packages/db/.env
cp services/auth-service/.env.example services/auth-service/.env
cp services/core-service/.env.example services/core-service/.env
cp workers/.env.example workers/.env
cp frontend/.env.example frontend/.env.local
# IMPORTANT: JWT_ACCESS_SECRET must be identical in auth-service/.env and core-service/.env

npm run docker:up          # Postgres + Redis only (docker compose from infra/, just the data layer)
npm run db:migrate
npm run db:seed            # super admin + ML seed data + knowledge base (check console for admin password)

# In separate terminals:
npm run dev:auth           # auth-service        → :5001
npm run dev:core           # core-service         → :5002
npm run dev:frontend       # Next.js frontend     → :3000
npm run dev:workers        # python workers  (all started at once from main root folder) 
   
# IMP : Use only python-3 version, not python-2 at all

# Python workers (if needed seperately, each is its own process, concurrency=1) — from workers/:
cd workers
python pdf_parser_worker/worker.py
python clause_extractor_worker/worker.py
python document_classifier_worker/worker.py
python rule_engine_worker/worker.py
python ml_classifier_worker/worker.py
python entity_extraction_worker/worker.py
python summary_worker/worker.py
python reminder_worker/worker.py

Either way: visit http://localhost:3000, sign up, verify the OTP (printed to the auth-service console/logs if SendGrid isn't configured), upload a PDF contract, and watch the dashboard update live as the eight workers process it end to end — status, document type, key facts, executive summary, and risk findings all populate progressively as each stage completes.

Testing notes from development

This project was developed in a sandboxed environment without access to binaries.prisma.sh, so prisma generate/prisma migrate couldn't be run through the Prisma CLI itself — the schema was hand-translated to raw SQL and applied directly to a real Postgres instance instead, which is a meaningful structural check (it created cleanly, all foreign keys and enums resolved) but isn't a substitute for running the real migration yourself first thing.

Everything else was verified for real, not just read over:

• Both Node services' TypeScript compiles cleanly (tsc --noEmit).
• The full Next.js app builds successfully in production mode across all 26 routes.
• A real Postgres + Redis + the actual bullmq package were stood up in this sandbox, and every Python worker's core logic (.handle() functions) was run against them with a real synthetically-generated PDF — pdfplumber extraction, boilerplate stripping, clause segmentation, document classification, rule matching (including compound-risk detection), ML training/calibration/classification, entity extraction, and TextRank summarization all ran end-to-end and produced correct output. Several real bugs were found and fixed this way (a regex that didn't handle "thirty (30) days" phrasing, an eagerly-connecting DB pool that would crash workers on slow Postgres startup, spaCy NER false-positives on capitalized clause headings, and a whitespace bug that let PDF line-wrapping leak into extracted governing-law text).

Run npm run db:generate && npm run db:migrate yourself early on — if anything doesn't apply cleanly against the real Prisma CLI, it's most likely a minor schema syntax adjustment, not a structural issue, given the hand-translated version was validated.