Skip to content
limitlessneverdiesPublic

About

Scam shield for Nepali migrant workers: checks recruiter promises against contracts, says STOP / CHECK / MATCH in Nepali. Gemini multimodal.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

रक्षक · Rakshak — Gemini Edition

Make the job prove itself.

Rakshak is a voice-first, multimodal evidence companion for safer foreign employment. It connects what a recruiter promised in voice notes, chats, advertisements and videos with what the worker received in contracts, demand letters, receipts, visas and permits.

Gemini 3.7 Flash deeply reads every source and links each extracted fact to an exact quote, page or timestamp. Rakshak then compares the sources, runs versioned Nepal-focused rules, and gives one understandable signal: STOP, CHECK or MATCH. Gemini 3.1 Flash TTS explains the result in natural Nepali or English.

The core idea

A worker should not have to understand legal language or remember every promise. Rakshak turns scattered evidence into one source-linked claim graph:

Voice + chat + ad + video           Contract + receipt + visa
             \                              /
              Gemini 3.7 deep multimodal reading
                 exact quote · page · time
                          |
                Promise-vs-paper claim graph
                          |
        Read-only tools + deterministic 15-rule shield
                          |
            STOP / CHECK / MATCH + next actions
                          |
             Gemini native Nepali/English voice

The hackathon “wow” moment

A recruiter says QAR 1,400, 8 hours, no fee and a labour visa. Rakshak connects that voice promise to a contract showing QAR 900, 12 hours and a different job, a receipt for Rs 170,000, and a visit visa. The result is not an unexplained AI opinion: every difference opens back to its source.

Why this is different

A multilingual contract checker is not new. Rakshak’s differentiation is the complete evidence journey:

  1. Promise-to-paper, not paper-only — links informal promises to formal documents.
  2. Multimodal by design — image, PDF, text, audio and video are first-class evidence.
  3. Source grounding — every non-empty fact needs an exact quote plus page/timestamp.
  4. Two-pass Gemini analysis — deep per-source reading, then bundle-wide reconciliation.
  5. Tools with honesty metadata — agency, destination, wage and fee checks expose snapshot completeness and freshness.
  6. Deterministic safety signal — Gemini discovers and explains; code owns STOP/CHECK/MATCH.
  7. Low-literacy action UX — natural Nepali/English voice, icons, large controls and one next action at a time.
  8. Worker-owned evidence — local hashing, export, vault, timeline and expiring-share architecture.

Gemini architecture

Pass 1 — Deep evidence reading

gemini-3.7-flash runs with thinking_level: "high", strict response JSON, and high media resolution for images/PDFs. It returns:

  • normalized descriptive facts;
  • exact source anchors;
  • speaker and timestamp segments for audio/video;
  • Nepali/English source summaries;
  • media quality and missing-page/inaudible warnings;
  • visible editing signals without declaring fraud;
  • prompt-injection detection for hostile text inside evidence;
  • focused questions for uncertain fields.

The request uses media-first ordering, no deprecated sampling parameters, header-only API authentication, and store:false.

Pass 2 — Whole-case reconciliation

Gemini receives only validated facts, anchors, deterministic signal and read-only tool outputs. It reasons across the complete case, identifies additional review candidates, generates minimal follow-up questions and prepares a short bilingual narration. It cannot change the deterministic signal or invent source IDs/quotes.

Verification tools

  • agency_registry — dated partial snapshot; absence never means unlicensed.
  • destination_snapshot — exposes whether the approved-country list is complete.
  • wage_reference — returns unavailable instead of inventing an unverified wage.
  • fee_calculator — auditable arithmetic; rules decide its meaning.

Safety shield

lib/rules.ts runs 15 versioned checks. lib/compare.ts finds promise-versus-paper conflicts. Low-confidence fields can only soften accusations, never strengthen them.

Native voice

POST /api/speak uses gemini-3.1-flash-tts-preview:

  • Nepali: Sulafat (warm)
  • English: Iapetus (clear)
  • 24 kHz mono WAV
  • private browser caching
  • device speech fallback when offline
  • separate TTS cost estimate

Gemini 3.7 Flash itself accepts audio/video but outputs text; native audio output correctly uses the dedicated Gemini TTS model.

Start in 30 seconds — offline showcase

Unzip Rakshak-Offline-Demo.zip and open index.html. Click ९० सेकेन्डको डेमो / Play 90-second demo, accept demo consent, and tap Compare promises with paper.

The offline demo is deterministic and never needs an API key. Native Gemini voice falls back to device speech when there is no server.

Live setup

Requirements: Node.js 20+, npm, and a billing-enabled Gemini project.

cp .env.example .env.local
# Put GEMINI_API_KEY in .env.local — never in source, frontend code or chat.
npm install
npm run verify:offline
npm run dev

Open http://localhost:3000.

Minimal .env.local:

RAKSHAK_PROVIDER=gemini
GEMINI_API_KEY=YOUR_SERVER_SIDE_KEY
RAKSHAK_MODEL=gemini-3.7-flash
RAKSHAK_TTS_MODEL=gemini-3.1-flash-tts-preview
RAKSHAK_TTS_VOICE_NE=Sulafat
RAKSHAK_TTS_VOICE_EN=Iapetus

Do not use an unpaid Gemini service for real contracts, passports or chats. Use a paid Gemini API project or Vertex AI and review the current data-governance terms.

Prepare the live demo evidence

Generated fictional papers are in docs/showcase-assets/.

To generate the fictional recruiter voice note with Gemini TTS after setting the key:

npm run demo:voice

It creates docs/showcase-assets/01-recruiter-promise-ne.wav. Every company, ID and violation in these assets is fictional and labelled DEMO / NOT VALID.

API

POST /api/analyze

Multipart fields:

  • repeated files;
  • manifest: JSON array in the same order with id, kind, role, labelNp, labelEn.

Current safe showcase limits: 8 files, 8 MB per media file, 1 MB text, 14 MB total. Use a short video under one minute. The production architecture document explains the large-file/Cloud Storage path.

Response includes:

  • validated evidence items and source anchors;
  • deterministic assessment and conflicts;
  • Gemini bundle review;
  • read-only tool checks and freshness;
  • per-stage token usage and estimated cost;
  • nonfatal review warnings.

POST /api/speak

{ "language": "ne", "text": "रोक्नुहोस्। अहिले पैसा नतिर्नुहोस्।" }

Returns a private-cacheable WAV. The key never reaches the browser.

Verification

npm run verify:offline

Current release gate covers:

  • 10 deterministic engine assertions;
  • home, intake, result and vault Chromium rendering;
  • API and library type checking;
  • 10 provider request/security assertions;
  • 7 Gemini TTS/WAV/cost assertions;
  • secret and archive integrity scans during packaging.

After installing all project dependencies, also run:

npm run typecheck
npm test
npm run build
npm run qa:layout
npm run qa:screens

Project map

app/api/analyze/route.ts      multimodal two-pass Gemini pipeline
app/api/speak/route.ts        native Gemini bilingual TTS
lib/deep-extraction.ts        strict schema, prompt and source validation
lib/gemini-review.ts          bundle-wide Gemini reconciliation
lib/verification-tools.ts     read-only checks with freshness
lib/providers.ts              Gemini Interactions API boundary
lib/compare.ts                promise-versus-paper conflict engine
lib/rules.ts                  deterministic R1–R15 shield
demo/                         installable PWA and offline single-file build
docs/showcase-assets/         fictional live-demo evidence
docs/HACKATHON_SHOWCASE_RUNBOOK.md

Fine-tuning reality

Gemini API/AI Studio currently do not offer fine-tuning for Gemini 3.7 Flash. Rakshak therefore uses the production-safe alternative: versioned system prompts, few-shot decision examples, strict schemas, source-grounding validation, deterministic tools, golden-case tests and prompt regression gates. Vertex AI tuning can be evaluated later only against a supported model and a consented labelled dataset.

Honest launch boundary

This is a polished hackathon/field-pilot release candidate, not a public legal certification. Before public launch:

  1. import and legally review the live DOFE agency/demand datasets;
  2. verify current fee, destination, wage and helpline data;
  3. add signed large-file uploads, malware/CDR scanning and automatic redaction;
  4. measure Nepali extraction on consented real evidence;
  5. complete field usability, screen-reader, penetration and legal reviews;
  6. contract trained human escalation partners.

See docs/HACKATHON_SHOWCASE_RUNBOOK.md for the exact pitch, click sequence, backup plan and judge Q&A.

About

Scam shield for Nepali migrant workers: checks recruiter promises against contracts, says STOP / CHECK / MATCH in Nepali. Gemini multimodal.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages