Make the job prove itself.
Rakshak is a voice-first, multimodal evidence companion for safer foreign employment. It connects what a recruiter promised in voice notes, chats, advertisements and videos with what the worker received in contracts, demand letters, receipts, visas and permits.
Gemini 3.7 Flash deeply reads every source and links each extracted fact to an exact quote, page or timestamp. Rakshak then compares the sources, runs versioned Nepal-focused rules, and gives one understandable signal: STOP, CHECK or MATCH. Gemini 3.1 Flash TTS explains the result in natural Nepali or English.
A worker should not have to understand legal language or remember every promise. Rakshak turns scattered evidence into one source-linked claim graph:
Voice + chat + ad + video Contract + receipt + visa
\ /
Gemini 3.7 deep multimodal reading
exact quote · page · time
|
Promise-vs-paper claim graph
|
Read-only tools + deterministic 15-rule shield
|
STOP / CHECK / MATCH + next actions
|
Gemini native Nepali/English voice
A recruiter says QAR 1,400, 8 hours, no fee and a labour visa. Rakshak connects that voice promise to a contract showing QAR 900, 12 hours and a different job, a receipt for Rs 170,000, and a visit visa. The result is not an unexplained AI opinion: every difference opens back to its source.
A multilingual contract checker is not new. Rakshak’s differentiation is the complete evidence journey:
- Promise-to-paper, not paper-only — links informal promises to formal documents.
- Multimodal by design — image, PDF, text, audio and video are first-class evidence.
- Source grounding — every non-empty fact needs an exact quote plus page/timestamp.
- Two-pass Gemini analysis — deep per-source reading, then bundle-wide reconciliation.
- Tools with honesty metadata — agency, destination, wage and fee checks expose snapshot completeness and freshness.
- Deterministic safety signal — Gemini discovers and explains; code owns STOP/CHECK/MATCH.
- Low-literacy action UX — natural Nepali/English voice, icons, large controls and one next action at a time.
- Worker-owned evidence — local hashing, export, vault, timeline and expiring-share architecture.
gemini-3.7-flash runs with thinking_level: "high", strict response JSON, and high media resolution for images/PDFs. It returns:
- normalized descriptive facts;
- exact source anchors;
- speaker and timestamp segments for audio/video;
- Nepali/English source summaries;
- media quality and missing-page/inaudible warnings;
- visible editing signals without declaring fraud;
- prompt-injection detection for hostile text inside evidence;
- focused questions for uncertain fields.
The request uses media-first ordering, no deprecated sampling parameters, header-only API authentication, and store:false.
Gemini receives only validated facts, anchors, deterministic signal and read-only tool outputs. It reasons across the complete case, identifies additional review candidates, generates minimal follow-up questions and prepares a short bilingual narration. It cannot change the deterministic signal or invent source IDs/quotes.
agency_registry— dated partial snapshot; absence never means unlicensed.destination_snapshot— exposes whether the approved-country list is complete.wage_reference— returns unavailable instead of inventing an unverified wage.fee_calculator— auditable arithmetic; rules decide its meaning.
lib/rules.ts runs 15 versioned checks. lib/compare.ts finds promise-versus-paper conflicts. Low-confidence fields can only soften accusations, never strengthen them.
POST /api/speak uses gemini-3.1-flash-tts-preview:
- Nepali: Sulafat (warm)
- English: Iapetus (clear)
- 24 kHz mono WAV
- private browser caching
- device speech fallback when offline
- separate TTS cost estimate
Gemini 3.7 Flash itself accepts audio/video but outputs text; native audio output correctly uses the dedicated Gemini TTS model.
Unzip Rakshak-Offline-Demo.zip and open index.html. Click ९० सेकेन्डको डेमो / Play 90-second demo, accept demo consent, and tap Compare promises with paper.
The offline demo is deterministic and never needs an API key. Native Gemini voice falls back to device speech when there is no server.
Requirements: Node.js 20+, npm, and a billing-enabled Gemini project.
cp .env.example .env.local
# Put GEMINI_API_KEY in .env.local — never in source, frontend code or chat.
npm install
npm run verify:offline
npm run devOpen http://localhost:3000.
Minimal .env.local:
RAKSHAK_PROVIDER=gemini
GEMINI_API_KEY=YOUR_SERVER_SIDE_KEY
RAKSHAK_MODEL=gemini-3.7-flash
RAKSHAK_TTS_MODEL=gemini-3.1-flash-tts-preview
RAKSHAK_TTS_VOICE_NE=Sulafat
RAKSHAK_TTS_VOICE_EN=Iapetus
Do not use an unpaid Gemini service for real contracts, passports or chats. Use a paid Gemini API project or Vertex AI and review the current data-governance terms.
Generated fictional papers are in docs/showcase-assets/.
To generate the fictional recruiter voice note with Gemini TTS after setting the key:
npm run demo:voiceIt creates docs/showcase-assets/01-recruiter-promise-ne.wav. Every company, ID and violation in these assets is fictional and labelled DEMO / NOT VALID.
Multipart fields:
- repeated
files; manifest: JSON array in the same order withid,kind,role,labelNp,labelEn.
Current safe showcase limits: 8 files, 8 MB per media file, 1 MB text, 14 MB total. Use a short video under one minute. The production architecture document explains the large-file/Cloud Storage path.
Response includes:
- validated evidence items and source anchors;
- deterministic assessment and conflicts;
- Gemini bundle review;
- read-only tool checks and freshness;
- per-stage token usage and estimated cost;
- nonfatal review warnings.
{ "language": "ne", "text": "रोक्नुहोस्। अहिले पैसा नतिर्नुहोस्।" }Returns a private-cacheable WAV. The key never reaches the browser.
npm run verify:offlineCurrent release gate covers:
- 10 deterministic engine assertions;
- home, intake, result and vault Chromium rendering;
- API and library type checking;
- 10 provider request/security assertions;
- 7 Gemini TTS/WAV/cost assertions;
- secret and archive integrity scans during packaging.
After installing all project dependencies, also run:
npm run typecheck
npm test
npm run build
npm run qa:layout
npm run qa:screensapp/api/analyze/route.ts multimodal two-pass Gemini pipeline
app/api/speak/route.ts native Gemini bilingual TTS
lib/deep-extraction.ts strict schema, prompt and source validation
lib/gemini-review.ts bundle-wide Gemini reconciliation
lib/verification-tools.ts read-only checks with freshness
lib/providers.ts Gemini Interactions API boundary
lib/compare.ts promise-versus-paper conflict engine
lib/rules.ts deterministic R1–R15 shield
demo/ installable PWA and offline single-file build
docs/showcase-assets/ fictional live-demo evidence
docs/HACKATHON_SHOWCASE_RUNBOOK.md
Gemini API/AI Studio currently do not offer fine-tuning for Gemini 3.7 Flash. Rakshak therefore uses the production-safe alternative: versioned system prompts, few-shot decision examples, strict schemas, source-grounding validation, deterministic tools, golden-case tests and prompt regression gates. Vertex AI tuning can be evaluated later only against a supported model and a consented labelled dataset.
This is a polished hackathon/field-pilot release candidate, not a public legal certification. Before public launch:
- import and legally review the live DOFE agency/demand datasets;
- verify current fee, destination, wage and helpline data;
- add signed large-file uploads, malware/CDR scanning and automatic redaction;
- measure Nepali extraction on consented real evidence;
- complete field usability, screen-reader, penetration and legal reviews;
- contract trained human escalation partners.
See docs/HACKATHON_SHOWCASE_RUNBOOK.md for the exact pitch, click sequence, backup plan and judge Q&A.