InterviewIQ

InterviewIQ is an AI mock interviewer that runs a live practice interview in the browser. It generates interview questions with Google Gemini, reads your webcam feed frame by frame to track body language, and returns written feedback on both your answers and your delivery.

3rd place at MadData 2025, UW-Madison's annual data science hackathon.

What it does

You pick an interview type, grant camera access, and the app walks you through a session in three stages.

1. Setup. You choose Technical, Behavioral, or System Design. The backend asks Gemini for five questions of that type and speaks the first one aloud.

2. Live interview. Your webcam feed plays in the browser while a timer runs. The browser captures frames and posts them to the backend, which analyses each frame and returns behavioural metrics that update live on screen. You answer in a chat box by typing or by pressing the microphone button, and each answer is sent to Gemini for immediate written feedback. You can move back and forth between the five questions.

3. Review. When you end the interview the backend sends your answers to Gemini and returns a structured performance analysis with an overall score. See the known limitations below for how that score is currently calculated.

Features

• Question generation for Technical, Behavioral, and System Design interviews using Gemini, with a hardcoded fallback question bank if the API call fails.
• Per-answer feedback from Gemini, delivered as chat messages and read aloud through text to speech.
• Real time computer vision analysis of the webcam feed using MediaPipe and OpenCV.
• Three live behavioural signals computed during the interview: smile intensity, posture, and hand fidgeting.
• Eye contact and crossed arms detection in the standalone detector, which the web app has a slot for but does not yet populate.
• Smile scoring built from several facial measurements at once: mouth width, mouth height, corner elevation, teeth visibility, and smile line depth, all compared against a baseline neutral expression captured at the start.
• Posture scoring from the vertical relationship between the nose and the shoulder midpoint, with cumulative slumped time tracked across the session.
• Fidget detection from frame to frame wrist movement, with a cooldown so a single gesture is not counted repeatedly.
• Speech to text through the SpeechRecognition library and text to speech through pyttsx3.
• Answer similarity scoring in backend/modules/evaluation_module.py that compares your answer against a Gemini generated ideal answer using TF-IDF cosine similarity, Jaccard overlap, fuzzy string matching, sentence embedding similarity from all-MiniLM-L6-v2, Flesch reading ease, and ROUGE-1 and ROUGE-L.
• Per frame metric storage in MongoDB, with automatic fallback to local JSON files when MongoDB is not reachable.
• End of session summary with an overall score and a structured written analysis.

Architecture

Backend

The backend is a Flask app in backend/app.py. It serves the frontend directly and exposes a small JSON API. CORS is enabled and the development server runs on port 5004.

Route	Method	Purpose
/	GET	Serves the single page frontend
/api/start_session	POST	Creates a session and returns five Gemini generated questions
/api/analyze_frame	POST	Takes a base64 JPEG frame and returns behavioural metrics
/api/evaluate_answer	POST	Sends one answer to Gemini and returns written feedback
/api/speech_to_text	POST	Records from the microphone and returns a transcript
/get_message	GET	Same speech to text capture, used by the microphone button
/api/print_summary	POST	Returns improvement tips for the session
/api/end_session	POST	Returns the final metrics, overall score, and Gemini analysis

The vision work lives in backend/modules/behavior_analysis.py. BehaviorAnalyzer combines three MediaPipe pipelines.

• FacialAnalyzer uses MediaPipe Face Mesh with refined landmarks to measure smile intensity.
• PostureAnalyzer uses MediaPipe Pose to classify upright versus slumped posture and accumulate slumped time.
• HandAnalyzer uses MediaPipe Hands to detect fidgeting from wrist movement between frames.

Supporting modules:

• backend/modules/question_generator.py wraps Gemini question generation and holds the fallback question bank.
• backend/modules/evaluation_module.py holds the answer similarity metrics and the feedback prompt templates.
• backend/modules/speech_processor.py wraps speech to text and text to speech.
• backend/modules/behavior_database.py writes frame records and session summaries to MongoDB, falling back to local JSON files.
• backend/modules/behavior_detector.py and backend/modules/main.py provide a standalone OpenCV window version of the behaviour tracking, useful for testing the vision pipeline without the web app.
• backend/modules/interview_manager.py and backend/database/db_manager.py hold a more structured session and scoring layer that app.py does not currently use.
• backend/config.py centralises configuration and thresholds read from environment variables.

Frontend

The frontend is plain HTML, CSS, and vanilla JavaScript ES modules. There is no framework and no build step, so the files are served as they are by Flask.

• frontend/templates/index.html holds all three stages of the interface as sections that are shown and hidden.
• frontend/static/js/app.js is the entry point and wires the pieces together.
• frontend/static/js/interview.js drives the session: camera access, timer, question navigation, chat, and the review screen.
• frontend/static/js/video-processor.js draws each video frame to an offscreen canvas, encodes it as a JPEG data URL, and posts it to /api/analyze_frame.
• frontend/static/css/styles.css holds all styling.

Running it locally

Prerequisites

• Python 3.8 or newer.
• A webcam and a microphone.
• A Google Gemini API key from Google AI Studio.
• MongoDB running locally on the default port, which is optional because the app falls back to local JSON files without it.

Setup

Clone the repository and create a virtual environment.

git clone https://github.com/Arunjay4213/AI_Interviewer.git
cd AI_Interviewer
python -m venv venv
source venv/bin/activate   # on Windows: venv\Scripts\activate

Install the dependencies.

pip install -r backend/requirements.txt

The pinned versions in backend/requirements.txt are from early 2025. On newer Python versions you may need to relax the pins, in particular for numpy, opencv-python, and mediapipe.

Environment variables

Create a .env file in the repository root.

GEMINI_API_KEY=your_gemini_api_key_here

GEMINI_API_KEY is the only required variable. The remaining variables below are read by backend/config.py and all have defaults.

Variable	Default	Purpose
GEMINI_API_KEY	none, required	Google Gemini API key
SECRET_KEY	your-secret-key-here	Flask secret key
FLASK_DEBUG	False	Enables Flask debug mode
MONGO_URI	mongodb://localhost:27017/	MongoDB connection string
DB_NAME	interview_db	MongoDB database name
MAX_INTERVIEW_DURATION	3600	Maximum interview length in seconds
FRAME_ANALYSIS_FPS	30	Target frame analysis rate
UPLOAD_FOLDER	uploads	Directory for uploaded files
MAX_CONTENT_LENGTH	16777216	Maximum request size in bytes
ENABLE_SPEECH_RECOGNITION	True	Feature flag for speech to text
ENABLE_VIDEO_ANALYSIS	True	Feature flag for webcam analysis
SMILE_THRESHOLD	0.3	Smile detection threshold
FIDGET_THRESHOLD	0.1	Fidget detection threshold
POSTURE_THRESHOLD	0.15	Posture detection threshold

Start the app

Run the Flask app from inside the backend directory, because the modules are imported as modules.* relative to that directory.

cd backend
python app.py

Open http://127.0.0.1:5004 in your browser and allow camera access when prompted.

Running the vision pipeline on its own

To watch the behaviour tracking without the web interface, run the standalone loop, which opens an OpenCV window and prints a session summary when you press q.

from modules.main import out
out()

Known limitations

This was built in a hackathon weekend, so a few rough edges are worth knowing about before you run it.

• Speech to text and text to speech run on the machine hosting the Flask server, not in the browser. Locally that is your own machine and it works, but the microphone button and spoken questions will not work if you deploy the backend to a remote host.
• The overall score returned by /api/end_session is computed from placeholder behavioural values hardcoded in app.py rather than from the metrics collected during the session. The live metrics shown during the interview are real, but they are not yet fed into the final score.
• /api/print_summary calls generate_improvement_tips with arguments that do not match its signature, so that route will fail.
• The web app uses BehaviorAnalyzer from behavior_analysis.py, which does not return an eye_contact value, so the eye contact readout in the interface stays empty. The eye contact logic exists in behavior_detector.py and is only reachable through the standalone loop in modules/main.py.
• Sessions are held in an in-process dictionary, so restarting the server drops any interview in progress.
• The frontend polls the analysis endpoint at roughly 30 frames per second, which is heavy for both the network and the MediaPipe pipeline. Lowering that interval in frontend/static/js/video-processor.js makes the app noticeably smoother.

Repository layout

backend/
  app.py                  Flask app and JSON API
  config.py               Configuration and thresholds
  requirements.txt        Python dependencies
  database/               Database helpers
  modules/
    behavior_analysis.py  MediaPipe face, pose, and hand analysis
    behavior_detector.py  Standalone detector variant
    behavior_database.py  MongoDB storage with file fallback
    evaluation_module.py  Answer similarity metrics and feedback prompts
    interview_manager.py  Session and scoring helpers
    main.py               Standalone OpenCV behaviour loop
    question_generator.py Gemini question generation and fallbacks
    speech_processor.py   Speech to text and text to speech
frontend/
  templates/index.html    Single page interface
  static/js/              Vanilla JavaScript ES modules
  static/css/styles.css   Styling

Acknowledgements

Built at MadData 2025, the annual data science hackathon at the University of Wisconsin-Madison, where it placed 3rd.