Skip to content
BranchesTags

Repository files navigation

Adaptive Real-Time AI Voice Interviewer & Skill Authoring Studio

An end-to-end adaptive AI voice interviewer and collaborative skill authoring studio. The platform combines real-time conversational voice interaction (barge-in / interruptions, sub-second latency) with a structured Markdown-based Skill Authoring subsystem (design.md ➔ AST Parsing ➔ Policy Validation ➔ Executable SKILL package).

🏛️ System Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                    Frontend Studio (React 19 + Vite)                    │
│    - Collaborative Design Studio (Design Agent Chat, Spec Modal)        │
│    - Interactive Markdown Editor & Outline Synchronization               │
│    - Real-Time Interview Simulation & RTVI WebRTC Audio Visualizer      │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │ HTTP Proxy (/api, /start, /sessions)
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                     FastAPI Voice & Authoring Server                    │
│                                                                         │
│  ┌──────────────────────────────┐     ┌──────────────────────────────┐  │
│  │   Skill Authoring Pipeline   │     │    Real-Time Voice Pipeline  │  │
│  │   - DesignSessionManager     │     │    - SmallWebRTC Transport   │  │
│  │   - AST Parser & Symbol Table│     │    - Silero VAD Analyzer     │  │
│  │   - PolicyValidator (Rules)  │     │    - Google Cloud STT        │  │
│  │   - Generator ➔ SkillStore   │     │    - ADK LLM Service (VQL)   │  │
│  │   - Dynamic Projections      │     │    - Google Cloud TTS (VQL)  │  │
│  └──────────────┬───────────────┘     └──────────────┬───────────────┘  │
│                 │                                    │                  │
│                 ▼                                    ▼                  │
│  ┌──────────────────────────────┐     ┌──────────────────────────────┐  │
│  │    Runtime Skill Store       │◀────│    Google ADK Interview      │  │
│  │    (InMemory / FileStore)    │     │    Agent (Gemini 3.5+ Flash) │  │
│  └──────────────────────────────┘     └──────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

Repository Structure

/
├── README.md                   # This root setup & deployment guide
├── AGENTS.md                   # Agent memory, guidance, and architecture contracts
├── InterviewAgent_BE/          # Backend (FastAPI, Pipecat WebRTC, Google ADK)
│   ├── server.py               # FastAPI WebRTC & REST server entrypoint
│   ├── demo.py                 # Multi-turn streaming CLI demo
│   ├── requirements.txt        # Backend dependencies (Python 3.12+)
│   ├── interview_agent/        # Core backend package
│   │   ├── agent.py            # ADK LlmAgent definition
│   │   ├── voice.py            # Pipecat voice pipeline (STT, TTS, VAD, WebRTC)
│   │   ├── runtime_policy.py   # Runtime policy enforcement layer
│   │   ├── skills/             # Runtime Skill & SkillStore models
│   │   └── skill_authoring/    # AST Parser, Validator, Orchestrator, Projections
│   └── tests/                  # Backend unit & integration test suite
└── InterviewAgent_FE/          # Frontend Studio (React 19, Vite, TypeScript, Tailwind)
    ├── package.json            # Frontend dependencies (Node 18+)
    ├── vite.config.ts          # Vite configuration with backend proxy
    └── src/                    # React UI components & authoring APIs

📋 Prerequisites & System Requirements

Before running or deploying the project, ensure you have:

  1. Python: Python 3.12, 3.13, or 3.14 installed.
  2. Node.js: Node.js 18.x or newer and npm installed.
  3. Google Cloud Project:
    • APIs enabled: Vertex AI API, Cloud Speech-to-Text API, and Cloud Text-to-Speech API.
    • Authenticated using Application Default Credentials (ADC) or a Gemini API key.
  4. Audio / Native Dependencies (for local voice pipeline):
    • Linux / Ubuntu: sudo apt-get install -y ffmpeg libasound2-dev libportaudio2 git
    • macOS: brew install ffmpeg portaudio git

💻 Part 1: Local Development Setup & Execution

Step 1: Backend Setup (InterviewAgent_BE)

  1. Open a terminal and navigate to the backend directory:

    cd InterviewAgent_BE

  2. Create and activate a Python virtual environment:

    # Linux / macOS
python3 -m venv .venv
source .venv/bin/activate

# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1

  3. Install backend dependencies:

    pip install --upgrade pip
pip install -r requirements.txt

  4. Configure environment variables in InterviewAgent_BE/.env:

    Option A: Vertex AI with Application Default Credentials (Recommended):

    gcloud auth application-default login

    Create InterviewAgent_BE/.env:

    GOOGLE_GENAI_USE_VERTEXAI=true
GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=us-central1
GEMINI_MODEL=gemini-2.5-flash

    Option B: Direct Gemini API Key: Create InterviewAgent_BE/.env:

    GOOGLE_API_KEY=your_gemini_api_key_here
GEMINI_MODEL=gemini-2.5-flash

  5. Start the backend server:

    python server.py --host 0.0.0.0 --port 7860

    The backend starts at http://localhost:7860.

Step 2: Frontend Setup (InterviewAgent_FE)

  1. Open a second terminal and navigate to the frontend directory:

    cd InterviewAgent_FE

  2. Install Node dependencies:

    npm install

  3. Start the Vite development server:

    npm run dev

    The frontend starts at http://localhost:3000.

Step 3: Accessing the Interfaces & Testing

Interface URL Purpose
Authoring & Simulation Studio http://localhost:3000 Full collaborative skill design studio, markdown editor, validation, and real-time interview simulator.
Standalone WebRTC Voice Client http://localhost:7860/client/ Built-in low-latency RTVI voice test client directly served by FastAPI.
REST & OpenAPI Docs http://localhost:7860/docs Interactive Swagger API documentation for authoring and skill management.

Running Backend Automated Tests

From InterviewAgent_BE/ with .venv active:

python -m unittest discover -s interview_agent/tests -v

Running the Multi-turn CLI Demo

To test dynamic multi-skill switching in a streaming console environment:

python demo.py

☁️ Part 2: Cloud Deployment Guide

Deploying a real-time WebRTC voice system to production requires three specific considerations:

  1. HTTPS (SSL/TLS): Browsers mandate HTTPS for microphone access (getUserMedia).
  2. WebRTC NAT Traversal (STUN / TURN): WebRTC media audio flows over UDP. In production across firewalls and cellular networks, a TURN server (e.g. Coturn or Twilio/Cloudflare TURN) is needed for relay fallback.
  3. Session Affinity / Single-Worker: In-memory WebRTC connection state requires 1 Uvicorn worker process per container instance (or sticky session routing).

Deployment Method A: Docker & Docker Compose (Recommended for VPS / Cloud VM)

This method sets up the Backend, Frontend, and Caddy (automatic Let's Encrypt SSL reverse proxy) on any cloud VM (AWS EC2, GCP Compute Engine, DigitalOcean, Hetzner).

1. Backend Dockerfile (InterviewAgent_BE/Dockerfile)

FROM python:3.12-slim AS builder

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential git ffmpeg libasound2-dev libportaudio2 \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN python -m venv /opt/venv && \
    /opt/venv/bin/pip install --no-cache-dir --upgrade pip && \
    /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim AS runner
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    ffmpeg libasound2 libportaudio2 curl \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1

RUN mkdir -p /data/skills
COPY . /app/

EXPOSE 7860
CMD ["python", "server.py", "--host", "0.0.0.0", "--port", "7860"]

2. Frontend Dockerfile (InterviewAgent_FE/Dockerfile)

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

3. Root docker-compose.yml

version: "3.8"

services:
  backend:
    build:
      context: ./InterviewAgent_BE
      dockerfile: Dockerfile
    restart: unless-stopped
    env_file:
      - ./InterviewAgent_BE/.env
    volumes:
      - skills_storage:/data/skills
    ports:
      - "7860:7860"

  frontend:
    build:
      context: ./InterviewAgent_FE
      dockerfile: Dockerfile
    restart: unless-stopped
    depends_on:
      - backend

  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      - backend
      - frontend

volumes:
  skills_storage:
  caddy_data:
  caddy_config:

4. Caddyfile (Reverse Proxy + Auto SSL)

Replace interview.yourdomain.com with your domain:

interview.yourdomain.com {
    # Proxy API and WebRTC signaling to Backend
    handle /api/* {
        reverse_proxy backend:7860
    }
    handle /start* {
        reverse_proxy backend:7860
    }
    handle /sessions/* {
        reverse_proxy backend:7860
    }
    handle /client/* {
        reverse_proxy backend:7860
    }

    # Serve Frontend UI for all other paths
    handle {
        reverse_proxy frontend:80
    }
}

5. Launch Stack

docker compose up -d --build

Deployment Method B: Google Cloud Run (Serverless Backend)

Cloud Run supports WebSockets and streaming gRPC, making it ideal for hosting the backend with Google Cloud Workload Identity.

  1. Build & Submit Container Image:

    gcloud builds submit InterviewAgent_BE \
  --tag gcr.io/YOUR_PROJECT_ID/interview-backend:latest

  2. Deploy to Cloud Run:

    gcloud run deploy interview-backend \
  --image gcr.io/YOUR_PROJECT_ID/interview-backend:latest \
  --platform managed \
  --region us-central1 \
  --allow-unauthenticated \
  --port 7860 \
  --cpu 2 \
  --memory 2Gi \
  --session-affinity \
  --set-env-vars GOOGLE_GENAI_USE_VERTEXAI=true,GOOGLE_CLOUD_PROJECT=YOUR_PROJECT_ID,GOOGLE_CLOUD_LOCATION=us-central1,GEMINI_MODEL=gemini-2.5-flash

  3. Deploy Frontend to Firebase Hosting or Cloudflare Pages:

    • Run npm run build in InterviewAgent_FE.
    • Configure rewrite rules or environment variables to point /api, /start, and /sessions to your Cloud Run service URL.

🛠️ Production Configuration & Hardening Checklist

Area Production Requirement Action / Setting
Authentication Google Cloud IAM Attach a service account with roles/aiplatform.user, roles/speech.client, and roles/texttospeech.client.
CORS Policy Restrict frontend origins In server.py, replace allow_origins=["*"] with your specific production domain (e.g. https://interview.yourdomain.com).
WebRTC Media Relay NAT / Firewall Traversal Provide STUN & TURN servers via /start response for candidates behind corporate firewalls.
Process Model WebRTC state affinity Use 1 Uvicorn worker process per container instance to ensure in-memory WebRTC state consistency.
Storage Persistence Custom skill persistence Mount a persistent volume at /data/skills for FileSkillStore to retain compiled skills across deployments.
Rate Limit Protection Gemini / Vertex AI quotas Monitor API quotas in GCP Console and handle rate limits (429) with exponential backoff.

❓ Troubleshooting

  • Microphone not working: Ensure the page is served over https:// (or http://localhost). Browsers strictly block audio capture on unencrypted HTTP.
  • WebRTC Connection Stuck: Check if the client or server is behind a symmetric NAT without a TURN server. Ensure STUN/TURN ICE candidate exchange succeeds.
  • Google Cloud Auth Errors (403 / 401): Run gcloud auth application-default login locally, or check that your production service account has roles/aiplatform.user attached.
  • Port Conflicts: Backend defaults to port 7860 and Frontend defaults to port 3000. Adjust with --port flag or Vite config if ports are occupied.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors