demo_video.mp4
An AI-powered screen guidance tool. You describe a goal, share your screen, and Dristi walks you through it step by step — analyzing screenshots, verifying progress, and answering questions along the way.
Stack: FastAPI + React (Vite/TypeScript)
AI: OpenAI gpt-5 for instructions, Google Gemini gemini-2.5-flash for verification
- User enters a goal (e.g. "Open Chrome and go to GitHub")
- Frontend captures the screen via the Screen Share API
- Backend analyzes the screenshot and returns the next instruction
- After the user acts, the frontend detects screen changes and sends before/after screenshots to verify the step was completed
- Cycle repeats until the goal is marked
done - User can ask questions at any point — the AI answers in context
- Python 3.12+
- Node.js 18+
- An OpenAI API key
- A Google Gemini API key
git clone <repo-url>
cd dristipython -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r backend/requirements.txtCreate a .env file in the project root:
OPENAI_API_KEY=your_openai_key_here
GEMINI_API_KEY=your_gemini_key_here
# Optional — defaults shown
OPENAI_MODEL=gpt-4o
GEMINI_MODEL=gemini-2.5-flashcd frontend
npm installOpen two terminals:
Terminal 1 — Backend
source .venv/bin/activate
uvicorn backend.main:app --reloadAPI runs at http://localhost:8000
Terminal 2 — Frontend
cd frontend
npm run devApp runs at http://localhost:5173
Base URL: http://localhost:8000
All request/response bodies are JSON.
Start a new guidance session.
Request
{
"goal": "Open Chrome and navigate to GitHub"
}Response
{
"session_id": "550e8400-e29b-41d4-a716-446655440000"
}Analyze the current screenshot and get the next instruction.
Request
{
"session_id": "550e8400-e29b-41d4-a716-446655440000",
"screenshot": "<base64-encoded PNG>"
}Response
{
"instruction": "Click the Chrome icon in the taskbar",
"step_number": 1,
"done": false
}When done is true, the goal has been completed.
Verify whether the last instruction was successfully completed by comparing before/after screenshots.
Request
{
"session_id": "550e8400-e29b-41d4-a716-446655440000",
"before_screenshot": "<base64-encoded PNG>",
"after_screenshot": "<base64-encoded PNG>"
}Response
{
"completed": true
}Ask a question in the context of the current session.
Request
{
"session_id": "550e8400-e29b-41d4-a716-446655440000",
"screenshot": "<base64-encoded PNG>",
"question": "Why is this step needed?"
}Response
{
"answer": "This step opens the browser so you can navigate to the website."
}End a session and clean up its state.
Response
{
"success": true
}dristi/
├── backend/
│ ├── main.py # FastAPI app, CORS config
│ ├── config.py # Loads env vars
│ ├── routes/
│ │ └── session.py # All API route handlers
│ ├── models/
│ │ └── schemas.py # Pydantic request/response models
│ ├── services/
│ │ ├── openai_service.py # Instruction generation & Q&A (OpenAI)
│ │ ├── gemini.py # Step verification (Gemini)
│ │ └── image_crop.py # Screenshot preprocessing
│ ├── prompts/
│ │ ├── action.py # Prompt for next-step instructions
│ │ ├── help.py # Prompt for user questions
│ │ └── index.py # Prompt builders
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── App.tsx
│ │ ├── api/
│ │ │ └── session.ts # API client functions
│ │ ├── hooks/
│ │ │ ├── useSession.ts # Core session state & polling logic
│ │ │ ├── useScreenShare.ts # Screen capture via browser API
│ │ │ └── useChangeDetection.ts # Detects meaningful screen changes
│ │ └── components/
│ │ └── StepOverlay.tsx # UI overlay showing instructions
│ └── package.json
└── .env # API keys (not committed)
Things not yet built — contributions welcome.
- YouTube video scraping — ingest a tutorial video and have Dristi teach it step by step, pausing and adapting to what's on screen
- Voice integration — receive spoken instructions instead of (or alongside) text overlays
- Session history & replay — save completed sessions so users can review or re-run past walkthroughs
- Custom AI provider — swap OpenAI/Gemini for a local model (e.g. Ollama) via a config flag
| Variable | Required | Default | Description |
|---|---|---|---|
OPENAI_API_KEY |
Yes | — | OpenAI API key |
GEMINI_API_KEY |
Yes | — | Google Gemini API key |
OPENAI_MODEL |
No | gpt-4o |
OpenAI model to use |
GEMINI_MODEL |
No | gemini-2.5-flash |
Gemini model to use |