Skip to content
ashishjamarkattelPublic

About

No description, website, or topics provided.

Resources

Stars

7 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Dristi

demo_video.mp4

An AI-powered screen guidance tool. You describe a goal, share your screen, and Dristi walks you through it step by step — analyzing screenshots, verifying progress, and answering questions along the way.

Stack: FastAPI + React (Vite/TypeScript)
AI: OpenAI gpt-5 for instructions, Google Gemini gemini-2.5-flash for verification

How It Works

  1. User enters a goal (e.g. "Open Chrome and go to GitHub")
  2. Frontend captures the screen via the Screen Share API
  3. Backend analyzes the screenshot and returns the next instruction
  4. After the user acts, the frontend detects screen changes and sends before/after screenshots to verify the step was completed
  5. Cycle repeats until the goal is marked done
  6. User can ask questions at any point — the AI answers in context

Prerequisites

  • Python 3.12+
  • Node.js 18+
  • An OpenAI API key
  • A Google Gemini API key

Installation

1. Clone the repo

git clone <repo-url>
cd dristi

2. Backend setup

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r backend/requirements.txt

Create a .env file in the project root:

OPENAI_API_KEY=your_openai_key_here
GEMINI_API_KEY=your_gemini_key_here

# Optional — defaults shown
OPENAI_MODEL=gpt-4o
GEMINI_MODEL=gemini-2.5-flash

3. Frontend setup

cd frontend
npm install

Running Locally

Open two terminals:

Terminal 1 — Backend

source .venv/bin/activate
uvicorn backend.main:app --reload

API runs at http://localhost:8000

Terminal 2 — Frontend

cd frontend
npm run dev

App runs at http://localhost:5173


API Reference

Base URL: http://localhost:8000
All request/response bodies are JSON.


POST /session/start

Start a new guidance session.

Request

{
  "goal": "Open Chrome and navigate to GitHub"
}

Response

{
  "session_id": "550e8400-e29b-41d4-a716-446655440000"
}

POST /session/analyze

Analyze the current screenshot and get the next instruction.

Request

{
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "screenshot": "<base64-encoded PNG>"
}

Response

{
  "instruction": "Click the Chrome icon in the taskbar",
  "step_number": 1,
  "done": false
}

When done is true, the goal has been completed.


POST /session/verify

Verify whether the last instruction was successfully completed by comparing before/after screenshots.

Request

{
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "before_screenshot": "<base64-encoded PNG>",
  "after_screenshot": "<base64-encoded PNG>"
}

Response

{
  "completed": true
}

POST /session/ask

Ask a question in the context of the current session.

Request

{
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "screenshot": "<base64-encoded PNG>",
  "question": "Why is this step needed?"
}

Response

{
  "answer": "This step opens the browser so you can navigate to the website."
}

DELETE /session/{session_id}

End a session and clean up its state.

Response

{
  "success": true
}

Project Structure

dristi/
├── backend/
│   ├── main.py               # FastAPI app, CORS config
│   ├── config.py             # Loads env vars
│   ├── routes/
│   │   └── session.py        # All API route handlers
│   ├── models/
│   │   └── schemas.py        # Pydantic request/response models
│   ├── services/
│   │   ├── openai_service.py # Instruction generation & Q&A (OpenAI)
│   │   ├── gemini.py         # Step verification (Gemini)
│   │   └── image_crop.py     # Screenshot preprocessing
│   ├── prompts/
│   │   ├── action.py         # Prompt for next-step instructions
│   │   ├── help.py           # Prompt for user questions
│   │   └── index.py          # Prompt builders
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.tsx
│   │   ├── api/
│   │   │   └── session.ts    # API client functions
│   │   ├── hooks/
│   │   │   ├── useSession.ts         # Core session state & polling logic
│   │   │   ├── useScreenShare.ts     # Screen capture via browser API
│   │   │   └── useChangeDetection.ts # Detects meaningful screen changes
│   │   └── components/
│   │       └── StepOverlay.tsx       # UI overlay showing instructions
│   └── package.json
└── .env                      # API keys (not committed)

Planned Features

Things not yet built — contributions welcome.

  • YouTube video scraping — ingest a tutorial video and have Dristi teach it step by step, pausing and adapting to what's on screen
  • Voice integration — receive spoken instructions instead of (or alongside) text overlays
  • Session history & replay — save completed sessions so users can review or re-run past walkthroughs
  • Custom AI provider — swap OpenAI/Gemini for a local model (e.g. Ollama) via a config flag

Environment Variables

Variable Required Default Description
OPENAI_API_KEY Yes — OpenAI API key
GEMINI_API_KEY Yes — Google Gemini API key
OPENAI_MODEL No gpt-4o OpenAI model to use
GEMINI_MODEL No gemini-2.5-flash Gemini model to use

About

No description, website, or topics provided.

Resources

Stars

7 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages