Skip to content

om-gupta-30/AgentOps-AI-Platform

BranchesTags

Repository files navigation

AgentOps AI Platform

A production-ready multi-agent AI system built with LangChain, LangGraph, and FastAPI. Features semantic memory with vector search, web tool integration, and full observability via LangSmith and Langfuse.

CI/CD Pipeline Python FastAPI Next.js License

Features

  • Multi-Agent Orchestration -- Supervisor, Research, Execution, and Evaluator agents coordinated through LangGraph
  • Tool Integration -- DuckDuckGo web search (extensible to custom tools)
  • Semantic Memory -- ChromaDB vector store for intelligent context reuse across tasks
  • Dual Observability -- LangSmith tracing + Langfuse analytics (both optional)
  • Self-Evaluation -- Automatic quality scoring (1-10) with retry on failure
  • Modern Frontend -- Next.js 14, TypeScript, responsive UI
  • Security First -- Zero hardcoded secrets, pre-push secret scanning, comprehensive .gitignore

Architecture

┌──────────────────────────────────────────────┐
│              Frontend (Next.js 14)            │
└──────────────────┬───────────────────────────┘
                   │ HTTP
┌──────────────────┴───────────────────────────┐
│              Backend (FastAPI)                │
└──────────────────┬───────────────────────────┘
                   │
      ┌────────────┴────────────┐
      │                         │
┌─────┴──────┐         ┌───────┴─────────┐
│  LangGraph │         │  Observability  │
│  Workflow  │         │ (LangSmith /    │
│            │         │  Langfuse)      │
└─────┬──────┘         └─────────────────┘
      │
      ├── Supervisor Agent  → Planning & routing
      ├── Research Agent    → Web search & context gathering
      ├── Execution Agent   → Output generation
      └── Evaluator Agent   → Quality control & scoring

Tech Stack

Layer Technology
Backend FastAPI, Uvicorn, Pydantic
Agents LangChain, LangGraph
LLM Google Gemini 2.0 Flash
Vector DB ChromaDB + Google Embeddings
Frontend Next.js 14, React 18, TypeScript
Observability LangSmith, Langfuse
Tools DuckDuckGo Search

Quick Start

Prerequisites

Setup

# Clone
git clone https://github.com/om-gupta-30/AgentOps-AI-Platform.git
cd AgentOps-AI-Platform

# Create virtual environment
make venv

# Configure environment
cp .env.example .env.local
# Edit .env.local — add your GOOGLE_API_KEY (required)

# Install dependencies
make install

# Start both backend + frontend
make dev
Service URL
Frontend http://localhost:3000
Backend API http://localhost:8000
API Docs (Swagger) http://localhost:8000/docs

Usage

Web Interface

  1. Open http://localhost:3000
  2. Enter a goal (e.g., "Explain vector databases")
  3. Click Execute
  4. View the result, evaluation score, and run history

REST API

# Execute a task
curl -X POST http://localhost:8000/run \
  -H "Content-Type: application/json" \
  -d '{"goal": "Explain vector databases"}'

# Get run history
curl http://localhost:8000/history

# Health check
curl http://localhost:8000/health

Example response:

{
  "final_output": "Vector databases are specialized systems...",
  "evaluation": {
    "passed": true,
    "score": 9,
    "reasons": ["Clear", "Accurate", "Well-structured"]
  },
  "memory_used": false
}

Project Structure

AgentOps-AI-Platform/
├── backend/                        # FastAPI REST API
│   ├── __init__.py
│   ├── main.py                     #   App entry, health check, CORS
│   └── routers/
│       ├── __init__.py
│       ├── run.py                  #   POST /run — task execution
│       └── history.py              #   GET /history — past runs
│
├── src/agentops_ai_platform/       # Core agent system
│   ├── __init__.py
│   ├── agents/
│   │   ├── __init__.py
│   │   ├── supervisor_agent.py     #   Planning & routing
│   │   ├── research_agent.py       #   Context gathering
│   │   ├── execution_agent.py      #   Output generation
│   │   └── evaluator_agent.py      #   Quality scoring
│   └── graphs/
│       ├── __init__.py
│       └── main_graph.py           #   LangGraph workflow
│
├── memory/                         # Memory system
│   ├── __init__.py
│   ├── memory_store.py             #   JSON storage & retrieval
│   └── vector_store.py             #   ChromaDB semantic search
│
├── tools/                          # Agent tools
│   ├── __init__.py
│   └── web_search.py               #   DuckDuckGo integration
│
├── observability/                  # Tracing & monitoring
│   ├── __init__.py
│   ├── langsmith.py                #   LangSmith client
│   ├── langfuse.py                 #   Langfuse client
│   └── trace_utils.py              #   Unified trace helpers
│
├── frontend/                       # Next.js 14 UI
│   ├── app/
│   │   ├── page.tsx                #   Main page
│   │   ├── layout.tsx              #   Root layout
│   │   ├── globals.css             #   Styles
│   │   └── components/
│   │       ├── AgentInput.tsx
│   │       ├── ResultDisplay.tsx
│   │       └── HistoryList.tsx
│   ├── package.json
│   └── tsconfig.json
│
├── tests/                          # Test suite
│   └── __init__.py
│
├── scripts/
│   └── check-secrets.sh            #   Pre-push security scanner
│
├── .github/                        # CI/CD & GitHub templates
│   ├── workflows/ci.yml
│   ├── PULL_REQUEST_TEMPLATE.md
│   └── ISSUE_TEMPLATE/
│
├── .env.example                    # Environment template (safe to commit)
├── .gitignore                      # Secrets & artifacts protection
├── .gitattributes                  # Line endings & diff settings
├── requirements.txt                # Python dependencies
├── Makefile                        # Dev commands
├── CONTRIBUTING.md
├── DEPLOYMENT.md
├── SECURITY.md
└── LICENSE                         # MIT

.env.local, memory/memory.json, and memory/chroma_db/ are gitignored and never pushed.

Agent Pipeline

User Goal → Supervisor → [Research] → Execution → Evaluator → Response
                              ↑                        │
                        (if needed)              retry if score < 7
Agent Role Description
Supervisor Planning Creates execution plan, decides if research is needed
Research Context Gathers information via web search (only when requested)
Execution Generation Produces the final output following the plan
Evaluator Quality Scores output 1-10, triggers one retry on failure
  • Model: Google Gemini 2.0 Flash (all agents)
  • Max retries: 1 retry on score < 7
  • Memory: Successful outputs (score >= 7) saved automatically for future context

Memory System

Feature Details
Storage Automatic for score >= 7
Search Semantic similarity via Google Embeddings
Retrieval Supervisor retrieves relevant past tasks during planning
Limits Max 100 memories, 90-day retention
Backend JSON file + ChromaDB vector index

Observability

Both integrations are optional. The system works without them.

LangSmith

Full tracing of agent execution, LLM calls, and timing.

# In .env.local
LANGSMITH_API_KEY=your_key
LANGSMITH_PROJECT=agentops-ai-platform
LANGCHAIN_TRACING_V2=true

Dashboard: https://smith.langchain.com/

Langfuse

Metrics, evaluation scores, and cost tracking.

# In .env.local
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

Dashboard: https://cloud.langfuse.com/

Makefile Commands

# Development
make dev              # Start backend + frontend (hot reload)
make backend          # Start backend only (port 8000)
make frontend         # Start frontend only (port 3000)

# Installation
make install          # Install all dependencies
make venv             # Create Python virtual environment

# Code Quality
make lint             # Run linters (Ruff + ESLint)
make format           # Auto-format code
make test             # Run test suite
make pre-push         # Format + lint + security scan

# Security
make security-check   # Scan for leaked secrets
make check            # Verify API keys and dependencies

# Build & Production
make build            # Build frontend for production
make prod             # Run in production mode

# Maintenance
make kill             # Kill processes on ports 8000/3000
make clean            # Clean build artifacts & caches

Environment Variables

Copy .env.example to .env.local and fill in your values. Never commit .env.local.

Variable Required Description
GOOGLE_API_KEY Yes Google Gemini API key
GEMINI_MODEL No Model name (default: gemini-2.0-flash)
LANGSMITH_API_KEY No LangSmith tracing key
LANGSMITH_PROJECT No LangSmith project name
LANGCHAIN_TRACING_V2 No Set true to enable tracing
LANGFUSE_SECRET_KEY No Langfuse secret key
LANGFUSE_PUBLIC_KEY No Langfuse public key
LANGFUSE_HOST No Langfuse host URL
OBSERVABILITY_ENABLED No Set 1 to enable
OFFLINE_MODE No Set 1 to skip LLM calls (testing)

For deployment, set these in your platform's dashboard (Vercel, GCP, etc.) -- never in code.

Deployment

See DEPLOYMENT.md for full deployment instructions.

Vercel (Recommended)

npm i -g vercel
vercel            # Deploy
vercel --prod     # Production deploy

Set environment variables in: Vercel Dashboard > Settings > Environment Variables

Google Cloud Platform

gcloud run deploy agentops-backend \
  --source . \
  --platform managed \
  --region us-central1 \
  --set-secrets GOOGLE_API_KEY=google-api-key:latest

Use Secret Manager for all sensitive values.

Security

  • All secrets loaded from environment variables -- zero hardcoded credentials
  • .gitignore covers env files, credentials, user data, and build artifacts
  • Pre-push security scanner (make security-check) checks for leaked keys
  • CI/CD pipeline runs automated security scan on every push
  • See SECURITY.md for full security policy and incident response

Quick security check before pushing:

make pre-push

Performance

Metric Value
Execution time 10-20 seconds per task
Cost per task ~$0.001 (Gemini 2.0 Flash)
Monthly cost (1000 tasks) ~$1
Architecture Stateless, horizontally scalable

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

# Fork → Clone → Branch → Code → Test → PR
git checkout -b feature/your-feature
make dev          # Test locally
make pre-push     # Verify before pushing
git push origin feature/your-feature

Troubleshooting

Problem Solution
Port already in use make kill then make dev
Missing GOOGLE_API_KEY make check to see what's set
Frontend not loading cd frontend && rm -rf node_modules .next && npm install
ChromaDB errors make clean then make dev

License

MIT License -- see LICENSE for details.

Issues · Pull Requests · Discussions

About

Platform for orchestrating, monitoring, and managing AI agents

agent-ops-ai-platform.vercel.app

Topics

python monitoring orchestration ai-agents llm agent-ops

Resources

Readme

License

MIT license

Contributing

Contributing

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages