I build backend systems and data infrastructure for AI. Large-scale scrapers and anti-bot research, agent systems that do real work, and retrieval that holds up in production. I run my own infrastructure on AWS with Docker and Kubernetes, and I take projects from design to running under load, not just to a demo.
The through line across my work is hard problems on the data and systems side: beating commercial bot defenses, keeping retrieval correct when the data moves, orchestrating agents, and shipping services that stay up.
build ──► scrape & extract at scale Playwright, TLS/JA3, drift detection
backend & data systems FastAPI, WebSocket, Postgres, Redis, queues
agents & retrieval multi-agent loops, RAG that stays current
own the infra AWS, Docker, Kubernetes, Caddy, Linux
|
A control plane for anti-bot research. It probes which browser fingerprints still beat Cloudflare, DataDome, and Akamai, detects drift with ML, and pushes proven fixes to scrapers over REST and WebSocket. FastAPI, Playwright, Docker, Kubernetes. |
A full web data-extraction platform: scraping API, browser extension, and dashboard. Self-hosted on Docker with Postgres, Redis, MinIO, and SearxNG. Large Python and TypeScript codebase built to run on my own servers. |
|
Re-rank RAG retrieval by time decay and source authority, so recent trusted sources beat stale but similar ones. A real scoring model that sits after your vector store. |
An autonomous multi-agent engineering orchestrator. Five agents research, design, adversarially review, implement, and hotfix in a continuous loop. |
|
A commerce AI platform built on a pluggable pipeline: every stage is a module wired by a registry, so adding a marketplace or a generator is one registration. Ingests products, builds brand and keyword intelligence, generates and validates listings. |
Scraper toolkit crawlsync semantic diffs between crawls, scrapeguard scraper health monitoring, and schemaweaver schema inference from raw extractions. |
Also: unsynth AI-text detection on PyPI · vahi invoice OCR for finance teams · threearrow AI career platform · save47 media downloader
LinkedIn · open an issue on any repo · pandeyvishwas51@gmail.com