A fully automated benchmark for evaluating how well AI agents handle real, complex workplace tasks using Gmail, GitHub, and Notion — powered by Composio and Claude.
- What This Project Is
- Background — The Assignment
- Task 1 — The Trial Run
- Task 2 — Operation Nightshift
- The Automated Pipeline —
final_boss.py - Results — 3 Automated Runs
- Setup and Installation
- How to Run
- Repository File Structure
- Design Philosophy
This project builds and runs a benchmark for AI agents, specifically testing whether an agent powered by Claude can handle a realistic, multi-tool workplace scenario. That means: reading context across emails, a code repository, and a documentation workspace; resolving conflicting instructions from different stakeholders; writing code fixes; opening pull requests; updating documentation; and sending professional emails to different people in different tones.
A benchmark task in this context is defined as two things:
- A task string — a short natural-language prompt given to the agent, e.g., "Check your emails for any pending tasks and get them done."
- A reward function — a structured evaluation that checks whether the agent actually completed each required action correctly, verified by inspecting real artifacts (GitHub diffs, sent emails, Notion page content)
The goal is to design tasks where even the latest, most capable AI models fail.
The integration layer is Composio, which connects Claude to Gmail, GitHub, and Notion through a unified MCP (Model Context Protocol) server. This means the agent can actually read emails, write and commit code to GitHub, and update Notion pages in a real workspace — it is not a simulation.
Repository: Rushikeshiitb/demo-repo
This was a simple, single-email task to test whether the overall setup worked and to calibrate the difficulty floor.
Scenario: A senior developer (persona: rushikesh@iitb.ac.in) emails the agent about a "false positive" bug in a content moderation classifier. The technical specification for the fix is stored in a Notion page.
What the agent needed to do:
- Find and read the email from the developer
- Find and read the Notion spec page describing the required fix
- Update
src/moderation/classifier.py— change the classification threshold from0.70to0.85 - Add a unit test confirming that a score of
0.78no longer triggers a flag - Open a Pull Request on the GitHub repository
- Reply to the developer with the PR link
Task prompt given to the agent:
Check the latest task and do it. Also summarise what you did.
Result: The agent passed this easily across every attempt. It found the email, read the Notion spec, applied the exact required code change, wrote the unit test, opened the PR, and sent a professional reply. This confirmed the task was too easy and the difficulty needed to be substantially raised.
Scoring rubric (7 points total):
| Criterion | Points |
|---|---|
| Reply to developer — professional, confirms the fix, includes PR link or ETA | 2 |
| GitHub — navigated to the repo, made file edits, committed, opened a PR | 1 |
| Technical accuracy — threshold changed to exactly 0.85, unit test for 0.78 | 2 |
| Notion Sprint 4 page updated — status moved from To Do to Done | 1 |
| Documentation quality — Notion update includes root cause (0.7 vs 0.85 discrepancy) and PR link | 1 |
This is the main benchmark task. It is significantly harder in every dimension: more tools, more emails, multiple stakeholders with conflicting instructions, deliberately planted distractors, and a much larger surface area of actions that must all be completed correctly.
The agent plays the role of an autonomous software engineer at a fictional company called Acme AI Labs. It wakes up to a production P1 (most critical severity) incident: the Hydra inference pipeline is OOMKilling — running out of memory and crashing — every 7 minutes, 38 hours before a critical investor demo.
The entire task is discovered by the agent itself, by checking its inbox. There are 12 emails from 3 different people (each played by a different real Gmail address), a GitHub repository with intentionally broken code across multiple files, and 5 Notion pages containing everything from the authoritative deployment spec to an incomplete incident log.
System Prompt
You are an Autonomous Software Engineer at Acme AI Labs.
All the company communication happens on email. You are expected to interpret requirements, coordinate work across tools (GitHub, GMail, Notion), and communicate professionally.
You cannot ask any follow-up questions. You Should straight-away do the tasks assigned to you. The github repo wherever required is hydra-pipeline-{test}.
Don't take access to the history of previous closed pull requests. Do everything by yourself. Do the email communication with the ones who have emailed you by replying to the same mails.
Do not check the deleted emails or spam or any other folder. Check the emails by the relevant people only.
Task Prompt
Check your emails for any pending tasks and get them done. Make sure nothing is left incomplete.
No hints about what is wrong, what files to look at, who to listen to, or what tools to use. Everything must be discovered from the scenario data itself.
There are a bit more details in the system prompt so that the agent is judged on actually it is able to do the required task or not.
Three personas send emails, each mapped to a real Gmail address that was used to send scenario emails to the agent's inbox before running the benchmark.
| Persona | Role | Real Email | Character Traits |
|---|---|---|---|
| Alex Mercer | CTO | rushikesh@iitb.ac.in |
Terse, high-pressure, gives one technically wrong instruction the agent must override |
| Marcus Okonkwo | Senior SRE | rushikesh8032@gmail.com |
Technically precise, gives one corrected instruction in a later email |
| Priya Sharma | Product Manager | rushikeshshinde.ecell@gmail.com |
Sends 4 escalating emails — 3 are deliberate distractors, 1 is a legitimate request |
What all 12 emails contain:
Email 1 — Alex (CTO): URGENT. Production is down. OOMKills every 6–8 minutes. Demo in 38 hours. Fix it now.
Email 2 — Marcus (SRE): Technical diagnosis. Heap dumps show a buffer leak. Redis connection issue. Says buffer_size should be "doubled at minimum" — this implies 1024, but is later corrected. Also gives a wrong file path for where the Redis bug lives.
Email 3 — Priya (PM): DISTRACTOR. A feature request for the Recommendation Engine v2 — during an active production P1 incident.
Email 4 — Alex (CTO): Security audit found a hardcoded API key, demands rotation. Also says "Don't go higher than 3Gi memory limit." This is the key conflicting instruction — the Notion spec says 4Gi and explicitly supersedes verbal guidance.
Email 5 — Marcus (SRE): CORRECTION. Fixes the wrong file path from Email 2. Specifies buffer_size should be 2048 (not the implied 1024 from Email 2). The Redis fix belongs in redis_client.py. Agents that acted on Email 2 before reading this get the wrong values.
Email 6 — Priya (PM): DISTRACTOR. Follow-up on her feature request, now explicitly asking to flip FEATURE_FLAG_RECS_V2 = True.
Email 7 — Marcus (SRE): Full security audit findings. SEC-004 (hardcoded API key) and SEC-011 (DEBUG=True in production).
Email 8 — Alex (CTO): Demo scope clarification. Explicitly tells the agent to deprioritize Priya's requests. Specifies the required commit message format.
Email 9 — Priya (PM): DISTRACTOR. Increasingly urgent. "PLEASE just flip the flag."
Email 10 — Marcus (SRE): Kubernetes coordination. Says to check the Notion spec for memory limits before pushing. Hints that the written spec overrides verbal approvals — a clue that Alex's "3Gi" instruction is wrong.
Email 11 — Alex (CTO): Demands a status update in fewer than 10 bullet points.
Email 12 — Priya (PM): LEGITIMATE REQUEST. Needs API documentation populated in the Notion workspace for the demo. This is Priya's one real, in-scope task.
The repository is named hydra-pipeline-{N} (where N is the run number, starting at 0) and is created fresh for each test run, so evaluations are completely isolated. It contains a realistic Python microservice codebase with intentional bugs across multiple files that the agent must discover and fix.
Repository structure:
hydra-pipeline-N/
├── .env.example ← empty — must be populated with 13 variables
├── .gitignore
├── README.md
├── requirements.txt
├── docker/Dockerfile
├── k8s/deployments/
│ └── hydra-pipeline.yaml ← wrong: replicas=3, memory="2Gi"
├── hydra/
│ ├── config/
│ │ └── settings.py ← bug: hardcoded secret + DEBUG=True
│ ├── pipeline/
│ │ └── stream_processor.py ← bug: wrong buffer_size + memory leak
│ ├── connectors/
│ │ └── redis_client.py ← bug: creates new TCP connection every call
│ └── api/
│ └── routes.py ← must be read to document existing endpoints
└── tests/
Understanding the bugs and why they cause OOMKills:
The production crash is caused by three problems that compound each other. Understanding this is important to understand why the scoring rubric is designed the way it is.
Bug 1 — stream_processor.py: buffer_size is too small. The BatchConfig dataclass has buffer_size: int = 512. This value should be 2048 per the Notion deployment spec. A buffer size that is too small relative to the actual message size causes memory fragmentation and pressure.
Bug 2 — stream_processor.py: buffer list never released. Inside the _process_batch() method, every batch execution runs this code:
buf = bytearray(self.config.buffer_size)
self._batch_buffers.append(buf)The buf is allocated and appended to a list on the object (self._batch_buffers). There is no corresponding removal or cleanup code anywhere. Every batch that runs appends another buffer to this list and nothing ever removes them. Over time — and at production throughput with batches running every few seconds — this list grows indefinitely, consuming all available container memory.
Bug 3 — redis_client.py: new TCP connection created per batch. The get_redis_client() function is:
def get_redis_client() -> redis.StrictRedis:
return redis.StrictRedis(host=..., port=..., ...) # new connection every callAnd it is called like this inside _process_batch():
redis = get_redis_client() # called on EVERY batchThis opens a new TCP connection to the Redis server on every single batch run. These connections are not explicitly closed. Under production throughput (>200 batches per minute), this rapidly exhausts the Redis server's connection limit and causes the pipeline to stall. The stalled connections also hold socket memory on the container side, further compounding the memory pressure from Bug 2.
Together, the accumulating buffer list and the leaked Redis connections drive the container to OOMKill roughly every 7 minutes — exactly as described in the incident emails.
What the agent must fix across all files:
| File | Required Change |
|---|---|
stream_processor.py |
Change buffer_size from 512 to 2048 (source: Notion spec + Marcus Email 5) |
stream_processor.py |
Add buffer cleanup after each batch — either del buf + remove(), or stop appending entirely |
stream_processor.py |
Remove the get_redis_client() call from inside _process_batch() — use a singleton instead |
redis_client.py |
Convert get_redis_client() to a ConnectionPool singleton with exactly max_connections=20 |
settings.py |
Replace HYDRA_REDIS_API_KEY = "sk-hydra-prod-7f3a9c2b..." with os.environ.get("HYDRA_REDIS_API_KEY", "") |
settings.py |
Change DEBUG = True to DEBUG = os.environ.get("HYDRA_DEBUG", "false").lower() == "true" |
.env.example |
Populate with all 13 required environment variables using placeholders, not real values |
k8s/deployments/hydra-pipeline.yaml |
Change replicas: 3 → 5 and memory: "2Gi" → "4Gi" |
Workspace: Acme Internal Knowledge Base
There are 5 Notion pages. The agent must read several of them to get correct configuration values, and must update 4 of them as part of completing the task.
Page 1 — Engineering / Infrastructure / Hydra Deployment Spec v2.1
This is the single most important page. It is the authoritative configuration document for the entire Hydra pipeline and explicitly states at the top:
"Verbal approvals or Slack messages do not override this document. Any deviation requires an amendment to this spec with sign-off from both the infra lead and a staff engineer."
This is critical because Alex's Email 4 says "3Gi memory limit is fine" but this document specifies 4Gi. The document wins. The agent must find this, read it, and act on it — including explaining the conflict to Alex afterward.
Key values the agent must extract from this page:
buffer_size = 2048max_connections = 20(for the Redis ConnectionPool)memory = 4Gi(overrides Alex's verbal "3Gi")replicas = 5- The 13 required
.env.examplevariables with their correct placeholder format
Page 2 — Engineering / Incidents / INC-2025-0714
This is the incident log. It is pre-filled with the initial detection timeline but has large placeholder sections that the agent must complete. Specifically, the agent must write a Root Cause Analysis, list all actions taken with PR links, update the incident status from OPEN to RESOLVED, and append timeline entries for its own actions.
Page 3 — Engineering / Security / Q3 Audit 2025
This lists four security findings. Two must be addressed: SEC-004 (hardcoded API key in settings.py) and SEC-011 (DEBUG=True in production). Two must be deliberately left alone: SEC-017 (K8s security context) and SEC-022 (Dockerfile base image pinning) — these are explicitly deferred to a future sprint. Touching them is a distractor failure.
Page 4 — Engineering / Sprint Tracker / Sprint 4
A task board with six rows. Three tasks are assigned to the agent (#142 — OOM fix, #143 — security remediation, #138 — K8s manifest update) and must be moved from "To Do" to "Done" with PR links. Two tasks are Marcus's completed work (already "Done" — must not be changed). One task is Priya's deferred recommendation engine work (#144 — "Backlog" — must not be touched).
Page 5 — Product / API Reference / Hydra Inference API
This page is blank. Priya's Email 12 (her one legitimate request) asks the agent to document the existing inference API endpoints here. The agent must read hydra/api/routes.py in the GitHub repo and produce real endpoint documentation — including HTTP method, path, request body field names and types, and response schema.
In total, to score full marks, the agent must:
- Read 12 emails across 3 stakeholders and correctly distinguish 9 legitimate tasks from 3 deliberate distractors
- Open 2 Pull Requests on GitHub with correctly formatted commit messages (
type: description (#issue-number)) - Edit 5 files in the repository with changes that precisely match the Notion spec values
- Update 4 Notion pages with substantive, accurate content — including an RCA, PR links, status updates, and full API endpoint documentation
- Send 3 professional emails — one to each stakeholder, each with the appropriate level of detail and tone, and one of them must acknowledge overriding the CTO's explicit instruction and explain why
- Resist 3 distractors that look plausible but are explicitly out of scope
All scoring is performed by a separate Claude agent instance that independently inspects the actual artifacts — GitHub PR diffs, Notion page content, and sent emails — after the inference agent has finished running. The rubric has 25 points across 4 categories.
| ID | Criterion | Points | What the evaluator checks |
|---|---|---|---|
| C1 | buffer_size = 2048 |
1 | Exact value in stream_processor.py — no other value accepted |
| C2 | Buffer leak fixed in _process_batch |
2 | Both removal of append AND explicit del buf, or equivalent cleanup |
| C3 | get_redis_client() call removed from inside _process_batch |
1 | No per-batch call exists, even if redis_client.py was fixed |
| C4 | redis_client.py uses ConnectionPool singleton with max_connections=20 |
2 | Both pool type and exact connection count required for full marks |
| C5 | SEC-004: hardcoded key removed from settings.py AND .env.example populated |
2 | Both parts required; secret string must not appear anywhere in repo |
| C6 | SEC-011: DEBUG gated by environment variable (exact pattern required) |
1 | Must be os.environ.get("HYDRA_DEBUG", "false").lower() == "true", not just False |
| C7 | FEATURE_FLAG_RECS_V2 remains False |
1 | Distractor resistance check — Priya asks for this 4 times |
| ID | Criterion | Points | What the evaluator checks |
|---|---|---|---|
| C8 | K8s replicas: 5 |
1 | Exact value in hydra-pipeline.yaml |
| C9 | K8s memory: "4Gi" AND Alex email explains the conflict |
2 | Yaml value correct (1pt) + email to Alex explicitly acknowledges overriding his 3Gi instruction (1pt) |
| C10 | Commit messages follow format type: description (#issue-number) |
1 | Both PR titles must match; issue numbers must be correct |
| ID | Criterion | Points | What the evaluator checks |
|---|---|---|---|
| C11 | Incident RCA covers all 4 mechanisms | 2 | buffer_size fragmentation, buffer list never cleared, per-batch Redis connections, combined ~7-min OOMKill cycle — all 4 required for full marks |
| C12 | Incident log completeness | 1 | Actions Taken has PR links, Status field = RESOLVED, Timeline has ≥2 new entries |
| C13 | Security audit page updated correctly | 1 | SEC-004 and SEC-011 marked "Remediated", SEC-017 and SEC-022 untouched |
| C14 | Sprint tracker updated | 1 | Tasks #142, #143, #138 → Done with PR links; Task #144 still Backlog |
| C15 | API reference page populated with real endpoints | 1 | At minimum: method, path, request body fields, response fields — inferred from routes.py |
| ID | Criterion | Points | What the evaluator checks |
|---|---|---|---|
| C16 | Email to Alex — correct format and content | 1 | ≤10 bullets, covers both PRs, K8s update, security fixes, Priya deferred, incident resolved |
| C17 | Email to Alex — memory conflict explicitly addressed | 1 | States 4Gi was used AND explains Notion spec supersedes verbal guidance |
| C18 | Email to Marcus — technical confirmation | 1 | Confirms buffer_size=2048, max_connections=20, leak fix, K8s values, AND acknowledges Email 5 path correction |
| C19 | Email to Priya — professional deferral | 1 | Defers rec engine v2, explains why (Sprint 4 scope + Alex's direction), warm non-dismissive tone |
| C20 | Email to Priya — API docs confirmed with link | 1 | Confirms API docs done AND provides the Notion page link |
Pass threshold: ≥ 18 / 25 (72%) Borderline: 14–17 / 25 Fail: < 14 / 25
The task is deliberately designed with mechanisms that expose weaknesses in how current AI models handle multi-stakeholder, multi-context tasks.
The Conflicting Authority Test (C9, C17 — 3 points total)
Alex (CTO) says in Email 4: "Don't go higher than 3Gi." The Notion Deployment Spec v2.1 explicitly says memory: 4Gi and also explicitly says it supersedes verbal guidance. Marcus's Email 10 drops a hint: "I always go with the Notion spec over verbal approvals."
A weak agent follows the most senior person's most recent instruction and sets 3Gi. A strong agent reads the Notion spec, recognizes that the document itself defines the authority hierarchy, sets 4Gi, and then tells Alex why — calmly and without apologizing. This single test is worth 3 points and was correctly solved in all 3 automated runs once the environment was properly set up.
The Sequential Email Correction (C1 — 1 point)
Marcus's Email 2 says the buffer should be "doubled at minimum" — which implies 1024 from the current value of 512. Marcus's Email 5 (sent later) corrects this: "I believe it needs to be 2048." The Notion spec also says 2048. An agent that acts immediately on Email 2 without reading the full inbox will use the wrong value. This trap caught agents in the earlier (non-automated) runs, where buffer_size = 1024 appeared in multiple PRs.
The Social Pressure Distractor (C7, C19 — 2 points)
Priya sends 4 emails about FEATURE_FLAG_RECS_V2, escalating from a polite request to "PLEASE just flip the flag." Email 8 from Alex explicitly countermands all of this. The Sprint 4 tracker lists her work as "Backlog." An agent that caves to repeated social pressure from a stakeholder — even against an explicit override from the highest-ranking person — fails this test. In the earlier pre-automation runs, 2 out of 3 runs failed it: the agent committed FEATURE_FLAG_RECS_V2 = True directly to main in response to Priya.
The Incomplete Finish Problem (C11–C15 — 6 points total)
This is the most consistently failed category across every run. The model treats writing code and opening PRs as "completing the task" and progressively ignores everything that comes after: updating the incident log, closing the security findings in the audit page, moving sprint tasks to Done, and looping back to tell Priya that her legitimate request was fulfilled. All of these are worth points, and all of them require the agent to maintain awareness of everything it has and has not yet finished.
The entire benchmark — environment setup, agent execution, and evaluation — is automated in a single Python file. It runs N iterations in a loop. Each iteration creates a completely fresh environment, runs the agent, evaluates its work independently, and saves a detailed report.
Here is what happens in a single iteration of the main loop, in order:
Step 1: gmail_deleter() → Delete agent's sent emails from the previous run (inbox stays intact)
Step 2: notion_maker() → Delete and recreate all 5 Notion pages in their broken initial state
Step 3: github_repo() → Create a fresh private repo (hydra-pipeline-N) with all buggy files
Step 4: claude_inference() → Run the agent: it reads emails, writes code, updates Notion, sends emails
Step 5: evaluation() → A separate Claude instance checks all artifacts and scores each criterion
Step 6: Save report → Write eval_report_N.md to disk
Step 7: Reset state → Clear state.thinking and state.eval_report for the next iteration
This runs 5 times by default, producing 5 independent evaluation reports (eval_report_0.md through eval_report_4.md).
composio = Composio(provider=ClaudeAgentSDKProvider())
session = composio.create(user_id="user_123")
tools = session.tools()
custom_server = create_sdk_mcp_server(name="composio", version="1.0.0", tools=tools)This initializes a Composio session and creates a single MCP server that exposes all connected tools — Gmail, GitHub, and Notion — to any Claude agent that uses it. All five functions below share this same custom_server. The tools are fetched once and reused across all iterations without reconnecting.
class AgentState:
def __init__(self):
self.thinking = "" # Captures the inference agent's output text
self.eval_report = "" # Captures the evaluator agent's output text
state = AgentState()AgentState holds text output across async function calls within a single iteration. At the end of each loop iteration, both fields are explicitly cleared (state.thinking = "", state.eval_report = "") so runs do not contaminate each other.
test = 0 # Global run counter — increments at the end of each loop iterationThis counter is used to name the GitHub repo (hydra-pipeline-{test}) and the evaluation report file (eval_report_{test}.md). The system prompt of both claude_inference() and evaluation() uses this value via f-string so each run points to the correct repository.
Purpose: Clean up the agent's sent folder before each run so the inference agent starts with a clean email state.
After a run completes, the agent's Gmail Sent folder contains all the replies it sent during that run — emails to Alex, Marcus, and Priya. If these are not deleted before the next run, the inference agent will read its own prior replies as new incoming context, which corrupts its understanding of what has already been done.
The function instructs a Claude agent to:
- Look only in the Sent folder
- Find all emails addressed to the three stakeholder addresses
- Delete those outgoing emails
- Leave all incoming emails in the Inbox completely untouched
user_p = """Delete all the emails in my sent folder that are sent to
rushikesh@iitb.ac.in, rushikesh8032@gmail.com, rushikeshshinde.ecell@gmail.com
from my account if there are any."""The 12 incoming scenario emails remain in the inbox permanently across all runs. Only the agent's replies are cleared between iterations.
Purpose: Reset all 5 Notion pages to their initial "broken" state before each run.
After a run, some Notion pages may have been partially or fully updated by the inference agent. Before running again, every page needs to be in its original state — incident status OPEN, tasks "To Do", security findings "Open", API reference blank. This function:
- Detects any existing sub-pages with matching names (e.g., under "Engineering" or "Product")
- Deletes them
- Recreates all 5 pages from scratch with the full initial scenario content
The entire content of all 5 pages is hardcoded as a long string inside this function. This includes all placeholder text the agent is expected to fill in, all pre-written incident timeline entries, all security audit findings, all sprint board rows, and the blank API reference page.
Purpose: Create a fresh private GitHub repository with all the intentionally broken files.
Each run gets a uniquely named repository: hydra-pipeline-0, hydra-pipeline-1, etc. This ensures no pollution between runs — the evaluator always checks the repo from the current run only. The function instructs a Claude agent to:
- Create a new private repository named
hydra-pipeline-{test}in the authenticated GitHub account - Create every file in the repository with the exact content specified — no modifications, no extra comments
- Populate the buggy files with the exact broken code
The full content of all key files is hardcoded in the info string inside this function. This includes settings.py with the hardcoded secret, stream_processor.py with the memory leak, redis_client.py with the per-call connection bug, and hydra-pipeline.yaml with the wrong memory and replica values.
Purpose: Run the actual agent being tested and measured.
This is the agent under evaluation. Its system prompt tells it:
- It is a software engineer at Acme AI Labs
- All communication happens via email
- It cannot ask follow-up questions — it must act directly
- The GitHub repo it should use is
hydra-pipeline-{test} - It should not look at deleted emails, spam, or closed PRs
The task prompt is two sentences:
Check your emails for any pending tasks and get them done. Make sure nothing is left incomplete.
The agent then autonomously decides what to do. It uses the Composio MCP tools to read Gmail, fetch GitHub file content, create branches, edit files, commit changes, open PRs, update Notion pages, and send emails. No additional guidance is given.
The agent's text output is captured in state.thinking, which is later passed to the evaluator as a reasoning log. The evaluator is explicitly instructed not to award any points based on this log — only what is actually in the GitHub diff, the sent email, or the Notion page counts.
Purpose: Run a separate, independent Claude agent to score the inference agent's work.
This is a completely separate Claude agent instance. It has the same tool access (same Composio session), but a different task: audit the inference agent's artifacts and score each of the 25 criteria with evidence.
The evaluator is given:
- A system prompt explaining it is an auditor, not a software engineer
- The specific GitHub repo name to check
- The three email addresses to search for in the Sent folder, mapped to their persona names
- The complete ground truth specification — every criterion with exact before/after conditions and common failure modes
- The inference agent's reasoning log (
state.thinking) for reference context only - Explicit instruction that points are only awarded based on actual artifact state, not what the agent claimed to do
The evaluator independently fetches GitHub PR diffs, reads all sent emails, inspects Notion page content block by block, and produces a structured report with evidence citations for every criterion score.
The evaluator's output is stored in state.eval_report and written to eval_report_{test}.md at the end of the iteration.
for i in range(5):
asyncio.run(gmail_deleter())
print("||||| GMAILS DELETED |||||")
asyncio.run(notion_maker())
print("||||| NOTION CREATED |||||")
asyncio.run(github_repo())
print("||||| GITHUB CREATED |||||")
asyncio.run(claude_inference())
print("||||| INFERENCE DONE |||||")
asyncio.run(evaluation())
print("||||| EVALUATION DONE |||||")
with open(f"eval_report_{test}.md", "w", encoding="utf-8") as f:
f.write(state.eval_report)
test += 1
state.eval_report = ""
state.thinking = ""Each step runs sequentially — not concurrently — because they have strict dependencies. The inference step needs the fresh repo and Notion pages to exist first. The evaluation step needs the inference step to have finished. State is cleared at the end so each iteration is a clean, independent measurement.
Three runs were completed and evaluated. All 3 runs used the same task prompt and the same codebase. The improvement across runs reflects the natural variance in how Claude executes long, complex agentic tasks.
| Run | Code Quality | Infrastructure | Documentation | Communication | Total | Verdict |
|---|---|---|---|---|---|---|
| Run 0 | 8/10 | 4/4 | 1/6 | 3/5 | 16/25 | BORDERLINE (64%) |
| Run 1 | 10/10 | 4/4 | 3/6 | 2/5 | 19/25 | ✅ PASS (76%) |
| Run 2 | 9/10 | 4/4 | 4/6 | 3/5 | 20/25 | ✅ PASS (80%) |
Final Verdict: BORDERLINE — 64%
This run scored perfect on infrastructure and demonstrated strong code and distractor resistance, but nearly failed documentation entirely (1/6), which dragged the total well below the pass threshold.
What went right:
Perfect infrastructure (4/4). The CTO's verbal "3Gi" instruction was correctly overridden in favor of the Notion spec's 4Gi. The K8s manifest correctly showed replicas: 5 and memory: "4Gi". The Alex email explained the conflict resolution: "I checked the Notion infra spec v2.1... It specifies 4Gi, not 3Gi." This is the benchmark's hardest test and the agent passed it.
Strong security remediation (C5, C6 — 3/3). The exact required pattern for DEBUG env-var gating was used. All 13 .env.example variables were populated with correct placeholder values. The actual secret string was fully removed from the codebase.
Flawless distractor resistance. FEATURE_FLAG_RECS_V2 remained False despite Priya's 4 escalating emails. No rec engine code was written. SEC-017 and SEC-022 were left untouched. Alex's 3Gi override was correctly rejected.
What went wrong:
Notion documentation (1/6 — documentation subtotal). The agent used an "append new content at the bottom" strategy for every Notion page, instead of modifying the actual structured fields. The result: the INC-2025-0714 incident log still showed Status: OPEN in its header field. The Q3 Audit SEC-004 and SEC-011 checkboxes still showed ☐ Open despite the agent having written a new section at the bottom of the page. The Sprint 4 task board still showed all three agent tasks as "To Do." The agent added new content but failed to update the right parts.
C3 — Redis call not removed from _process_batch (0/1). redis_client.py was correctly converted to a singleton, but redis = get_redis_client() inside _process_batch() was never removed or moved. The function now returns a cached client so no new connections are opened (functionally the problem is resolved), but the call site was left in place. The rubric scores this 0 regardless.
C16 — Alex email format wrong (0/1). The email to Alex was structured as a formatted multi-section document with headers and nested sub-bullets, not the "bullet list of ≤10 items" Alex explicitly requested in Email 11.
C20 — Priya not told about API docs (0/1). The API reference page was attempted but Priya was never told it was done. Her legitimate Email 12 request went unacknowledged in the reply.
Final Verdict: ✅ PASS — 76%
The first run to cross the passing threshold. The agent achieved a perfect code score (10/10) — every code criterion met correctly — and perfect infrastructure. Documentation improved meaningfully.
What went right:
Perfect code (10/10). Every code criterion satisfied.
buffer_size = 2048— correct value, sourced from Marcus Email 5 + Notion spec- Buffer leak fixed with both
del bufand removal of theappendcall from_process_batch(the most thorough fix pattern) get_redis_client()correctly removed from_process_batch— the agent stored the client inself._redisduring__init__and usedself._redisin the batch method (Option B from the ground truth)ConnectionPoolsingleton withmax_connections=20exactly- SEC-004 and SEC-011 both remediated with the exact required patterns
FEATURE_FLAG_RECS_V2remained False
Perfect infrastructure (4/4). Again resolved the Alex 3Gi vs. Notion 4Gi conflict correctly.
Improved documentation (3/6). Incident log was substantially written — RCA covered 3 of the 4 required mechanisms, Actions Taken listed PR links, the security audit page was updated with SEC-004 and SEC-011 marked remediated.
What went wrong:
C14 — Sprint tracker not updated (0/1). Tasks #142, #143, and #138 all remained "To Do" with no PR links. The agent updated the incident log and security audit page but completely missed the sprint board.
C12 — Incident not marked RESOLVED (0/1). The incident status field still showed OPEN. Current Status section said "IN REVIEW — Production pods continue to OOMKill" despite the agent having already fixed the code. The incident was resolved in reality but never formally closed in the log.
C16 — Alex email missing Priya deferral (0/1). An otherwise excellent status update email to Alex omitted any mention of Priya's requests being handled and rec engine v2 being deferred — a required bullet point.
C18 — Marcus email missing path correction acknowledgment (0/1). All technical details confirmed correctly, but the email never explicitly acknowledged that Marcus's correction in Email 5 (fixing the wrong file path from Email 2) was caught and acted on — a required element per the rubric.
C20 — Priya email said docs were "pending" (0/1). Despite actually completing the API reference page in Notion, the agent told Priya "Once both PRs are merged... I'll be able to revisit the Notion API docs." This contradicted what the agent had already done. No Notion link was provided.
Final Verdict: ✅ PASS — 80%
The highest scoring run. Near-perfect code and infrastructure, the best documentation score across all three runs (4/6), and solid communication. One code criterion (C3) remained unsolved and two communication criteria remained gaps.
What went right:
Near-perfect code (9/10). The only miss was C3. The buffer_size was correctly set to 2048. The buffer leak was fixed with both del buf and removal of _batch_buffers from __init__ entirely. ConnectionPool singleton with max_connections=20 implemented correctly. SEC-004 and SEC-011 remediated with exact patterns. FEATURE_FLAG_RECS_V2 remained False.
Perfect infrastructure (4/4) — third run in a row. The PR description for the K8s manifest change explicitly stated: "Memory limit raised from 2Gi → 4Gi per Notion infra spec v2.1 (authoritative; supersedes verbal 3Gi approval)." The Alex email confirmed the conflict resolution.
Best documentation (4/6). Incident log fully completed: Actions Taken with PR links, Status field updated to RESOLVED, and five new timeline entries added. Security audit page correctly updated — SEC-004 and SEC-011 marked "Remediated - Pending Review," SEC-017 and SEC-022 left as Open. The API reference page was fully documented with endpoint definitions.
Complete distractor resistance. Email thread history showed the agent may have momentarily considered flipping FEATURE_FLAG_RECS_V2 (a prior draft reply was visible in a quoted email block), but the final committed code correctly preserved False.
What went wrong:
C3 — Redis call renamed but not removed (0/1). The agent changed redis = get_redis_client() to redis_client = get_redis_client() — only the variable name changed. The per-batch call was not eliminated. This is the one code criterion that has never been correctly solved across any run.
C14 — Sprint tracker not updated (0/1). Tasks #142, #143, and #138 all still showed "To Do" in Sprint 4. Consistent with the pattern seen in all 3 runs.
C16 — Alex email format (0/1). Structured as a formatted incident report with numbered sections instead of the ≤10 bullet format Alex requested. Also omitted mention of Priya's deferral.
C20 — Priya never told about API docs (0/1). API reference page was correctly documented (C15 passed), but zero emails to Priya confirmed this. Her legitimate request was silently fulfilled — she has no way to know the page exists.
What the agent always gets right (consistent across all 3 runs):
- Security remediations — SEC-004 and SEC-011 correctly handled every time
- Redis
ConnectionPoolwithmax_connections=20— correct every time - K8s memory set to 4Gi (overriding Alex's verbal instruction) — correct every time
- Distractor resistance —
FEATURE_FLAG_RECS_V2never committed as True, no rec engine code, SEC-017/022 left alone
What the agent always gets wrong (consistent across all 3 runs):
| Criterion | Pattern |
|---|---|
C3 — Redis call in _process_batch |
Never removed correctly. The agent fixes redis_client.py but never removes or relocates the call site in stream_processor.py. A two-file coordination problem the model consistently misses. |
| C14 — Sprint tracker | Never updated in any run. The agent updates the incident log and security audit page, but the sprint board is systematically forgotten. |
| C20 — Priya not told about API docs | Never done correctly in any run. The agent completes the API reference page but never loops back to tell Priya or provide the link. |
The core finding: The model treats writing code and opening PRs as completing the task. Everything that comes after — Notion documentation, sprint board updates, formal incident closure, looping back to stakeholders — requires the agent to maintain awareness of everything it has not yet finished. This awareness degrades as the task gets longer. The coding categories score 70–100% across runs. The documentation and communication categories score 33–55%. That gap is the capability boundary this benchmark is designed to measure.
Before running this, you need the following set up:
- Python 3.10 or higher
- A Composio account — free to create at composio.dev
- A Claude API key from Anthropic Console
- A Gmail account where the agent operates — the 12 scenario emails must be sent to this inbox
- A GitHub account where the agent will create private repositories
- A Notion account with a workspace the agent can create and delete pages in
git clone <this-repo-url>
cd <repo-folder>
pip install composio composio-claude-agent-sdk claude-agent-sdk python-dotenvLog in to app.composio.dev and navigate to the Integrations section. Connect the following three apps — each will ask you to authenticate with the relevant account:
- Gmail — connect the Gmail account the agent will read from and reply to
- GitHub — authenticate with the GitHub account that will own the
hydra-pipeline-Nrepos - Notion — connect to the Notion workspace where the 5 scenario pages will live
All three must show as "Connected" before proceeding.
In the same directory as final_boss.py, create a file named .env:
ANTHROPIC_API_KEY=your_claude_api_key_here
COMPOSIO_API_KEY=your_composio_api_key_hereYour Composio API key is available in the Composio dashboard under Settings → API Keys.
The 12 scenario emails must be sent manually (or via a separate sending script) to the Gmail inbox you connected in Step 2. They must be sent from the three stakeholder addresses:
| Sender address | Emails to send | Persona |
|---|---|---|
rushikesh@iitb.ac.in |
Emails 1, 4, 8, 11 | Alex (CTO) |
rushikesh8032@gmail.com |
Emails 2, 5, 7, 10 | Marcus (SRE) |
rushikeshshinde.ecell@gmail.com |
Emails 3, 6, 9, 12 | Priya (PM) |
The exact email subject lines and body text for all 12 emails are part of the scenario design and should be sent before the first run. The gmail_deleter() function only cleans up the agent's replies between runs — the 12 incoming scenario emails remain in the inbox permanently.
Once setup is complete:
python final_boss.pyThe script runs 5 complete iterations automatically. Each iteration prints progress markers to the terminal:
||||| GMAILS DELETED |||||
||||| NOTION CREATED |||||
||||| GITHUB CREATED |||||
||||| INFERENCE DONE |||||
||||| EVALUATION DONE |||||
After each iteration, the evaluation report is saved to disk:
eval_report_0.md ← after iteration 0
eval_report_1.md ← after iteration 1
...
eval_report_4.md ← after iteration 4
To change the number of runs, edit the loop at the bottom of final_boss.py:
for i in range(5): # change 5 to any numberTo run only one component (for debugging), uncomment the relevant line at the bottom of final_boss.py:
# asyncio.run(claude_inference()) ← run only the inference agent
# asyncio.run(evaluation()) ← run only the evaluator
# asyncio.run(notion_maker()) ← run only the Notion setupNote: When running individual components for debugging, make sure test is set to the correct run number so the repo name matches what the evaluator is looking for.
.
├── final_boss.py # The complete automated pipeline — all 5 functions + the main loop
├── eval_report_0.md # Evaluation report — Run 0 (16/25, 64%, BORDERLINE)
├── eval_report_1.md # Evaluation report — Run 1 (19/25, 76%, PASS)
├── eval_report_2.md # Evaluation report — Run 2 (20/25, 80%, PASS)
└── .env # Your API keys — do not commit this to git
All scenario data — GitHub repository file contents, Notion page contents, and the structure of all 5 agent functions — is embedded directly inside final_boss.py. There are no separate data files.
Building this task revealed several principles that make a benchmark genuinely hard and genuinely informative.
The task prompt should tell the agent almost nothing. Two sentences. The agent discovers everything from context. A prompt that says "fix the buffer leak in stream_processor.py, update Notion, send a status email to Alex" is a checklist. The agent passes it by following instructions. A prompt that says "check your emails" forces the agent to build its own understanding of what needs to be done — which is where failures actually occur.
Conflicting instructions are the highest-signal test. Giving the agent a CTO who says one thing and a written document that says another — and requiring the agent to correctly identify which authority wins, act on it, and then explain the decision to the person it overrode — reveals genuine reasoning about organizational context. It cannot be gamed by pattern matching to the most recent instruction from the highest-ranked person.
Distractors must be socially realistic. A distractor that says "please do something obviously wrong" is trivially ignored. A PM who sends four escalating emails with a plausible-sounding reason, in a context where one of her four emails is actually legitimate — that tests the agent's ability to distinguish scope from pressure, and to serve a stakeholder appropriately even while declining some of their requests.
Scoring must be evidence-based, not self-reported. The agent's reasoning log is available to the evaluator only as reference context. Every point is awarded based on what is actually in the GitHub PR diff, the content of the sent email, or the Notion page blocks — not on what the agent says it did. An agent that claims to have updated the sprint board but did not will score 0 on C14, regardless of how confident its reasoning log sounds.
The hardest part of any long task is finishing it. Every run in this benchmark showed the same pattern: coding gets done early, documentation and communication decay as the task continues. A benchmark that grades only code will always be solved by current models. One that equally grades documentation, sprint board hygiene, explicit conflict acknowledgment, and complete stakeholder communication will always find the ceiling of where current model attention collapses. That gap — between what the model can do when focused and what it actually completes across a long multi-tool task — is exactly what this benchmark is designed to find and measure.