diff --git a/README.md b/README.md index 46d690a..a27060d 100644 --- a/README.md +++ b/README.md @@ -42,12 +42,14 @@ Input Task | Math – pure expression | `AST_EVAL` | AST (deterministic) | 0 | | Math – word problem | `API_MATH` | `kimi-k2p7-code` / `minimax-m3` | 768 | | Sentiment Classification | `LOCAL_SENTIMENT` | Qwen2.5-3B | 20 (local) | -| Text Summarization (≤6k) | `LOCAL_GENERAL` | Qwen2.5-3B | 250 (local) | -| Text Summarization (>6k) | `API_LONG_CONTEXT` | `gemma-4-26b-a4b-it` | 200 | +| Text Summarization (≤6k chars) | `LOCAL_GENERAL` | Qwen2.5-3B | 250 (local) | +| Text Summarization (>6k chars) | `API_LONG_CONTEXT` | `gemma-4-26b-a4b-it` | 200 | | Named Entity Recognition | `LOCAL_NER` | Qwen2.5-3B | 300 (local) | -| Code Debugging | `API_CODE` | `kimi-k2p7-code` | 400 | +| Code Debugging | `API_CODE` (local-first) | Qwen2.5-3B → `kimi-k2p7-code` | 400 | | Logical Reasoning | `API_LOGIC` | `kimi-k2p7-code` / `minimax-m3` | 768 | -| Code Generation | `API_CODE` | `kimi-k2p7-code` | 500 | +| Code Generation | `API_CODE` (local-first) | Qwen2.5-3B → `kimi-k2p7-code` | 500 | + +> **Local-first code strategy**: Code debugging and code generation tasks use a difficulty classifier (`code_utils.py`). Easy/medium tasks attempt Qwen2.5-3B locally first, validated with `ast.parse` + completeness checks — only falling back to the remote API if local output is invalid. Hard tasks go directly to the remote API to avoid wasting wall-clock time. ### Semantic Classifier (L2) @@ -57,12 +59,27 @@ Layer 2 uses **`all-MiniLM-L6-v2`** (sentence-transformers) combined with a **Su - **Consolidated Training**: Trained on a diverse combined dataset of **3,235 tasks** (covering standard, adversarial, and conversational phrasings). - **Accuracy**: Achieves **100.00% classification accuracy** across all task categories, including tricky inputs with overlapping keywords (e.g., historical numbers or code snippets). - **Efficiency**: Runs entirely local with **0 Fireworks API tokens** and extremely low memory footprint (weights are only ~100 KB). +- **Auto-training**: If pre-trained weights are missing, the classifier automatically trains from `tests/fixtures/task.json` at startup. + +### Remote Model Selection + +The agent dynamically selects the best model from `ALLOWED_MODELS` (injected at runtime by the harness) using per-category priority preferences: + +| Category | Preferred Models (in priority order) | Fallback | +|---|---|---| +| `API_CODE` | `kimi-k2p7-code` → `gemma-4-31b-it` | First available | +| `API_MATH` | `kimi-k2p7-code` → `minimax-m3` | First available | +| `API_LOGIC` | `kimi-k2p7-code` → `minimax-m3` | First available | +| `API_LONG_CONTEXT` | `gemma-4-26b-a4b-it` → `gemma-4-31b-it-nvfp4` | First available | +| `LOCAL_GENERAL` (escalation) | `minimax-m3` → `kimi-k2p7-code` | First available | +| `LOCAL_SENTIMENT` (escalation) | `minimax-m3` → `kimi-k2p7-code` | First available | +| `LOCAL_NER` (escalation) | `minimax-m3` → `kimi-k2p7-code` | First available | ### Prompt Compression Before every remote API call, the prompt goes through two transforms: 1. **Filler strip** — removes phrases like *"Can you please explain..."*, *"I would like you to..."* -2. **Output suffix** — appends a concise constraint (e.g., `" Output ONLY the final numeric answer."`) +2. **Output suffix** — appends a concise constraint per category (e.g., `" Return ONLY raw code."` for code tasks) This reduces input + output tokens on every remote call. @@ -72,54 +89,83 @@ This reduces input + output tokens on every remote call. ``` ├── agent/ -│ ├── schemas.py # Pydantic Task & Result models -│ ├── cache.py # SHA-256 semantic dedup cache (thread-safe) -│ ├── ast_eval.py # Safe deterministic math evaluator (AST whitelist) -│ ├── classifier.py # Semantic embedding classifier (all-MiniLM-L6-v2) -│ ├── router.py # AgentRouter — orchestrates all 4 layers -│ └── watchdog.py # Daemon thread: fires at 570s, flushes partial output +│ ├── __init__.py +│ ├── schemas.py # Pydantic Task & Result models +│ ├── cache.py # SHA-256 semantic dedup cache (thread-safe) +│ ├── ast_eval.py # Safe deterministic math evaluator (AST whitelist) +│ ├── classifier.py # Supervised PyTorch classifier (all-MiniLM-L6-v2 + MLP) +│ ├── supervised_model.pt # Pre-trained classifier weights (~100 KB) +│ ├── router.py # AgentRouter — orchestrates all 4 layers +│ └── watchdog.py # Daemon thread: fires at 570s, flushes partial output │ ├── engines/ -│ ├── local_slm.py # llama-cpp-python wrapper (Qwen2.5-3B Q4_K_M) -│ └── remote_llm.py # Async Fireworks API client (aiohttp + tenacity retry) +│ ├── __init__.py +│ ├── local_slm.py # llama-cpp-python wrapper (Qwen2.5-3B Q4_K_M) +│ └── remote_llm.py # Async Fireworks API client (aiohttp + tenacity retry) +│ +├── handlers/ # One handler file per capability domain +│ ├── __init__.py +│ ├── _base.py # Shared load_prompt_template utility +│ ├── code_utils.py # Code difficulty classifier + extract/validate helpers +│ ├── factual.py # → local SLM +│ ├── sentiment.py # → local SLM (Positive / Negative / Neutral) +│ ├── ner.py # → local SLM (JSON list output) +│ ├── summarization.py # → local SLM +│ ├── math_handler.py # → remote (CoT + numeric extraction, max 768 tokens) +│ ├── debug.py # → local-first, API fallback (max 400 tokens) +│ ├── code_gen.py # → local-first, API fallback (max 500 tokens) +│ ├── logic.py # → remote (max 768 tokens) +│ ├── local_handlers.py # Backwards-compat composite LocalGeneralHandler +│ └── remote_handlers.py # RemoteGeneralHandler (escalation fallback) │ -├── handlers/ # One handler file per capability domain -│ ├── _base.py # Shared load_prompt_template utility -│ ├── factual.py # → local SLM -│ ├── sentiment.py # → local SLM (Positive / Negative / Neutral) -│ ├── ner.py # → local SLM (JSON list output) -│ ├── summarization.py # → local SLM -│ ├── math_handler.py # → remote kimi-k2p7-code / minimax-m3 (max 768 tokens) -│ ├── debug.py # → remote kimi-k2p7-code (max 400 tokens) -│ ├── code_gen.py # → remote kimi-k2p7-code (max 500 tokens) -│ ├── logic.py # → remote kimi-k2p7-code / minimax-m3 (max 768 tokens) -│ └── remote_handlers.py # RemoteGeneralHandler (escalation fallback) +├── prompts/ # System prompt templates (.txt) +│ ├── factual.txt +│ ├── sentiment.txt +│ ├── ner.txt +│ ├── summarization.txt +│ ├── remote_math.txt # CoT with few-shot examples → ANSWER: +│ ├── remote_logic.txt # Direct answer only, no explanation +│ ├── remote_code.txt # Raw Python code output only +│ ├── remote_general.txt # Escalation fallback prompt +│ ├── local_code_gen.txt # Local code generation (no markdown) +│ └── local_code_debug.txt # Local debug (corrected code only) +│ +├── models/ # Bundled GGUF weights (~1 GB, not tracked in git) │ -├── prompts/ # System prompt templates (.txt) -├── models/ # Bundled GGUF weights (~1 GB, not tracked in git) ├── tests/ │ ├── fixtures/ -│ │ ├── task.json # 3,235 consolidated tasks (standard + tricky + diverse + practice + sample) -│ │ └── expected_results.json # Baseline expected answers for sample tasks +│ │ ├── task.json # 3,235 consolidated tasks (classifier training data) +│ │ ├── expected_results.json # Baseline expected answers for sample tasks +│ │ ├── sample_tasks.json # Practice tasks from the hackathon guide +│ │ └── test_cases_60.json # 60-task evaluation subset +│ ├── results/ # Saved test run outputs +│ ├── eval_200_tasks.json # 200-task evaluation dataset +│ ├── eval_results.json # Evaluation results (200 tasks) +│ ├── evaluation_report.md # Detailed performance report (93% accuracy) │ ├── test_ast_eval.py │ ├── test_cache.py │ ├── test_classifier.py +│ ├── test_local_slm.py │ ├── test_remote_llm.py │ ├── test_router.py │ └── test_integration.py │ ├── scripts/ -│ ├── setup.sh # 🚀 First-time setup (install everything) -│ ├── run.sh # ▶️ Run agent with custom input/output -│ ├── test_local.py # 🧪 Run practice tasks locally + token stats -│ ├── download_model.sh # Download GGUF weights from HuggingFace -│ └── simulate_grading.sh # Docker run with 4GB RAM / 2 CPU constraints +│ ├── setup.sh # 🚀 First-time setup (install everything) +│ ├── run.sh # ▶️ Run agent with custom input/output +│ ├── test_local.py # 🧪 Run practice tasks locally + token stats +│ ├── prompt_benchmark.py # 📊 Benchmark 5 prompting strategies (sentiment + summarization) +│ ├── download_model.sh # Download GGUF weights from HuggingFace +│ └── simulate_grading.sh # Docker run with 4GB RAM / 2 CPU constraints │ -├── output/ # Generated results (git-ignored) -├── main.py # Entrypoint -├── Dockerfile -├── .env.example # Template — copy to .env and fill in credentials -└── requirements.txt +├── output/ # Generated results (git-ignored) +├── main.py # Entrypoint — async task processing with watchdog +├── Dockerfile # Python 3.12-slim + uv package manager +├── entrypoint.sh # Loads .env if present, then runs main.py +├── .env.example # Template — copy to .env and fill in credentials +├── pyproject.toml # ruff + mypy + pytest configuration +├── requirements.txt +└── requirements-dev.txt ``` --- @@ -185,6 +231,14 @@ ANSWER : The capital of Australia is Canberra... ⚠️ Đây là LOCAL tokens (0 Fireworks tokens) ``` +### Benchmark prompting strategies + +```bash +PYTHONPATH=. python scripts/prompt_benchmark.py +``` + +Benchmarks 5 strategies (baseline, zero-shot strict, few-shot, chain-of-thought, self-consistency 3×) across sentiment and summarization — outputs CSV + markdown summary to `output/`. + ### Chạy unit tests ```bash @@ -192,6 +246,12 @@ ANSWER : The capital of Australia is Canberra... PYTHONPATH=. pytest tests/test_ast_eval.py tests/test_cache.py \ tests/test_remote_llm.py tests/test_router.py -v +# Classifier test (requires all-MiniLM-L6-v2 + task.json) +PYTHONPATH=. pytest tests/test_classifier.py -v + +# Local SLM test (requires GGUF model) +PYTHONPATH=. pytest tests/test_local_slm.py -v + # Full integration test (loads local SLM) PYTHONPATH=. python tests/test_integration.py ``` @@ -220,6 +280,12 @@ cat output_test/results.json docker push /develarper-agent:latest ``` +**Docker image features:** +- Uses `entrypoint.sh` (loads `.env` if present, then runs `main.py`) +- Sets `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1` — all model weights are pre-cached at build time, no runtime downloads +- Pre-caches `all-MiniLM-L6-v2` sentence-transformer during build +- Bundles `Qwen2.5-3B Q4_K_M` GGUF (~986 MB) in `/app/models/` + --- ## Environment Variables @@ -260,19 +326,40 @@ Subject to: accuracy ≥ 80% (binary gate — phải pass trước) ``` - **Local execution = 0 Fireworks tokens** → maximize local handling +- **Local-first code strategy** → easy/medium code tasks solved locally with AST validation → only hard tasks or failed validations use API tokens - **Supervised PyTorch Classifier** → 100.00% routing accuracy → zero misroutes → zero unnecessary API token waste - **Prompt compression** → strip filler phrases + output suffix → giảm tokens mỗi remote call - **Per-category `max_tokens` budgets** → giới hạn output dài không cần thiết - **Semantic cache** → dedup identical/similar prompts +- **Category-aware model selection** → pick best available model per task type from `ALLOWED_MODELS` +- **Escalation model preferences** → local escalations prefer `minimax-m3` for cost efficiency + +--- + +## Evaluation Results + +Local evaluation on 200 tasks (100 Factual + 100 Summarization) achieved: + +| Metric | Value | +|---|---| +| **Global Accuracy** | 93.00% (186/200) | +| **Avg Latency** | 505.1 ms | +| Factual Knowledge | 89.00% accuracy, 598.6 ms avg | +| Text Summarization | 97.00% accuracy, 411.5 ms avg | + +See [`tests/evaluation_report.md`](tests/evaluation_report.md) for detailed analysis. --- ## Development Notes -- **Python version**: 3.10 (Docker) / 3.11+ (host dev) +- **Python version**: 3.12 (Docker) / 3.11+ (host dev) +- **Package manager**: `uv` (in Docker), `pip` (host dev) - **Classifier**: `all-MiniLM-L6-v2` (SentenceTransformer) + PyTorch MLP head — trained locally on 3,235 consolidated tasks (including test suite prompts) - **Local SLM**: `Qwen2.5-3B-Instruct Q4_K_M` via `llama-cpp-python` -- **Linting**: `ruff check .` -- **Type checking**: `mypy .` -- **Pre-commit**: `pre-commit run --all-files` +- **Remote API**: `aiohttp` + `tenacity` retry (3 attempts, exponential backoff) +- **Math prompting**: CoT with few-shot examples, handles fractions/decimals, answer extraction via regex +- **Linting**: `ruff check .` (target: Python 3.11, line-length: 150) +- **Type checking**: `mypy .` (strict mode, Python 3.12) +- **Pre-commit**: `pre-commit run --all-files` (ruff + ruff-format + mypy + file checks) - Không cần API key để chạy local SLM tasks và unit tests diff --git a/tests/fixtures/test_cases_60.json b/tests/fixtures/test_cases_60.json new file mode 100644 index 0000000..57818b0 --- /dev/null +++ b/tests/fixtures/test_cases_60.json @@ -0,0 +1,422 @@ +[ + { + "task_id": "task-0001", + "category": "factual", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "What is the capital of France? Answer concisely." + }, + { + "task_id": "task-0002", + "category": "factual", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "What is the chemical symbol for water?" + }, + { + "task_id": "task-0003", + "category": "factual", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "How many continents are there on Earth?" + }, + { + "task_id": "task-0004", + "category": "factual", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "Who painted the Mona Lisa and in which century was it painted?" + }, + { + "task_id": "task-0005", + "category": "factual", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "What is the boiling point of water at sea level in degrees Celsius?" + }, + { + "task_id": "task-0006", + "category": "factual", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "What is the largest planet in our solar system and what is its main composition?" + }, + { + "task_id": "task-0007", + "category": "factual", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "In what year did the Berlin Wall fall, and which event did it symbolize?" + }, + { + "task_id": "task-0008", + "category": "factual", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "Explain the difference between TCP and UDP in one sentence, mentioning reliability and connection state." + }, + { + "task_id": "task-0009", + "category": "factual", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "What is the Heisenberg uncertainty principle and what two physical quantities does it relate?" + }, + { + "task_id": "task-0010", + "category": "factual", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "Who wrote 'One Hundred Years of Solitude' and what literary genre is the work most associated with?" + }, + { + "task_id": "task-0011", + "category": "sentiment", + "difficulty": "easy", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'I love this product, it's amazing!'" + }, + { + "task_id": "task-0012", + "category": "sentiment", + "difficulty": "easy", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'This is the worst experience ever, I want a refund.'" + }, + { + "task_id": "task-0013", + "category": "sentiment", + "difficulty": "easy", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'The package arrived on Tuesday.'" + }, + { + "task_id": "task-0014", + "category": "sentiment", + "difficulty": "medium", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'While the build quality is excellent, the customer service was disappointing.'" + }, + { + "task_id": "task-0015", + "category": "sentiment", + "difficulty": "medium", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'I'm not sure how I feel about this update; some features are great but others are confusing.'" + }, + { + "task_id": "task-0016", + "category": "sentiment", + "difficulty": "medium", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'It's okay, nothing special but not terrible either.'" + }, + { + "task_id": "task-0017", + "category": "sentiment", + "difficulty": "medium", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'Despite the long wait, the food was absolutely delicious and worth it.'" + }, + { + "task_id": "task-0018", + "category": "sentiment", + "difficulty": "hard", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'The movie had stunning visuals and a powerful score, yet the plot felt hollow and the ending left me frustrated.'" + }, + { + "task_id": "task-0019", + "category": "sentiment", + "difficulty": "hard", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'I suppose it's fine if you don't mind paying premium prices for mediocre quality.'" + }, + { + "task_id": "task-0020", + "category": "sentiment", + "difficulty": "hard", + "expected_route": "LOCAL_SENTIMENT", + "prompt": "Classify the sentiment of this text as Positive, Negative, or Neutral: 'Slightly disappointed but not surprised given the brand's recent track record.'" + }, + { + "task_id": "task-0021", + "category": "summarization", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text in one sentence: 'The sun rises in the east and sets in the west. This happens every single day because the Earth rotates on its axis once every 24 hours.'" + }, + { + "task_id": "task-0022", + "category": "summarization", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text: 'Photosynthesis is the process by which plants use sunlight, water, and carbon dioxide to produce glucose and oxygen. This process is essential for plant growth and for providing oxygen to the atmosphere.'" + }, + { + "task_id": "task-0023", + "category": "summarization", + "difficulty": "easy", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text: 'The water cycle describes how water evaporates from oceans, forms clouds through condensation, falls back to Earth as precipitation, and collects in rivers and oceans, then repeats continuously.'" + }, + { + "task_id": "task-0024", + "category": "summarization", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text in two sentences: 'Renewable energy technologies such as solar panels, wind turbines, and hydroelectric dams are expanding rapidly across the globe. Governments are investing heavily in these technologies to reduce dependence on fossil fuels and to meet international climate targets. Solar energy in particular has seen dramatic cost reductions over the past decade, making it competitive with traditional energy sources in many regions. Wind farms, both onshore and offshore, are now a common sight in countries committed to lowering their carbon emissions.'" + }, + { + "task_id": "task-0025", + "category": "summarization", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text: 'The Industrial Revolution began in Britain in the late 18th century and spread throughout Europe and North America. It marked a shift from agrarian, handcraft-based economies to industrial, machine-driven production. The invention of the steam engine, the mechanization of textiles, and improvements in iron production were central to this transformation. The revolution brought urbanization, as people moved from farms to cities to work in factories, and it fundamentally reshaped social structures, labor, and the global economy.'" + }, + { + "task_id": "task-0026", + "category": "summarization", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text: 'Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It relies on algorithms that identify patterns in data and use those patterns to make predictions or decisions. Common approaches include supervised learning, where models learn from labeled examples, and unsupervised learning, where models find structure in unlabeled data. Deep learning, a more advanced technique, uses neural networks with many layers and has driven recent breakthroughs in image recognition, natural language processing, and speech translation.'" + }, + { + "task_id": "task-0027", + "category": "summarization", + "difficulty": "medium", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text: 'Climate change is causing noticeable effects across the planet, including rising global temperatures, more frequent extreme weather events, and shifting precipitation patterns. Coastal cities face growing risks from sea level rise, while inland regions experience prolonged droughts and intense heatwaves. Ecosystems are also affected, with many species struggling to adapt to rapidly changing conditions. Scientists warn that without significant reductions in greenhouse gas emissions, these impacts will worsen over the coming decades, threatening food security, water supplies, and human settlements worldwide.'" + }, + { + "task_id": "task-0028", + "category": "summarization", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize this text in exactly one sentence: 'The global economy is facing unprecedented challenges due to shifting trade routes, supply chain disruptions, and geopolitical tensions between major powers. Inflation has risen to levels not seen in decades, prompting central banks to raise interest rates aggressively, which in turn has slowed investment and consumer spending. Energy markets remain volatile as countries transition away from fossil fuels while still relying on them for short-term stability. Labor markets are tight in some sectors but weakening in others, creating an uneven economic landscape. Policymakers must balance the need to control inflation against the risk of triggering a deep recession, all while addressing long-term issues such as aging populations and climate-related financial risks.'" + }, + { + "task_id": "task-0029", + "category": "summarization", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "TLDR: 'The recent policy overhaul introduced by the regulatory body imposes stricter reporting requirements on financial institutions, mandates higher capital reserves for mid-sized banks, and establishes a new framework for assessing systemic risk. Proponents argue that these measures will prevent a repeat of the banking instability seen earlier this decade and restore public confidence in the financial system. Critics, however, contend that the increased compliance costs will disproportionately affect smaller community banks, potentially reducing lending to small businesses and rural communities. The policy will be phased in over a three-year period, with periodic reviews to assess its effectiveness and unintended consequences. International coordination with foreign regulators is also planned to avoid regulatory arbitrage.'" + }, + { + "task_id": "task-0030", + "category": "summarization", + "difficulty": "hard", + "expected_route": "LOCAL_GENERAL", + "prompt": "Summarize the key arguments in under three sentences: 'Proponents of universal basic income argue that it provides a safety net in an era of automation, reduces poverty, and gives individuals the freedom to pursue education or entrepreneurship. They point to pilot programs showing improvements in health and wellbeing among recipients. Critics counter that unconditional cash payments may discourage work, inflate prices, and place an unsustainable burden on public finances. They also argue that targeted welfare programs are more efficient at helping those most in need. A middle position suggests that a modest basic income, combined with existing social programs, could balance these concerns while reducing bureaucratic overhead. Ultimately, the debate hinges on empirical questions about human behavior, economic conditions, and the design of the policy itself.'" + }, + { + "task_id": "task-0031", + "category": "ner", + "difficulty": "easy", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities and return them as a JSON list: 'Barack Obama visited Paris last summer.'" + }, + { + "task_id": "task-0032", + "category": "ner", + "difficulty": "easy", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities (Persons, Organizations, Locations) as a JSON list: 'Apple Inc. is headquartered in Cupertino, California.'" + }, + { + "task_id": "task-0033", + "category": "ner", + "difficulty": "easy", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities as a JSON list: 'Albert Einstein was born in Germany and later worked at Princeton.'" + }, + { + "task_id": "task-0034", + "category": "ner", + "difficulty": "medium", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities (Persons, Organizations, Locations) as a JSON list: 'Microsoft acquired LinkedIn in 2016 for $26.2 billion, strengthening its presence in the professional networking space.'" + }, + { + "task_id": "task-0035", + "category": "ner", + "difficulty": "medium", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities as a JSON list: 'The United Nations held a summit in New York City where Secretary-General Antonio Guterres addressed delegates from over 190 countries.'" + }, + { + "task_id": "task-0036", + "category": "ner", + "difficulty": "medium", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities (Persons, Organizations, Locations) as a JSON list: 'Tesla CEO Elon Musk announced a new gigafactory in Berlin, aiming to compete with BMW and Volkswagen in the European market.'" + }, + { + "task_id": "task-0037", + "category": "ner", + "difficulty": "medium", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities as a JSON list: 'In 1969, NASA's Apollo 11 mission landed Neil Armstrong and Buzz Aldrin on the Moon, a moment watched by millions across the United States.'" + }, + { + "task_id": "task-0038", + "category": "ner", + "difficulty": "hard", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities with their types as a JSON list of objects: 'The Treaty of Versailles, signed in 1919, formally ended World War I between the Allied Powers and Germany, redrawing borders across Europe and the Middle East.'" + }, + { + "task_id": "task-0039", + "category": "ner", + "difficulty": "hard", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities (Persons, Organizations, Locations) as a JSON list: 'During his tenure at Stanford University, Dr. Andrew Ng co-founded Coursera and led the Google Brain project, later joining Baidu to direct its artificial intelligence research lab.'" + }, + { + "task_id": "task-0040", + "category": "ner", + "difficulty": "hard", + "expected_route": "LOCAL_NER", + "prompt": "Extract all named entities (Persons, Organizations, Locations, Dates) as a JSON list: 'On January 6, 2021, the World Health Organization announced that the COVID-19 vaccine developed by Pfizer and BioNTech showed 95% efficacy in trials conducted across the United States, Brazil, and South Africa.'" + }, + { + "task_id": "task-0041", + "category": "code_debug", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef add(a, b):\n return a - b" + }, + { + "task_id": "task-0042", + "category": "code_debug", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef count_to_ten():\n for i in range(1, 10):\n print(i)" + }, + { + "task_id": "task-0043", + "category": "code_debug", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef greet(name):\n return 'Hello, ' + nam" + }, + { + "task_id": "task-0044", + "category": "code_debug", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef binary_search(arr, target):\n low, high = 0, len(arr)\n while low <= high:\n mid = (low + high) // 2\n if arr[mid] == target:\n return mid\n elif arr[mid] < target:\n low = mid + 1\n else:\n high = mid - 1\n return -1" + }, + { + "task_id": "task-0045", + "category": "code_debug", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef reverse_string(s):\n result = ''\n for i in range(len(s)):\n result += s[i]\n return result" + }, + { + "task_id": "task-0046", + "category": "code_debug", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef factorial(n):\n if n == 0:\n return 1\n return n * factorial(n)\n" + }, + { + "task_id": "task-0047", + "category": "code_debug", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python function. Return ONLY the corrected code without explanation.\ndef is_palindrome(s):\n left, right = 0, len(s)\n while left < right:\n if s[left] != s[right]:\n return False\n left += 1\n right -= 1\n return True" + }, + { + "task_id": "task-0048", + "category": "code_debug", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this Python LRU cache implementation. Return ONLY the corrected code without explanation.\nclass LRUCache:\n def __init__(self, capacity):\n self.capacity = capacity\n self.cache = {}\n def get(self, key):\n if key in self.cache:\n return self.cache[key]\n return -1\n def put(self, key, value):\n if key in self.cache:\n self.cache[key] = value\n else:\n if len(self.cache) >= self.capacity:\n self.cache.pop(next(iter(self.cache)))\n self.cache[key] = value" + }, + { + "task_id": "task-0049", + "category": "code_debug", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this linked list cycle detection function. Return ONLY the corrected code without explanation.\ndef has_cycle(head):\n slow = head\n fast = head.next\n while slow != fast:\n if fast is None or fast.next is None:\n return False\n slow = slow.next\n fast = fast.next\n return True" + }, + { + "task_id": "task-0050", + "category": "code_debug", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Find and fix the bug in this merge sort implementation. Return ONLY the corrected code without explanation.\ndef merge_sort(arr):\n if len(arr) <= 1:\n return arr\n mid = len(arr) // 2\n left = merge_sort(arr[:mid])\n right = merge_sort(arr[mid:])\n return merge(left, right)\n\ndef merge(left, right):\n result = []\n i = j = 0\n while i < len(left) and j < len(right):\n if left[i] < right[j]:\n result.append(left[j])\n i += 1\n else:\n result.append(right[i])\n j += 1\n result.extend(left[i:])\n result.extend(right[j:])\n return result" + }, + { + "task_id": "task-0051", + "category": "code_gen", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Write a Python function that takes two numbers and returns the larger one. Return ONLY raw code." + }, + { + "task_id": "task-0052", + "category": "code_gen", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Write a Python function that returns True if a number is even, False otherwise. Return ONLY raw code." + }, + { + "task_id": "task-0053", + "category": "code_gen", + "difficulty": "easy", + "expected_route": "API_CODE", + "prompt": "Write a Python function that reverses a string. Return ONLY raw code." + }, + { + "task_id": "task-0054", + "category": "code_gen", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Write a Python function that checks whether a given string is a palindrome. Return ONLY raw code." + }, + { + "task_id": "task-0055", + "category": "code_gen", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Write a Python function that computes the nth Fibonacci number iteratively. Return ONLY raw code." + }, + { + "task_id": "task-0056", + "category": "code_gen", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Write a Python function that merges two sorted lists into one sorted list. Return ONLY raw code." + }, + { + "task_id": "task-0057", + "category": "code_gen", + "difficulty": "medium", + "expected_route": "API_CODE", + "prompt": "Write a Python function that finds the first non-repeating character in a string and returns it, or None if none exists. Return ONLY raw code." + }, + { + "task_id": "task-0058", + "category": "code_gen", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Implement Dijkstra's shortest path algorithm in Python. Given a weighted graph as an adjacency dict and a start node, return a dict of shortest distances to all nodes. Return ONLY raw code." + }, + { + "task_id": "task-0059", + "category": "code_gen", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Implement an LRU cache in Python with get(key) and put(key, value) methods, both operating in O(1) time. Return ONLY raw code." + }, + { + "task_id": "task-0060", + "category": "code_gen", + "difficulty": "hard", + "expected_route": "API_CODE", + "prompt": "Write a Python function that detects whether a singly linked list has a cycle using Floyd's tortoise and hare algorithm. Return ONLY raw code." + } +]