A natural-language music recommender built on top of a content-based scoring engine. The user types a free-text request like "I want something chill for late-night studying"; an agentic loop running on Claude Haiku 4.5 plans, ranks, evaluates, broadens if needed, retrieves song-specific lore from a TF-IDF knowledge base, and writes a warm, grounded final response that names three picks and explains why each fits — using concrete details from the retrieved passages, not just genre/mood labels.
This project extends the original CodePath Applied AI Systems Engineering recommender with three production-grade additions: an agentic workflow (5 tools, plan→act→check→fix), retrieval-augmented generation (TF-IDF cosine similarity over per-song lore documents), and a reliability harness (12-case gold evaluation set + 103 pytest tests).
Music Recommender Simulation. A small Python content-based filtering engine that loaded an 18-song CSV catalog and scored each song against a user taste profile using a weighted formula over genre, mood, energy, valence, danceability, acousticness, and tempo. It shipped a CLI runner (src/main.py) and a Tkinter desktop UI (src/ui.py), and its scoring logic was deterministic and explainable — every recommendation came with a reason string showing which features earned which points.
The original system, while transparent, had three structural limitations that motivated this Module 4 rebuild: it could only accept structured slider input (no natural-language understanding), it had no notion of "what a song actually sounds like" beyond seven numeric features, and there was no way to test whether the system was getting better — you could check that the math was right, but not that the recommendations were any good. The agentic + RAG + eval rebuild addresses each of these directly.
flowchart LR
User[User: free-text request]
Catalog[(songs.csv · 18 songs)]
Lore[(song_lore/ · 18 TF-IDF docs)]
User --> Parse
subgraph Agent["Agentic Loop — Claude Haiku 4.5"]
direction TB
Parse[parse_preferences] --> Rank[rank_songs]
Rank --> Evaluate[evaluate_quality]
Evaluate --> Decide{needs<br/>refinement?}
Decide -->|yes ≤2×| Adjust[adjust_preferences]
Adjust --> Rank
Decide -->|no| Retrieve[retrieve_song_context<br/><b>RAG step</b>]
Retrieve --> Respond[final response<br/>grounded in lore]
end
Rank -.scoring.-> Catalog
Retrieve -.cosine sim.-> Lore
Respond --> User
Pytest[[pytest · 103 offline tests]] -.tests.-> Agent
EvalHarness[[eval/ · 12 gold cases]] -.tests.-> Agent
The agent runs a six-step plan-act-check loop. After parsing the user's request into structured preferences, it ranks the 18-song catalog, then calls a quality evaluator that reports issues like "no songs matched target genre" or "top score below threshold." If the quality is poor, the agent calls adjust_preferences (drop genre, drop mood, relax numeric targets, or drop both categoricals), re-ranks, and re-evaluates — up to two retries. Once the ranking is acceptable, the agent calls retrieve_song_context, which performs TF-IDF cosine similarity between the user's request and the per-song lore documents, returning the most relevant passages for the top 3 recommendations. The final response is required to reference concrete details from those passages, which is what makes the RAG step integrated rather than decorative.
A separate evaluation harness (eval/) runs the full agent over a gold test set of 12 cases and computes five per-case metrics including the critical retrieval_called flag — a case fails outright if the agent skipped the RAG step, even when the recommendation is otherwise good. This is what wires the eval to the rubric's "feature must meaningfully change behavior" requirement.
Every tool call is logged as JSON Lines to logs/recommendations.jsonl for offline inspection, and a guardrail layer (src/guardrails.py) clamps numeric inputs, normalises categorical values, and drops unknown genres/moods so a malformed LLM tool call can never crash the loop.
You will need Python 3.11+ and an Anthropic API key.
git clone <your-fork-url>
cd applied-ai-systems-engineering
python3.11 -m venv .venv
source .venv/bin/activate # Mac / Linux
# .venv\Scripts\activate # Windows
pip install -r requirements.txt
cp .env.example .env
# edit .env and paste your real ANTHROPIC_API_KEYThen run any of:
# Original deterministic recommender (Module 1-3)
python -m src.main
python -m src.ui # optional Tkinter UI
# Agentic + RAG entry point (Module 4)
python run_agent.py "I want something chill for late-night studying"
python run_agent.py --verbose "high-energy workout music" # show full tool trace
python run_agent.py # interactive demo loop
# Test suites
pytest # 103 offline tests, ~0.6s
python -m eval.run_eval --limit 3 # cheap eval smoke (3 cases, ~$0.10)
python -m eval.run_eval # full eval (12 cases, ~$0.30–$0.60)Three real transcripts pulled from eval/results/submission_v1_pre_fix.json — the report the evaluation harness wrote during the 12-case gold run. Output is verbatim from the agent.
Top 3 picks: Midnight Coding (lofi/chill, 3.24) · Library Rain (lofi/chill, 3.23) · Spacewalk Thoughts (ambient/chill, 3.23)
Perfect! Here are my top 3 chill picks for your late-night study session:
1. "Midnight Coding" by LoRoom — This hushed lofi track is built for exactly what you need. It layers a sleepy boom-bap drum loop with dusty piano chords and soft vinyl crackle, creating an instrumental groove that fades into the background without demanding attention. The mood is calm, introspective, and unhurried — perfect for late-night focus sessions and concentration.
2. "Library Rain" by Paper Lanterns — Imagine sitting in a cozy library while rain taps on the window. This track pairs a soft jazz-piano loop with gentle falling-rain ambience, creating that perfect cozy study atmosphere. With warm chords and brushed drums, it's designed specifically for deep focus and reading.
3. "Spacewalk Thoughts" by Orbit Bloom — If you want something even more minimal and meditative, this ambient piece breathes through long synth pads and distant bell tones with almost no rhythm. It's weightless and dreamy, letting your mind float freely while you study.
Phrases like "boom-bap drum loop," "vinyl crackle," "rain ambience," "long synth pads," and "distant bell tones" come directly from the retrieved lore documents — a generic LLM would not produce them. This is what RAG-grounding looks like in practice. (5 iterations · 4 tool calls · 11.3s)
Top 3 picks: Iron Anthem (metal/rebellious, 5.54) · Storm Runner (rock/intense, 2.21) · Night Drive Loop (synthwave/moody, 1.99)
Perfect! Here are your angry, rebellious metal recommendations:
Iron Anthem by Black Summit is exactly what you're looking for. This furious metal track is absolutely packed with drop-tuned guitars, blast-beat drums, and a roared chorus about defiance — it's all about breaking systems and refusing to bend. It's a wall-of-sound heavyweight designed for moments when you need to scream into a pillow with a soundtrack.
Storm Runner by Voltline brings that high-octane rock energy if you want a close second. It features overdriven guitars and double-time drums with lyrics pushing themes of defiance and breaking through chaos.
Night Drive Loop by Neon Echo offers an alternative if you want something darker — glossy synthwave with a brooding bassline that evokes 1980s nostalgia. While it's less of a scream-along anthem, it's perfect if you want to sit with that anger in a more atmospheric vibe.
The agent honestly downgrades the second and third picks ("close second," "alternative") because their scores (2.21, 1.99) are well below Iron Anthem's 5.54. It also broadens beyond pure metal into rock and synthwave because the catalog has only one metal song — and tells the user it's doing so. (5 iterations · 4 tool calls · 18.0s)
Top 3 picks: Blue Neon Smoke (blues/soulful, 2.79) · Iron Anthem (metal/rebellious, 0.86) · Storm Runner (rock/intense, 0.72)
For a heartbreak night, I'd recommend "Blue Neon Smoke" by Velvet Haze — this is exactly what you're looking for. It's a smoky, slow-burn blues track built on rolling Hammond organ and weeping guitar bends, with a soulful vocal that aches in all the right places. The lyrics tell of late-night rooms and lost love, making it ideal for those intimate, melancholic moments.
"Iron Anthem" is a heavy metal track that could work if you want to channel your heartbreak into something more cathartic and defiant.
"Storm Runner" is a driving rock anthem that might appeal if you're looking to process heartbreak through motion and energy rather than deep introspection.
But honestly, "Blue Neon Smoke" is your best match — it's specifically crafted for those heartbreak nights when you need a soundtrack that understands the ache.
Notice how the agent leans hard on "Hammond organ" and "weeping guitar bends" — phrases unique to the lore document for this song. The other two picks are framed as cathartic alternatives rather than equals, because their scores show they're not really blues matches. (6 iterations · 5 tool calls · 24.5s)
Agentic loop over a single LLM call. A single prompt-completion would have to do parsing, ranking, broadening, and grounding all at once, with no observability into where the reasoning went wrong. The 5-tool loop separates concerns cleanly: each tool is a pure Python function whose inputs and outputs are inspectable in the JSON-Lines log. When something fails, the trace tells you exactly which step misbehaved. The trade-off is latency — the loop costs 5–10 seconds and 4–8 LLM round-trips — but for a recommender used in a homework / portfolio setting, that's a fair price for being able to debug the system.
TF-IDF over embedding-based retrieval. The catalog has 18 documents. Sentence-transformer embeddings would pull in PyTorch (~2 GB download) and require a model warm-up, with no measurable retrieval-quality gain at this scale because the lore vocabulary is distinctive (genre names, instrument names, vibe descriptors). TF-IDF with 1-2-grams gives a 1291-token vocabulary that lands the right answer in the top-3 for every query in the gold set, runs in under a millisecond, and is fully deterministic — which makes the eval harness reproducible. The trade-off is no semantic-synonym matching ("upbeat" doesn't auto-match "energetic" unless both words appear in the lore), which I mitigated by writing each lore blurb with deliberately rich, synonym-heavy vocabulary.
Keyword-based lore-grounding metric instead of LLM-as-judge. I authored the lore documents myself, so I know exactly which phrases ("boom-bap," "drop-tuned," "Hammond organ," "lap steel," "dembow") are vanishingly unlikely to appear in a generic LLM answer about chill music or workout music. Their presence in the response is therefore strong evidence that the retrieved passages shaped the output. This metric is deterministic, free, and reproducible, which is what a homework reliability rubric rewards. It is a proxy for grounding rather than a direct semantic measure — listed as future work in the model card.
retrieval_called is a hard fail condition. The eval treats a missing retrieve_song_context call as an automatic case failure, even when the recommendation is otherwise correct. This wires the eval directly to the rubric's "feature must meaningfully change behavior" gate: a run that skips RAG and answers from priors cannot count as an integrated RAG system, regardless of how good the answer looks.
Defensive tool executors over strict validation. When the LLM passes a malformed argument (a recommendations array missing a score field, a song_ids value that's a string instead of a list), the executors return a warning payload rather than raising. The agent loop sees the warning, can react, and progresses. The alternative — strict schema validation — would crash the whole run on a single LLM transcription error, which the live eval showed happens often enough to matter (see Testing Summary).
The project ships with two complementary test layers. Together they cover the codepaths a deployed agent actually exercises.
Layer 1 — pytest (103 deterministic offline tests, runs in ~0.6 s). Covers every public function in src/recommender.py, src/guardrails.py, src/retriever.py, every tool executor in src/agent.py (including the RAG executor and its 8 input-validation paths), every metric in eval/metrics.py, and a full integration test of the agent loop with a mocked Anthropic client (tests/test_agent_loop.py) that proves the dispatcher, message-history bookkeeping, end-turn termination, MAX_ITERATIONS bail-out, and unknown-tool resilience all work without making any API calls.
Layer 2 — Live eval harness (python -m eval.run_eval). Runs the full agent over 12 gold cases — chill_study, workout_pump, nostalgic_acoustic, party_dance, jazz_brunch, metal_anger, classical_meditation, synthwave_drive, latin_summer, blues_heartbreak, vague_happy, quiet_morning. For each, it captures per-case metrics (retrieval-call, genre hit, mood hit, optional song-id hit, lore-grounding keyword check) and writes a timestamped JSON report.
Submission report — 10 / 12 passed (83.3 %), threshold 70 %. Aggregate metrics from eval/results/submission_v1_pre_fix.json: retrieval-call rate 83.3 %, genre hit rate 83.3 %, mood hit rate 83.3 %, lore-grounded rate 83.3 %, mean iterations 4.92, mean tool calls 4.08, mean latency 16.3 s.
The two failing cases caught a real production bug. Both synthwave_drive and quiet_morning crashed with KeyError (one for score, one for mood) inside _exec_evaluate_quality. Pytest had passed every test up to that run — because pytest only ever passed well-formed inputs. The live eval discovered that Claude occasionally re-types a recommendations array from memory and drops a field, which my code was not defensive against. I fixed _exec_evaluate_quality to use .get() with safe defaults and surface missing-field warnings as informational issues (not refinement triggers), and added six regression tests covering missing score, missing mood, non-dict entries, all-malformed input, non-numeric scores, and the "informational issues don't force refinement" rule. The v1 report is preserved at eval/results/submission_v1_pre_fix.json as evidence the eval did its job.
This is the strongest argument I can make for why the eval harness exists: pytest tests the code you wrote against the inputs you imagined; the live eval tests the full system against the inputs the LLM actually produces. They catch different bugs.
The biggest takeaway from this rebuild: the boundary between "code that works" and "code that's reliable" is the eval harness. I had 97 passing pytest tests before the live eval ran, and I would have shipped a system with two KeyError crash paths in production-relevant codepaths. The eval found them in 12 runs.
The second takeaway: RAG with a small, locally-authored knowledge base is shockingly effective for grounding. The cosine-similarity over 18 short markdown documents costs nothing to run, requires zero model weights, and meaningfully changes the texture of every response — picks 1 and 2 in Example 3 above quote phrases ("Hammond organ," "weeping guitar bends") that the agent had no way to invent without retrieval.
The full reflection — including AI-collaboration moments where the assistant gave both a useful and a flawed suggestion — is in model_card.md.
python run_agent.py --verbose for 3 prompts and a python -m eval.run_eval --limit 3 end-to-end]
applied-ai-systems-engineering/
├── README.md ← this file
├── model_card.md ← reflections, biases, AI collaboration story
├── reflection.md ← original Module 1-3 recommendation comparisons
├── requirements.txt
├── run_agent.py ← main CLI entry point (Module 4)
├── .env.example
├── .gitignore
│
├── data/
│ ├── songs.csv ← 18-song catalog (Module 1-3)
│ └── song_lore/ ← 18 TF-IDF lore documents (Module 4)
│
├── src/
│ ├── agent.py ← agentic loop, 5 tool executors
│ ├── recommender.py ← deterministic content-based scorer (Module 1-3)
│ ├── retriever.py ← TF-IDF lore retriever (Module 4)
│ ├── guardrails.py ← input validation + clamping
│ ├── logger.py ← JSON-Lines structured logger
│ ├── main.py ← original CLI for the deterministic recommender
│ └── ui.py ← original Tkinter desktop UI
│
├── tests/ ← 103 offline pytest tests
│ ├── conftest.py
│ ├── test_agent.py ← guardrails, all 5 tool executors, defensiveness regressions
│ ├── test_agent_loop.py ← full agent loop with mocked Anthropic client
│ ├── test_eval_metrics.py ← every metric + the critical "passed requires retrieval" rule
│ ├── test_recommender.py ← original deterministic scorer
│ └── test_retriever.py ← TF-IDF retriever, lore loading, edge cases
│
├── eval/ ← live evaluation harness (Module 4)
│ ├── README.md ← detailed eval design + metric definitions
│ ├── test_cases.py ← 12 EvalCase records (the gold set)
│ ├── metrics.py ← pure-function metrics, no LLM
│ ├── run_eval.py ← CLI runner (--limit, --case, --no-color, --out)
│ └── results/
│ └── submission_v1_pre_fix.json ← live eval report (10/12, caught two bugs)
│
├── scripts/ ← dev/smoke utilities (not deliverables)
│ ├── README.md
│ ├── smoke_retriever.py
│ └── smoke_agent_tool.py
│
├── assets/ ← original CLI screenshots
└── logs/
└── recommendations.jsonl ← runtime structured logs (gitignored)