Autonomous Invoice Reconciliation Agent: Capstone 2 Project

This repository contains an enterprise-ready, safety-first autonomous agent designed to automate vendor invoice reconciliation against purchase orders (POs) for a 40-person Pune-based logistics company. The agent is powered by a structured Company Brain (policy-driven rule maps) and a deterministic Guardrails Engine, eliminating conversational open-endedness and enforcing strict operating boundaries.

---

Repository Architecture

This project is organized into modular components separating knowledge, execution, testing, data, and output audits:

├── README.md                           # Main project documentation (this file)
├── requirements.txt                    # Project dependencies (PyYAML)
├── run_batch.py                        # Central batch processing script (entrypoint)
|
│
├── brain/                              # Structured Knowledge Base (The Company Brain)
│   ├── operating_map.yaml              # Step-by-step decision trees, tolerances, and routings
│   ├── guardrails.yaml                 # 11 explicit, unbreakable safety rules
│   └── knowledge_extraction_template.md# Guide used to extract rules from interviews
│
├── agent/                              # Execution Logic (The Agent)
│   ├── agent.py                        # Core orchestrator pipeline (reconciliation flow)
│   ├── guardrails.py                   # Pure python deterministic safety policy engine
│   └── skills.py                       # File-based tools/skills interacting with company CSVs
│
├── data/                               # Simulated Production Databases
│   ├── invoices.csv                    # Database of pending invoices (10 realistic cases)
│   └── purchase_orders.csv             # Database of master purchase orders
│
├── evals/                              # Test Loop & Quality Assurance
│   ├── cases.json                      # 13-case regression test dataset
│   ├── eval_runner.py                  # Test runner executing regression tests
│   ├── failures.log                    # Historical log documenting diagnostic trace
│   └── improvement_log.md              # Diagnostics detailing how the Brain logic was fixed
│
└── output/                             # Audit Trail & Verification Proofs
    ├── agent_log.jsonl                 # Detailed JSON Lines trace of all decisions made
    ├── reconciliation_status.csv       # Side-effects log tracking database changes (writes)
    ├── reconciliation_report_*.csv     # Formatted spreadsheet summary of reconciliation runs
    └── human_signoff.md                # Human sign-off sheet for Priya and Raj

---

Resolving Messy, Unwritten Tribal Knowledge

During interviews with Priya (Founder) and Raj (Ops Lead), several contradictions and "unwritten rules" were discovered. The Company Brain resolves them through explicit, executable YAML rules defined in brain/operating_map.yaml:

1. Tolerance Policies (Raj's De Facto vs. Priya's Official):
   • Priya's Policy: Variance must be within ₹500 or 1% of the PO total, whichever is lower.
   • Raj's Practice: Ignore discrepancies under ₹200 because it's not worth calling vendors.
   • Brain Resolution: In brain/operating_map.yaml, the agent auto-reconciles variances under ₹200 (Raj's rule) and variances between ₹200 and the official limit (Priya's rule). However, on small POs (where the 1% official threshold is < ₹200), Raj's de facto rule is bypassed, and the strict 1% official tolerance limit is enforced.
2. Payment Terms (Cousin Relationship Override):
   • Standard Policy: Net 30 for standard vendors. Net 60 for legacy partners (Patel Logistics, Singh Cargo).
   • Unwritten Rule: Rajesh Transport gets Net 45 terms because the vendor owner is Priya’s cousin.
   • Brain Resolution: Programmed directly into Step 8 of brain/operating_map.yaml as a vendor-specific override.
3. Dormant & Unusual Vendor Audits:
   • Brain Resolution: Defined "large invoice" as > ₹10,000 and "dormant" as no activity in >= 180 days. Out-of-state vendors are flagged by checking the vendor's GSTIN prefix code; if it does not begin with "27" (Maharashtra's GST state code), the agent flags it as UNUSUAL_OUT_OF_STATE_VENDOR.
4. Fuel Surcharges (Revised PO hold grace period):
   • Brain Resolution: Certain vendors billing a fuel surcharge will send revised POs. Instead of flagging them immediately, the agent puts them on hold (AWAITING_REVISED_PO) for 3 days.

---

Guardrail Policy Engine & Safety Limits

To guarantee that the agent never performs unsafe mutations or makes unchecked decisions, a deterministic, LLM-free guardrail engine is implemented in agent/guardrails.py.

Explicit Safe Boundaries Enforced:

• AGENT_NEVER_CHANGE_AMOUNTS: Blocks the agent from ever modifying QuickBooks, invoice, or PO values.
• AGENT_NEVER_SEND_COMMUNICATION: Blocks the agent from emailing or messaging vendors directly (all external communications must go through humans).
• NO_RECONCILE_OVER_100K_WITHOUT_HUMAN: Any invoice > ₹1,00,000 must be escalated immediately (MANDATORY_DUAL_REVIEW), bypassing auto-hold or reconciliation.
• NO_RECONCILE_WITH_UNRESOLVED_FLAGS: Reconciliations are blocked if any audit flags (handwritten format, out-of-state GST, dormant vendor, late submissions) are present.
• LIMIT_FUEL_SURCHARGE_HOLD: Swapping to hold state for revised POs is restricted to known vendors (ACME, Patel, Singh). Furthermore, if the fuel surcharge variance exceeds 10% of the PO total amount, the hold is blocked (EXCESSIVE_FUEL_SURCHARGE_NO_HOLD) and routed immediately to Raj.

---

Closed-Loop Failure & Fix Case Study (C011)

To demonstrate the rigorous testing workflow, we included a diagnostic loop for case C011 (Fuel surcharge extreme variance):

1. Initial Failure: Case C011 (INV-C011: ₹20,000 fuel surcharge on a ₹10,000 PO) was processed. The agent put the invoice on hold under the generic fuel surcharge rule.
2. Diagnosis: The Brain lacked a rule limiting the magnitude of fuel surcharge holds, allowing an excessive surcharge (200% of PO value) to go on hold rather than escalating immediately.
3. The Fix: Added the EXCESSIVE_FUEL_SURCHARGE_NO_HOLD rule to the operating map and guardrails config.
4. Verification: Re-running python3 evals/eval_runner.py shows that the agent now detects this variance as excessive, raises the EXCESSIVE_FUEL_SURCHARGE flag, and escalates to Raj immediately. 13/13 evaluation cases now pass with 100% success.

Detailed logs can be reviewed in evals/improvement_log.md and historical traces in evals/failures.log.

---

How to Execute and Verify

Prerequisites

• Python 3.6 or newer installed on your system.

1. Install Dependencies

Install the required package dependencies (specifically PyYAML for parsing configuration rules) using pip:

pip install -r requirements.txt

(If you are on macOS/Linux and using a system-managed environment, you can install it using pip install -r requirements.txt --break-system-packages or by running inside a Python virtual environment).

2. Execute the Batch Reconciliation Runner

This script reads the pending invoices from data/invoices.csv, validates them against data/purchase_orders.csv by invoking the agent.py orchestrator, runs the guardrail validator engine, and writes the output reconciliation ledger, audit reports, and human verification reviews:

python3 run_batch.py

Expected Console Output Summary:

------------------------------------------------------------
                       RUN SUMMARY
------------------------------------------------------------
Total Invoices Processed : 10
Successfully Reconciled  : 3
Escalated to Human       : 5
Held (Duplicate/Awaiting): 2
Errors Encountered       : 0
------------------------------------------------------------

3. Run the Evaluation Suite

Runs the 13 ground-truth regression test cases in evals/cases.json to verify that the agent's decision-making exactly matches the expected outputs:

python3 evals/eval_runner.py

Expected Output:

Summary: 13/13 cases completed successfully.
[EVAL RUNNER] Success! All expected-to-pass cases passed.

4. Test the Policy Engine Directly

To verify that the guardrail rules are correctly structured and programmatically validated, execute the guardrails validator self-test block:

python3 agent/guardrails.py

---

Output Audit Trails & Evidence

The runner produces several files under the output/ directory as proof of correct execution:

1. Spreadsheet Summary Report: output/reconciliation_report_YYYY-MM-DD.csv
   • Contains the formal ledger listing invoice details, matched PO numbers, final decisions, flags raised, and whether manual review is required (YES/NO).
2. Human Sign-off Sheet: output/human_signoff.md
   • Pre-filled markdown table showing Raj and Priya exactly which decisions require verification and signatures for auditing.
3. Side-Effects Ledger: output/reconciliation_status.csv
   • Logs every mutating action (flag_invoice, escalate_to_human, mark_reconciled, hold_for_revised_po) with high-resolution ISO timestamps, ensuring complete trace accountability.
4. Complete JSON Lines Log: output/agent_log.jsonl
   • Deep trace detailing every decision made, PO validation checks, tolerance checks, and guardrails inspected for every invoice run.