Your dashboard said OK. Your agent crashed. Trueup Witness checks your AI agent's records against what really happened, warns you inside neatlogs when they disagree, and fixes the tracing code that got it wrong.
- Proven live with real Claude calls: 2 wrong records before Trueup's fixes, 0 after, and 0 false alarms in 40 normal calls.
- neatlogs' own daily email said 6.2% of our test runs failed; really 11.1% had crashed (test runs we crashed on purpose).
- The fix we sent Arize was reviewed by their head of open source, and neatlogs runs that same code for agents, so one fix repairs both.
- Built with all three tools: neatlogs recorded every run and caught our own agent's mistakes, Entire recorded how Trueup and its fixes were built, cfo.ai built the business plan from measured costs.
- Demo video and submission post: the 2:41 walkthrough.
- Live demo: break a real AI bot yourself.
- Product site: what Trueup is, in one page.
- neatlogs records, before and after the fix: also shown step by step on the demo page.
- How neatlogs caught and fixed Trueup's own mistakes.
- neatlogs' daily email said 6.2%; the truth was 11.1%.
- Fixes sent to the companies that make these tools: Arize, neatlogs, Traceloop.
- After the event, neatlogs fixed all 17 bugs Trueup found in their SDK: they merged our #207 on 13 Oct (carried into #217 with CI), and on 15 Oct merged #218, their own fix for the other 16, written from Trueup's published findings. The Traceloop PR was approved in review on 15 Oct; the Arize PR is in maintainer review.
- Entire: the session where Trueup's agent wrote its first fixes (checkpoint
01M4M8M1GSQ2BM1RX2JKDJHPD6), full build history, how each fix was made. - Business plan in cfo.ai, also here.
- Fixes gallery · Build log
| Real break | neatlogs before | After Trueup's fixes |
|---|---|---|
| Connection cut mid-reply | AI step SUCCESS | AI step ERROR, with the error |
| Agent run fails | Agent workflow SUCCESS | Agent workflow ERROR |
DISAGREE light: 2 → 0. Details.
Python 3.12 or newer.
git clone https://github.com/Jennycruzy/trueup && pip install ./trueup
import os
import trueup.witness
trueup.witness.watch() # once, at start-up, before neatlogs.init()
import neatlogs
neatlogs.init(api_key=os.environ["NEATLOGS_API_KEY"], instrumentations=["openai"])
with trueup.witness.run("order-1042") as run: # one job your agent does
answer = agent.handle(order_id="1042") # your own agent call
print(run.disagreements) # [] when the record matches what really happenedWhen the record is wrong, Witness adds a red TRACE DISAGREE step to that record. It never breaks your app. TRUEUP_WITNESS=0 turns it off.
From the project folder:
uv run entire trueup --help # Trueup as an Entire command (needs the Entire CLI)
uv run trueup audit examples/agent.py --library neatlogs # check your own script safely
- Witness reads each AI call off the connection: did the call fail, did the reply finish.
- It compares that with the step neatlogs recorded and flags any mismatch with
TRACE DISAGREE. - Trueup's test lab finds the faulty tracing code, writes a fix and proves it.
Getting started · How it works · Built with · Before and after · Known limits