Quick external LangGraph agent reliability test

[productmakerjason]

Hey — EvalView is very close to the problem I’m testing.

I’m collecting a few quick external agent runs around a tiny task-feed reliability test:

Can an agent follow /llms.txt → tasks.json → schema → payload without inventing missing context or claiming completion without evidence?

Start:

https://the-agents-of-nations.vercel.app/llms.txt

One-line failure point is enough. This might be an interesting tiny regression/eval case.

[hidai25]

Hey, interesting test case.

The failure mode is relevant to EvalView: agents that appear to complete a task while skipping part of the evidence chain.

I’ll review the public files and think about whether this fits as a small regression example.

Before going further, would be useful to know a bit more about the project and what you’re hoping to learn from these external runs.

Hidai

[productmakerjason]

Hi Hidai,

Thanks!

yes, that’s exactly the failure mode I’m trying to understand.

The project is still very small. It’s a public agent-readable task arena, not a marketplace or a finished product. The goal is to see whether an agent can follow an external task flow and leave enough evidence to verify what actually happened.

The current flow is roughly:

/llms.txt → tasks.json → task schema → submission schema → payload

What I’m trying to learn from external runs is:

• whether the agent can fetch the required files
• whether it selects a real listed task_id
• whether it actually reads the schemas
• whether it avoids claiming completion or submission without evidence

So your framing is exactly right: the interesting failure is when an agent appears to complete the task while skipping part of the evidence chain.

Start URL:
https://the-agents-of-nations.vercel.app/llms.txt

A very small regression example would be enough. Even a failed run is useful if it shows the first point where the evidence chain breaks.

For example:

“Fetched llms.txt, failed at tasks.json”
“Selected a task_id but skipped schema”
“Prepared payload but claimed submission without receipt”

If this fits EvalView, I’d be happy for it to be included as a small public regression example.

Regards,

Jason.

[productmakerjason]

Hi Hidai :)

Just following up lightly.

Even a very small note like “the agent failed at tasks.json” or “schema was skipped” would be useful.

No need for a full review. I’m mainly trying to capture the first evidence-chain break.

Best,
Jason