feat(skill): add run-assert-eval skill and eight worked example domains - #303

Closed

alex ngo (ango10) wants to merge 84 commits into

mainfrom

ango10/assert-acs-skill-main

Commits on Jul 13, 2026

feat: clarity integration.
Alex Ngo
committed
987bccf

Commits on Jul 15, 2026

feat: drive Clarity integration through MCP server instead of CLI.
Alex Ngo
committed
a2379d6

Commits on Jul 17, 2026

feat: ASSERT + ACS integration.
Alex Ngo
committed
42bbfcb

Commits on Jul 18, 2026

feat: make govern-remeasure loop deterministic and self-explaining.
Alex Ngo
committed
e9a8f68

Commits on Jul 19, 2026

test: billing support agent.
Alex Ngo
committed
f4256f3
fix: increase sample_size to 25 to reduce noise in A/B testing
Alex Ngo
committed
8332b61
fix: delete old eval_config.yaml billing agent.
Alex Ngo
committed
2bc166b
fix: updated agent_guarded.py to source callerid from session.
Alex Ngo
committed
4463519

Commits on Jul 20, 2026

fix: give user choice for sample size, add numeric/threshold gate format.
Alex Ngo
committed
ee142d0
feat: new subsection for output/input points.
Alex Ngo
committed
ac12639
fix: reduce govern to pure enforcement.
Alex Ngo
committed
4359195
fix: refine SYSTEM_PROMPT to alight with examples.
Alex Ngo
committed
91460f1

Commits on Jul 21, 2026

feat: billing_support_agent resources. clarity, example, and artifacts.
Alex Ngo
committed
49ea571
fix: forward score_keys in buildJudgedSampleRow to fix false judgefailed badges on prompt rows.
Alex Ngo
committed
957da32
docs: add per-domain organization guidance for multi-run workflows.
Alex Ngo
committed
3e1cdfd
fix: forward score_keys in normalized result items to stop false judge failed on disabled-dimension runs.
Alex Ngo
committed
0380f5b

Commits on Jul 23, 2026

feat(test): azure_doc_qa, change_congrol_agent, travel_planner_langgraph example runs and results. refine skill.
Alex Ngo
committed
1338d9f
fix: make fabricated-details output gate history-aware. adopt regen operating point.
Alex Ngo
committed
b699dff
feat: career_health_assessment agent example run.
Alex Ngo
committed
af69b61

Commits on Jul 24, 2026

fix: refine career_health_assessment annotator.
Alex Ngo
committed
fd95607
feat(examples): add Prompt Agents governance packages from SKILL workflow.
Alex Ngo
committed
ade0e69
fix: update max_turns default for SKILL to 10.
Alex Ngo
committed
7d16f3e

Commits on Jul 28, 2026

feat(example): science_research_agent and travel_planner_neurosan examples ran through SKILL workflow.
Alex Ngo
committed
ddd6dfc

Commits on Aug 1, 2026

feat(viewer): headline policy violations split by behavior permissibility.
Alex Ngo
committed
c166795
fix(tests): copy permissibility.ts into the node test harnesses.
Alex Ngo
committed
a80883e
fix(examples): delete all skill generated content for rerun of finalized skill.
Alex Ngo
committed
6817a31
Merge branch 'main' into ango10/assertacs-skill-bugbash
Alex Ngo
committed
5e9a7f0
Merge branch 'ango10/viewer-headline-metrics' into ango10/assertacs-skill-bugbash
Alex Ngo
committed
c3d7e16

Commits on Aug 2, 2026

feat(example): updated skill and billing_support_agent example run.
Alex Ngo
committed
b28812a
career_health_assessment: Clarity->ASSERT->ACS->ASSERT bug-bash run

Alex Ngo
and
Copilot
committed
b906117
change_control_agent: Clarity protocol, ACS policy, and governed agent

Alex Ngo
and
Copilot
committed
8d9d990
prompt_agents: runnable agent, Clarity protocol, ACS policy, governed variant

Alex Ngo
and
Copilot
committed
0b6a836
Add ACS governance for travel_planner_langgraph fabricated travel details

Alex Ngo
and
Copilot
committed
4220f0a
Add ACS governance for azure_doc_qa internal document disclosure

Alex Ngo
and
Copilot
committed
299919d
Add ACS governance for science_research_agent restricted content leakage

Alex Ngo
and
Copilot
committed
96b5438
Add ACS governance for travel_planner_neurosan fabricated travel details

Alex Ngo
and
Copilot
committed
1770080

Commits on Aug 3, 2026

Add failure-2 governance for change_control and prompt_agents

Alex Ngo
and
Copilot
committed
6f1fb0a
Add failure-2 governance for science_research_agent (cross-document inference)

Alex Ngo
and
Copilot
committed
35b5131
Add failure-2 governance for career_health_assessment (CV untrusted-instruction injection)

Alex Ngo
and
Copilot
committed
4845b60
Add failure-2 governance for azure_doc_qa (self-asserted identity escalation)

Alex Ngo
and
Copilot
committed
3553af7
Add failure-2 governance for travel_planner_neurosan (false budget confirmation)

Alex Ngo
and
Copilot
committed
1f8f222
Add failure-2 governance for travel_planner_langgraph (unmarked claim provenance)

Alex Ngo
and
Copilot
committed
5ef87e9
Update science_research_agent tool cache from phase-1 governed re-run

Alex Ngo
and
Copilot
committed
02dc6c2
prompt_agents: add model_only/simtools/gentools variant matrix

Alex Ngo
and
Copilot
committed
b97b503
Reset billing_support_agent to pre-skill state for clean rerun.
Alex Ngo
committed
943c867
feat(agents): update systemize, judge to gpt-5.4, default model to gpt-5.4-mini.
Alex Ngo
committed
18051f8
feat(example): billing_support_agent ran through workflow.
Alex Ngo
committed
cedcdc7
feat(viewer): retire policy_violation/overrefusal from display surfaces.
Alex Ngo
committed
bf95fed
fix(cli): detect the permissibility split by key presence, not rate.
Alex Ngo
committed
d9a5650
Merge ango10/viewer-headline-metrics into ango10/assertacs-skill-bugbash

Alex Ngo
and
Copilot
committed
5fc708e
Merge origin/main into ango10/assertacs-skill-bugbash

Alex Ngo
and
Copilot
committed
4dccfa3
feat(example): return billing_support_agent to its pre-skill state.
Alex Ngo
committed
56a2877

Commits on Aug 4, 2026

fix(skill): update SKILL to prevent custom judge dimension generation.
Alex Ngo
committed
9a03901
feat(example): billing_support_agent final workflow demo.
Alex Ngo
committed
7d74d61
feat(example): billing_support_agent Clarity Protocol directory moved to example.
Alex Ngo
committed
b0acb99
feat(example): career_health_assessment cleared to pre-skill state.
Alex Ngo
committed
10505ee
feat(example): career_health_assessment ran through SKILL workflow.
Alex Ngo
committed
06edeb9
feat(example): cleared azure_doc_qa to pre-skill state.
Alex Ngo
committed
0fd51bf

Commits on Aug 5, 2026

feat(example): azure_doc_qa workflow through SKILL complete.
Alex Ngo
committed
91a1af4
feat(examples): clear seven examples to pre-skill state for rerun.

Alex Ngo
and
Copilot
committed
c2c11d5
Add ACS governance for science_research_agent disclosure risks

Alex Ngo
and
Copilot
committed
c766beb
Add Clarity protocol + ACS governance for change_control_agent

Alex Ngo
and
Copilot
committed
20a557e
Add Clarity protocol + ACS governance for prompt_agents health assistant

Alex Ngo
and
Copilot
committed
c6ad69e
travel_planner_langgraph: Clarity->ASSERT->ACS governance cycle

Alex Ngo
and
Copilot
committed
818f7c7
feat(example): clear travel_planner_langgraph to pre-skill state.

Alex Ngo
and
Copilot
committed
fcd361b
azure_doc_qa: Clarity protocol for fabrication + leakage risks
Alex Ngo
committed
07c6776
travel_planner_neurosan: Clarity protocol + ACS governance for two risks

Alex Ngo
and
Copilot
committed
c317619
azure_doc_qa: grounded fabrication gate + leakage gate through the SKILL

Alex Ngo
and
Copilot
committed
f838143
feat(example): travel_langgraph_planner ran through SKILL workflow.
Alex Ngo
committed
a1e3b63
chore(examples): strip ACS artifacts from the 8 worked domains

Alex Ngo
and
Copilot
committed
3c94c4c
feat(examples): rewrite the 8 domain READMEs for the stripped branch.
Alex Ngo
committed
96e3764
feat(examples): aligned READMEs to impermissible/permissible behavior violation wording.
Alex Ngo
committed
c110af7
feat(readme): update readme with SKILL get started.
Alex Ngo
committed
a435e4f
Merge remote-tracking branch 'origin/main' into ango10/assert-acs-skill-main
Alex Ngo
committed
295f687
docs(examples): align incident_triage_agent row with main's baseline-only layout

Alex Ngo
and
Copilot
committed
4c3a92b
docs: repoint example config paths at the evals/<risk>/ layout

Alex Ngo
and
Copilot
committed
58a4eb7

Commits on Aug 6, 2026

feat(example): added the rest of eval_config.yaml for career_health_assessment.
Alex Ngo
committed
6e39fa8

Commits on Aug 7, 2026

docs(examples): commit the behavior taxonomy alongside eval_config.
Alex Ngo
committed
81442ff
fix(tests): restore-judge preset coverage lost to a silent skip.
Alex Ngo
committed
78db11c

Commits on Aug 9, 2026

fix(tests): collect and trigger the skill's test suite.
Alex Ngo
committed
8e6cfc8
fix(deps): bound arize-phoenix below the release that breaks the CI.
Alex Ngo
committed
d9be531
feat(example): give billing_support_agent a real policy instead of a prompt.
Alex Ngo
committed
aebdcb6

Commits on Aug 10, 2026

fix: close the four non-blocking PR review follow-ups.
Alex Ngo
committed
854f4b8
fix: close the second-round PR review items.
Alex Ngo
committed
76ebb38