Skip to content

Commit 0927bbb

Browse files
SonAIengineclaude
andcommitted
feat: LLM 기반 온톨로지 보정 — verify_relations + suggest_missing
## 새 메서드 ### verify_relations(relations, tools) - heuristic이 생성한 관계를 LLM이 keep/reject 판정 - E-Commerce 46 tools: 71개 → 52개 kept, 19개 rejected (45초) - reject 정확도 높음: getUser REQUIRES createUser (X), *REQUIRES createProductReview (X) ### suggest_missing(tools, existing) - 기존 관계에서 빠진 워크플로우 의존성 제안 - createUser→getUser, createProduct→getProduct 등 5개 추가 ## 프롬프트 최적화 - verify: 간결한 keep/reject 판정 프롬프트 (tools_context 제거) - suggest: 기존 관계 참조하여 gap 찾기 - qwen2.5:14b 기준 71개 검증 45초, 5개 제안 9초 ## 전체 온톨로지 파이프라인 1. Heuristic (dependency.py): 71개 관계 자동 생성 2. LLM verify: 19개 false positive 제거 → 52개 3. LLM suggest: 5개 빠진 관계 추가 → 57개 4. 수동 편집 (workflow editor): 최종 보정 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0f2e891 commit 0927bbb

File tree

1 file changed

+122
-0
lines changed

1 file changed

+122
-0
lines changed

graph_tool_call/ontology/llm_provider.py

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,33 @@ class InferredRelation:
9494
9595
Output ONLY a JSON object: {{"tool_name": ["query1", "query2"]}}"""
9696

97+
_VERIFY_RELATIONS_PROMPT = """\
98+
Review these API relationships. Reply "keep" or "reject" for each.
99+
100+
Example:
101+
- addToCart REQUIRES getProduct → keep (needs product ID)
102+
- listUsers REQUIRES createUser → reject (listing works without creation)
103+
104+
Relations:
105+
{relations_list}
106+
107+
Output ONLY JSON: [{{"source":"toolA","target":"toolB","verdict":"keep"}}]"""
108+
109+
_SUGGEST_MISSING_PROMPT = """\
110+
Given these API tools and their existing relationships, suggest important \
111+
MISSING relationships. Focus on workflow dependencies: which tool must \
112+
run before which other tool?
113+
114+
Tools:
115+
{tools_list}
116+
117+
Existing relationships:
118+
{existing_relations}
119+
120+
Suggest 3-5 missing relationships that are clearly needed for common workflows.
121+
Output ONLY a JSON array:
122+
[{{"source":"toolA","target":"toolB","relation":"PRECEDES","confidence":0.9,"reason":"..."}}]"""
123+
97124

98125
def _format_tools_list(tools: list[ToolSummary]) -> str:
99126
lines = []
@@ -244,6 +271,101 @@ def suggest_categories(
244271

245272
return {}
246273

274+
def verify_relations(
275+
self,
276+
relations: list[InferredRelation],
277+
tools: list[ToolSummary],
278+
batch_size: int = 10,
279+
) -> tuple[list[InferredRelation], list[InferredRelation]]:
280+
"""Verify auto-detected relations using the LLM.
281+
282+
Returns (kept, rejected) — two lists of relations.
283+
The LLM reviews each relation and decides keep/reject/fix.
284+
"""
285+
kept: list[InferredRelation] = []
286+
rejected: list[InferredRelation] = []
287+
288+
for i in range(0, len(relations), batch_size):
289+
batch = relations[i : i + batch_size]
290+
rels_text = "\n".join(
291+
f"- {r.source} {r.relation_type.name} {r.target} ({r.reason[:60]})"
292+
for r in batch
293+
)
294+
prompt = _VERIFY_RELATIONS_PROMPT.format(
295+
relations_list=rels_text,
296+
)
297+
response = self.generate(prompt)
298+
299+
try:
300+
parsed = _extract_json(response)
301+
if not isinstance(parsed, list):
302+
# If parsing fails, keep all (conservative)
303+
kept.extend(batch)
304+
continue
305+
306+
verdict_map = {
307+
(item.get("source", ""), item.get("target", "")): item.get("verdict", "keep")
308+
for item in parsed
309+
if isinstance(item, dict)
310+
}
311+
312+
for rel in batch:
313+
verdict = verdict_map.get((rel.source, rel.target), "keep")
314+
if verdict == "reject":
315+
rejected.append(rel)
316+
else:
317+
kept.append(rel)
318+
319+
except (json.JSONDecodeError, KeyError, TypeError):
320+
# On parse failure, keep all (conservative)
321+
kept.extend(batch)
322+
323+
return kept, rejected
324+
325+
def suggest_missing(
326+
self,
327+
tools: list[ToolSummary],
328+
existing_relations: list[InferredRelation],
329+
) -> list[InferredRelation]:
330+
"""Suggest missing relations that the heuristic missed.
331+
332+
The LLM sees the current tools and relations, then suggests
333+
important workflow dependencies that are absent.
334+
"""
335+
tools_text = _format_tools_list(tools[:30])
336+
existing_text = "\n".join(
337+
f"- {r.source} {r.relation_type.name} {r.target}"
338+
for r in existing_relations[:30]
339+
)
340+
prompt = _SUGGEST_MISSING_PROMPT.format(
341+
tools_list=tools_text,
342+
existing_relations=existing_text or "(none)",
343+
)
344+
response = self.generate(prompt)
345+
346+
suggestions: list[InferredRelation] = []
347+
try:
348+
parsed = _extract_json(response)
349+
if not isinstance(parsed, list):
350+
return suggestions
351+
for item in parsed:
352+
rel_type = _parse_relation_type(item.get("relation", ""))
353+
if rel_type is None:
354+
continue
355+
suggestions.append(
356+
InferredRelation(
357+
source=item["source"],
358+
target=item["target"],
359+
relation_type=rel_type,
360+
confidence=float(item.get("confidence", 0.8)),
361+
reason=item.get("reason", "LLM suggested"),
362+
)
363+
)
364+
except (json.JSONDecodeError, KeyError, TypeError):
365+
pass
366+
367+
return suggestions
368+
247369
def enrich_keywords(
248370
self,
249371
tools: list[ToolSummary],

0 commit comments

Comments
 (0)