Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion evaluation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@

通常の `npm test` はタイトル分類や原文照合などの回帰テストです。投資判断の有用性は測りません。従来の実 PDF スクリプトは空ページと指定語の有無を確認します。事実要約の9件については、主要数値・単位・期間・物理ページの期待値を別のfixtureに記録しています。

今回の決算・診断の操作ストーリー、保持上限、26頁の固定応答再生は[結果整合性の検証記録](earnings-result-consistency.md)を参照。

## 検証の選択と停止条件

恒久テストの追加・削除は[開発ガイドのテスト戦略](../docs/development.md#テスト戦略)に従う。今回の変更で証明が必要な責務を先に選び、成功済みの同条件検証を機械的に繰り返さない。
Expand Down Expand Up @@ -55,7 +57,6 @@ v93は表配置ではなく意味の契約を変更するため、World/BlueMeme

**v92の実施結果。** 11実行/33APIで終了。A/B/Cでは共通の原文単位・表記・複数行範囲の不具合を発見して停止、残り3回だけDで実施した。履歴全体の数値期待値は7/11、表示10/11、説明partial9/unavailable1/未保存1。条件が異なるため最終版の成功率とはしない。Dは数値期待値/表示3/3、説明partial2/unavailable1。記録のある10回の補足生成/点検は出力打切り・期限到達0(最大48.849秒/4,778出力token)。Remixは3要求後の表示照合で例外終了し、生応答/usageが最終保存前に失われた。元の失敗を保持し、追加試行せず、原PDFの10数量と5範囲の表示対応をAPI不要で再確認した。評価器は要求ごとのチェックポイントと表示失敗の記録を追加した。`results/local/summary-structure-v92-{plan,ledger}.json`と時刻付きfact-summary JSONが正本。最終画面は実応答の固定再生によるWorld/BlueMeme/月次/CFと説明unavailableの保存復元・診断コピーを確認し、モデルの欠落を成功へ読み替えない。詳細と配置先は[開発ガイド](../docs/development.md)のv92最終確認へ記録した。


v97は、確定済みの出来事を補足の成否に依存せず「確認済み事項(原文)」として表示し、冒頭の理由・条件は点検済み説明と根拠IDで対応させる。同額の別セルを統合せず、位置で同一性を証明できる行内数量だけを物理IDに一本化する。過去資料の採点は事実抽出だけを呼び出す。本文number/rangeの指示も数量kindへ合わせた。今回の検証は固定応答・保存復元と通常のローカル検査であり、追加の実モデル評価は行っていない。v96の実モデル記録はv97の成功実績へ読み替えない。

v98はPR #27の境界監査に基づき、出来事の別注記条件、3期比較の中間行、分割単位の完全性と物理セルの別名を修正する。担当の固定応答・表示・保存検査を使い、実API・実拡張画面・配置は未実施。表示数量の生成入力が変わる資料について実モデル品質を保証しない。詳細は[開発ガイドのレビュー境界監査](../docs/development.md#pr-27のレビュー境界監査2026-10-06v98)に記録する。
Expand Down
198 changes: 198 additions & 0 deletions evaluation/earnings-result-consistency.md

Large diffs are not rendered by default.

47 changes: 47 additions & 0 deletions evaluation/scripts/check-earnings-story.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/** Local-only 26-page replay; no PDF download, API key, or external network. */
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { readFile, mkdir, writeFile } from 'node:fs/promises';
import { getDocument, OPS } from 'pdfjs-dist/legacy/build/pdf.mjs';
import { extractPdfPageLayout } from '../../src/lib/pdf-layout';
import { earningsStorySource, replayEarningsStory } from './earnings-story-fixture';
const file = process.argv[2];
assert.ok(
file,
'Usage: node --import tsx evaluation/scripts/check-earnings-story.ts /path/to/140120261008547647.pdf'
);
const bytes = await readFile(file);
assert.equal(
createHash('sha256').update(bytes).digest('hex'),
earningsStorySource.sha256,
'Use the exact independently checked public PDF'
);
const document = await getDocument({
data: new Uint8Array(bytes),
cMapUrl: `${process.cwd()}/node_modules/pdfjs-dist/cmaps/`,
cMapPacked: true,
useWorkerFetch: false,
isEvalSupported: false,
useSystemFonts: true,
}).promise;
const pages = [];
try {
assert.equal(document.numPages, 26);
for (let n = 1; n <= document.numPages; n++) {
const page = await document.getPage(n);
pages.push(await extractPdfPageLayout(page, n, OPS));
page.cleanup();
}
} finally {
await document.destroy();
}
assert.ok(pages.every((page) => page.status === 'ok'));
const { html, markdown, result, ...evidence } = await replayEarningsStory(pages);
await mkdir('evaluation/results/local', { recursive: true });
await writeFile(
'evaluation/results/local/earnings-user-story.json',
JSON.stringify({ pdfSha256: earningsStorySource.sha256, ...evidence, result }, null, 2)
);
await writeFile('evaluation/results/local/earnings-user-story.html', html);
await writeFile('evaluation/results/local/earnings-user-story.md', markdown);
console.log(JSON.stringify({ pdfSha256: earningsStorySource.sha256, ...evidence }, null, 2));
10 changes: 7 additions & 3 deletions evaluation/scripts/check-extension-smoke.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { validateSavedFacts } from '../../src/lib/fact-cache';
import { validatePresentation } from '../../src/lib/summary-presentation';
import { SUMMARY_TRACE_KEY } from '../../src/lib/summary-trace';
import { SUMMARY_DIAGNOSTICS_KEY } from '../../src/lib/summary-trace';
import {
preflightExtensionSmoke,
SMOKE_API,
Expand Down Expand Up @@ -162,10 +162,13 @@ export async function checkExtensionSmoke(args: string[]) {
worker.evaluate(async (traceKey: string) => {
const data = await chrome.storage.local.get();
return {
trace: data[traceKey],
trace: data[traceKey]?.traces?.find(
(trace: { pdfUrl: string }) =>
trace.pdfUrl === 'https://www.release.tdnet.info/inbs/fixture-extension-smoke.pdf'
),
caches: Object.entries(data).filter(([key]) => key.startsWith('summaryCacheV2:')),
};
}, SUMMARY_TRACE_KEY);
}, SUMMARY_DIAGNOSTICS_KEY);
const stored = await readStored();
assert.equal(
stored.trace.buildDigest,
Expand Down Expand Up @@ -217,6 +220,7 @@ export async function checkExtensionSmoke(args: string[]) {
outcome === 'success' ? 'ready' : 'unavailable'
);
assert.equal(saved.resultId, stored.trace.resultId);
assert.equal(saved.diagnosticRunId, stored.trace.runId);
assert.equal(saved.metadata.generationCalls, preflight.requests);
const visible = await summary.innerText();
assert.match(visible, /100/);
Expand Down
117 changes: 71 additions & 46 deletions evaluation/scripts/check-extension.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { pathToFileURL } from 'node:url';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import type { LLMConfig } from '../../src/lib/llm-client';
import { SUMMARY_DIAGNOSTICS_KEY, type SummaryTrace } from '../../src/lib/summary-trace';
import type { CandidateFact } from '../../src/lib/fact-contract';
import { candidateResponse } from '../../src/lib/fixtures/candidate-test-source';
import { extractPageLayout } from '../../src/lib/pdf-layout';
Expand All @@ -27,6 +28,41 @@ import {
renderedFactErrors,
type Case as BrowserCase,
} from './fact-summary-expectations';
type TraceMatch = {
key: string;
pdfUrl: string;
runId?: string;
resultId?: string;
model?: string;
};
async function storedTrace(
worker: {
evaluate: (
fn: (match: TraceMatch) => Promise<SummaryTrace | undefined>,
match: TraceMatch
) => Promise<SummaryTrace | undefined>;
},
pdfUrl: string,
expected: Omit<Partial<TraceMatch>, 'key' | 'pdfUrl'> = {}
): Promise<SummaryTrace> {
const trace = await worker.evaluate(
async (match) => {
const data = await chrome.storage.local.get(match.key);
return (data[match.key]?.traces as SummaryTrace[] | undefined)
?.filter(
(trace) =>
trace.pdfUrl === match.pdfUrl &&
(!match.runId || trace.runId === match.runId) &&
(!match.resultId || trace.resultId === match.resultId) &&
(!match.model || trace.model === match.model)
)
.sort((a, b) => b.startedAt.localeCompare(a.startedAt))[0];
},
{ key: SUMMARY_DIAGNOSTICS_KEY, pdfUrl, ...expected }
);
assert.ok(trace, 'Expected result-linked diagnostic is unavailable');
return trace;
}
async function builtDigest(): Promise<string> {
const digest = createHash('sha256');
for (const directory of ['dist', 'dist/assets', 'dist/.vite']) {
Expand Down Expand Up @@ -180,7 +216,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
if (args.includes('--fixed-api') && !manualReviewFlags.some((flag) => args.includes(flag)))
throw new Error(
'固定APIスモークは node --import tsx evaluation/scripts/check-extension-smoke.ts を使用してください。' +
' --preflight-only はブラウザー不要、--smoke-outcome success|partial|failure で対象を選べます。'
' --preflight-only はブラウザー不要、--smoke-outcome success|partial|failure で対象を選べます。'
);
const arg = (flag: string) => args[args.indexOf(flag) + 1];
if (!args.includes('--browser-module') || !args.includes('--browser-executable'))
Expand Down Expand Up @@ -256,7 +292,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
!fixtureSource ||
reviewCase ||
!narrativeReplay.result ||
narrativeReplay.presentation?.version !== 6 ||
narrativeReplay.presentation?.version !== 7 ||
!narrativeReplay.presentation?.organization ||
narrativeReplay.item.id !== item.id)
)
Expand Down Expand Up @@ -637,9 +673,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
await summary
.getByText('TDnetのPDF URLではありません', { exact: false })
.waitFor({ timeout: 10000 });
const trace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const trace = await storedTrace(worker, pdfUrl);
assert.equal(trace.pdfUrl, pdfUrl);
assert.equal(trace.outcome, 'failure');
assert.equal(trace.resultId, null);
Expand Down Expand Up @@ -695,24 +729,21 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
{ timeout: 330000 }
);
if (reviewSettingsChange) {
const currentTrace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const currentTrace = await storedTrace(worker, pdfUrl, { model: 'fixture-next' });
assert.equal(currentTrace.model, 'fixture-next');
assert.ok(['firstSuccess', 'repairSuccess'].includes(currentTrace.outcome));
releaseFirstResponse();
// The bounded settle window supplements the deterministic deferred-request integration tests.
await page.waitForTimeout(1000);
const afterOld = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const afterOld = await storedTrace(worker, pdfUrl, {
runId: currentTrace.runId,
resultId: currentTrace.resultId!,
});
assert.deepEqual(afterOld, currentTrace);
evidence.stages.push('new result finished before old response → current trace retained');
}
evidence.stages.push('button → result/error HTML');
const completedTrace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const completedTrace = await storedTrace(worker, pdfUrl);
evidence.trace = completedTrace;
assert.equal(
completedTrace?.buildDigest,
Expand All @@ -728,12 +759,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
await summary
.getByRole('heading', { name: '開示の要点', exact: true })
.waitFor({ timeout: 330000 });
Object.assign(
completedTrace,
await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
)
);
Object.assign(completedTrace, await storedTrace(worker, pdfUrl));
evidence.stages.push('smart completeness failure without API → explicit full retry');
}
if (!fixedFailure && completedTrace.outcome === 'failure')
Expand All @@ -750,9 +776,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
)
);
if (reviewFixture) assert.ok(!evidence.rendered.includes('売上高: 100百万円'));
const trace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const trace = await storedTrace(worker, pdfUrl);
evidence.trace = trace;
assert.equal(trace.outcome, 'failure');
assert.deepEqual(
Expand Down Expand Up @@ -815,7 +839,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
return entry ? { key: entry[0], value: entry[1] } : null;
}, ANALYSIS_SCHEMA_VERSION);
assert.ok(stored?.value?.facts?.version === FACT_SCHEMA_VERSION);
assert.ok(stored.value.presentation?.version === 6);
assert.ok(stored.value.presentation?.version === 7);
if (narrativeReplay) {
// UI replay must preserve the model evaluation outcome, including warnings.
// It is not a way to turn a failed model assessment into a success.
Expand Down Expand Up @@ -870,7 +894,9 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
assert.ok(
await toggles.evaluateAll((nodes: HTMLDetailsElement[]) => nodes.every((n) => !n.open))
);
await page.screenshot({ path: `evaluation/results/local/${item.id}-v${ANALYSIS_SCHEMA_VERSION}-summary-top.png` });
await page.screenshot({
path: `evaluation/results/local/${item.id}-v${ANALYSIS_SCHEMA_VERSION}-summary-top.png`,
});
for (const [title, suffix] of [
['事業別業績', 'business'],
['キャッシュフロー', 'cash-flow'],
Expand All @@ -888,7 +914,9 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
}
await summary.evaluate((node: HTMLElement) => node.ownerDocument.defaultView!.scrollTo(0, 0));
await page.setViewportSize({ width: 600, height: 800 });
await page.screenshot({ path: `evaluation/results/local/${item.id}-v${ANALYSIS_SCHEMA_VERSION}-summary-narrow.png` });
await page.screenshot({
path: `evaluation/results/local/${item.id}-v${ANALYSIS_SCHEMA_VERSION}-summary-narrow.png`,
});
await toggles.evaluateAll((nodes: HTMLDetailsElement[]) =>
nodes.forEach((n) => (n.open = true))
);
Expand All @@ -913,9 +941,10 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
evidence.metadata = stored.value.metadata;
evidence.reading = reading;
evidence.pdfHash = pdfHash;
const trace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const trace = await storedTrace(worker, pdfUrl, {
runId: stored.value.diagnosticRunId,
resultId: stored.value.resultId,
});
assert.equal(
trace.outcome,
narrativeReplay.result.unverified.length ||
Expand Down Expand Up @@ -1206,9 +1235,10 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
}
assert.match(stored.value.metadata.documentHash, /^[a-f0-9]{64}$/);
if (pdfHash) assert.equal(stored.value.metadata.documentHash, pdfHash);
const trace = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const trace = await storedTrace(worker, pdfUrl, {
runId: stored.value.diagnosticRunId,
resultId: stored.value.resultId,
});
evidence.trace = trace;
assert.ok(trace?.attempts.length);
assert.equal(trace.buildDigest, sourceDigest, '実行Workerと現在の製品ソースが一致しません');
Expand Down Expand Up @@ -1482,9 +1512,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
.waitFor({ timeout: 10000 });
const failed = await copiedDiagnostic(summary, page);
assert.equal(apiCalls, callsBefore);
const failure = await worker.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
);
const failure = await storedTrace(worker, pdfUrl);
assert.notEqual(failure.runId, trace.runId);
assert.equal(failure.outcome, 'failure');
assert.equal(failure.error, 'APIキーが設定されていません');
Expand Down Expand Up @@ -1643,16 +1671,13 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
);
await worker.evaluate(async () => chrome.storage.sync.set({ experimentalScoring: false }));
await row.getByRole('button', { name: '閉じる', exact: true }).click();
await worker.evaluate(
async (key: string) => {
const entry = { key, value: (await chrome.storage.local.get(key))[key] };
if (!entry.value) throw new Error('現在の設定で復元したキャッシュがありません');
await chrome.storage.local.set({
[entry.key]: { ...entry.value, facts: { ...entry.value.facts, version: 3 } },
});
},
restoredKey
);
await worker.evaluate(async (key: string) => {
const entry = { key, value: (await chrome.storage.local.get(key))[key] };
if (!entry.value) throw new Error('現在の設定で復元したキャッシュがありません');
await chrome.storage.local.set({
[entry.key]: { ...entry.value, facts: { ...entry.value.facts, version: 3 } },
});
}, restoredKey);
const previousCalls = apiCalls;
await row.getByRole('button', { name: '表示', exact: true }).click();
await page.waitForFunction(
Expand All @@ -1677,7 +1702,7 @@ export async function checkExtension(item: BrowserCase, config: LLMConfig, args:
const w = context.serviceWorkers()[0];
if (w)
evidence.trace = await w.evaluate(
async () => (await chrome.storage.local.get('summaryLastRunV1')).summaryLastRunV1
async () => (await chrome.storage.local.get('summaryDiagnosticsV1')).summaryDiagnosticsV1
);
}
throw error;
Expand Down
Loading
Loading