[Hackathon] harvard-phd: EigenTrust plugin with checkable invariants#2
Open
mariagorskikh wants to merge 2 commits into
Open
[Hackathon] harvard-phd: EigenTrust plugin with checkable invariants#2mariagorskikh wants to merge 2 commits into
mariagorskikh wants to merge 2 commits into
Conversation
EigenTrust (Kamvar et al., WWW 2003) computes global trust as the stationary distribution of a teleporting random walk over the local trust graph: t = (1 - alpha) * C^T * t + alpha * p. The reference score_average plugin is Sybil-vulnerable and untransitive; EigenTrust fixes both via a pre-trusted seed distribution and power iteration to convergence. What's checkable (not just commented): * simplex: sum_i t_i == 1, all t_i >= 0 * row-stochasticity: every row of C sums to 1 * fixed point: ||(1-a) C^T t + a p - t||_inf < tol * Sybil upper bound: t_sybil <= alpha * p_sybil for unendorsed Sybils * weak monotonicity in honest positive evidence All five properties are asserted in test_eigentrust.py, including hypothesis-driven property tests over randomly generated trust graphs. Plugin is wired into the registry under (trust, eigentrust) so it can be swapped into any scenario via `layers.trust: eigentrust`.
Reviewer's GuideAdds an EigenTrust-based trust plugin as a first-class alternative to score_average, fully wired into the plugin registry, with a pure-Python implementation and an extensive invariant-focused test suite (unit + property-based) to validate its probabilistic, fixed-point, and Sybil-resistance properties. Sequence diagram for scoring with the EigenTrust trust pluginsequenceDiagram
actor Scenario
participant TrustLayer
participant EigenTrust
participant _compute_trust_vector
participant _power_iterate
Scenario->>TrustLayer: request_score(agent)
TrustLayer->>EigenTrust: score(agent)
EigenTrust->>EigenTrust: _agents.add(agent)
EigenTrust->>_compute_trust_vector: _compute_trust_vector()
_compute_trust_vector->>_compute_trust_vector: _pretrusted_distribution(agents)
_compute_trust_vector->>_compute_trust_vector: _local_trust_raw(agents)
_compute_trust_vector->>_compute_trust_vector: _normalize_local_trust(raw, agents, p)
_compute_trust_vector->>_power_iterate: _power_iterate(c, p, alpha, agents, max_iter, tol)
_power_iterate-->>_compute_trust_vector: t, iters, residual
_compute_trust_vector->>EigenTrust: trust_vector t
EigenTrust->>EigenTrust: set last_iters, last_residual
EigenTrust-->>TrustLayer: ReputationScore
TrustLayer-->>Scenario: ReputationScore
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
report, silently overridingevidence.subjectwith theagentparameter when they differ could hide upstream bugs; consider either asserting they match or consistently using one source of truth so mismatches are surfaced early. - Both
_normalize_local_trustand_power_iteratematerialize dense rows over all agents for each reporter, which is fine for small scenarios but becomes O(n²) in memory/time; if you expect larger graphs it may be worth keepingCsparse by only storing non-zero entries and iterating over those.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `report`, silently overriding `evidence.subject` with the `agent` parameter when they differ could hide upstream bugs; consider either asserting they match or consistently using one source of truth so mismatches are surfaced early.
- Both `_normalize_local_trust` and `_power_iterate` materialize dense rows over all agents for each reporter, which is fine for small scenarios but becomes O(n²) in memory/time; if you expect larger graphs it may be worth keeping `C` sparse by only storing non-zero entries and iterating over those.
## Individual Comments
### Comment 1
<location path="packages/nest-plugins-reference/nest_plugins_reference/trust/eigentrust.py" line_range="375" />
<code_context>
+ # No self-trust. An agent rating itself would let it
+ # short-circuit the random walk.
+ continue
+ val = float(pos_row.get(j, 0) * _POS_WEIGHT - neg_row.get(j, 0) * _NEG_WEIGHT)
+ if val > 0.0:
+ row[j] = val
</code_context>
<issue_to_address>
**issue (bug_risk):** Negative/positive weights are effectively applied twice, which may not match the intended semantics.
In `report()` each event is already scaled by `_POS_WEIGHT` / `_NEG_WEIGHT` when updating `_pos`/`_neg`. Multiplying by the same constants again here makes the effective weighting `weight ** 2` instead of `weight`. With both set to 1 this currently has no effect, but changing (say) `_NEG_WEIGHT = 3` would unintentionally give a 9x impact.
If the intent is to apply weighting only once when forming the raw local-trust score, either:
- increment by `1` in `report()` and keep the multiplication here, or
- keep the increments as they are and remove the extra multiplication here.
</issue_to_address>
### Comment 2
<location path="packages/nest-plugins-reference/nest_plugins_reference/trust/eigentrust.py" line_range="275-276" />
<code_context>
+ sample_count = sum(
+ len(self._pos[i]) + len(self._neg[i]) for i in self._agents
+ )
+ confidence = 0.0 if self._tol <= 0 else max(
+ 0.0, min(1.0, 1.0 - self.last_residual / max(self._tol, 1e-12)),
+ )
+ return ReputationScore(
</code_context>
<issue_to_address>
**nitpick:** The confidence computation includes dead branches and redundant safeguards.
Because the constructor already enforces `tol > 0.0` (`if tol <= 0.0: raise ValueError`), both `self._tol <= 0` and `max(self._tol, 1e-12)` are unreachable/unused cases. You can simplify to:
```python
confidence = max(0.0, min(1.0, 1.0 - self.last_residual / self._tol))
```
This preserves behavior under the existing class invariant while making the confidence calculation more direct.
</issue_to_address>
### Comment 3
<location path="packages/nest-plugins-reference/tests/test_eigentrust.py" line_range="370" />
<code_context>
+ return agents, reports
+
+
+class TestEigenTrustProperties:
+ @settings(
+ max_examples=80,
</code_context>
<issue_to_address>
**suggestion (testing):** Consider varying the number of pretrusted agents in the property-based tests
Right now the property-based tests always use `pretrusted = [agents[0]]`, but the behavior of `p` and invariants like pretrusted mass lower bound and Sybil resistance depend on the structure of the pretrusted set. It would be useful to exercise several shapes of `pretrusted`.
You could, for example, extend `_trust_graph` (or add a small strategy) to draw `pretrusted` like:
```python
pretrusted = draw(
st.lists(
st.sampled_from(agents),
min_size=1,
max_size=min(3, len(agents)),
unique=True,
)
)
```
and reuse this in the property tests that reason about `p` to broaden coverage without much added complexity.
Suggested implementation:
```python
for _ in range(edge_count):
i = draw(st.integers(min_value=0, max_value=n - 1))
j = draw(st.integers(min_value=0, max_value=n - 1))
kind = draw(st.sampled_from(["positive", "negative"]))
if i == j:
continue
reports.append((f"a{i}", f"a{j}", kind))
pretrusted = draw(
st.lists(
st.sampled_from(agents),
min_size=1,
max_size=min(3, len(agents)),
unique=True,
)
)
return agents, reports, pretrusted
```
```python
@settings(
max_examples=80,
deadline=None,
suppress_health_check=[HealthCheck.function_scoped_fixture],
)
@given(_trust_graph())
@pytest.mark.asyncio
async def test_simplex_and_row_stochastic(
self,
graph: tuple[list[AgentId], list[tuple[str, str, str]], list[AgentId]],
) -> None:
agents, reports, pretrusted = graph
```
1. Update all other usages of `_trust_graph()` in this file to unpack three values (`agents, reports, pretrusted = graph`) and fix any type hints accordingly.
2. In property tests that currently construct `pretrusted = [agents[0]]` (or similar), remove that construction and instead use the `pretrusted` drawn from `_trust_graph()` so that the tests exercise varying pretrusted sets.
3. If there are helper functions that accept `(agents, reports)` from `_trust_graph()`, update their signatures (and call sites) to also accept `pretrusted` or to accept a single `graph` tuple and unpack internally.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| # No self-trust. An agent rating itself would let it | ||
| # short-circuit the random walk. | ||
| continue | ||
| val = float(pos_row.get(j, 0) * _POS_WEIGHT - neg_row.get(j, 0) * _NEG_WEIGHT) |
There was a problem hiding this comment.
issue (bug_risk): Negative/positive weights are effectively applied twice, which may not match the intended semantics.
In report() each event is already scaled by _POS_WEIGHT / _NEG_WEIGHT when updating _pos/_neg. Multiplying by the same constants again here makes the effective weighting weight ** 2 instead of weight. With both set to 1 this currently has no effect, but changing (say) _NEG_WEIGHT = 3 would unintentionally give a 9x impact.
If the intent is to apply weighting only once when forming the raw local-trust score, either:
- increment by
1inreport()and keep the multiplication here, or - keep the increments as they are and remove the extra multiplication here.
Comment on lines
+275
to
+276
| confidence = 0.0 if self._tol <= 0 else max( | ||
| 0.0, min(1.0, 1.0 - self.last_residual / max(self._tol, 1e-12)), |
There was a problem hiding this comment.
nitpick: The confidence computation includes dead branches and redundant safeguards.
Because the constructor already enforces tol > 0.0 (if tol <= 0.0: raise ValueError), both self._tol <= 0 and max(self._tol, 1e-12) are unreachable/unused cases. You can simplify to:
confidence = max(0.0, min(1.0, 1.0 - self.last_residual / self._tol))This preserves behavior under the existing class invariant while making the confidence calculation more direct.
| return agents, reports | ||
|
|
||
|
|
||
| class TestEigenTrustProperties: |
There was a problem hiding this comment.
suggestion (testing): Consider varying the number of pretrusted agents in the property-based tests
Right now the property-based tests always use pretrusted = [agents[0]], but the behavior of p and invariants like pretrusted mass lower bound and Sybil resistance depend on the structure of the pretrusted set. It would be useful to exercise several shapes of pretrusted.
You could, for example, extend _trust_graph (or add a small strategy) to draw pretrusted like:
pretrusted = draw(
st.lists(
st.sampled_from(agents),
min_size=1,
max_size=min(3, len(agents)),
unique=True,
)
)and reuse this in the property tests that reason about p to broaden coverage without much added complexity.
Suggested implementation:
for _ in range(edge_count):
i = draw(st.integers(min_value=0, max_value=n - 1))
j = draw(st.integers(min_value=0, max_value=n - 1))
kind = draw(st.sampled_from(["positive", "negative"]))
if i == j:
continue
reports.append((f"a{i}", f"a{j}", kind))
pretrusted = draw(
st.lists(
st.sampled_from(agents),
min_size=1,
max_size=min(3, len(agents)),
unique=True,
)
)
return agents, reports, pretrusted @settings(
max_examples=80,
deadline=None,
suppress_health_check=[HealthCheck.function_scoped_fixture],
)
@given(_trust_graph())
@pytest.mark.asyncio
async def test_simplex_and_row_stochastic(
self,
graph: tuple[list[AgentId], list[tuple[str, str, str]], list[AgentId]],
) -> None:
agents, reports, pretrusted = graph- Update all other usages of
_trust_graph()in this file to unpack three values (agents, reports, pretrusted = graph) and fix any type hints accordingly. - In property tests that currently construct
pretrusted = [agents[0]](or similar), remove that construction and instead use thepretrusteddrawn from_trust_graph()so that the tests exercise varying pretrusted sets. - If there are helper functions that accept
(agents, reports)from_trust_graph(), update their signatures (and call sites) to also acceptpretrustedor to accept a singlegraphtuple and unpack internally.
mariagorskikh
pushed a commit
that referenced
this pull request
May 26, 2026
…dian `_build_consensus` was reporting `sum(per-dim medians)` as the score, while `total_median` in the JSON used `median_low(per-judge totals)`. In any non-degenerate case these diverge — e.g., PR #2 was showing "scored 20.0/30" in prose while `median: 21.0` in JSON, confusing downstream consumers. - Plumb `total_median` (the value that actually gets written to JSON) through to `_build_consensus` so the prose uses the same aggregation. - Add a regression test (`test_consensus_uses_total_median_not_sum_of_medians`) that constructs three judges with divergent score patterns where the old buggy computation (20) and the new correct computation (21) differ; it asserts the consensus string contains `f"{total_median:.1f}/30"`. - Re-bootstrap `docs/hackathon/scores.json` with the fixed code; all 10 entries now have the median field and consensus prose in agreement.
mariagorskikh
pushed a commit
that referenced
this pull request
May 26, 2026
…l-diff fixture - judge_pr.py: skip temperature kwarg for gpt-5.x models (they only accept default=1) - fixtures/hackathon-prs-2026-05-26.json: replace stub diffs with real git diffs (688-2002 lines per PR, fetched via git diff origin/main...origin/<head_ref>) - docs/hackathon/scores.json: 10 PRs scored by 3 gpt-5.5 judges each via OpenAI.
mariagorskikh
added a commit
that referenced
this pull request
May 26, 2026
Integration of 5 platform tracks built in parallel by specialist agents: - platform/ci-hygiene (PR #12): Makefile + pre-commit + idempotent CI feedback bot + CONTRIBUTING Definition of Done - platform/open-problems (PR #13): 10 differentiated open problems across 10 layers, charter, judging doc - platform/judge-panel (PR #14): rubric, anthropic + openai providers, run_all CLI, real-diff fixture, live gpt-5.5 scoreboard for PRs #2-#11 - platform/research-harness (PR #15): conditions matrix, claude-CLI live runner, collect + analyze, dry-run fixtures + tests - platform/marketplace-ui (PR #16): /hackathon Next.js section with author tags, judge scores, layer browser; Python data adapter Schema reconciled end-to-end (rubric -> scores.json -> adapter -> TS types -> UI) on the 6-dim 1-5 scale with totals in [6, 30]. Local CI: 341 passed, 1 skipped (matplotlib gated), 1 deselected (live marker). Live judge scoreboard top: #2 harvard-phd trust 26.0/30 (EigenTrust + checkable invariants) #7 coinbase-crypto payments 26.0/30 (HTLC escrow) #6 stanford-ml-phd trust 25.0/30 #11 google-staff transport 25.0/30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which piece
Trust layer. Adds
eigentrustas a bundled alternative to thedefault
score_averageplugin, registered under(trust, eigentrust)innest_core.plugins.Why
The trust layer is the most interesting "what could go wrong"
surface in NEST: the bundled
score_averageis a running mean withno Sybil resistance, no transitivity, no decay. The docs even point
to EigenTrust as "a good fit to test here". This PR delivers it —
not as an academic curiosity, but as a plugin whose correctness
properties are asserted in tests, not promised in comments.
Core idea
EigenTrust (Kamvar, Schlosser, Garcia-Molina, WWW 2003) computes the
global trust vector as the stationary distribution of a teleporting
random walk over the local-trust graph:
where
Cis the row-normalized local-trust matrix(
c_ij = max(0, pos_ij - neg_ij) / row_sum, fallback topwhena row sums to zero — the "naive walker"), and
pis the seed distribution over pre-trusted peers (uniformfallback when no pre-trusted set is declared).
Solved by power iteration; pure Python, no numpy dependency, in
line with the rest of the reference plugin pack.
What's checkable, not just commented
The whole point of doing this in a verification-flavored way: every
property the algorithm claims is asserted in
test_eigentrust.py.sum_i t_i == 1,t_i >= 0)_assert_on_simplex, hand-rolled + hypothesisC_assert_row_stochastic, hand-rolled + hypothesis< tol_assert_fixed_point, hand-rolled + hypothesist_sybil <= alpha * p_sybiltest_sybil_lower_boundtest_weak_monotonicity_of_positive_evidencealpha = 1recoverspexactlytest_alpha_one_recovers_pretrusted_distributiont_i >= alpha * p_iHypothesis is already a dev dependency, so the property tests cost
nothing to add.
How to test
Or wire it into a scenario:
The plugin is drop-in compatible with the existing
Trustprotocol,so no scenario YAML changes are required to test it under the
reputationscenario or anywhere else.Key assumptions
identity (the
Evidence.reporterfield). A Sybil cannot forge anhonest agent's report; if NEST's auth/identity layers fail upstream,
no trust layer can save you.
pisuniform and the Sybil-resistance bound degrades to the uniform
case. The test for the Sybil upper bound makes this explicit by
passing
pretrusted=[a0].c_iiare forced to zero soagents can't short-circuit the random walk by endorsing
themselves.
free of heavy deps. Power iteration runs in
O(iters * |E|)overthe local-trust graph; with
DEFAULT_MAX_ITER=300(sized for(1 - alpha)^k < tol) and the agent counts NEST actuallysimulates, this is sub-millisecond.
Persona
Harvard CS PhD candidate in formal methods — invariants beat
comments; a property you can check beats a promise that one holds.
Future work
validate_reputation_sybil_boundvalidator that reads a traceand asserts the Sybil upper bound directly, on top of the existing
reputation_scoring/reputation_warningschecks.extension of the same fixed-point formulation.
Python implementation; left for a follow-up because a 1-PR
hackathon is the wrong place for it.
https://claude.ai/code/session_01C5j2D4MgCkPgsjSCqBVpWW
Generated by Claude Code
Summary by Sourcery
Add an EigenTrust-based trust plugin as a bundled alternative to score_average, document it, and verify its core invariants with targeted and property-based tests.
New Features:
Enhancements:
Documentation:
Tests: