Skip to content

feat(signals): add deterministic copycat containment engine for gate.copycat (#1969)#5129

Open
carlh7777 wants to merge 2 commits into
JSONbored:mainfrom
carlh7777:feat/copycat-containment-engine
Open

feat(signals): add deterministic copycat containment engine for gate.copycat (#1969)#5129
carlh7777 wants to merge 2 commits into
JSONbored:mainfrom
carlh7777:feat/copycat-containment-engine

Conversation

@carlh7777

Copy link
Copy Markdown
Contributor

Summary

Adds the deterministic copycat/plagiarism containment engine (src/signals/copycat.ts) — the detection primitive the already-scaffolded but currently-inert gate.copycat.mode / gate.copycat.minScore config (#4140) was threaded for. It's a natural sibling of the deterministic anti-slop signal (src/signals/slop.ts) and duplicate-cluster adjudication (src/signals/duplicate-winner.ts), and follows the house style of the miner-side self-plagiarism throttle (packages/gittensory-engine/src/governor/self-plagiarism.ts).

Given a PR's added code and one piece of prior art, it:

  • Scores containment (0-100) — the asymmetric percentage of the PR's added-code shingles found in the prior art (answers "how much of THIS PR is copied FROM prior art", without the dilution a symmetric Jaccard suffers against a large corpus). Multi-line (3-line) shingling so incidental single-line coincidences (}, return null;) don't inflate the score; whitespace/case-normalized so pure reformatting never reads as copied content.
  • Resolves copy direction by submission timestamp — the earlier submission is the original, so only the later one can be the copier. Any missing/unparseable timestamp, or an exact tie, is ambiguous.
  • Maps to the configured tier (warn → info / label → warning / block → critical) into a public-safe finding.

Precision-first / false-accusation-averse (the issue's hard requirement): a finding is emitted only when the mode is non-off, the score clears the (resolved) threshold, and the candidate is unambiguously the later submission. An earlier-submitted victim — or any ambiguous timing — is never flagged. The engine is pure: no IO, no Date.now(), no randomness; identical inputs always yield the identical verdict.

Scoped to the engine + direction + tier mapping. The gate call-site that feeds real added-code/prior-art in and turns a wouldAct verdict into an actual label/block + the cross-repo "strikes" escalation is a deferred follow-up — mirroring the config's own documented "CURRENTLY INERT … later, separate PRs against #1969" plan, and how self-plagiarism.ts "gates nothing on its own."

Closes #1969

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (Closes Plagiarism / copycat detection #1969) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally (focused): npx vitest run test/unit/copycat.test.ts — 21 pass, v8 coverage 100% statements / 100% branches / 100% functions / 100% lines on src/signals/copycat.ts (every changed line and branch). Will run full npm run test:ci + npm audit before push.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New behavior has unit tests for every branch: shingling (empty / sub-width / sliding / normalization), containment (empty candidate, empty prior, 0 / partial / 100%), direction (later / earlier / tie / missing / unparseable), threshold clamping/rounding/default, tier→severity mapping, and each wouldAct guard (off, sub-threshold, victim-direction).

If any required check was skipped, explain why:

New self-contained pure module under src/signals/ plus its unit test. It touches no workflow, MCP, OpenAPI, DB/migration, or UI surface, so actionlint, build:mcp/test:mcp-pack, ui:*, and ui:openapi:check are N/A. It's a new file with no engine-package twin, so test:engine-parity does not apply. Full test:ci + npm audit will run before push.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics. The finding's publicText reports the containment score + threshold only — never raw code, filenames, or any contributor identity — and is accusation-neutral ("please confirm originality or attribution").
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — pure signal module, not exposed via API/OpenAPI/MCP.)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes.)
  • Visible UI changes include a UI Evidence section below. (N/A — no visible UI changes.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (N/A.)

UI Evidence

N/A — no UI/frontend/docs changes in this PR.

Notes

  • Precision-first defaults: DEFAULT_COPYCAT_MIN_SCORE = 85 (mirrors the 0.85 spirit of the self-plagiarism throttle); an out-of-range/non-numeric gate.copycat.minScore clamps to 0-100 and falls back to the default.
  • The score is always computed (for observability), even when the mode is off or the direction is ambiguous — only the finding is gated on the precision guards.
  • Deferred to a follow-up on Plagiarism / copycat detection #1969: the gate call-site wiring + the "strikes" escalation via the existing cross-repo banned-contributors ledger.

… scoring and direction assessment

This commit introduces the `copycat` module, which provides a deterministic mechanism for detecting code plagiarism by measuring the containment of added code in prior art. Key features include:

- Calculation of containment scores based on shingling of code lines.
- Determination of copy direction based on submission timestamps.
- Public-safe findings that report potential overlaps without revealing sensitive information.

Additionally, unit tests for the `copycat` functionality have been added to ensure accuracy and reliability of the detection mechanisms.
@carlh7777 carlh7777 requested a review from JSONbored as a code owner July 11, 2026 21:40
@superagent-security

Copy link
Copy Markdown
Contributor

🚨 Contributor flagged. Click here for more info: Superagent Dashboard

@superagent-security superagent-security Bot added the contributor:flagged Contributor flagged for review by trust analysis. label Jul 11, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.32%. Comparing base (a416632) to head (0f1ffa2).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5129   +/-   ##
=======================================
  Coverage   94.32%   94.32%           
=======================================
  Files         471      472    +1     
  Lines       39821    39861   +40     
  Branches    14533    14547   +14     
=======================================
+ Hits        37560    37600   +40     
  Misses       1583     1583           
  Partials      678      678           
Flag Coverage Δ
shard-1 46.30% <0.00%> (-0.05%) ⬇️
shard-2 34.37% <0.00%> (-0.29%) ⬇️
shard-3 31.25% <0.00%> (+0.26%) ⬆️
shard-4 32.69% <0.00%> (-0.19%) ⬇️
shard-5 33.52% <0.00%> (-0.13%) ⬇️
shard-6 45.12% <100.00%> (+0.25%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/signals/copycat.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gittensory-orb gittensory-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 11, 2026
@gittensory-orb

gittensory-orb Bot commented Jul 11, 2026

Copy link
Copy Markdown

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-11 21:58:40 UTC

2 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · unstable

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This adds a new, isolated, pure detection engine (src/signals/copycat.ts) plus a thorough unit test suite, computing asymmetric shingle-based containment, resolving copy direction by timestamp, and mapping to tier severities without emitting a finding unless mode is non-off, score clears threshold, and direction is unambiguously 'candidate_copied'. The logic is internally consistent and fail-safe (ties/missing timestamps → ambiguous → never flagged), and tests exercise the branch matrix (thresholds, mode mapping, direction, min-score clamping) well. The PR explicitly scopes itself to the engine primitive only, deferring the actual gate call-site wiring to a later slice, which is consistent with its own description.

Blockers

  • src/signals/copycat.ts:51: `codeShingles` returns a `Set`, and `containmentScore` divides by `candidate.size`, so a candidate with repeated shingles is measured as distinct-shingle overlap rather than the advertised percentage of the candidate's added-code shingles; change the scoring path to keep candidate shingles as an array/multiset and only use a `Set` for prior lookup, e.g. `const candidate = codeShingleList(candidateLines); const prior = new Set(codeShingleList(priorArtLines)); const contained = candidate.filter((shingle) => prior.has(shingle)).length; return Math.round((contained / candidate.length) * 100);`.
Nits — 5 non-blocking
  • The fixed-size shingling means any candidate snippet shorter than SHINGLE_SIZE (3 lines) collapses to a single whole-block token that can only match an identically-short prior-art block, so a short passage copied verbatim out of a longer prior-art file will score 0 containment — worth a comment noting this is an accepted false-negative tradeoff rather than an oversight (src/signals/copycat.ts:47-55).
  • DEFAULT_COPYCAT_MIN_SCORE (src/signals/copycat.ts:23) and SHINGLE_SIZE (src/signals/copycat.ts:29) are both hardcoded module constants with no override point besides minScore — fine for this slice, but confirm the deferred gate call-site will actually be able to configure shingle width if that's ever needed.
  • buildFinding's publicText/detail always includes the raw score and threshold — confirm that's intentionally acceptable to surface publicly per the 'public-safe' design goal, since containment percentage combined with PR content could narrow down which file was flagged.
  • Add one test explicitly documenting the short-snippet (< SHINGLE_SIZE lines) false-negative case discussed above, since it's a deliberate but non-obvious limitation of the shingle approach.
  • Consider a follow-up doc/comment cross-referencing where the deferred gate call-site work will land so reviewers of the next slice can find this engine's contract quickly.
Flagged checks (non-blocking)
  • Contributor trust — Contributor flagged for review
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #1969
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 182 registered-repo PR(s), 114 merged, 17 issue(s).
Contributor context ✅ Confirmed Gittensor contributor carlh7777; Gittensor profile; 182 PR(s), 17 issue(s).
Gate result ✅ Passing No configured blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence.
Review context
  • Author: carlh7777
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, JavaScript, TypeScript, Cuda, HTML, MDX, Rust
  • Official Gittensor activity: 182 PR(s), 17 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@gittensory-orb gittensory-orb Bot added the manual-review Gittensor contributor context label Jul 11, 2026
@gittensory-orb

Copy link
Copy Markdown

An AI reviewer flagged a likely defect, but its confidence was below this repository's configured close-confidence floor, so this is held for a maintainer to confirm instead of closing automatically. Resolve the flagged defect (see the review notes), or ask a maintainer to override.

This commit refactors the `copycat` module to improve the shingle generation process by introducing a new function, `codeShingleList`, which preserves duplicates in the shingles. The existing `codeShingles` function is updated to return a distinct set of shingles. Additionally, the containment scoring logic is adjusted to work with the new shingle list format, ensuring accurate scoring based on the total shingle count. These changes enhance the module's ability to detect code plagiarism more effectively.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor:flagged Contributor flagged for review by trust analysis. gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plagiarism / copycat detection

1 participant