Repository navigation
feat(ai): score recommendations against a fixed maximum and measure the result - #146
Merged
ma7moudalysalem merged 5 commits intoSep 18, 2026
Merged
Conversation
…ation harness The recommender and the eligibility checker both compared a student's field of study against a listing by exact string equality, across two vocabularies that never met: listings are written from a fifteen-item canonical list, while the seeder wrote fine-grained specialisms and the student profile takes free text. A student interested in "Software Engineering" could never match a listing open to "Engineering". Scoring - The match score is now the share of the attainable weight a listing earns across the criteria it actually constrains, rather than a fixed point sum. A criterion the listing leaves open drops out of both sides of the ratio, so an award open to every discipline is no longer punished for that openness. - Field comparison is graded: an exact match counts in full, a shared significant word counts as a half. - The explanation now names the criteria that matched instead of restating the score band, and says plainly when nothing matched. Eligibility - The field criterion could never report "not met" — its final branch returned "partially met", so a law student was told a computer science award was partially met. Unrelated fields now report "not met" and "partially met" is reserved for a genuinely adjacent field. Data - The seeder writes FieldsOfStudyJson on generated listings and states student preferences in the same canonical vocabulary. Measured on a seeded database, listings carrying a field went from 0 of 336 to 733 of 763, and the share of (student, listing) pairs sharing a field from 0.0% to 12.2%. - The curated external dataset carries fieldsOfStudy; five entries restrict discipline and nineteen are explicitly open to any field. Evaluation - server/tools/ScholarPath.Eval runs the delivered checker over a seeded database with a fixed seed and reports per-criterion outcomes. On 3,600 pairs, fields with nothing in common reported as "partially met" fell from 93.5% to zero. Also enables the SignalR Redis backplane when Redis is configured, and adds WebKit and Edge to the Playwright projects, so both claims hold in fact. Tests: 951 passing (18 new).
…he result The match score was the share of *attainable* weight a listing earned, so a listing that constrained nothing was judged against nothing and scored full marks. Against the seeded corpus that made 728 of 750 recommendations score exactly 100, left the ranking arbitrary for 142 of 150 students, and inverted the score's meaning: the listings scoring 100 were ineligible 96% of the time while those scoring 80-92 were eligible every time. Every criterion is now scored against the same fixed maximum. A listing that places no restriction on a criterion takes neutral credit -- it does not exclude the student, but offers no evidence of fit either -- while a profile that states no preference earns nothing, because a score invented from silence is worse than no score. Academic level becomes a gate rather than a weight: a listing for a level the student is not at can never be one they are eligible for, so it is excluded rather than ranked down. The funding bonus applies only above a floor of fit, so a large award cannot carry a listing the student does not suit. The explanation now names the criteria that matched instead of restating the score band, so a student can see why a listing was offered. Adds recommendation and retrieval modes to the evaluation harness, which is how the defect above was found and how the fix is shown to hold: precision@5 of 4.3% against a 0.3% random baseline over 150 sampled students, and an oracle pass that bounds what the corpus makes reachable. Refs #144
…ve results for The harness measures whether the advisory features report what they were specified to report. It cannot measure whether a student finds the platform useful, and no figure the project has speaks to that. This adds the instrument for the study that would: protocol, participant information sheet and consent form in English and Arabic, screening questions, six task scenarios drawn from the student journey with success criteria and a five-minute cap, the System Usability Scale in both languages with its scoring rule written out, a debrief script, the data sheet, and the analysis script. The analysis script is committed before the first session so that the way a number is computed cannot be adjusted once the number is visible. It refuses to compute a usability result from a row whose role is `author`: a score given by someone who built the system measures their familiarity with it rather than the system, and pooling such a row with participant data would misreport the study. Pilot rows live in a separate file and print timings only. No results are included, because the study has not been run.
The before/after comparison of the recommendation ranking had been measured by swapping the whole service, so the "before" ranker was judged by its own, older eligibility checker -- the one that called almost everything partially eligible -- and the two columns were not on one ground truth. --ranker legacy runs a verbatim copy of the pre-redesign rule (additive points, no level gate, ties by title) through the current checker, so the redesign is measured for what it changed rather than for a change in how it was judged. Judged that way the shipped, intermediate and current rules place a usable listing in 4.4, 5.1 and 4.4 percent of slots: the same. What differs is the score itself, which the mode now reports -- slots spent on a level the student is not at (8 -> 0 of 750), lists whose five scores all tie (55% / 95% / 55%), distinct score values, and the usable share by score band.
A review of the paper against this repository found three places where the harness could not reproduce what the paper reports, and two where what it reported was weaker than it looked. - --ranker intermediate: the scoring rule of 68afdda, transcribed verbatim, so the third column of the recommendation table no longer depends on swapping a source file by hand. - Chance baselines are now exact expectations from the exhaustive pass rather than one realised random draw, which on this catalogue was two events. A second baseline draws only at the student's academic level: against it the ranker is 5.5x better than chance, against a uniform draw 26x. --oracle sets how many sampled students get the exhaustive pass. - Raw counts are printed beside every percentage. - The population header prints how many open listings declare a discipline and what share of all (student, listing) pairs share a declared field exactly. - Eligibility mode counts verdicts reported 'met' although no accepted field equals the student's, which explains the fall in 'met' after the correction (the shipped rule matched on containment). - Retrieval mode prints the size of the unique-combination population it samples from, and adds a BM25 row. The shipped hashing fallback is weaker than that lexical baseline (26.7% vs 48.3% recall@4), which is worth knowing. User-study instrument: the usability-scale ranges are now those Bangor, Kortum and Miller (2008) actually published (<50, 50-70, >70) rather than a scheme misattributed to them; the sample-size rationale cites what supports it; and the screening and debrief sheets exist in Arabic as well as English.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The match score was a sum of points — 30 for a matching level, up to 40 for exact-string field overlap, up to 25 for country overlap, 5 for a large award — with no gate on academic level, ties broken alphabetically by title, and an explanation chosen by score band. A listing at a level the student is not at could still be recommended (it can never be one they are eligible for), a wording difference between "Software Engineering" and "Engineering" scored zero, and the explanation said "strong match" without saying what matched.
What changed
What the evaluation says — measured, not asserted
tools/ScholarPath.Evalgainsrecommendationandretrievalmodes, and--ranker legacy(a verbatim copy of the pre-change rule) so that the old and new rankers are judged by the same eligibility checker. 150 sampled students, five slots each, seed 20260918, oracle over 40 students × 763 listings:So, honestly: on this corpus the redesign does not move precision or recall — the oracle pass finds only 1.2 usable listings per student among 763, and any rule that gets the level right and prefers an exact field match reaches the same handful. What it changes is what the score means: no slot is wasted on a wrong-level listing, the score is monotonic with eligibility (every listing ≥ 80 is usable), and the explanation is checkable.
An intermediate revision on this branch (68afdda) that normalized by the attainable weight was rejected by the same harness: 142 of 150 lists fully tied, mean score 99.3, 733 of 750 scoring ≥ 80 — the number told the student nothing.
Ties remain for 55% of students because most seeded listings restrict nothing but level; breaking ties by deadline or award size is a follow-up, not part of this PR.
Also in this PR
docs/user-study/— the full usability-study instrument (protocol, consent EN/AR, screening, six tasks EN/AR, SUS EN/AR, debrief, data sheet,analyze.py). No results; the study has not been run, and the script refuses rows whoseroleisauthor.Tests
11 tests pin the scoring contract. Full suite: 952 passed, 0 failed on a clean tree.
Refs #144