Motivation
The search pipeline applies scoring weights and screening decisions that are not recorded anywhere, so a search run cannot be reproduced or audited afterwards. The weights in particular are currently magic numbers.
Proposal
- Log every query string, source, timestamp, and result count to a search manifest.
- Record inclusion and exclusion decisions with the reason attached.
- Document the scoring weights and add a sensitivity check showing how the ranking shifts under alternative weightings.
Done when:
Scope
- In scope: manifest logging, decision reasons, weight documentation and sensitivity check.
- Out of scope: expanding to additional bibliographic sources; automating the manual screening step.
Open questions
- Is the search pipeline still in scope for this repository at all, or should it be split into its own tool? If the harness is the deliverable, this may belong elsewhere.
- Should exclusion reasons come from a fixed vocabulary or free text? Fixed is analysable; free text is faster to apply.
Related
- Close as won't-fix if the search pipeline moves out of scope.
- Wohlin (2014), Guidelines for snowballing in systematic literature studies.
Motivation
The search pipeline applies scoring weights and screening decisions that are not recorded anywhere, so a search run cannot be reproduced or audited afterwards. The weights in particular are currently magic numbers.
Proposal
Done when:
Scope
Open questions
Related