Modernization, Storage Migration & UI/UX Improvements - #14
Open
MiskaWasTaken wants to merge 4 commits into
Open
Conversation
- Migrate PDF storage from MongoDB blobs to filesystem-backed storage - Introduce storage abstraction layer and static file serving - Add migration and verification utilities with dry-run and integrity checks - Remove legacy GridFS/blob storage and simplify document retrieval - Implement type-aware metadata extraction for specimen papers, examiner reports and grade thresholds - Improve indexing pipeline with ahead-of-time metadata and directory generation - Add fuzzy search, phrase boosting and metadata-aware Elasticsearch ranking - Optimize backend by eliminating N+1 database queries and reducing Elasticsearch payload sizes - Modernize frontend with responsive layout, dark mode, advanced filters and improved search UX - Add split-view PDF mode and improve result presentation - Improve caching, compression and static asset delivery - Modernize MongoDB usage and replace deprecated APIs where safe - Expand documentation with architecture, storage and indexing guides - Improve test suite reliability and make database setup idempotent - Add storage verification, migration auditing and performance tooling This update significantly reduces MongoDB storage requirements, improves search relevance and performance, modernizes the frontend, and prepares the project for future PDF.js modernization while preserving backward compatibility and existing functionality.
…UI and layout components.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The following message is written by AI and and was reviewed by me to ensure accuracy.
There was just so much that has been done, I could not accurately remember.
Rest assured everything has been reviewed.
Overview
This PR contains the major modernization and maintenance work completed across SchSrch, including dependency upgrades, UI/UX improvements, performance optimizations, database/storage changes, search/indexing improvements, and the migration tooling required to move PDF storage from MongoDB to the filesystem.
The most significant backend change is the migration of PDF binaries from the legacy pastpaperpaperblobs MongoDB collection to filesystem-backed storage using pastpaperdocs.relativePath and SHA-256 integrity verification.
The changes have been tested against a restored production database dump and a production-like application environment, including migration, verification, indexing, search, PDF serving, and legacy blob cleanup.
What's Included
Dependency & Infrastructure Updates
UI/UX Improvements
Various frontend improvements have been made throughout schsrch to improve usability and presentation.
Improvements include:
These changes are intended to make the existing functionality easier to use without changing the core purpose of schsrch
Split-view paper/search experience
Improved PDF workflow
Responsive layout
The UI was adjusted to better accommodate different viewport sizes, particularly around the split-view/search/PDF workflow.
Dark UI/Search result presentation
The application UI has been modernized around the existing dark presentation, including the search and document-viewing workflow.
Search improvements
The search system has been improved at both the Elasticsearch and application layers.
Elasticsearch document structure
The Elasticsearch mapping currently contains:
This keeps Elasticsearch focused on searchable/indexable data instead of storing the actual PDF binaries.
A typical indexed document is:
Phrase/relevance handling
Search ranking has been improved to better distinguish meaningful phrase matches from documents that merely contain the individual words somewhere in their text.
Metadata-aware search
Subject and paper information is available to Elasticsearch as keyword fields, allowing search and filtering to operate on structured metadata as well as full-text content.
Elasticsearch is independent of PDF storage
A critical architectural property of the migration is:
Once text has been extracted, Elasticsearch does not need to reopen the PDF to perform searches.
Therefore, moving PDF binaries from MongoDB to the filesystem does not require Elasticsearch to be rebuilt merely because the storage backend changed.
pastpaperindexesand Elasticsearch architectureThe system now clearly separates the roles of the two indexing layers.
pastpaperindexes
This is the MongoDB persistence layer for extracted PDF text.
Each entry contains information such as:
It acts as the durable source from which Elasticsearch can be rebuilt.
Elasticsearch
Elasticsearch provides the actual full-text search layer.
The relationship is:
Elasticsearch rebuild
reIndexElasticSearch.bin.js is the tool intended for rebuilding Elasticsearch.
It can rebuild Elasticsearch from the existing pastpaperindexes collection without needing to re-read the physical PDFs.
This is fundamentally different from doIndex.bin.js.
doIndex.bin.js clarified
doIndex.bin.js is an ingestion/importer, not an Elasticsearch rebuild tool.
Its workflow is approximately:
It accepts filesystem PDFs as input.
It does not depend on pastpaperpaperblobs.
It can therefore be used after the filesystem storage migration for future paper ingestion.
It also provides a way to reprocess PDFs when extracted/indexed data needs to be regenerated.
Backend performance improvements
Backend performance improvements
Backend data access was optimized to reduce repeated database queries inside iteration paths.
The problematic pattern is effectively:
which turns one logical request into potentially dozens or hundreds of database operations.
The relevant code paths were changed to reduce unnecessary repeated lookups and reuse already available data where possible.
This is particularly relevant for paper/search pages where many documents can be displayed at once.
MongoDB count API modernization
Deprecated MongoDB counting patterns were replaced with explicit document-count APIs such as:
instead of relying on the older:
This removes deprecated API usage and makes the intended counting behavior explicit.
Database access modernization
MongoDB/Mongoose configuration and access patterns were updated as part of the dependency/runtime modernization.
Static asset performance
Static resource delivery was optimized to reduce repeat downloads.
Long-lived caching
Static resources can be cached for longer periods where appropriate, reducing network traffic on subsequent visits.
PDF HTTP caching
Filesystem-backed PDF delivery can use HTTP cache validation mechanisms including:
This allows clients to avoid retransmitting unchanged PDFs.
Font loading
Important fonts are preloaded to reduce the delay before the UI reaches its intended typography.
Deferred JavaScript
Non-critical JavaScript resources are loaded without unnecessarily blocking initial page rendering.
PDF storage migration
Previous architecture
The old architecture stored PDF binary fragments in:
pastpaperpaperblobsThe restored production database contained:
The blob collection occupied approximately:
This is a significant amount of database storage dedicated to PDF binaries.
New filesystem storage architecture
The new architecture stores physical PDF files under STORAGE_ROOT.
MongoDB retains the document metadata and filesystem reference:
The binary itself is no longer required to live inside MongoDB.
Migration bug fixed: metadata was not unique
The original migration implementation used:
as a filesystem signature.
That assumption is invalid.
Production data contained multiple MongoDB documents with the same metadata but different binary contents.
For example:
had two distinct MongoDB documents.
One had:
9937c3ca808d5a84ad66098d4f17ff03a1edd05291b384a4481a8d5f74f3a36b
while the other had:
4acc63a0fbb74e2eb8aaa4cdc59ec68fa308651a5c368fa101ec8124f8adb549
Both cannot safely occupy the same physical file.
Collision-safe path generation
The new migration no longer treats metadata signatures as globally unique filesystem identifiers.
Path ownership is tracked using the document identity and calculated content hash.
If the canonical path is already occupied by a different document/content hash, a deterministic collision path is generated:
<filename>__<MongoDB document ID>.pdfFor example:
new_imports/0620/s06/0620_s06_5_0_ci.pdfand:
new_imports/0620/s06/0620_s06_5_0_ci__588bfd02bf262205b65ccc5f.pdfThis ensures that both documents survive migration without overwriting each other.
The migration identified 8 path collisions in the production dataset.
All 8 were resolved without data loss.
SHA-256 integrity verification
The migration does not simply write the reconstructed PDF and assume everything worked.
For every migrated document:
The database is only updated with the new filesystem path/hash after the filesystem copy has been verified.
The invariant is:
Independent migration verification
migrate-storage.js --verify performs an independent verification pass.
It checks:
Database → filesystem
For every pastpaperdocs document:
relativePath exists
referenced file exists
file is readable
SHA-256 matches fileHash
Filesystem → database
The filesystem is also scanned so that unreferenced PDFs can be identified as orphans.
Collision verification
The verification process checks that multiple database documents are not incorrectly assigned to the same physical file.
This is deliberately separate from the migration's own bookkeeping so that the migration isn't simply grading its own homework.
Resume removed
The old --resume approach based on the original metadata-signature assumptions was intentionally not relied upon for the production migration.
The production migration was performed as a clean migration against the restored database.
The recommended production procedure is therefore:
rather than depending on an ambiguous partially migrated state.
Dry-run validation
The migration can be first tested using:
Legacy blob cleanup
The migration intentionally does not delete pastpaperpaperblobs automatically.
This is deliberate.
Migration and deletion are separate operations:
The cleanup path:
Runs migration verification.
Refuses to delete blobs if blocking integrity failures exist.
Reports the number of legacy blob documents.
Reports estimated storage reclamation.
Requires --confirm-cleanup.
Drops pastpaperpaperblobs.
Verifies that the collection no longer exists.
Writes cleanup-report.json.
It leaves:
untouched.
A --force mode was added for cases where an operator has explicitly reviewed and accepted known migration exceptions.
--force cannot be used by itself.
This combination is required:
--cleanup-blobs
--confirm-cleanup
--force
Without --confirm-cleanup, the operation aborts.
MIGRATION STEPS FOR PRODUCTION
mongodump --uri="<PRODUCTION_MONGODB_URI>"Also ensure the filesystem destination is backed up or otherwise recoverable.
Do not continue if unexpected errors are reported. (~8 collisons may be experienced, this is okay continue)
This does not delete pastpaperpaperblobs. It simply rebuilds the pdf to storage
Expected:
(or the 8 files)
Orphaned files should be reviewed separately.
As part of the production migration, Elasticsearch should be cleared and rebuilt after the filesystem migration.
This provides a clean index generated from the migrated PDF set and ensures that Elasticsearch is synchronized with the final pastpaperdocs/filesystem state.
Production indexing procedure
After the PDF migration has completed successfully:
curl -s 'http://127.0.0.1:9200/_cat/indices?v'Delete the existing pastpaper index:
curl -X DELETE 'http://127.0.0.1:9200/pastpaper'Verify its deleted
curl -s 'http://127.0.0.1:9200/_cat/indices?v'The old pastpaper index should no longer be present.
Rebuild indexing from the migrated PDFs
Run doIndex.bin.js against the migrated filesystem storage:
doIndex.bin.js processes the PDFs and rebuilds the application/indexing data from the actual filesystem papers.
Verify Index
curl -s 'http://127.0.0.1:9200/_cat/indices?v'Then
curl -s 'http://127.0.0.1:9200/pastpaper/_count'Verify that Elasticsearch contains indexed documents.
Start SchSrch against the migrated database and verify:
After successful migration, verification and application testing:
pastpaperpaperblobsSummary of migration