Skip to content

Search Overhaul - Phase 1 - #26

Merged
dave-mills merged 14 commits into
devfrom
search-review
Jul 20, 2026
Merged

Search Overhaul - Phase 1#26
dave-mills merged 14 commits into
devfrom
search-review

Conversation

@dave-mills

Copy link
Copy Markdown
Member

High-level overview: Full rewrite of the BrowseAll page, search / tag filtering. Now all searching + filtering happens on Meilisearch, rather than searching on Meilisearch and filtering via Eloquent queries.

Thoughts on the Search Engine / Meilisearch

I thought about whether or not to pull back from Meilisearch and go with a more "generic" tool that people can use with their own search engine choices, but honestly Meilisearch is one of the only robust tools like this that is fully open source and self-hostable. ElasticSearch is another potential option, but requires a huge amount more setup / learning, and is somewhat overkill here. So instead this work leans into Meilisearch, and makes the 'search' feature explicitly dependent on it.

Implementation Details

  • Added settings in config/scout.php so that Trove and Collection indexes can be sorted and filtered as well as searched.
  • Updated Trove and Collection toSearchableArray() methods to include the tags and trove types needed to make filtering work.
  • Search code is now packaged up into Services\Search\ and the BrowseAll page uses an interface. This means we can have 2 versions of the search service (one Meilisearch, and one Database only, for if Meilisearch is ever unreachable) - so if Meilisearch goes offline, the user will still be able to browse and filter (just not do a keyword search, though we can potentially add that back by using MySQL full-text search in the DatabaseLibrarySearch class...)
  • The BrowseAll page does the following:
    • triggers the search by calling out to the app's registered SearchesLibrary::class,
    • hydrates the list of Troves and Collections into the component based only on the current page of results ($results) - which means only a small amount of data is loaded into the page at any one time.

'new' Features:

  • Tags and trove type filters show the count of results in that facet based on the other search terms (0 entries are greyed out).
  • The current query is stored in the url, so the exact state can be shared or returned to.
  • When meilisearch is offline, users can still browse and filter by tag/type. (searching is not possible).
  • A daily scheduled task runs the scout import commands to make sure the meilisearch indexes remain in sync (catches errors in sync during publishing etc).

Trove and Collection searchable arrays gain tag_ids, trove_type_ids,
locales and a shared sort_date attribute; collections aggregate tags
and types from published canonical members only. Filterable, sortable
and searchable attributes are declared in config/scout.php so
scout:sync-index-settings versions them.
…tions

MeilisearchLibrarySearch issues one federated multi-search request
(shared filter expressions, sort_date ordering for blank queries,
federation-level pagination and merged facets). DatabaseLibrarySearch
is the portable-SQL fallback for null-driver installs and outages.
The container binds the contract by scout driver.

UsesCustomSearchOptions is still referenced by the old BrowseAll and
will be deleted with its rewrite.
BrowseAll is now a single full-page component holding only scalar
query/filter/page state (all in the URL), issuing one SearchesLibrary
request per render and hydrating one page of models in hit order. An
engine outage flags a notice and re-runs through the database
fallback. Filter tag types are fetched and sorted in PHP (the
MySQL-only ISNULL/JSON_EXTRACT ordering is gone), so the whole render
path now runs on SQLite. SearchBar and UsesCustomSearchOptions are
deleted; the search input lives inline in the blade.
…, alt text

The sidebar gains a resource-type filter group; every tag/type/language
checkbox shows a facet count badge (zero-count options muted and
disabled unless already selected; no badges in fallback mode).
Pagination is windowed First/Prev/window/Next/Last buttons with
aria-current and real disabled attributes, replacing the Alpine
entangle machinery. Card images get alt text / aria-labels.
…cies

Tags and member-trove data are denormalised into the index, so changes
outside the trove-save path now reindex the affected documents: a
TagObserver reindexes a renamed/deleted tag's published troves and
their collections (member ids captured at deleting, before the pivot
cascade); the AllTrovesTable attach/detach/bulk-attach actions and the
TrovesRelationManager detach actions reindex the collection; and
TrovePublisher reindexes affected collections after publish (both
sides of a membership change), unpublish and delete, deferred to after
commit. Also inlines the publisher's single-use NON_CONTENT constant.
The hardcoded themes/topics slugs were a Stats4SD leftover; the only
consumer was the old BrowseAll (whose cards hide tags anyway). Sweep
for dangling references came back clean; full suite green.
- integration tests for meilisearch engine
- documentation
- update facet counts so they don't count their own tag type filters
@dave-mills
dave-mills requested a review from ciara-mc July 16, 2026 13:25
@dave-mills

Copy link
Copy Markdown
Member Author

@ciara-mc - debugged and fixed the issues with the CI test runner:

  • it was testing against php 8.3 instead of 8.4 (when some of our composer requirements now explicitly ask for 8.4)
  • npm is now added so Vite can run and front-end tests work
  • fixed an issue where the Meilisearch docker container was never showing as 'healthy' due to a mismatched local IP.

All tests now passing here on GitHub, just like locally 👍

@ciara-mc

Copy link
Copy Markdown
Contributor

I think the messaging here is unclear

image

@ciara-mc ciara-mc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After upgrading my meilisearch - this works well

@dave-mills

Copy link
Copy Markdown
Member Author

I think the messaging here is unclear

Agreed; I've updated the text to say:

Text-based search is temporarily unavailable. You can still browse resources below, and use the filters to narrow your search.

@dave-mills
dave-mills merged commit 4c051f6 into dev Jul 20, 2026
1 of 2 checks passed
@ciara-mc
ciara-mc deleted the search-review branch July 20, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants