Skip to content

Optimise indexing of linked resources #1014

Description

@PascalRepond

How it works

When indexing a resource, SONAR dumps the full content of every linked
resource into the Elasticsearch _source.

Root cause: INDEXER_REPLACE_REFS = True (Invenio default, not overridden).
At index time, _prepare_record runs data = deepcopy(record.replace_refs()),
so all $ref links are fully resolved and the entire linked resource is
embedded into the indexed document.

For a user, this means:

  • organisation.$ref → the whole organisation is embedded;
  • subdivision.$ref → the whole subdivision is embedded, and since the
    subdivision itself contains organisation.$ref, the organisation is
    embedded a second time under subdivision.organisation.

The ES mapping only declares the useful sub-fields (e.g. organisation.pid,
organisation.name), but the stored _source still carries the whole
resolved graph, and ES dynamically indexes the undeclared embedded fields.

Example: https://sonar.rero.ch/api/users/?q=309638

Affected indexes

Index Heavy resolved refs embedded Notable duplicate
users organisation, subdivision subdivision.organisation
deposits user, organisation, documents, projects, collections, subdivisions organisation via subdivision
documents organisation, projects, collections, subdivisions, related documents organisation via subdivision
subdivisions organisation
collections organisation
projects organisations, users

organisation is the main offender: embedded in almost every index, and
duplicated through subdivisions.

What should be done

Trim the dumped linked data so that only the fields useful for search/display
are kept (typically pid + name). Follow the rero-ils pattern, split by
module style:

  1. Legacy modules (sonar/modules/*: users, deposits, documents,
    subdivisions, collections, organisations) — add/extend a
    before_record_index receiver per resource that prunes the resolved refs
    after resolution (same mechanism as the existing add_full_name receiver,
    mirroring rero-ils enrich_*_data listeners). Drop nested duplicates such
    as subdivision.organisation.

  2. New-style resources (sonar/resources/projects) — use a record
    dumper (cf. rero-ils MultiDumper / per-module dumpers) to control
    exactly what enters the _source.

  3. Harden the mappings — set dynamic: false on the linked objects
    (organisation, subdivision, etc.) in the ES mappings so undeclared
    embedded fields are no longer dynamically indexed. Requires a reindex.

For each resource, the useful linked fields must be decided explicitly
(default: pid + name); anything not used for search, facets, display or
permissions should not be dumped.

Acceptance criteria

  • For each affected index, only the agreed useful sub-fields of linked
    resources are present in _source
  • No duplicated organisation (no subdivision.organisation full dump)
  • Mappings set dynamic: false on linked objects
  • Search, facets, sort and permissions still work (regression tests)
  • A reindex of all affected indexes is performed/documented

To verify during implementation

  • Confirm, per index, which linked sub-fields are actually consumed (search
    query fields, aggregations in config.py, serializers, permissions) before
    trimming.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementEnhancement of an existing feature

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions