How it works
When indexing a resource, SONAR dumps the full content of every linked
resource into the Elasticsearch _source.
Root cause: INDEXER_REPLACE_REFS = True (Invenio default, not overridden).
At index time, _prepare_record runs data = deepcopy(record.replace_refs()),
so all $ref links are fully resolved and the entire linked resource is
embedded into the indexed document.
For a user, this means:
organisation.$ref → the whole organisation is embedded;
subdivision.$ref → the whole subdivision is embedded, and since the
subdivision itself contains organisation.$ref, the organisation is
embedded a second time under subdivision.organisation.
The ES mapping only declares the useful sub-fields (e.g. organisation.pid,
organisation.name), but the stored _source still carries the whole
resolved graph, and ES dynamically indexes the undeclared embedded fields.
Example: https://sonar.rero.ch/api/users/?q=309638
Affected indexes
| Index |
Heavy resolved refs embedded |
Notable duplicate |
| users |
organisation, subdivision |
subdivision.organisation |
| deposits |
user, organisation, documents, projects, collections, subdivisions |
organisation via subdivision |
| documents |
organisation, projects, collections, subdivisions, related documents |
organisation via subdivision |
| subdivisions |
organisation |
— |
| collections |
organisation |
— |
| projects |
organisations, users |
— |
organisation is the main offender: embedded in almost every index, and
duplicated through subdivisions.
What should be done
Trim the dumped linked data so that only the fields useful for search/display
are kept (typically pid + name). Follow the rero-ils pattern, split by
module style:
-
Legacy modules (sonar/modules/*: users, deposits, documents,
subdivisions, collections, organisations) — add/extend a
before_record_index receiver per resource that prunes the resolved refs
after resolution (same mechanism as the existing add_full_name receiver,
mirroring rero-ils enrich_*_data listeners). Drop nested duplicates such
as subdivision.organisation.
-
New-style resources (sonar/resources/projects) — use a record
dumper (cf. rero-ils MultiDumper / per-module dumpers) to control
exactly what enters the _source.
-
Harden the mappings — set dynamic: false on the linked objects
(organisation, subdivision, etc.) in the ES mappings so undeclared
embedded fields are no longer dynamically indexed. Requires a reindex.
For each resource, the useful linked fields must be decided explicitly
(default: pid + name); anything not used for search, facets, display or
permissions should not be dumped.
Acceptance criteria
To verify during implementation
- Confirm, per index, which linked sub-fields are actually consumed (search
query fields, aggregations in config.py, serializers, permissions) before
trimming.
How it works
When indexing a resource, SONAR dumps the full content of every linked
resource into the Elasticsearch
_source.Root cause:
INDEXER_REPLACE_REFS = True(Invenio default, not overridden).At index time,
_prepare_recordrunsdata = deepcopy(record.replace_refs()),so all
$reflinks are fully resolved and the entire linked resource isembedded into the indexed document.
For a user, this means:
organisation.$ref→ the whole organisation is embedded;subdivision.$ref→ the whole subdivision is embedded, and since thesubdivision itself contains
organisation.$ref, the organisation isembedded a second time under
subdivision.organisation.The ES mapping only declares the useful sub-fields (e.g.
organisation.pid,organisation.name), but the stored_sourcestill carries the wholeresolved graph, and ES dynamically indexes the undeclared embedded fields.
Example: https://sonar.rero.ch/api/users/?q=309638
Affected indexes
subdivision.organisationorganisationis the main offender: embedded in almost every index, andduplicated through subdivisions.
What should be done
Trim the dumped linked data so that only the fields useful for search/display
are kept (typically
pid+name). Follow the rero-ils pattern, split bymodule style:
Legacy modules (
sonar/modules/*: users, deposits, documents,subdivisions, collections, organisations) — add/extend a
before_record_indexreceiver per resource that prunes the resolved refsafter resolution (same mechanism as the existing
add_full_namereceiver,mirroring rero-ils
enrich_*_datalisteners). Drop nested duplicates suchas
subdivision.organisation.New-style resources (
sonar/resources/projects) — use a recorddumper(cf. rero-ilsMultiDumper/ per-module dumpers) to controlexactly what enters the
_source.Harden the mappings — set
dynamic: falseon the linked objects(
organisation,subdivision, etc.) in the ES mappings so undeclaredembedded fields are no longer dynamically indexed. Requires a reindex.
For each resource, the useful linked fields must be decided explicitly
(default:
pid+name); anything not used for search, facets, display orpermissions should not be dumped.
Acceptance criteria
resources are present in
_sourcesubdivision.organisationfull dump)dynamic: falseon linked objectsTo verify during implementation
query fields, aggregations in
config.py, serializers, permissions) beforetrimming.