Sometimes deposits are not marked as validated even when document is created

[PascalRepond]

Bug description

When validating (approving) a deposit, the system creates the document but
sometimes does not mark the deposit as validated. The moderator then retries,
which creates the document a second time (double validation).

Example: document https://folia.unifr.ch/manage/records/documents/detail/324501
was created, but the corresponding deposit stayed "to validate":
https://folia.unifr.ch/manage/deposit/324490/create

Root cause

The approve flow is neither atomic nor idempotent.

In DepositRecord.create_document()
(sonar/modules/deposits/api.py):

1. DocumentRecord.create(metadata, dbcommit=True, ...) commits the document
   to the DB immediately (independently of the deposit).
2. Files are added, then document.reindex() runs — this dumps all resolved
   fields and, if there are more than 1000 fields, ES throws here.
3. self["document"] = {"$ref": ...} (the deposit→document link) is only set
   after the reindex.

In the review endpoint (sonar/modules/deposits/rest.py):
create_document() → deposit["status"] = VALIDATED → email → final
db.session.commit().

So if document.reindex() (or any later step) raises:

• the document is already committed (step 1),
• the deposit→document link (step 3) and status = VALIDATED + final commit
  never run → the deposit stays "to validate",
• the request returns 500.

Result: document created and committed, deposit not validated, no link
between them. On retry, create_document() runs again with no guard checking
self.get("document") → a second document is created.

The >1000-fields ES error is only the trigger that was observed; the
underlying defect is the non-atomic, non-idempotent design, which any
post-create_document() failure (email, network to ES, DB hiccup, a future
mapping change) can re-trigger.

Relation to #1014

#1014 (trim dumped linked fields) removes the specific >1000-fields trigger,
and the ES field limit was already raised in prod. Those reduce the
occurrences but do not fix the structural fragility addressed here.

Expected behavior

Validating a deposit either fully succeeds (document created and deposit
marked validated, atomically) or fully fails (nothing created, deposit
unchanged) — and can never create a duplicate document.

What should be done

• Atomicity (full rollback on failure): make document creation +
  deposit status update a single transaction. If any step fails (including
  indexing), roll everything back so neither a document nor a validated
  deposit remains; the moderator can retry cleanly. Do not commit the
  document independently before the deposit status is ready to commit.
• Idempotency guard: in create_document(), if the deposit already has a
  linked document (self.get("document")), do not create another one.

Acceptance criteria

• On a successful approve: document created, deposit validated, and the
  deposit.document link set — all committed together
• On a failure during approve (incl. indexing error): no document is
  persisted and the deposit stays to_validate
• Re-running approve on a deposit that already has a linked document does
  not create a second document
• Test reproducing a failure mid-validation and asserting consistent state

Server: production (folia) — v1.8.2

[PascalRepond]

This seems to not be linked to the data and is difficult to reproduce. Maybe we should add a security on deposit validation.

[PascalRepond]

Some deposits have no link to documents in the data, maybe it is linked:

[
    '312965', '312988', '313007', '313010', '313095',
    '313097', '313107', '313114', '313118', '313146',
    '313149', '313199', '313200', '313258', '313259',
    '313260', '313261', '313262', '313264', '313265',
    '313266', '313267', '313268', '313269', '313270',
    '313271', '313272', '313273', '313274', '313275',
    '313276', '313277', '313278', '313279', '313280',
    '313282', '313283', '319486', '319487', '319488',
    '319489', '319490', '319491', '319492', '319493',
    '319494', '319495', '319496', '319497', '319498',
    '319499', '319500', '319501', '319502', '319503',
    '319504', '319505', '319596', '319627', '319702',
    '319817', '320093', '320114', '320387', '320389',
    '320506', '320507', '320508', '320509', '320510',
    '320511', '320512', '320513', '320514', '320515',
    '320516', '320517', '320518', '320519', '320520',
    '320521', '320522', '320523', '320524', '320525',
    '320526', '320527', '320528', '320529', '320543',
    '320697', '320931'
]

[PascalRepond]

This could be caused by an error during deposit reindexing, see SONAR-N0 (Sentry).
On validation, the system dumps all document resolved index fields. If there are more than 1000 fields, there is an ES error. The document is created, but the deposit is not validated!

We increased the fields limit in ES prod, but this dump mechanism might still be improved to be more robust on the app level.