Fix nested message object in API responses

[ErykKul]

What this PR does / why we need it:

Fixes the ok(String msg, JsonObjectBuilder bld) method in AbstractApiBean.java which incorrectly wraps the message in a nested object:

// Before (incorrect):
{"status":"OK","message":{"message":"actual message"},"data":{...}}

// After (correct):
{"status":"OK","message":"actual message","data":{...}}

The bug was introduced in commit f311312c34 on May 12, 2020, as part of #4813.

Which issue(s) this PR closes:

• Closes API returns malformed JSON with nested message object instead of string in ok(String, JsonObjectBuilder) responses #12096
• Closes Dataverse API responses inconsistent in JSON structure #11667

Special notes for your reviewer:

This is a one-line fix in AbstractApiBean.java line 1000:

// From:
.add("message", Json.createObjectBuilder().add("message",msg))
// To:
.add("message", msg)

The following 7 API endpoints are affected:

File	Line	API Endpoint
Datasets.java	3087	POST /api/datasets/{id}/add (duplicate file warning)
Admin.java	233	PUT /api/admin/settings
Dataverses.java	754	PUT /api/dataverses/{id}
Dataverses.java	788	PUT /api/dataverses/{id}/inputLevels
SavedSearches.java	161	POST /api/admin/savedsearches
HarvestingClients.java	309	PUT /api/harvest/clients/{nickName}
HarvestingServer.java	228	PUT /api/harvest/server/oaisets/{specname}

Suggestions on how to test this:

1. Upload a file to a dataset via API
2. Upload the same file again (duplicate)
3. Check that the response message field is a string, not a nested object:

curl -X POST "http://localhost:8080/api/datasets/:persistentId/add?persistentId=doi:..." \
  -H "X-Dataverse-key: $API_TOKEN" \
  -F "file=@test-file.txt"
# Upload again to trigger duplicate warning
curl -X POST "http://localhost:8080/api/datasets/:persistentId/add?persistentId=doi:..." \
  -H "X-Dataverse-key: $API_TOKEN" \
  -F "file=@test-file.txt"

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No

Is there a release notes update needed for this change?:

Yes, included in this PR: doc/release-notes/12096-fix-ok-message-nested-object.md

Additional documentation:

N/A

[coveralls]

coverage: 24.964% (+0.01%) from 24.951% — 12096-fix-ok-message-nested-object into develop

[pdurbin]

A couple quick comments.

Also, @poikilotherm noticed this double message bug, which lead to this API v2 proposal: https://docs.google.com/document/d/1XtXPF_PZCuhPbm4HCu28yRge_4_--ah604NhigPdjPk/edit?usp=sharing . And these related issues:

• #11836
• IQSS/dataverse-pm#445

[ErykKul]

@pdurbin Thanks for the quick review and the context about the API v2 proposal!

I was a bit hesitant to make this change since it could break existing API consumers that have implemented workarounds for the nested message object. However, I think it's worth fixing. The current behavior is clearly a bug, and the affected endpoints are relatively niche (the bug was actually discovered through duplicate file uploads, while regular file additions work fine).

That said, I can see the argument for keeping this for API v2 if you think the breaking change risk is too high. Let me know your thoughts - happy to go either way.

I've fixed the integration tests and added the changelog entry as requested. The docs build should pass now too (had a small RST syntax issue with the reference).

[poikilotherm]

While the commits look good overall, I am reluctant to give this a pass as-is.
I discovered more problems with inconsistencies in #11667 and as @pdurbin pointed out, this led to the v2 API proposal.

I agree with @ErykKul that changing this is very useful. And for 6 of the 7 affected endpoints I am agreeing that changing them may cause minor inconveniences for administrators (fine, just deal with it).

Yet with the Dataset endpoint to add files warning about duplicates and which are duplicated - this may seriously affect some integration with Dataverse out there. This may actually be a breaking change causing trouble for users. IMHO we shouldn't do this.

Also, if we fix that, let's do it all in one sweep and fix any other inconsistencies as well.

There may be some middle ground here if we make this opt-in. Using a feature flag marking it as experimental, we can leave a choice to the admin. They can test with their integrations and see what happens and return to the default at any time. (This reminds me of mentioning https://openfeature.dev somewhere - making feature flags activate based on more criteria than mere configuration...)
A way forward may be to make this experimental for now, then deprecate it in a future version, then drop it. This way this may not even need an API v2 (only regarding this message - the mileage for the other inconsistencies may vary!)

[ErykKul]

Hi @pdurbin, @poikilotherm,

I've addressed the feedback in this PR:

• Merged the latest develop into this branch.
• Added a feature flag dataverse.feature.api-message-field-legacy as @poikilotherm suggested. The fix is now the default behavior, and admins who discover issues with their integrations can set this flag to true to revert to the previous (buggy) format. This way we get the fix out without risking breakage for anyone.
• Updated the release notes and API changelog accordingly.

I ran the full test suite locally (unit tests + integration tests) and everything passes. However, I'm unable to access the Jenkins server from here — it seems to be restricted to internal networks. @pdurbin, if Jenkins fails again on this latest commit, could you share the relevant log output or a screenshot? That would help me figure out what's going on. Thanks!

[ErykKul]

All tests passed, I reverted the unrelated changes so tests fail again. However, the fix is in #11983

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[poikilotherm]

I'm not sure why @ErykKul re-assigned this to me - is there something missing, do I need to do anything? In my head I marked this one as "done here" for myself. As I have skin in this PR now, I figured I should not be the one to review it. So I'm taking myself of the PR (again) and put it in "Ready for Review". Thanks guys!

Tante Edit: I did take a quick look at the changes @ErykKul pushed to get the latest developmerged and catch a few small things. Nothing stood out as a particular risk, thanks for fixing and merging all of these in!

[sekmiller]

Hi
@ErykKul, sorry this has been sitting around for so long. I went to pick it up today and saw that there are merge conflict. Please resolve them and assign to me when you are done. Thanks!

[pdurbin]

@ErykKul just a reminder about merge conflicts! Great seeing you in Barcelona!

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:12096-fix-ok-message-nested-object

ghcr.io/gdcc/configbaker:12096-fix-ok-message-nested-object

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[ErykKul]

Conflict resolved.