Skip to content

CBG-5731 Finish a database delete when another node already deleted it - #8834

Merged
bbrks merged 3 commits into
mainfrom
CBG-5731
Sep 29, 2026
Merged

bbrks merged 3 commits into
mainfrom
CBG-5731

Conversation

@torcolvin

@torcolvin torcolvin commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

CBG-5731 Finish a database delete when another node already deleted it

When DELETE /db/ runs on a node after a peer deleted the database, the node now removes the database locally and returns 200 instead of 500. If the config doc is already gone when the node deletes it, the node still finalizes the registry, so no delete stays marked in progress.

Registry finalize now removes the entry only if it is still the delete that this node marked. Before, a peer could recreate the database after the config doc was deleted, and finalize removed the new registry entry.

Add LeakyBootstrapConnection with hooks around DeleteMetadataDocument, so that tests can simulate a peer that writes during a delete.

I don't think this blocks this commit but some tests are skipped on rosmar due to #8832

Pre-review checklist

  • Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
  • Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

When DELETE /db/ runs on a node after a peer deleted the database, the
node now removes the database locally and returns 200 instead of 500.
If the config doc is already gone when the node deletes it, the node
still finalizes the registry, so no delete stays marked in progress.

Registry finalize now removes the entry only if it is still the delete
that this node marked. Before, a peer could recreate the database after
the config doc was deleted, and finalize removed the new registry entry.

Add LeakyBootstrapConnection with hooks around DeleteMetadataDocument,
so that tests can simulate a peer that writes during a delete.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@torcolvin
torcolvin requested review from a team and a balanced review from Copilot September 29, 2026 00:55
@factory-droid

factory-droid Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Droid finished @torcolvin's task —— View job


Review summary

The persistent config delete handling looks good and the test coverage for concurrent deletion/recreation cases is strong. The main follow-up is keeping the new leaky bootstrap connection behavior consistent across all test harness code paths that rebuild bootstrap connections or add nodes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Interrupted-delete recovery returns ErrConfigRegistryRollback, which remains mapped to HTTP 500.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Improves multi-node database deletion recovery and prevents deletion finalization from removing a recreated registry entry.

Changes:

  • Treats missing config documents as successful concurrent deletion.
  • Guards registry finalization using the original config version.
  • Adds bootstrap hooks and multi-node concurrency tests.
File Description
base/​leaky_bootstrap_connection.go Adds test hooks around metadata deletion.
rest/​admin_api.go Handles peer-deleted database configs.
rest/​config_manager.go Finalizes concurrent deletes safely.
rest/​persistent_config_test.go Adds deletion race and recovery tests.
rest/​server_context.go Installs the leaky bootstrap wrapper.
rest/​utilities_testing.go Exposes wrapper configuration to tests.
rest/​utilities_testing_rest_tester_cluster.go Supports per-node test hooks.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rest/admin_api.go
Comment thread rest/utilities_testing.go
Comment thread rest/utilities_testing_rest_tester_cluster.go
@bbrks
bbrks merged commit 8d44b3d into main Sep 29, 2026
49 checks passed
@bbrks
bbrks deleted the CBG-5731 branch September 29, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants