test(dursto): add long-running Registry test

[victor-dumitrescu]

Closes TZX-180.

What

The Registry long test is structured similarly to the Database long test: it is based on running a number of independent "test epochs" (property-based test cases) on top of a base, which is advanced after each epoch. When a test case fails, the failure artifact can be persisted so that only the failing epoch needs to be replayed to reproduce it.

Specifically in the case of the Registry long tests:

• The test is set up with a number of "permanent" databases which cannot be moved, cleared, or copied into. This is to guarantee that a certain number of databases will steadily grow during the course of the test.
• In addition, a --keep-stable-size mode adjusts the probabilities of sampling registry grow and shrink operations so that the size of the registry tends towards 2 * permanent (as opposed to equal probabilities which yield an ever growing registry). This resembles more closely the way in which Etherlink/Tezos X is expected to use the NDS.
• The --keep-epochs option requires a slightly more complicated snapshot pruning approach compared to the Database tests.

For now, a 10-minute run of the Registry long test is added to PR CI. Next:

• feat(ci): add Registry long tests to nightly tests #1122

Why

To gain a higher degree of confidence in the correctness and consistency of the durable storage.

How

Most of the large diff of this PR comes from the first 2 commits (notably from moves in the first commit):

• refactor(test): extract shared long-test harness:
  • change directory structure, move shared helpers to long_test/harness.rs and database-specific ones to long_test/database/*
  • test Base becomes Base<LongTestModel> to enable sharing
  • slight refactor of the Database key generation strategy (pooled_key_strategy) for clarity
• test(dursto): add long-running Registry test:
  • adds the Registry long test in long_test/registry/*
  • adds database and registry subcommands to the long test binary

The other 4 commits should hopefully be more self-explanatory.

Manually Testing

make all

Tasks for the Author

• Link all Linear issues related to this MR using magic words (e.g. part of, relates to, closes).
• Eliminate dead code and other spurious artefacts introduced in your changes.
• Document new public functions, methods and types.
• Make sure the documentation for updated functions, methods, and types is correct.
• Add tests for bugs that have been fixed.
• Explain changes to regression test captures when applicable.
• Write commit messages in agreement with our guidelines.
• Self-review your changes to ensure they are high-quality.
• Complete all of the above before assigning this MR to reviewers.

[codecov]

Codecov Report

❌ Patch coverage is 80.10033% with 238 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (vdum/tzx-178@3da33a3). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
durable-storage/src/long_test/database/mod.rs	72.36%	54 Missing and 35 partials ⚠️
durable-storage/src/long_test/registry/mod.rs	76.47%	42 Missing and 42 partials ⚠️
durable-storage/src/bin/long_test.rs	0.00%	29 Missing ⚠️
durable-storage/src/long_test/harness.rs	64.28%	12 Missing and 3 partials ⚠️
durable-storage/src/long_test/registry/gc.rs	88.28%	5 Missing and 8 partials ⚠️
durable-storage/src/long_test/registry/run_case.rs	96.87%	3 Missing and 1 partial ⚠️
durable-storage/src/long_test/registry/model.rs	95.12%	1 Missing and 1 partial ⚠️
durable-storage/src/registry.rs	83.33%	0 Missing and 1 partial ⚠️
durable-storage/src/storage/in_memory.rs	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##             vdum/tzx-178    #1121   +/-   ##
===============================================
  Coverage                ?   89.85%           
===============================================
  Files                   ?      143           
  Lines                   ?    30146           
  Branches                ?    30146           
===============================================
  Hits                    ?    27087           
  Misses                  ?     2181           
  Partials                ?      878           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.