Skip to content

cloud: qualify hybrid mode operationally with a backlog larger than the disk, complete disk loss, and sampled peak disk #337

Description

@smiggleworth

Gap cloud-23 · missing-coverage · confidence: verified · tracked in #242
Baseline: Midge 7d39f86 (0.1.1) vs Pants 2507c66. Midge paths are relative to the midge repo, Pants paths to this repo. Midge behavior is used as the proxy for the not-yet-extracted lsm-spec behavioral contract.

Prior: related to G02 and #128/#130 (closed)

Midge: docs/development/cloud-operational-qualification.md describes an ignored Sqrzl campaign
(tests/cloud_provider_engine_qualification.rs::operational::should_recover_cloud_backlog_after_complete_local_disk_loss). Its phases:

  • Delete the local directory and crash after the first recovery checkpoint.
  • Delete it again, wait out the lease, and fully recover.
  • Run a mixed workload with an injected SST-upload outage and observe retained charges.
  • Delete the directory again and verify every value.
  • Local file bytes are sampled every 5 ms, and the JSON evidence records HTTP cost, RSS and phases.
  • There are also enforced ext4/memory-limited profiles (resource-bound-qualification.md).

Pants: SqrzlCloudProviderQualificationTests.ShouldRecoverAfterCacheLossGiven* and CloudProviderEngineQualificationTests.*AfterLocalCacheLoss
verify correctness after cache loss on small data. Nothing runs a WAL backlog larger than the local budget, samples
peak local bytes, or injects an upload outage with a pressure check. PantsHybridStorageTests.ShouldKeepLocalUsageBoundedAcrossBurstyFlushCycles
checks final or periodic usage only.

Acceptance criteria: an opt-in Sqrzl campaign with configurable WAL, disk and memory sizes that crashes mid-recovery, recovers after complete disk loss, sustains an upload outage, and records sampled peak local bytes at or below the budget plus request counts. This depends on cloud-10-cloud-14.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:cloudReal cloud provider backendsarea:governanceRepository policy, review, and merge controlsparity-gapFeature not yet at parity with Midgepriority:P2Normal: bounded hardening, coverage, maintainability, or non-critical behavior gap

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions