[v3] Add dragonfly backend error handling and E2E tests

[bergwolf]

This PR implements proper error handling for Dragonfly backend with differentiated retry policies for prefetch vs on-demand reads. Adds configurable retry budgets, fallback throttling, and proper error classification, plus a new E2E test workflow to validate the changes.

• Prefetch reads retry up to 10 times before failing without fallback, while on-demand reads retry 3 times then fall back to origin registry. Rate limiting (429) triggers immediate fallback for on-demand reads and immediate failure for prefetch. Forbidden errors (403) are terminal for both read types.

• Adds fallback limiter to throttle consecutive origin requests with configurable interval. Implements proper error propagation with throttled marker for rate limiting scenarios. Updates telemetry metrics to track backend read attribution between proxy and origin.

Example workflow run: https://github.com/bergwolf/nydus/actions/runs/32847782111

[Copilot]

Pull request overview

Adds Dragonfly-aware retry, fallback, throttling, telemetry, and E2E coverage for registry reads.

Changes:

• Differentiates prefetch and on-demand Dragonfly failure policies.
• Adds delayed prefetch rescheduling and fallback telemetry.
• Introduces Dragonfly service configurations and CI workflows.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
.github/workflows/e2e-dragonfly.yml	Adds Dragonfly E2E CI jobs.
docs/nydus.md	Documents policies, configuration, and metrics.
misc/dragonfly/dfdaemon.yaml	Configures the test dfdaemon.
misc/dragonfly/manager.yaml	Configures the Dragonfly manager.
misc/dragonfly/scheduler.yaml	Configures the Dragonfly scheduler.
misc/dragonfly/nydus-sdk-fallback.yaml	Defines SDK fallback mode.
misc/dragonfly/nydus-sdk-strict.yaml	Defines SDK strict mode.
misc/dragonfly/nydus-proxy-error-fallback.yaml	Configures fallback error tests.
misc/dragonfly/nydus-proxy-error-strict.yaml	Configures strict error tests.
nydus-backend/src/lib.rs	Adds throttling markers and read attribution.
nydus-backend/src/registry/dragonfly.rs	Classifies Dragonfly SDK failures.
nydus-backend/src/registry/http.rs	Exposes a non-retrying HTTP client.
nydus-backend/src/registry/mod.rs	Implements retry and fallback policy.
nydus-config/src/lib.rs	Adds policy and rescheduling settings.
nydus-core/src/lib.rs	Manages the prefetch stop flag.
nydus-storage/src/prefetch.rs	Reschedules throttled prefetches.
nydus-telemetry/src/metrics.rs	Adds policy and fallback metrics.
nydus/src/bin/nydus/fuse.rs	Passes prefetch settings and stops workers.
tests/e2e/dragonfly_test.go	Adds Dragonfly integration tests.
tests/e2e/fanotify_test.go	Uses the shared environment helper.
tests/e2e/harness.go	Adds the shared environment helper.
tests/e2e/uffd_fault_test.go	Prevents GC deadlocks during UFFD tests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

Suppressed comments (6)

Previously missed (5) — in code that hasn't changed since the last review.

nydus/src/bin/nydus/fuse.rs:340

• The stop flag is raised only when session.serve() returns Ok; any outer error from signal setup, controller-thread creation, or result-channel receive exits via ? first and leaves the detached prefetcher rescheduling indefinitely. Capture the serve result, raise the flag on both paths, and only then propagate the error.

        // The filesystem is unmounted: wind down the detached prefetch thread
        // (it may be sleeping towards a rescheduled throttled-blob retry).
        if let Some(stop) = prefetch_stop {
            stop.store(true, std::sync::atomic::Ordering::Relaxed);
        }

nydus-config/src/lib.rs:196

• max_retries is a retry budget in addition to the initial attempt: the fallback loop stops when attempts > max_retries, and the test with max_retries: 1 expects two origin requests. The current “3 attempts” wording is off by one; the default permits four total attempts.

    /// The maximum number of retry attempts per request, applied by the HTTP
    /// client's retry middleware on direct origin requests — including origin
    /// requests issued as Dragonfly fallbacks, so the default of 3 is what
    /// bounds "origin failing 3 attempts" before a fallback read errors out.

docs/nydus.md:826

• This table describes http.max_retries as the total number of failed origin attempts, but the implementation performs one initial attempt plus that many retries. Clarify the total to avoid understating origin traffic and fallback latency.

	| Proxy `429` | No retry, no origin fallback; the blob's prefetch fails and is rescheduled after a random `prefetch.retry_delay_min`–`prefetch.retry_delay_max` delay | No Dragonfly retry; fall back to the origin through the fallback throttle; the origin failing `http.max_retries` attempts → IO error |

nydus-config/src/lib.rs:308

• The canonical config/registry.example.yaml is now stale: it still says http.max_retries controls Dragonfly SDK retries, says all Dragonfly failures fall back, and does not expose any of these new policy fields. Update that example with the differentiated policy and configurable budgets so users are not guided to ineffective settings.

    #[serde(
        default = "default_dragonfly_fallback_interval",
        with = "humantime_serde"
    )]
    pub fallback_interval: Duration,

.github/workflows/e2e-dragonfly.yml:368

• The newly added misc/dragonfly/nydus-proxy-error-strict.yaml fixture is never referenced; this job always supplies only the fallback fixture. Add the strict fixture to a matrix/run if it is intended coverage, or remove the dead configuration.

            NYDUS_CONFIG=${{ github.workspace }}/misc/dragonfly/nydus-sdk-fallback.yaml \
            NYDUS_CACHE_DIR=/tmp/nydus-dragonfly/cache-proxy-fallback \
            NYDUS_PROXY_ERROR_CONFIG=${{ github.workspace }}/misc/dragonfly/nydus-proxy-error-fallback.yaml \

nydus-backend/src/registry/mod.rs:705

• A fallback earlier in the same logical read sets READ_SERVED_BY to Origin, but a subsequent Dragonfly success (for example during a 401 auth handshake or after evicting an expired redirect) does not restore it. The metered read and any CRC error are then incorrectly attributed to origin even though Dragonfly supplied the final bytes. Record Proxy on this success path.

            let err = match self.request_dragonfly(dragonfly, url, headers.clone(), context) {
                Ok(response) => return Ok(response),