Retry transient Proton block failures without restarting the whole file

[ClaudiuSchuster]

Problem

A transient Proton Storage 502 currently aborts the complete file-upload
attempt even though the bridge still holds the encrypted blocks from the
current batch in memory. rclone's VFS then retries the file from the beginning.
For a large file this can retransmit tens of gigabytes and turn an otherwise
bounded upload into a many-hour operation.

PDrive's generation-bound recovery guard correctly avoids a service restart
while payload bytes continue to move. That guard cannot preserve work already
discarded inside one rclone file attempt, so this needs to be fixed in the
Proton API bridge upload path.

Sanitized live evidence

The issue was reproduced with PDrive's checksum-pinned rclone beta containing
the fresh-stream retry fix and the block-worker drain fix:

• one 30.4 GiB VFS upload with a 4.8 MiB/s payload limit;
• the same rclone process remained active for more than seven hours;
• aggregate transfer accounting exceeded 65 GB while the current file attempt
  was still at roughly 16 GB / 49%;
• six separated same-process 502 POST .../storage/blocks cycles were
  observed;
• no systemd service restart occurred after the worker-drain build became
  active;
• after the latest 502, the same PID resumed sustained payload progress.

This shows that the service-level recovery guard is behaving conservatively,
but whole-file VFS retries still discard substantial completed work.

Root cause

uploadAndCollectBlockData encrypts blocks in batches of eight 4 MiB blocks.
uploadPendingBlocks requests signed upload links, starts the block workers and
drains every result. If any worker returns an error, the function returns that
error immediately after draining the batch. The error propagates through
UploadFileByReader, and the caller can only retry the complete file stream.

The underlying API client retries rate limiting, 503, dial failures and
dropped connections, but it does not retry the observed 502. rclone's
low-level retry setting does not wrap this CallNoRetry upload path.

Expected behavior

• Retry only transiently failed encrypted blocks from the current batch.
• Request fresh signed upload links for the failed block indexes and create a
  fresh reader for every block attempt.
• Retain successful blocks instead of replaying the complete batch or file.
• Use a small bounded attempt count with context-aware backoff.
• Drain every started worker before deciding whether to retry or return.
• Return non-retryable errors immediately after the batch is drained.
• Return the final concrete error when the bounded retry budget is exhausted.
• Keep retry logs free of signed URLs, tokens, link IDs and file paths.

Safety boundaries

• Stay inside the existing draft, revision and block indexes.
• Do not delete cache data, drafts, revisions or remote files.
• Do not retry authentication, draft-conflict or other client-side 4xx
  failures as storage transients.
• Never loop indefinitely and always honor context cancellation.
• Preserve the existing global upload-worker semaphore and worker-drain
  guarantee.

Acceptance criteria

• A mixed batch retries only the failed transient blocks.
• A later successful retry completes without rereading the file stream.
• A non-retryable block failure is returned without another attempt.
• Exhausted transient retries return the last useful error.
• Cancellation during backoff returns promptly.
• Concurrent failures do not leak upload-worker semaphore slots.
• go test ./..., go test -race ./... and the repository linter pass.
• PDrive continues to treat measurable payload progress as non-actionable and
  does not restart a healthy moving upload.

The bridge repositories currently have their GitHub issue trackers disabled,
so this cross-repository dependency issue is tracked here and implemented in a
focused bridge pull request.

Implementation status

• oss-singularity/Proton-API-Bridge#2
  is merged at c059d7573afb. It combines the worker-drain dependency with the
  bounded failed-block retry in the maintained bridge branch.
• PDrive rclone pdrive-v1.76.0-beta.10204.2
  publishes the tested static Linux amd64 dependency at source commit
  d83aeb8f; SHA-256:
  85da18d19fd5e4a0969ed0fcc1ec43d00a90dfa7a06ce5483fd681c619db390d.
• proton-drive-linux#45
  pins that reviewed build and documents the recovery behavior for PDrive
  0.7.0.
• Upstream rclone/Proton-API-Bridge#8
  remains the worker-drain dependency. The bounded failed-block retry will be
  proposed as a separate focused follow-up after Create and configure repos social-media-preview 'repo card' #8 lands, avoiding a duplicate
  stacked upstream change.