Skip to content

Add shared HTTP retry adapter for sub-indexers - #1862

Merged
YoshihitoAso merged 1 commit into
dev-26.9from
improve-retry-s3
Aug 28, 2026
Merged

Add shared HTTP retry adapter for sub-indexers#1862
YoshihitoAso merged 1 commit into
dev-26.9from
improve-retry-s3

Conversation

@YoshihitoAso

@YoshihitoAso YoshihitoAso commented Aug 28, 2026

Copy link
Copy Markdown
Member

📌 Description

Improve resilience when fetching public information lists from S3 URLs that temporarily return HTTP 503 responses.

✅ Related Issues

None

🔄 Changes

  • Added a shared HTTP retry adapter for public information fetches.
  • Retry HTTP 500, 502, 503, and 504 responses for GET requests.
  • Retry configuration:
    • total=3: retry up to three times after the initial request, for a maximum of four requests in total.
    • backoff_factor=1.0: use exponential backoff between retries, resulting in approximately 0, 2, and 4 seconds of delay.
    • backoff_jitter=0.5: add a small random delay to avoid simultaneous retries from multiple processes.
    • respect_retry_after_header=True: honor the wait time specified by the S3 response.
    • raise_on_status=False: return the final HTTP response after retries are exhausted so the indexer can handle it consistently.
    • status_forcelist=(500, 502, 503, 504): retry temporary server-side errors.
    • allowed_methods=("GET",): limit retries to idempotent GET requests.
  • Applied the retry policy to Company List, Token List, and Public Account List indexers.
  • Preserve ERROR-level exception logging when all retry attempts fail.
  • Added test coverage for the retry configuration and failed request logging.

📌 Checklist

  • I have added tests where necessary.
  • I have updated the documentation where necessary.

@github-actions

Copy link
Copy Markdown

Coverage

Coverage Report •
FileStmtsMissBranchBrPartCoverMissing
batch/lib
   http.py4000100% 
batch/sub_indexers
   indexer_Company_List.py72712289%52->53, 53–54, 95->105, 105–106, 109–111
   indexer_PublicInfo_PublicAccountList.py75412293%52->53, 53–54, 104->105, 105, 108
   indexer_PublicInfo_TokenList.py6758192%53->54, 54–55, 99–101
tests/batch/sub_indexers
   indexer_Company_List_test.py558000100% 
TOTAL374601470352464795% 

Tests Skipped Failures Errors Time
1281 0 💤 0 ❌ 0 🔥 9m 30s ⏱️

@YoshihitoAso
YoshihitoAso merged commit 571f168 into dev-26.9 Aug 28, 2026
13 checks passed
@YoshihitoAso
YoshihitoAso deleted the improve-retry-s3 branch August 28, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants