perf(solana): faster getBlock + smarter RPC failover

CLAUDE.md

@@ -0,0 +1,51 @@
+# CLAUDE.md
+
+Guidance for AI agents working in this repo. Keep changes small, tested, and consistent with existing patterns.
+
+## What this is
+
+Multi-chain blockchain transaction indexer (Go 1.25). Watches configured chains, fetches blocks, extracts transfers for monitored addresses, and emits events. Supported chains live in `internal/indexer/` (EVM, Solana, Bitcoin, Tron, Sui, Cosmos, Aptos, TON, XRP, Stellar).
+
+## Commands
+
+```bash
+make build                                          # -> ./indexer
+./indexer index --chain=<name> --catchup --debug    # run one chain (name = config key, e.g. solana_mainnet)
+./indexer index --catchup                            # run all enabled chains
+make stop                                            # pkill the running indexer
+go test ./...                                        # all tests (some hit live RPC / need network)
+go test ./pkg/adaptive/ -race                        # a single package, with race detector
+go fmt ./...                                          # format before committing
+```
+
+- Config path defaults to `configs/config.yaml`. **`configs/config.yaml` is gitignored** (real keys/endpoints live there, local only). The tracked template is `configs/config.example.yaml` — update it with **placeholders** (`${HELIUS_KEY}`), never real keys.
+- Running needs NATS + Redis reachable (`nats:` / `redis:` blocks in config). Without them the indexer fails at startup.
+
+## Architecture
+
+- `internal/indexer/` — per-chain `Indexer` implementations (`indexer.go` defines the interface: `GetBlock`, `GetBlocks`, `GetBlocksByNumbers`, `GetLatestBlockNumber`, `IsHealthy`). Each parses raw RPC blocks into `types.Transaction` transfers.
+- `internal/worker/` — worker modes over an indexer, built once per chain in `factory.go` and shared across modes (`types.go`): `regular` (real-time head), `catchup` (backfill ranges), `rescanner` (retry failed blocks), `manual`, `mempool`. `base.go` holds shared block-handling/emit logic.
+- `internal/rpc/` — `Failover[T]` provider pool: health tracking, blacklisting, latency-based rotation, error classification (`analyzeError`). `internal/rpc/<chain>/` has the concrete RPC clients.
+- `pkg/adaptive/` — AIMD concurrency limiter used to pace RPC calls to observed latency/errors.
+- `pkg/store/`, `pkg/kvstore/`, `pkg/repository/` — persistence (latest block, failed blocks, catchup ranges). `pkg/events/` — emission. `pkg/ratelimiter/` — shared per-chain RPS limiter. `pkg/common/config/` — config types.
+
+## Conventions
+
+- **Write minimal comments.** Prefer self-explanatory code (clear names, small functions) over comments. Only comment non-obvious rationale — a gotcha, a "why", something that would bite the next reader. Never restate what the code plainly does. No doc-comment boilerplate on every function.
+- **Adding/changing a chain:** implement `indexer.Indexer` in `internal/indexer/<chain>.go`, wire an RPC client in `internal/rpc/<chain>/`, add a `build<Chain>Indexer` in `internal/worker/factory.go`, and a config block.
+- **Failover tuning lives in code**, not yaml — `rpc.DefaultFailoverConfig()`. Every `NewFailover` call passes `nil` and gets those defaults. Don't reintroduce a yaml `failover:` block.
+- **Solana specifics:** `getBlock` uses `encoding=json` (not `jsonParsed`) — cheaper. The parser resolves accounts by index + base58 data and must append `meta.loadedAddresses` (v0 ALT accounts: static + writable + readonly) via `solanaEffectiveAccountKeys`. Skipped slots are normal (`ErrorTypeBlockNotFound`); never treat them as failed blocks.
+- **Free public RPCs cannot sustain Solana getBlock** at slot rate — expect 429s/lag without a keyed node. This is capacity, not code.
+- **Errors:** classify recoverable RPC failures in `analyzeError` (rate_limit, timeout, chain_unavailable, ...) so blacklist/cooldown policy is consistent.
+
+## Testing
+
+- Unit tests are deterministic and offline; prefer them. Some indexer tests hit live mainnet RPC (network-dependent — a DNS/429 failure there is environmental, not your change).
+- Use `-race` for anything with goroutines/locks (e.g. `pkg/adaptive`).
+- After edits: `go build ./...` then run the affected package's tests.
+
+## Git
+
+- Branch off `main`; never commit directly to it.
+- Conventional commits (`feat(scope):`, `fix(scope):`, `refactor:`, `chore:`).
+- Stage only the files you changed — do not `git add -A` (unrelated WIP files may be dirty in the tree).

configs/config.example.yaml

@@ -163,14 +163,19 @@ chains:
     start_block: 0
     poll_interval: "2s"
     nodes:
+      # STRONGLY RECOMMENDED: put at least one keyed node first. Free public
+      # nodes cannot sustain getBlock at Solana's slot rate — expect heavy 429s
+      # and lag without a paid/free-tier-with-key endpoint.
+      # - url: "https://mainnet.helius-rpc.com/?api-key=${HELIUS_KEY}"          # recommended
+      # - url: "https://solana-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"        # optional
+      # - url: "https://solana-mainnet.g.quicknode.pro/${QUICKNODE_KEY}/"      # optional
       - url: "https://solana-rpc.publicnode.com"
       - url: "https://api.mainnet.solana.com"
-      - url: "https://solana.drpc.org"
       - url: "https://solana.leorpc.com/?api_key=FREE"
+      - url: "https://solana-mainnet.gateway.tatum.io"
       - url: "https://solana.api.pocket.network"
       - url: "https://public.rpc.solanavibestation.com"
-      # - url: "https://solana-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY}"  # optional
-      # - url: "https://solana-mainnet.g.quicknode.pro/${QUICKNODE_KEY}/"  # optional
+      # NOTE: solana.drpc.org does NOT serve Solana on the free plan — removed.
     client:
       timeout: "15s"
       max_retries: 2
@@ -189,8 +194,13 @@ chains:
     start_block: 0
     poll_interval: "2s"
     nodes:
-      - url: "https://api.devnet.solana.com"
+      # Keyed nodes (Ankr/Chainstack free tiers with a key) lead — verified to
+      # serve getBlock reliably, unlike the unkeyed public devnet endpoints.
+      # - url: "https://rpc.ankr.com/solana_devnet/${ANKR_KEY}"
+      # - url: "https://solana-devnet.core.chainstack.com/${CHAINSTACK_KEY}"
       - url: "https://solana-devnet.api.onfinality.io/public"
+      - url: "https://solana-devnet.gateway.tatum.io/"
+      - url: "https://api.devnet.solana.com"
     client:
       timeout: "12s"
       max_retries: 2

internal/indexer/solana.go

@@ -10,6 +10,7 @@ import (
 
 	"github.com/fystack/multichain-indexer/internal/rpc"
 	"github.com/fystack/multichain-indexer/internal/rpc/solana"
+	"github.com/fystack/multichain-indexer/pkg/adaptive"
 	"github.com/fystack/multichain-indexer/pkg/common/config"
 	"github.com/fystack/multichain-indexer/pkg/common/constant"
 	"github.com/fystack/multichain-indexer/pkg/common/enum"
@@ -25,6 +26,8 @@ type SolanaIndexer struct {
 	config      config.ChainConfig
 	failover    *rpc.Failover[solana.SolanaAPI]
 	pubkeyStore PubkeyStore
+	// shared across worker modes: one congestion controller per chain.
+	limiter *adaptive.Limiter
 }
 
 func NewSolanaIndexer(
@@ -33,7 +36,25 @@ func NewSolanaIndexer(
 	failover *rpc.Failover[solana.SolanaAPI],
 	pubkeyStore PubkeyStore,
 ) *SolanaIndexer {
-	return &SolanaIndexer{chainName: chainName, config: cfg, failover: failover, pubkeyStore: pubkeyStore}
+	maxConc := cfg.Throttle.Concurrency
+	if maxConc <= 0 {
+		maxConc = 1
+	}
+	limiter := adaptive.New(adaptive.Config{
+		Max:            maxConc,
+		Min:            1,
+		LowLatency:     1200 * time.Millisecond,
+		HighLatency:    2500 * time.Millisecond,
+		AdjustInterval: time.Second,
+		GrowStreak:     10,
+	})
+	return &SolanaIndexer{
+		chainName:   chainName,
+		config:      cfg,
+		failover:    failover,
+		pubkeyStore: pubkeyStore,
+		limiter:     limiter,
+	}
 }
 
 func (s *SolanaIndexer) GetName() string                  { return strings.ToUpper(s.chainName) }
@@ -150,36 +171,30 @@ func (s *SolanaIndexer) GetBlocksByNumbers(ctx context.Context, blockNumbers []u
 		return []BlockResult{}, nil
 	}
 
-	maxConc := s.config.Throttle.Concurrency
-	if maxConc <= 0 {
-		maxConc = 1
-	}
-
 	results := make([]BlockResult, len(blockNumbers))
 
 	eg, egCtx := errgroup.WithContext(ctx)
-	sem := make(chan struct{}, maxConc)
 
 	for i, slot := range blockNumbers {
 		i := i
 		slot := slot
 		eg.Go(func() error {
-			select {
-			case sem <- struct{}{}:
-				defer func() { <-sem }()
-			case <-egCtx.Done():
-				return egCtx.Err()
+			if err := s.limiter.Acquire(egCtx); err != nil {
+				return err
 			}
+			defer s.limiter.Release()
 
 			var (
 				b    *solana.GetBlockResult
 				berr error
 			)
+			fetchStart := time.Now()
 			berr = s.failover.ExecuteWithRetry(egCtx, func(c solana.SolanaAPI) error {
 				blk, err := c.GetBlock(egCtx, slot)
 				b = blk
 				return err
 			})
+			s.limiter.Observe(time.Since(fetchStart), berr == nil)
 
 			if berr != nil {
 				results[i] = BlockResult{Number: slot, Error: &Error{ErrorType: ErrorTypeUnknown, Message: berr.Error()}}
@@ -464,6 +479,23 @@ func solanaParseTokenTransfer(ix solana.Instruction, accountKeys []solana.Accoun
 	}
 }
 
+// solanaEffectiveAccountKeys appends v0 ALT accounts (static + writable + readonly)
+// so instruction/token-balance indices resolve under encoding=json.
+func solanaEffectiveAccountKeys(static []solana.AccountKey, loaded *solana.LoadedAddresses) []solana.AccountKey {
+	if loaded == nil || (len(loaded.Writable) == 0 && len(loaded.Readonly) == 0) {
+		return static
+	}
+	out := make([]solana.AccountKey, 0, len(static)+len(loaded.Writable)+len(loaded.Readonly))
+	out = append(out, static...)
+	for _, pk := range loaded.Writable {
+		out = append(out, solana.AccountKey{Pubkey: pk, Writable: true})
+	}
+	for _, pk := range loaded.Readonly {
+		out = append(out, solana.AccountKey{Pubkey: pk})
+	}
+	return out
+}
+
 func (s *SolanaIndexer) extractSolanaTransfers(networkID string, slot uint64, ts uint64, b *solana.GetBlockResult) []types.Transaction {
 	out := make([]types.Transaction, 0)
 	for txIdx, tx := range b.Transactions {
@@ -478,7 +510,7 @@ func (s *SolanaIndexer) extractSolanaTransfers(networkID string, slot uint64, ts
 		}
 		txHash := tx.Transaction.Signatures[0]
 		fee := decimal.NewFromInt(int64(tx.Meta.Fee))
-		accountKeys := tx.Transaction.Message.AccountKeys
+		accountKeys := solanaEffectiveAccountKeys(tx.Transaction.Message.AccountKeys, tx.Meta.LoadedAddresses)
 
 		// Build token-account -> (owner, mint) lookup from token balance metadata.
 		// This isn't used to infer transfers; only to map SPL token accounts to owners/mints.

internal/indexer/solana_test.go

@@ -6,6 +6,7 @@ import (
 	"time"
 
 	"github.com/fystack/multichain-indexer/internal/rpc/solana"
+	"github.com/fystack/multichain-indexer/pkg/adaptive"
 	"github.com/fystack/multichain-indexer/pkg/common/config"
 	"github.com/fystack/multichain-indexer/pkg/common/constant"
 	"github.com/fystack/multichain-indexer/pkg/common/types"
@@ -24,6 +25,7 @@ func newTestSolanaIndexer() *SolanaIndexer {
 		chainName:   "solana",
 		config:      config.ChainConfig{NetworkId: "solana-mainnet"},
 		pubkeyStore: nil, // no filtering
+		limiter:     adaptive.New(adaptive.Config{Max: 4}),
 	}
 }
 
@@ -370,3 +372,29 @@ func TestParseSquadsMultisigTransfer(t *testing.T) {
 		tokenTransfer.FromAddress, tokenTransfer.ToAddress,
 		tokenTransfer.Amount, tokenTransfer.AssetAddress)
 }
+
+// TestSolanaEffectiveAccountKeys: v0 ALT accounts append as static+writable+readonly.
+func TestSolanaEffectiveAccountKeys(t *testing.T) {
+	static := []solana.AccountKey{{Pubkey: "S0"}, {Pubkey: "S1"}}
+
+	// No loaded addresses: returns the static slice unchanged (jsonParsed path).
+	assert.Equal(t, static, solanaEffectiveAccountKeys(static, nil))
+	assert.Equal(t, static, solanaEffectiveAccountKeys(static, &solana.LoadedAddresses{}))
+
+	loaded := &solana.LoadedAddresses{
+		Writable: []string{"W0", "W1"},
+		Readonly: []string{"R0"},
+	}
+	got := solanaEffectiveAccountKeys(static, loaded)
+	require.Len(t, got, 5)
+
+	pubkeys := make([]string, len(got))
+	for i, k := range got {
+		pubkeys[i] = k.Pubkey
+	}
+	assert.Equal(t, []string{"S0", "S1", "W0", "W1", "R0"}, pubkeys,
+		"order must be static + loaded writable + loaded readonly")
+	assert.True(t, got[2].Writable, "loaded writable accounts must be marked writable")
+	assert.True(t, got[3].Writable)
+	assert.False(t, got[4].Writable, "loaded readonly accounts must not be writable")
+}