Problem: Getting no available backend

[morenonaitana]

I'm seeing the following error every day, it seems the backend is not stable, not sure if that is something currently being worked on but I prefer to create a ticket in case to rise the concern. Here is what I see:

 ERROR  [Balance] Failed to fetch TT2 bridge history: { address: '0xe364cc00c399930247ef70411f637c8e3f01549b',
  error:
   Error: {"jsonrpc":"2.0","id":22,"error":{"code":-900,"message":"no available backend"}}
       at checkRes (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/api_client/client.mjs:245:13)
       at NodeKwil.callClient (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/api_client/client.mjs:193:12)
       at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
       at async NodeKwil.baseCall (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/client/kwil.mjs:340:24)
       at async NodeKwil.call (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/client/node/nodeKwil.mjs:29:12)
       at async Object.handler (/Users/moreno/truflation/website/server/api/balance/truf-tt2-history.get.ts:82:1)
       at async file:///Users/moreno/truflation/website/node_modules/h3/dist/index.mjs:1962:19
       at async Object.callAsync (file:///Users/moreno/truflation/website/node_modules/unctx/dist/index.mjs:72:16)
       at async Server.toNodeHandle (file:///Users/moreno/truflation/website/node_modules/h3/dist/index.mjs:2249:7) }

I'm using those:
BILLING_TN_PROVIDER_HOODI=https://gateway.testnet.truf.network
BILLING_TN_CHAIN_ID_HOODI=testnet-v1

used this way:

const signerWallet = new Wallet(privateKey);

  cachedClient = new NodeTNClient({
    endpoint,
    signerInfo: {
      address: signerWallet.address,
      signer: signerWallet,
    },
    chainId,
    timeout: 30000,
  });

cc: @markholdex @vinarmani @MicBun

[MicBun]

Log: updated the instance size to t3a.large and tweaking memory on the server. Will keep monitoring now

[vinarmani]

@MicBun so you have identified this as a validator system resources issue?

[MicBun]

Yeah, one of it, the leader node keep crashing mid point and then automatically restarts. I previously set the automatic restart, but it is not enough, so I decided it is time to increase from t3.medium.