Skip to content

Nightly snippet run failed #179

Description

@github-actions

The nightly snippet run failed on c1cb04dfa41e8dbca73d0069ffc5a94070936d92.

Run log: https://github.com/allora-network/docs/actions/runs/31239309367

Snippet run report

  • snippets discovered: 15
  • executed: 13
  • failed: 4

snippets/forecaster.py

exited with code 1

Traceback (most recent call last):
  File "/tmp/allora-snippets-CCJhnn/runs/forecaster.py/forecaster.py", line 38, in <module>
    asyncio.run(main())
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/runs/forecaster.py/forecaster.py", line 31, in main
    async for result in worker.run():
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/worker/worker.py", line 498, in run
    await self._ensure_initialized()
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/worker/worker.py", line 332, in _ensure_initialized
    self._chain_id = await self.client.raise_for_chain_id_mismatch()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/client.py", line 211, in raise_for_chain_id_mismatch
    chain_id = await self.chain_id()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/client.py", line 218, in chain_id
    node_info_resp = await self.tendermint.query.get_node_info(tendermint_v1beta1.GetNodeInfoRequest())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/allora_sdk/rpc_client/protos/cosmos/base/tendermint/v1beta1/__init__.py", line 619, in get_node_info
    return await self._unary_unary(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/betterproto2/grpc/grpclib_client.py", line 69, in _unary_unary
    response = await stream.recv_message()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 426, in recv_message
    await self.recv_initial_metadata()
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 394, in recv_initial_metadata
    self._raise_for_grpc_status(status, message, details)
  File "/tmp/allora-snippets-CCJhnn/venv/lib/python3.11/site-packages/grpclib/client.py", line 346, in _raise_for_grpc_status
    raise GRPCError(status, message, details)
grpclib.exceptions.GRPCError: (<Status.UNAVAILABLE: 14>, 'upstream connect error or disconnect/reset before headers. reset reason: connection termination', None)

snippets/migrate_worker.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:23:58,986 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489480469174 ALLO
2026-08-08 04:23:59,011 INF 🔄 Starting polling worker
2026-08-08 04:23:59,035 INF    Topic 69: unfulfilled nonces: {10418255}
2026-08-08 04:23:59,036 INF    Our unfulfilled nonces: {10418255}
2026-08-08 04:23:59,049 INF 👉 Found new nonce 10418255 for topic 69, submitting... account_seq=40
2026-08-08 04:23:59,073 WRN ⚠️⚠️⚠️  SANITY CHECK WARNING: Your prediction (123.450000) is 328.4 standard deviations from the network consensus (mean: 64332.131497, std: 195.490681). Please verify you're predicting the correct target variable and using the right units.
2026-08-08 04:24:09,251 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418255 code=1
2026-08-08 04:24:09,275 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489466969174 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=A5C12FEE328A53D97D064CD5F59AF132FC6E733A4EC4648E32FE9EB2FD3DB9BB failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:24:09,286 INF Received signal 15, initiating graceful shutdown

snippets/sdk_py_worker.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:27:13,473 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489453469164 ALLO
2026-08-08 04:27:13,499 INF 🔄 Starting polling worker
2026-08-08 04:27:13,524 INF    Topic 69: unfulfilled nonces: {10418309}
2026-08-08 04:27:13,524 INF    Our unfulfilled nonces: {10418309}
2026-08-08 04:27:13,549 INF 👉 Found new nonce 10418309 for topic 69, submitting... account_seq=42
2026-08-08 04:27:15,691 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418309 code=1
2026-08-08 04:27:15,714 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489439969164 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=F75780859D3CBBCC37B364E201BA156A8ED31CDD5047298EBFF7B47E2FE67629 failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:27:15,726 INF Received signal 15, initiating graceful shutdown

snippets/worker_minimal.py

printed an error it swallowed (/Inference worker error:/)

_    _     _     ___  ____      _
      / \  | |   | |   / _ \|  _ \    / \
     / _ \ | |   | |  | | | | |_) |  / _ \
    / ___ \| |___| |__| |_| |  _ <  / ___ \        Chain:   allora-testnet-1
   /_/   \_\_____|_____\___/|_| \_\/_/   \_\       Topic:   PLAYGROUND: 1 day BTC/USD Price Prediction (ID: 69)
   __        _____  ____  _  _______ ____          Address: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe
   \ \      / / _ \|  _ \| |/ / ____|  _ \         Role:    INFERER
    \ \ /\ / / | | | |_) | ' /|  _| | |_) |
     \ V  V /| |_| |  _ <| . \| |___|  _ <
      \_/\_/  \___/|_| \_\_|\_\_____|_| \_\

2026-08-08 04:27:20,913 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489439969164 ALLO
2026-08-08 04:27:20,942 INF 🔄 Starting polling worker
2026-08-08 04:27:20,968 INF    Topic 69: unfulfilled nonces: {10418309}
2026-08-08 04:27:20,968 INF    Our unfulfilled nonces: {10418309}
2026-08-08 04:27:20,994 INF 👉 Found new nonce 10418309 for topic 69, submitting... account_seq=43
2026-08-08 04:27:39,246 INF ⚠️ Already submitted for this epoch: topic_id=69 nonce=10418309 code=1
2026-08-08 04:27:39,270 INF    Worker wallet: allo12vaa2z44uyse3sagxvrya8mavahguamqfqxefe  ||  Balance: 100002.395366489426469164 ALLO
Inference worker error: TxError: codespace=undefined code=1 tx_hash=25701F2DCC58D983A48C97DB0A85A967445232DC2C0963AF309A81336F98AC27 failed to execute message; message index: 0: Error planning inference admission: inference already submitted
2026-08-08 04:27:39,283 INF Received signal 15, initiating graceful shutdown

Metadata

Metadata

Assignees

No one assigned

    Labels

    snippets-nightlyOpened by the nightly snippet run

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions