genesis proof - panic: json: cannot unmarshal number into Go value

[ThomasBlock]

i try to launch testnet. last tries did stop at 46329 . now i started from scratch

POKTROLLD_IMAGE_TAG_OFF_CHAIN=0.0.10
POKTROLLD_IMAGE_TAG_VALIDATOR=0.0.10

now it fails instantly:

full-node            | 3:11PM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
full-node            | 3:11PM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
full-node            | 3:11PM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
full-node            | 3:11PM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
full-node            | 3:11PM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
full-node            | 3:11PM INF service start impl=EventBus module=events msg="Starting EventBus service"
full-node            | 3:11PM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
full-node            | 3:11PM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
full-node            | 3:11PM INF ABCI Handshake App Info hash=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 height=0 module=consensus protocol-version=0 software-version=0.0.10
full-node            | 3:11PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
full-node            | 3:11PM INF InitChain chainID=poktroll initialHeight=1 module=server
full-node            | 3:11PM INF initializing blockchain state from genesis.json module=server
full-node            | 3:11PM INF asserting crisis invariants inv=1/13 module=x/crisis name=group/Group-TotalWeight
full-node            | 3:11PM INF asserting crisis invariants inv=2/13 module=x/crisis name=staking/module-accounts
full-node            | 3:11PM INF asserting crisis invariants inv=3/13 module=x/crisis name=staking/nonnegative-power
full-node            | 3:11PM INF asserting crisis invariants inv=4/13 module=x/crisis name=staking/positive-delegation
full-node            | 3:11PM INF asserting crisis invariants inv=5/13 module=x/crisis name=staking/delegator-shares
full-node            | 3:11PM INF asserting crisis invariants inv=6/13 module=x/crisis name=distribution/nonnegative-outstanding
full-node            | 3:11PM INF asserting crisis invariants inv=7/13 module=x/crisis name=distribution/can-withdraw
full-node            | 3:11PM INF asserting crisis invariants inv=8/13 module=x/crisis name=distribution/reference-count
full-node            | 3:11PM INF asserting crisis invariants inv=9/13 module=x/crisis name=distribution/module-account
full-node            | 3:11PM INF asserting crisis invariants inv=10/13 module=x/crisis name=transfer/total-escrow-per-denom
full-node            | 3:11PM INF asserting crisis invariants inv=11/13 module=x/crisis name=gov/module-account
full-node            | 3:11PM INF asserting crisis invariants inv=12/13 module=x/crisis name=bank/nonnegative-outstanding
full-node            | 3:11PM INF asserting crisis invariants inv=13/13 module=x/crisis name=bank/total-supply
full-node            | 3:11PM INF asserted all invariants duration=0.690781 height=0 module=x/crisis
full-node            | 3:11PM INF Closing application.db module=server
full-node            | 3:11PM INF Closing snapshots/metadata.db module=server
full-node            | panic: json: cannot unmarshal number into Go value of type map[string]json.RawMessage
full-node            | 
full-node            | goroutine 1 [running]:
full-node            | github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshalJSON(0xc001a09e40?, {0xc00032eb40?, 0xc0017fb680?, 0xc001efcd58?}, {0x4416c20?, 0xc00137b3b0?})
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/codec/proto_codec.go:219 +0x3b
full-node            | github.com/pokt-network/poktroll/x/proof/module.AppModule.InitGenesis({{{0x4451aa0, 0xc000207450}}, {{0x4451aa0, 0xc000207450}, {0x43f1600, 0xc0010ad540}, {0x443c018, 0xc0003e3b90}, {0xc001425680, 0x2b}, ...}, ...}, ...)
full-node            | 	/home/runner/work/poktroll/poktroll/x/proof/module/module.go:128 +0x94
full-node            | github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x4432260, 0x64cf660}, {0x4452450, 0xc003222f40}, {{0x0, 0x0}, {0xc0021e8258, 0x8}, 0x0, ...}, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/types/module/module.go:505 +0x694
full-node            | github.com/cosmos/cosmos-sdk/runtime.(*App).InitChainer(_, {{0x4432260, 0x64cf660}, {0x4452450, 0xc003222f40}, {{0x0, 0x0}, {0xc0021e8258, 0x8}, 0x0, ...}, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/runtime/app.go:189 +0xdf
full-node            | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc0017f6d88, 0xc001fb1490)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/baseapp/abci.go:105 +0x6a2
full-node            | github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.InitChain(...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/cmt_abci.go:32
full-node            | github.com/cometbft/cometbft/abci/client.(*localClient).InitChain(0xc000a9a700?, {0x4432570?, 0x64cf660?}, 0x7f43690bc108?)
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/abci/client/local_client.go:120 +0xc7
full-node            | github.com/cometbft/cometbft/proxy.(*appConnConsensus).InitChain(0xc001a9bd10, {0x4432570, 0x64cf660}, 0xc001fb1490)
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/proxy/app_conn.go:78 +0x15e
full-node            | github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0x36b5d0e, 0x7}}, {0xc0021e8258, 0x8}, 0x1, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/consensus/replay.go:336 +0xfb3
full-node            | github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc001effa28, {0x44324c8, 0xc001a94780}, {0x4457e78, 0xc0012b61c0})
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/consensus/replay.go:274 +0x425
full-node            | github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0x36b5d0e, 0x7}}, {0xc0021e8258, 0x8}, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/node/setup.go:186 +0x1a9
full-node            | github.com/cometbft/cometbft/node.NewNodeWithContext({0x44324c8, 0xc001a94780}, 0xc00139a500, {0x440ce10, 0xc00019cbe0}, 0xc001f6b780, {0x43f12a0, 0xc001fa0588}, 0xc0008ae5d0, 0x3f896c0, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.10/node/node.go:360 +0x630
full-node            | github.com/cosmos/cosmos-sdk/server.startCmtNode({0x44324c8, 0xc001a94780}, 0xc00139a500, {0x4484390, 0xc000abb308}, 0xc000794b00)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:377 +0x41d
full-node            | github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{{0xc00109a5f0, 0x6}, 0x0, {0xc00109ab10, 0x7}, {0x3718852, 0x1}, {0x3718852, 0x1}, ...}, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:323 +0x17c
full-node            | github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x445c338, 0xc00179baa0}, 0x0, {0x0, 0x0}, {0x44842c8, ...}, ...}, ...)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:240 +0x2c7
full-node            | github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:198 +0x63
full-node            | github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc000794b00, 0xc0008af9f8)
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:570 +0x1b3
full-node            | github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0xc0015bd508, {0xc00179b470?, 0x0?, 0x3?})
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/start.go:197 +0x212
full-node            | github.com/spf13/cobra.(*Command).execute(0xc0015bd508, {0xc00179b3e0, 0x3, 0x3})
full-node            | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
full-node            | github.com/spf13/cobra.(*Command).ExecuteC(0xc0012e5808)
full-node            | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
full-node            | github.com/spf13/cobra.(*Command).Execute(...)
full-node            | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
full-node            | github.com/spf13/cobra.(*Command).ExecuteContext(...)
full-node            | 	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
full-node            | github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc0012e5808, {0x0, 0x0}, {0xc001290c18, 0x16})
full-node            | 	/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.9/server/cmd/execute.go:34 +0x187
full-node            | main.main()
full-node            | 	/home/runner/work/poktroll/poktroll/cmd/poktrolld/main.go:15 +0x33
full-node            | Error: exit status 2

[okdas]

Hey @ThomasBlock - thanks for the issue.
POKTROLLD_IMAGE_TAG_VALIDATOR should not have been changed to 0.0.10 and should be 0.0.9 instead.

Now that we have two separate TestNets, I'm working on a change to support both networks in #28 - so that parameter will go away soon.

Let me change that value in the main branch in a meanwhile.

[okdas]

@ThomasBlock

The path to fix your current deployment:

1. Change POKTROLLD_IMAGE_TAG_VALIDATOR in .env to 0.0.9
2. Stop the full node.
3. Clean up old state:

     # Remove existing data
     rm -rf poktrolld-data/config/addrbook.json poktrolld-data/config/genesis.json poktrolld-data/config/genesis.seeds poktrolld-data/data/ poktrolld-data/cosmovisor/ poktrolld-data/config/node_key.json poktrolld-data/config/priv_validator_key.json

4. Start the full node back

[ThomasBlock]

@okdas Thank you. so i stopped and deleted everything. did git clone . chose this config

NETWORK_NAME=testnet-validated
POKTROLLD_IMAGE_REPO=ghcr.io/pokt-network/poktrolld
POKTROLLD_IMAGE_TAG_OFF_CHAIN=0.0.10
POKTROLLD_IMAGE_TAG_VALIDATOR=0.0.9

but again crashing:

full-node            | 9:21PM INF Stored block hash at height 46330 EndBlock=SessionModuleEndBlock module=x/session
full-node            | 9:21PM INF found 2 expiring claims at block height 46330 method=SettlePendingClaims module=x/tokenomics
full-node            | 9:21PM INF claim does not require proof due to compute units (5) being less than the threshold (20) and random sample (0.84) being greater than probability (0.25) method=proofRequirementForClaim module=server
full-node            | 9:21PM WRN Claim by supplier pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7 EXCEEDS LIMITS for application pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87. Max claimable amount < Claim amount: 0upokt < 210 application=pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87 claim_settlement_upokt=210 helper=ensureClaimAmountLimits method=ProcessTokenLogicModules module=x/tokenomics num_relays=5 service_id=0021 session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM INF About to start processing TLMs for (5) relays, equal to (0upokt) claimed actual_settlement_upokt=0upokt application=pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87 claim_settlement_upokt=210 method=ProcessTokenLogicModules module=x/tokenomics num_relays=5 service_id=0021 session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM INF Starting TLM processing: "TLMRelayBurnEqualsMint" actual_settlement_upokt=0upokt application=pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87 claim_settlement_upokt=210 method=ProcessTokenLogicModules module=x/tokenomics num_relays=5 service_id=0021 session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM INF updated application "pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87" stake to 14upokt method=TokenLogicModuleRelayBurnEqualsMint module=x/tokenomics
full-node            | 9:21PM INF burned (0upokt) from the application module account method=TokenLogicModuleRelayBurnEqualsMint module=x/tokenomics
full-node            | 9:21PM INF minted (0upokt) coins in the supplier module method=TokenLogicModuleRelayBurnEqualsMint module=x/tokenomics
full-node            | 9:21PM INF sent 0upokt from the supplier module to the supplier shareholder with address "pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7" method=distributeSupplierRewardsToShareHolders module=x/tokenomics
full-node            | 9:21PM INF distributed 0 uPOKT to supplier "pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7" shareholders method=distributeSupplierRewardsToShareHolders module=x/tokenomics
full-node            | 9:21PM INF sent (0upokt) from the supplier module to the supplier account with address "pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7" method=TokenLogicModuleRelayBurnEqualsMint module=x/tokenomics
full-node            | 9:21PM INF Finished TLM processing: "TLMRelayBurnEqualsMint" actual_settlement_upokt=0upokt application=pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87 claim_settlement_upokt=210 method=ProcessTokenLogicModules module=x/tokenomics num_relays=5 service_id=0021 session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM INF Starting TLM processing: "TLMGlobalMint" actual_settlement_upokt=0upokt application=pokt1gsufws4c5dhemd0z3njfh859xl0624xyyw5m87 claim_settlement_upokt=210 method=ProcessTokenLogicModules module=x/tokenomics num_relays=5 service_id=0021 session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM ERR error processing token logic modules for claim "084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775": TLM "TLMGlobalMint": mint amount cannot be zero: failed to process TLM [/home/runner/go/pkg/mod/cosmossdk.io/errors@v1.0.1/errors.go:155] module=server num_claim_compute_units=5 num_relays_in_session_tree=5 proof_requirement=NOT_REQUIRED session_id=084deab39f3d7ff81426853ea186561e8ac1746987cf55edea22a98eff35e775 supplier_operator_address=pokt1jvsuszgvsjyp4lxnfxqmrf7lqh4jutplcw8wx7
full-node            | 9:21PM ERR could not settle pending claims due to error TLM "TLMGlobalMint": mint amount cannot be zero: failed to process TLM [/home/runner/go/pkg/mod/cosmossdk.io/errors@v1.0.1/errors.go:155] method=EndBlocker module=x/tokenomics
full-node            | panic: runtime error: invalid memory address or nil pointer dereference
full-node            | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15cdfa9]

restarting i still see 9-3

full-node | 9:24PM INF running app args=["start","--p2p.external-address=:26656","--log_level=info","--p2p.seeds=838625e3cfc7a2e347a1afc769957157441f649b@35.184.192.176:26656"] module=cosmovisor path=/home/pocket/.poktroll/cosmovisor/upgrades/v0.0.9-3/bin/poktrolld

[Olshansk]

@ThomasBlock Appreciate your patience here!

Things are in flux (code, releases, docs, etc...). What I would suggest is waiting until next week and string to sync with our beta testnet then.

The docs at [1] should be updated by then.

[1] https://dev.poktroll.com/

[okdas]

@Olshansk this issue is on Alpha TestNet.

Now that the new Cosmovisor is released we can simplify the process. I'll investigate on how to automate previously manual commands next week.