Skip to content

Upgrade pyquarkchain from Python 3.8 to Python 3.13#969

Draft
ping-ke wants to merge 16 commits intomasterfrom
upgrade-py3-13
Draft

Upgrade pyquarkchain from Python 3.8 to Python 3.13#969
ping-ke wants to merge 16 commits intomasterfrom
upgrade-py3-13

Conversation

@ping-ke
Copy link
Contributor

@ping-ke ping-ke commented Mar 10, 2026

Summary

  • Upgrade all dependencies, asyncio patterns, and removed stdlib APIs to support Python 3.13
  • Replace third-party jsonrpcserver/jsonrpcclient with lightweight local implementations
  • All 348 tests pass cleanly

Dependencies (requirements.txt)

  • jsonrpcserver / jsonrpcclient → removed (replaced with local implementations)
  • aiohttp 3.1 → ≥3.9
  • cryptography 2.3 → ≥42
  • numpy 1.15 → ≥1.26
  • websockets 8.0 → ≥12
  • pytest 3.6 → ≥8, pytest-asyncio 0.9 → ≥0.23
  • coincurve, ecdsa, eth-utils, eth-keys, eth-bloom, py_ecc, rlp, psutil, decorator, cachetools — all bumped to current versions
  • upnpclient → async-upnp-client
  • Added httpx (for the new JSON-RPC client)
  • Removed pyethash and async_armor

asyncio API migration

  • asyncio.get_event_loop() → asyncio.get_running_loop() or _get_or_create_event_loop() helper (depending on sync/async context) across cluster, p2p, and protocol modules
  • asyncio.ensure_future() / loop.create_task() → asyncio.create_task() in synchronous methods where no running event loop is guaranteed (root_state.py, shard.py, shard_state.py, master.py)
  • Removed deprecated loop= parameter from asyncio.Future, asyncio.Queue, asyncio.Event, asyncio.ensure_future, asyncio.start_server, and all connection/cancel-token classes in p2p/
  • Updated asyncio.CancelledError handling (now subclass of BaseException since Python 3.9)

JSON-RPC library replacement

  • Added quarkchain/cluster/jsonrpcserver.py — minimal JSON-RPC 2.0 server with method registration, dispatch, context injection, batch requests, and notification support
  • Added quarkchain/jsonrpc_client.py — lightweight async JSON-RPC client using httpx
  • Updated quarkchain/cluster/jsonrpc.py HTTP and WebSocket handlers to use the new dict-based dispatch API

Python 3.12+ removed APIs

  • unittest.TestCase.assertRaisesRegexp → assertRaisesRegex (17 occurrences in test_shard_state.py and test_root_state.py)
  • unittest.TestCase.assertDictContainsSubset → assertTrue(expected.items() <= actual.items())

p2p module

  • Removed all loop= parameters from CancelToken, connections, and server classes
  • asyncio.get_event_loop() → asyncio.get_running_loop() in services and utilities
  • Removed stale loop parameters from NullConnection, VirtualConnection, P2PConnection, ClusterConnection

Test stability

  • Added quarkchain/cluster/tests/conftest.py with autouse fixture that cancels all pending asyncio tasks after each test, preventing inter-test contamination from background tasks
  • Added task cleanup in shutdown_clusters() test utility
  • simple_network.py: start_server() and shutdown() made properly async

Code formatting

  • Applied consistent formatting across the codebase

Test plan

  • python -m pytest — 348 passed, 0 failures (217s)

@ping-ke ping-ke marked this pull request as draft March 10, 2026 09:46
ping-ke and others added 12 commits March 10, 2026 23:26
- adjust_difficulty.py: replace bare `import monitoring` with
  `from quarkchain.tools import monitoring` (breaks when run from
  outside the tools directory)
- adjust_difficulty.py: replace jsonrpc_async.Server with
  AsyncJsonRpcClient and use .call() method
- monitoring.py: replace jsonrpc_async.Server with AsyncJsonRpcClient
  and use .call() method; close session via .close()
- jsonrpc_client.py: fix AsyncJsonRpcClient.call signature to use
  *params (variadic) to match JsonRpcClient.call, fixing callers
  that pass positional arguments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix send_request in test_jsonrpc.py to unpack list params correctly
  instead of double-wrapping them (e.g. [["0x..."]] → ["0x..."])
- Add call_with_dict_params to AsyncJsonRpcClient for named params
- Implement JSONRPCWebsocketServer.shutdown() to actually close the
  server, fixing test isolation hangs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track and cancel all fire-and-forget asyncio tasks that were leaking
across tests, causing resource exhaustion and timeout failures.

- AbstractConnection: add _loop_task and _handler_tasks tracking;
  use try/finally in active_and_loop_forever to ensure cleanup on
  cancellation; cancel _loop_task in close()
- master.py: track SlaveConnection loop task and __init_cluster task;
  cancel _init_task on shutdown
- slave.py: track MasterConnection, SlaveConnection, PeerShardConnection
  loop tasks and __start_server task
- shard.py: track PeerShardConnection loop task
- miner.py: track and cancel mining task in disable()
- simple_network.py: track Peer loop task and connect_seed task
- test_utils.py: restructure shutdown_clusters with try/finally to
  guarantee task cleanup; await server.wait_closed() for slave servers
- conftest.py: multi-round task cancellation; reset aborted_rpc_count
- Fix AttributeError: replace undefined self.external_port/self.internal_port/self.protocol with self.port
- Fix _discover timeout: wrap async_search with asyncio.wait_for instead of waiting after it completes
- Fix _run: guard _add_port_mapping with self._service check to avoid AttributeError
- Fix _delete_port_mapping: delete both TCP and UDP mappings to match AddPortMapping
- Remove dead code: _refresh_task and _running fields that were never used
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant