Skip to content

Difficulty plot helpers - original PR - #1967

Closed
ImplOfAnImpl wants to merge 4 commits into
masterfrom
difficulty_plot_helpers
Closed

ImplOfAnImpl wants to merge 4 commits into
masterfrom
difficulty_plot_helpers

Conversation

@ImplOfAnImpl

@ImplOfAnImpl ImplOfAnImpl commented Sep 10, 2025

Copy link
Copy Markdown
Contributor

Edit: this has been superseded by #1974 (the difficulty plot helpers themselves) and #1971 (minor improvements and cleanup).

The original description goes below.


Here I add a couple of python scripts to plot block targets and block time differences - collect_data.py collects the data from the API server and show_plots.py makes the plots using matplotlib.
(previously we would collect the data from node's logs and use Mathematica to make the plots, which was inconvenient).

The plots can be improved further I guess, but even in the current state they are good enough to observe the chain performance.

In order to be able to obtain block targets and timestamps from the API server "in bulk", I had to add a new endpoint "/chain" which returns an array of "block infos" containing block's id, height, timestamp and target.
For consistency, I also changed the "/chain/tip" endpoint, so that it also returns the timestamp and the target.

P.S. actually now I'm not sure if using the API server to collect the data was a good idea. Probably it'd be better if collect_data.py was accessing the local node's chainstatedb directly instead, this way we'd be able to add more interesting visualizations in the future (e.g. visualize the block tree stored inside the chainstatedb).
But I guess the changes in the API server are not completely useless on their own, so I'd keep it this way for now.

P.S.2 I also had to "cargo update" tracing-subscriber to address the vulnerability https://rustsec.org/advisories/RUSTSEC-2025-0055

P.S.3 I've modified set_mainchain_block in the postgres storage so that it always updates all columns on block id conflict (the reason being that it's a low-level function that shouldn't make assumptions about the data). Also I fixed some inconsistencies between the postgres and in-memory storage implementations.

Comment thread .dockerignore

# The cache for docker container dependency
.cargo
.cargo/*

ghost Sep 10, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I have no idea what "The cache for docker container dependency" means and suspect that this line is not actually needed. But for now I've just made it consistent with .gitignore

@ImplOfAnImpl
ImplOfAnImpl marked this pull request as ready for review September 10, 2025 15:06
@ImplOfAnImpl
ImplOfAnImpl marked this pull request as draft September 10, 2025 16:51
@ImplOfAnImpl
ImplOfAnImpl force-pushed the difficulty_plot_helpers branch from e83e0dd to 9cdb184 Compare September 11, 2025 09:28
@ImplOfAnImpl
ImplOfAnImpl marked this pull request as ready for review September 11, 2025 13:01
genesis_info = self.get("chain/genesis", {})

block_infos_by_height = {}
starting_height = 1

ghost Sep 11, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also be nice if we can update an existing file instead of always fetching the entire chain, i.e. we can check the output file if it exists and is valid, get the last height (as it is not stored, maybe number of lines I guess) and start from there, and append to it only the new blocks.

ghost Sep 22, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also be nice if we can update an existing file instead of always fetching the entire chain, i.e. we can check the output file if it exists and is valid, get the last height (as it is not stored, maybe number of lines I guess) and start from there, and append to it only the new blocks.

Well, even in this PR the whole chain retrieval was taking under 20s (at least when the API server was being run locally), so the improvement wouldn't be that useful IMO.

And with the new approach (using the chainstate dumper) it takes just a few seconds.

@ImplOfAnImpl ImplOfAnImpl changed the title Difficulty plot helpers Difficulty plot helpers - original PR Sep 22, 2025
@ImplOfAnImpl
ImplOfAnImpl deleted the difficulty_plot_helpers branch September 22, 2025 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants