Adding stratum-work first-seen timestamps

[bboerst]

Here are some timestamp-related data exports from my https://stratum.work/ dataset (see #30).

These exports have timestamps for when the first mining.notify message arrived at my collector for each height. The collector is physically located in us-east (Ashburn, VA), so expect a small bit of geo-latency in these timestamps between the stratum server -> stratum.work collector.

• data/stratum_work_empty.csv - first empty job seen
• data/stratum_work_not_empty.csv - first non-empty job seen

The format is:

height,hash,timestamp

closes #30

[bboerst]

Seems to slot in nicely with the existing data:

[bboerst]

Thinking about this more... I should be incorporating prev_hash from the mining.notify. This would give me timings for blocks that become stale too. I'll fix this later today.

[0xB10C]

Thanks for adding your timestamps! I think this is a valuable contribution because pools are usually the first ones to learn about a new block.

However, I prefer to have all csv files in the same format. So ideally: height,hash,timestamp

I saw that the stratum-work timestamps are in nanoseconds and that you've changed them to ms.

Instead of hash, you have the pool_name in your CSV files. I think the data should be uniform, so probably best to just use the timestamp of the first height-hash combination and ignore the pool here. An alternative could be to have a per-pool CSV file, but that might become a lot.

I see the problem with the height of the jobs needing to be height - 1, and I think this should already be correct in the commited CSV file. Researchers / users of this dataset might not be aware of this, and run into problems.

[0xB10C]

I've just merged #32. If you rebase your branch, you don't need to update the mermaid gantt chart anymore (since I removed it from the README, we have this on the website now). Additionally, I've updated the block header timestamps file. Depending on the height at which your timestamps end, we might need to update it once more. Let me known if you need help!

[bboerst]

Thanks for the guidance. This is ready for a re-review:

• Fixed the dataset to reflect height-1
• Removed pool_name column. Added hash column
• Adjusted the export to grab the first timestamp per height and prev_hash. Now it's grabbing eventual stales as well, which is great. Spot-checked 939832 and it looks like that is working as expected.

[0xB10C]

I'll do the following rebase to squash the commits:

from:

pick 3118382 # Adding stratum-work first-seen timestamps
pick ae79b9c # Fixing some timezome of some timestamps and switching to ms instead of ns
pick 3690865 # Fixing stats gen to account for stratum_work templates
pick 8545243 # Fixing variable name to be more descriptive.
pick b645afd # Adding to contrib readme
pick b1e9eb6 # Revert generate_stats change
pick 8610694 # Refactored datasets
pick 578c9b0 # Truncing dataset at 942199 to match block-timestamps

to:

pick 3118382 # Adding stratum-work first-seen timestamps
squash ae79b9c # Fixing some timezome of some timestamps and switching to ms instead of ns
squash 8610694 # Refactored datasets
drop 3690865 # Fixing stats gen to account for stratum_work templates
drop 8545243 # Fixing variable name to be more descriptive.
pick b645afd # Adding to contrib readme
drop b1e9eb6 # Revert generate_stats change
drop 578c9b0 # Truncing dataset at 942199 to match block-timestamps

And I'll also add the missing block header timestamps to make sure we have the full data.

[0xB10C]

There seems to be an issue with a handfull of timestamps. Maybe you've restarted stratum-work or similar. Otherwise, this looks good!

$ python3 contrib/combine.py
reading CSV file data/stratum_work_not_empty.csv
reading CSV file data/0xb10c_peer-observer-erin.csv
reading CSV file data/0xb10c_memo-old.csv
reading CSV file data/0xb10c_peer-observer-charlie.csv
reading CSV file data/n-thumann.csv
reading CSV file data/0xb10c_peer-observer-frank.csv
reading CSV file data/0xb10c_rs2.csv
reading CSV file data/0xb10c_peer-observer-alice.csv
reading CSV file data/0xb10c_peer-observer-bob.csv
reading CSV file data/0xb10c_monitoring1.csv
reading CSV file data/0xb10c_memo.csv
reading CSV file data/darosior_node0.csv
reading CSV file data/stratum_work_empty.csv
reading CSV file data/KIT_monitorB.csv
reading CSV file data/vostrnad_node1.csv
reading CSV file data/offing-gcp.csv
reading CSV file data/KIT_monitor1.csv
reading CSV file data/KIT_monitor2.csv
reading CSV file data/0xb10c_peer-observer-dave.csv
writing combined CSV file timestamps.csv

$ python3 qa/block-timestamps/check-block-timestamps.py
Checking that timestamps don't deviate too much from block header timestamps
Imported 942358 timestamps from qa/block-timestamps/block-timestamps.csv
Checking timestamps.csv...
ERROR: Timestamp for block 880226 (000000000000000000007a710abe4302be29d9c188d1e700251cc04125b63d59) from source stratum_work_empty deviates more that 7200.0s into the future: 25210.631s
ERROR: Timestamp for block 880225 (0000000000000000000207fa3335ef2bd21c46d6c8797b427a4d158a25b9a18a) from source stratum_work_not_empty deviates more that 7200.0s into the future: 25208.232s
ERROR: Timestamp for block 880225 (0000000000000000000207fa3335ef2bd21c46d6c8797b427a4d158a25b9a18a) from source stratum_work_empty deviates more that 7200.0s into the future: 25207.781s
ERROR: Timestamp for block 876890 (000000000000000000027d1906a426f753ebaec5ae83565d5463f7b97480e9ae) from source stratum_work_not_empty deviates more that 7200.0s into the future: 9282.057s
ERROR: Timestamp for block 876890 (000000000000000000027d1906a426f753ebaec5ae83565d5463f7b97480e9ae) from source stratum_work_empty deviates more that 7200.0s into the future: 9282.022s
ERROR: Timestamp for block 876889 (0000000000000000000225876ebbb8e260d061d50011899fc2418622cef02178) from source stratum_work_not_empty deviates more that 7200.0s into the future: 10198.654s
ERROR: Timestamp for block 876889 (0000000000000000000225876ebbb8e260d061d50011899fc2418622cef02178) from source stratum_work_empty deviates more that 7200.0s into the future: 10198.605s
ERROR: Timestamp for block 876888 (000000000000000000010fe123e3027a3624eb762449dd9a864a7710c29d4cba) from source stratum_work_not_empty deviates more that 7200.0s into the future: 10283.537s

Statistics about offsets from block timestamps:
source                    count      min        max          mean         stdev        quantiles[s]
stratum_work_not_empty    95472      -3594s     25208s       13.55s       109.36s      [6.987, 14.063, 23.368]
stratum_work_empty        95426      -3595s     25211s       13.23s       132.09s      [6.615, 13.698, 22.942249999999998]
0xb10c_peer-observer-erin 109944     -2746s     2111s        14.31s       41.71s       [7.611, 15.01, 24.655]
0xb10c_peer-observer-charlie 110674     -2746s     2661s        14.40s       41.24s       [7.768750000000001, 15.1325, 24.856500000000004]
0xb10c_peer-observer-bob  110228     -2746s     2115s        14.20s       40.70s       [7.558, 14.99, 24.587]
0xb10c_peer-observer-dave 108206     -2746s     1831s        14.22s       40.56s       [7.60375, 15.0, 24.672250000000002]
0xb10c_peer-observer-alice 109233     -2746s     1804s        14.23s       41.77s       [7.5515, 14.997, 24.615]
0xb10c_peer-observer-frank 110819     -2744s     2546s        49.77s       137.20s      [10.32, 18.542, 30.332]
0xb10c_rs2                120391     -3440s     2739s        22.20s       52.07s       [11.0, 20.0, 29.0]
vostrnad_node1            40107      -489s      2767s        20.99s       39.50s       [12.0, 21.0, 30.0]
n-thumann                 55358      -3439s     2399s        53.02s       61.88s       [31.0, 45.0, 62.0]
darosior_node0            42628      -493s      2091s        20.50s       31.52s       [11.0, 20.0, 29.0]
0xb10c_monitoring1        149103     -3440s     1672s        21.37s       28.49s       [10.645, 19.875, 29.127]
KIT_monitorB              238464     -3440s     2898s        20.96s       29.48s       [9.835, 19.269, 29.133]
KIT_monitor2              433620     -4418s     3918s        20.09s       44.29s       [9.663, 19.5045, 30.287]
KIT_monitor1              430091     -4418s     5571s        20.12s       45.92s       [9.639, 19.472, 30.243]
0xb10c_memo               51491      -1708s     1442s        27.06s       45.43s       [11.0, 20.0, 30.0]
offing-gcp                10927      -268s      635s         21.47s       31.92s       [10.0, 19.0, 28.0]
0xb10c_memo-old           72693      -2130s     682s         20.90s       30.47s       [9.0, 19.0, 30.0]

Check not successful.
The listed timestamps might have been recorded during IBD or the system clock was wrong?

Should we drop them? Do you have a guess why they are wrong?

[bboerst]

This check-block-timestamps sanity test has been super useful to me. I'm going to add something similar on my end to catch outliers. It helped me to correct some data issues, so thanks for that.

The export is looking better now.

• I had a power outage between heights 876882 -> 876902 and my data collection was intermittent during this range. I'm now excluding this range from this export.
• 880225 -> 880226 was apparently around the time when I decided to do some timezone work. I fixed 880225 and going to exclude 880226 for now while I fix my underlying data. I'll re-include it in a future data update.

check-block-timestamps.py looks like it's passing now:

python3 qa/block-timestamps/check-block-timestamps.py
Checking that timestamps don't deviate too much from block header timestamps
Imported 942358 timestamps from qa/block-timestamps/block-timestamps.csv
Checking timestamps.csv...
No offset problems found.

Statistics about offsets from block timestamps:
source                    count      min        max          mean         stdev        quantiles[s]
stratum_work_not_empty    95522      -3594s     1777s        12.85s       39.64s       [6.98475, 14.061, 23.364]
stratum_work_empty        95479      -3595s     1238s        12.38s       38.87s       [6.613, 13.696, 22.938]
0xb10c_peer-observer-erin 109944     -2746s     2111s        14.31s       41.71s       [7.611, 15.01, 24.655]
0xb10c_peer-observer-charlie 110674     -2746s     2661s        14.40s       41.24s       [7.768750000000001, 15.1325, 24.856500000000004]
0xb10c_peer-observer-bob  110228     -2746s     2115s        14.20s       40.70s       [7.558, 14.99, 24.587]
0xb10c_peer-observer-dave 108206     -2746s     1831s        14.22s       40.56s       [7.60375, 15.0, 24.672250000000002]
0xb10c_peer-observer-alice 109233     -2746s     1804s        14.23s       41.77s       [7.5515, 14.997, 24.615]
0xb10c_peer-observer-frank 110819     -2744s     2546s        49.77s       137.20s      [10.32, 18.542, 30.332]
0xb10c_rs2                120391     -3440s     2739s        22.20s       52.07s       [11.0, 20.0, 29.0]
vostrnad_node1            40107      -489s      2767s        20.99s       39.50s       [12.0, 21.0, 30.0]
n-thumann                 55358      -3439s     2399s        53.02s       61.88s       [31.0, 45.0, 62.0]
darosior_node0            42628      -493s      2091s        20.50s       31.52s       [11.0, 20.0, 29.0]
0xb10c_monitoring1        149103     -3440s     1672s        21.37s       28.49s       [10.645, 19.875, 29.127]
KIT_monitorB              238464     -3440s     2898s        20.96s       29.48s       [9.835, 19.269, 29.133]
KIT_monitor2              433620     -4418s     3918s        20.09s       44.29s       [9.663, 19.5045, 30.287]
KIT_monitor1              430091     -4418s     5571s        20.12s       45.92s       [9.639, 19.472, 30.243]
0xb10c_memo               51491      -1708s     1442s        27.06s       45.43s       [11.0, 20.0, 30.0]
offing-gcp                10927      -268s      635s         21.47s       31.92s       [10.0, 19.0, 28.0]
0xb10c_memo-old           72693      -2130s     682s         20.90s       30.47s       [9.0, 19.0, 30.0]

Check successful.

[0xB10C]

This check-block-timestamps sanity test has been super useful to me. I'm going to add something similar on my end to catch outliers. It helped me to correct some data issues, so thanks for that.

Awesome!

Looks good, thank you for the contribution!

I'll re-include it in a future data update.

If you used any special scripts or steps you want to document for a potential next time, feel free to just leave a comment here on what you did this time. I usually find this very useful and thank my past self.