Augment graph with market data post traverse #65
Merged
Conversation
…ding performance.
…& from post-process.
VJalili
marked this pull request as ready for review
June 8, 2026 00:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces the post-import graph augmentation process.
Previously, we were reading nodes and edges directly from Neo4j, computing economic indicators, and updating the block nodes back in the database. Iterating through all nodes and edges this way is an extremely slow process, taking ~30 minutes just to process 2 million out of roughly 10 billion edges. There are various reasons for this performance issue, a primary one being the massive database I/O overhead of iterating over edges.
To solve this, we are moving the calculation to a post-traverse/pre-import step. The augmentation now runs on the generated TSV files before the data is imported into the graph database.
The algorithm implemented in this PR runs in a few hours, significantly faster than the previous implementation that took ~30min to process 0.02% of data.
Details
The challenge with many financial indicators is that they are computed based on the UTxO lifecycle (e.g., its fiat value at creation, its fiat value when spent, how many UTxOs are unspent at block$b$ , and if sold now, the unrealized profit [NUP] or loss [NUL]). Computing these metrics for a single new block is computationally cheap on the graph. The real challenge is computing these metrics for all historic blocks. You must iterate over all edges, track when they are created and spent, and at each block, know exactly which UTxOs are unspent to compute their fiat values.
Keeping track of every active UTxO requires significant memory, making it infeasible at scale. You would need to keep track of their intervals and evaluate them batch by batch. For each block, you must iterate over all currently active UTxOs—which easily reaches tens of millions per block—resulting in an$O(n \times m)$ iteration bottleneck.
The algorithm implemented in this PR is a two-step process:
1. Parallel Aggregation:
Instead of tracking millions of individual UTxO IDs, we aggregate the data by the block height where the coins were minted. We build a ledger of how many Satoshis were created and spent at each block height. In other words, we only track the sum of satoshis created and the sum of satoshis spent at any given block.
2. Single-Pass Sweep:
We iterate through the blocks; as we sweep, we maintain a running tally of active, unspent satoshis categorized solely by the block they were originally minted in (e.g., tracking that "we currently have 500 active satoshis that were created in Block 10"). Because all satoshis created in Block 10 share the exact same fiat cost basis, we can calculate the Realized Cap, Unrealized Profit, and Unrealized Loss for that entire group at once against pre-computed fiat prices.