Skip to content

Augment graph with market data post traverse #65

Merged
VJalili merged 15 commits into
B1AAB:mainfrom
VJalili:extend-post-traverse
Jun 8, 2026
Merged

Augment graph with market data post traverse #65
VJalili merged 15 commits into
B1AAB:mainfrom
VJalili:extend-post-traverse

Conversation

@VJalili

@VJalili VJalili commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

This PR replaces the post-import graph augmentation process.

Previously, we were reading nodes and edges directly from Neo4j, computing economic indicators, and updating the block nodes back in the database. Iterating through all nodes and edges this way is an extremely slow process, taking ~30 minutes just to process 2 million out of roughly 10 billion edges. There are various reasons for this performance issue, a primary one being the massive database I/O overhead of iterating over edges.

To solve this, we are moving the calculation to a post-traverse/pre-import step. The augmentation now runs on the generated TSV files before the data is imported into the graph database.

The algorithm implemented in this PR runs in a few hours, significantly faster than the previous implementation that took ~30min to process 0.02% of data.

Details

The challenge with many financial indicators is that they are computed based on the UTxO lifecycle (e.g., its fiat value at creation, its fiat value when spent, how many UTxOs are unspent at block $b$, and if sold now, the unrealized profit [NUP] or loss [NUL]). Computing these metrics for a single new block is computationally cheap on the graph. The real challenge is computing these metrics for all historic blocks. You must iterate over all edges, track when they are created and spent, and at each block, know exactly which UTxOs are unspent to compute their fiat values.

Block Timeline:         [ Block 1 ] ------- [ Block 2 ] ------- [ Block 3 ] ------- [ Block 4 ]
                             |                   |                   |                   |
UTxO A (10 Sats):        [Created] ----------------------------- [Spent]                 |
                             |                   |                   |                   |
UTxO B (50 Sats):            |               [Created] ----------------------------- [Spent]
                             |                   |                   |                   |
Total Active Unspent:     10 Sats             60 Sats             50 Sats             0 Sats

Keeping track of every active UTxO requires significant memory, making it infeasible at scale. You would need to keep track of their intervals and evaluate them batch by batch. For each block, you must iterate over all currently active UTxOs—which easily reaches tens of millions per block—resulting in an $O(n \times m)$ iteration bottleneck.

The algorithm implemented in this PR is a two-step process:

1. Parallel Aggregation:
Instead of tracking millions of individual UTxO IDs, we aggregate the data by the block height where the coins were minted. We build a ledger of how many Satoshis were created and spent at each block height. In other words, we only track the sum of satoshis created and the sum of satoshis spent at any given block.

2. Single-Pass Sweep:
We iterate through the blocks; as we sweep, we maintain a running tally of active, unspent satoshis categorized solely by the block they were originally minted in (e.g., tracking that "we currently have 500 active satoshis that were created in Block 10"). Because all satoshis created in Block 10 share the exact same fiat cost basis, we can calculate the Realized Cap, Unrealized Profit, and Unrealized Loss for that entire group at once against pre-computed fiat prices.

@VJalili
VJalili marked this pull request as ready for review June 8, 2026 00:06
@VJalili
VJalili merged commit c3479f9 into B1AAB:main Jun 8, 2026
5 checks passed
@VJalili
VJalili deleted the extend-post-traverse branch June 8, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant