Resolve Txo spending post-traverse #43
Merged
Merged
Conversation
…id to block height.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Given a Txo, Bitcoin Core does not provide details about its spending (only an indirect method to determine whether it is spent or unspent). However, for many quant-related features, we need details such as the block height where a Txo is spent (see #41 for details).
There are multiple ways to track Txo spending. In earlier versions of EBA, we used PostgreSQL to track created and spent Txos, which were populated as part of the traverse process. However, using this database had a significant performance penalty at traverse time, so that feature was dropped. As part of #41 we tried relying on the graph imported in Neo4j, and used Neo4j to track Txo spending. The graph schema provides an efficient way to track the spending; however, for many quant-related features (e.g., NUPL) we need to update the graph with spending information, or else we will need to track all Txo spending for feature computation. And, the drawback of this method lies at the updating step, where updating tens of billions of edges takes a considerable amount of time.
This PR adds a method that runs post-traverse and pre-graph-import. Hence, it does not add a performance penalty to traverse, and does not need a mass database update post-import. This method creates a per-batch file that contains all the Txos spent in that batch by reading all the
*_edges_Script_Redeems_Tx.csv.gzfiles, then updates all the edges in the*_edges_Tx_Credits_Script.csv.gzfiles. Since the Txo spending is recorded per-batch, it can perform an update in memory, without a need for a database-based solution.