Implements memory-based dynamic cache flushing#7
Closed
macols77 wants to merge 3 commits into
Closed
Conversation
Adds transaction size and weight information to the AddressTxOut data structure and populates it during index lookups. This allows clients to have direct access to the size and weight of a transaction associated with an address output, improving the granularity of data available to clients.
Replaces the fixed block commit interval with a dynamic memory-based flushing mechanism. The indexer now monitors memory usage and flushes the cache when it reaches a certain ratio of the total available memory. This helps prevent out-of-memory errors and improves stability. A new `MemoryLimiter` struct is introduced to manage memory snapshots and determine when to flush. The flush ratio is configurable via the `memory_flush_ratio` option. Removes the old `commit_interval` setting as it is no longer used.
Introduces a memory limiter to dynamically flush the cache when memory usage exceeds a high watermark. Adds configurable options for memory management: - `memory_flush_ratio`: The memory usage ratio at which the cache is flushed. - `memory_resume_ratio`: The memory usage ratio at which processing resumes after a flush. - `memory_refresh_ms`: The minimum interval between memory usage checks. Synchronously flushes the cache and waits for persistence to complete when the high watermark is reached, ensuring memory pressure is relieved. Processing is paused until memory usage drops below the low watermark to prevent thrashing. Avoids OOM errors during indexing by proactively managing memory usage.
|
Hey @macols77. Is it possible to pass to Titan, exactly (in GBs) how much memory it should use, instead of it figuring out itself (by checking on which server it runs and then calculating ratio)? The use case is, for instance, when the Titan runs on k8s or some other PaaS platform where it sees the whole node memory instead of how much it is allocated to it (i.e. when k8s kills the pod). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the fixed block commit interval with a dynamic memory-based flushing mechanism.
The indexer now monitors memory usage and flushes the cache when it reaches a certain ratio of the total available memory. This helps prevent out-of-memory errors and improves stability.
A new
MemoryLimiterstruct is introduced to manage memory snapshots and determine when to flush.The flush ratio is configurable via the
memory_flush_ratiooption.Removes the old
commit_intervalsetting as it is no longer used.