Runs out of memory during syncing and crashes #2

Activity

1. tsjk commented on Feb 4, 2025

   

   tsjk

   on Feb 4, 2025

   More actions

   Did you manage to work around this issue?
2. HalFinneyIsMyHomeBoy commented on Feb 4, 2025

   

   HalFinneyIsMyHomeBoy

   on Feb 4, 2025

   Author

   More actions

   No, never got around this issue. Gave up on it a few months back.
   Has there been any fix since then?
3. tsjk commented on Feb 4, 2025

   

   tsjk

   on Feb 4, 2025

   More actions

   No, never got around this issue. Gave up on it a few months back. Has there been any fix since then?

   No, there hasn't. But I experienced the same issue. I think the problem is that computing the scalars goes much faster than updating the unspents, and so
   https://github.com/louisinger/silentiumd/blob/master/internal/application/syncer_service.go#L217-L219
   becomes a very long memory consuming queue. I don't know any go, really - just looked at the code quickly. The code base is small.
   One can change this to become a blocking call by just

   --- a/internal/application/syncer_service.go
   +++ b/internal/application/syncer_service.go
   @@ -214,11 +214,8 @@ func (s *syncer) computeBlockScalars(block *btcutil.Block) {
                   logrus.Error(err)
           }
    
   -       go func() {
   -               s.updateUnspentsCh <- block
   -       }()
   -
           logrus.Debugf("[%d] compute scalars done", block.Height())
   +       s.updateUnspentsCh <- block
    }
    
    func (s *syncer) updateUnspents(block *btcutil.Block) {
   

   and this makes it not eat all your memory. But syncing is very slow.
   I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.
4. tsjk commented on Feb 4, 2025

   

   tsjk

   on Feb 4, 2025

   More actions

   Perhaps there is a middle-ground somewhere, like limiting the queue size, but as I don't know golang I have no concrete ideas.
5. setavenger commented on Feb 4, 2025

   

   setavenger

   on Feb 4, 2025

   More actions

   I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.

   I assume you are referencing my implementation with the BlindBit suite. https://github.com/setavenger/blindbit-oracle
   I'm not aware of any memory issues with Oracle, so give it a try if you want to.

   😄React with 😄1Reacted by Tamas Jantvik
6. tsjk commented on Feb 4, 2025

   

   tsjk

   on Feb 4, 2025

   Last edited by tsjk

   More actions

   I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.

   I assume you are referencing my implementation with the BlindBit suite. https://github.com/setavenger/blindbit-oracle I'm not aware of any memory issues with Oracle, so give it a try if you want to.

   Indeed I was. Will look at that more closely as well.