reduce memory usage

[tschimbr]

The current memory usage tends to get very high after some time. Ist there anything that could be garbage collected?

[asittampalam]

Without having double checked it, I suspect that it might be, because I'm storing a little bit of data about each visited page in memory (e.g. the hash value of its content). It might be that this, when accumulated, turns out to be a lot of data. I could solve that problem by bucketing all the information into domain-buckets (e.g. "www.eonum.ch/about" etc. will be in the bucket "www.eonum.ch") and then only holding the currently used bucket in memory and storing everything else on disk. I think this will reduce the memory usage but increase disk access (which shouldn't be a huge problem).

[asittampalam]

I just remembered that I'm caching requests as well. I have to check if I'm already deleting them after they are used or not. If I'm not, that'll be the main problem.