Enumerating over pypxlib.Table causes memory leak

I may be doing something incorrectly, but when using enumerate() over a Table object, memory balloons out of control and isn't released. I don't immediately have a fix, but I wanted to document it for others that might struggle with the same. Fantastic library, a million thanks for it.

[jmcarter17]

I have the same issue as well.
Not using enumerate, but using a generator (with a generator expression) to iterate over a table.

Is there a way to iterate over a table that does not cause a memory leak?
If so, that would help us tremendously.

Million thanks for this library, very useful.

[jmcarter17]

Here is an example of code that I use to iterate over the table (which causes memory leaks)

with Table(path_table) as table:
    data = (row.x, row.y, row.z for row in table if row.z > 0) 
    process(data)  # this saves the data to some postgresql database on the cloud

When running this snippet for multiple tables one after the other, the memory usage keeps increasing until we run out of memory and the program crashes.

I would be interested in learning of a work around that does not have the memory leak side effect.

[fievelmousekewitz]

I had the same issue and never found a way to stop the memory ballooning. It's been quite a while but if I remember right, this isn't actually an issue with pypxlib, but more an issue of how python does garbage collection with CTypes.

I DID get around the problem though. I switched to a 64-bit python interpreter and allowed my program to allocate more memory. GC does eventually catch up--my py script iterates over 16 million paradoxdb records and converts them to mssql in about an hour, without any memory related crashes.

Hope that helps, and I'm sorry you're in the same boat as me, dealing with a paradox database in 2022!

[jmcarter17]

@fievelmousekewitz thanks.
Unfortunately, the code has to run on windows xp, with python2.7, with only 2 GB of ram (of which only 500MB are available). I might have to write a solution in pure C, maybe. Thanks for the info, though.

[m-m-baranov]

def free_row(row):
    p_doc = row._table.pxdoc
    doc = p_doc.contents
    free = lambda p: doc.free(p_doc, p)
    for i in range(doc.px_head.contents.px_numfields):
        p_val = row._pxvals[i]
        val = p_val.contents
        if val.type in (1, 12, 13, 14, 15, 16, 23, 24):
            free(val.value.str.val.raw)
        free(p_val)
    free(row._pxvals)

import pypxlib, sys
with pypxlib.Table(sys.argv[1]) as table:
    for c in range(int(sys.argv[2])):
        for row in table:
            free_row(row)

Execute it as "python3 test.py table.DB 1000" and see that no memory is leaking. For a detailed explanation, see my gist at https://gist.github.com/m-m-baranov/362fef0da86d87beddcb0d3b09e01833.