Skip to content

memory usage can be bad #98

@andportnoy

Description

@andportnoy

This issue groups three now closed memory-related issues.

  1. (improve memory usage with lmer #66)lmer fitting is memory expensive. At least three copies of the data are created when using multiple processes: in the parent process, in the child process, in the R process embedded in the child process. A solution is to use maxtasksperchild to purge R memory regularly. Another solution is to fit channel by channel, which has the same result.

  2. (chunk data when running in parallel #67) multiprocessing can't transfer objects > 2 Gb in size. Potential solution is to ensure the chunks are under 2 Gb.

  3. (memory #49) Objects can in general be very large. We can del certain attributes (like .data) and conserve a lot of memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    optimizationspeeds things up, but isn't crucial

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions