Reduce memory usage #82
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hmgaudecker
approved these changes
Jul 16, 2025
Member
hmgaudecker
left a comment
hmgaudecker
left a comment
There was a problem hiding this comment.
Apologies for taking so long to look at this. Very, very nice -- thanks a lot!!!
hmgaudecker
added a commit
that referenced
this pull request
Jan 8, 2026
* Revert to using 64-bit precision. * Add function to check whether the model has investments. * [WIP] Refactor: Include stuff in process_model / process_data. * Backup commit before removing fill_list() feature. * Fix tests, model processing seems to work with additional periods now. Missing data and constraints. * Remove fill_list(). * [WIP] re-write process_data s.t. it returns a dictionary. Not working yet with investments. * [WIP] Test for augmenting data with investments. * Stack data for investments. * Fix interfaces. * Readability. * Update environment. * Jax 0.5 installed via pypi, only way that works right now. * [WIP] implement fixed constraints right away. * Refactor, enforce bounds on fixed parameters. * Refactor test for augmented data so we can re-use the fixture. * Implement restrictions for augmented periods. Allow stages + investments. * Update pixi on GHA. * Fix requirements for MacOS. * Correct misnomer. * Update & run hooks. * In case investments are present, leave out entire 'raw' period (=2 skillmodels periods) for transitions. * Fast QR Factorization on GPU (#81) * Implement QR Factorization * Move QR and add tests * Fix Seed, Split tests * Jax via conda. * Fix (?) periods for shocks. * Remove the last two internal periods for transitions if investments are present. * Do not accidentally modify the params_template when enforcing constraints. * Use local optimagic. * Clip at -1e30 in case we end up running in 32bit mode. Use bounds_distance instead of 0, stick to dicts for the moment though code for custom constraints is there. * Reduce memory usage (#82) * Move Checkpoints * Add splitting for gradient * Speed up QR * Update hooks and GHA pipeline. * Use optimagic from GitHub. * Fix heatmaps, update hooks etc. * Fix heatmaps also for factor correlations. * Got started updating transition equations. * Be pedantic about using 'periods' for interface and 'aug_periods' for internal usage. * Implement Janos' suggestion to only ever talk of endogenous factors. * Get rid of deprecation warning. * Use 'aug_period' as the index. * More changes of periods -> aug_periods. * Use 'aug_period' also in DataFrame. * Use 'aug_period' as index also in update_info. * More fixes to aug_period / period. * Do not include correction factors as dependend variables in transition plots by default. * Fix tests, basically allowing non-augmented models again. * Update actions/pyproject. * Use codecov token again. * Fix docstring. * Update GHA, pre-commit hooks. * Add CLAUDE.md * Move from mypy -> ty. Allow anchoring and endogenous factors at the same time. --------- Co-authored-by: Max Jahn <max.jahn45@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces multiple measures to reduce the memory usage of the gradient computation in skillmodels.