Skip to content

Speeding up reduce - #178

Open
martinjohndyer wants to merge 11 commits into
masterfrom
speedup
Open

Speeding up reduce#178
martinjohndyer wants to merge 11 commits into
masterfrom
speedup

Conversation

@martinjohndyer

Copy link
Copy Markdown
Member

This has been going for ages, finally time to get it done before going through again to remove PGPLOT.

This branch includes moving some of the key profile fitting functions (which was found to take ~66% of the time when running reduce according to #142) into C++ code, specifically using pybind11 over Cython. At the same time I also moved the existing "support" code over to pybind11, which is basically the single avgstd function used in combine.

The old versions of the functions are still kept in reduce.py for comparison and fallback, I've also added a simple test script to compare and time the old and new versions. The output tests all pass, and on my laptop I get a variable but consistent speedup e.g.

$ python hipercam/tests/fitting_test.py 
All checks passed: 20 cases, rtol=1.0e-09, atol=1.0e-11

Benchmarking 20 cases over 100 runs:
-------------------------------------------------------------------------------------
function                         Numba              C++                Speedup 
-------------------------------------------------------------------------------------
moffat                           0.080±0.011 ms     0.105±0.355 ms      0.77x
gaussian                         0.036±0.007 ms     0.014±0.002 ms      2.55x
dmoffat[dfwhm=False,dbeta=False] 0.165±0.015 ms     0.033±0.008 ms      4.96x
dmoffat[dfwhm=False,dbeta=True]  0.198±0.021 ms     0.166±0.841 ms      1.20x
dmoffat[dfwhm=True,dbeta=False]  0.170±0.019 ms     0.033±0.005 ms      5.21x
dmoffat[dfwhm=True,dbeta=True]   0.205±0.025 ms     0.041±0.005 ms      5.04x
dgaussian[dfwhm=False]           0.093±0.047 ms     0.019±0.008 ms      4.96x
dgaussian[dfwhm=True]            0.068±0.010 ms     0.026±0.030 ms      2.59x

Doing some overall timing tests on reduce, with the current master branch:

1000 frames, 7 runs:
49.56 ± 0.95 s per run (20.18 Hz)
0.050 ± 0.028 s per frame (20.18 Hz)

And after these changes:

1000 frames, 7 runs:
31.05 ± 1.32 s per run (32.20 Hz)
0.031 ± 0.014 s per frame (32.20 Hz)

Now my laptop is clearly slower than @StuartLittlefair's Mac that was getting 130Hz in #142 (comment) before any optimisations, but I'm hoping the level of speed up is maintained.

I didn't go for the idea of a quick reduce preset in genred that we talked about, although I still think that's something worth considering. I did look at adding in extra options like the least squares tolerances, but right now adding extra parameters to the reduce file means old files fail to be read. I think that could be improved, but that'll have to be touched on when PGPLOT is removed so I'll save it until then.

And a note for completeness, I'm not really a C++ coder so translating the existing functions over and looking for optimisations was very much AI-assisted. At least with the test scripts the outputs can be confirmed, but there may be some oddities in there. Based on the lcurve experience though maybe in the end we'd rather pivot to Rust anyway...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant