Skip to content

Do not filter or smooth if data len is too small#201

Closed
taldcroft wants to merge 1 commit intomasterfrom
fix-kernel-volume-warnings
Closed

Do not filter or smooth if data len is too small#201
taldcroft wants to merge 1 commit intomasterfrom
fix-kernel-volume-warnings

Conversation

@taldcroft
Copy link
Copy Markdown
Member

@taldcroft taldcroft commented Mar 12, 2026

Description

This should fix the frequent errors we get in the aca high background monitor like:

Errors in files:
/proj/sot/ska3/aca/data/aca_hi_bgd_mon/logs/aca_hi_bgd.log
** ERROR - line 23: /proj/sot/ska3/aca/lib/python3.13/site-packages/chandra_aca/dark_subtract.py:77: UserWarning: kernel_size exceeds volume extent: the volume will be zero-padded.
...

However, I am not sure I entirely understand what is happening. Here is the current code:

    if median_window > 0:
        # median_window=3 by default
        yin = scipy.signal.medfilt(yin, kernel_size=median_window) 

    if smooth_window > 0:
        # smooth_window=30 by default
        yin = smooth(yin, window_len=smooth_window)  

The smooth function actually raises an exception if the data length < window_len, so I don't get why this is working at all unless exceptions are being caught and ignored.

For reference:

>>> import scipy.signal
>>> import ska_numpy
>>> import numpy as np
>>> scipy.signal.medfilt(np.arange(3), 3)
array([0, 1, 1])
>>> scipy.signal.medfilt(np.arange(2), 3)
<ipython-input-5-83c986713dae>:1: UserWarning: kernel_size exceeds volume extent: the volume will be zero-padded.
  scipy.signal.medfilt(np.arange(2), 3)
array([0, 0])
>>> ska_numpy.smooth(np.arange(30), 30)
array([ 0.06876338,  1.13592383,  2.19995339,  3.25947048,  4.31330451,
        5.36055066,  6.4006121 ,  7.433228  ,  8.45848565,  9.47681639,
       10.48897548, 11.49600676, 12.49919381, 13.5       , 14.5       ,
       15.5       , 16.49919381, 17.49600676, 18.48897548, 19.47681639,
       20.45848565, 21.433228  , 22.4006121 , 23.36055066, 24.31330451,
       25.25947048, 26.19995339, 27.13592383, 28.06876338])
>>> ska_numpy.smooth(np.arange(29), 30)
Traceback (most recent call last):
  Cell In[7], line 1
    ska_numpy.smooth(np.arange(29), 30)
  File ~/miniconda3-arm/envs/ska3/lib/python3.13/site-packages/ska_numpy/Numpy.py:272 in smooth
    raise ValueError("Input vector needs to be bigger than window size.")
ValueError: Input vector needs to be bigger than window size.

Interface impacts

Testing

Unit tests

  • Mac
(ska3) ➜  chandra_aca git:(fix-kernel-volume-warnings) git rev-parse --short HEAD
981d461
(ska3) ➜  chandra_aca git:(fix-kernel-volume-warnings) pytest
========================================== test session starts ==========================================
platform darwin -- Python 3.13.11, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/aldcroft/git
configfile: pytest.ini
plugins: anyio-4.12.1, timeout-2.4.0
collected 263 items                                                                                     

chandra_aca/tests/test_aca_image.py .....................                                         [  7%]
chandra_aca/tests/test_all.py ........................                                            [ 17%]
chandra_aca/tests/test_attitude.py .............................................................  [ 40%]
chandra_aca/tests/test_dark_model.py ............                                                 [ 44%]
chandra_aca/tests/test_dark_subtract.py .........                                                 [ 48%]
chandra_aca/tests/test_drift.py ..............................................                    [ 65%]
chandra_aca/tests/test_manvr_mon_images.py .......                                                [ 68%]
chandra_aca/tests/test_maude_decom.py .........................                                   [ 77%]
chandra_aca/tests/test_planets.py .................                                               [ 84%]
chandra_aca/tests/test_psf.py ...                                                                 [ 85%]
chandra_aca/tests/test_residuals.py .....                                                         [ 87%]
chandra_aca/tests/test_star_probs.py .................................                            [100%]

========================================= 263 passed in 39.81s ==========================================

Independent check of unit tests by Jean

  • osx
======================================================================================= test session starts ========================================================================================
platform darwin -- Python 3.13.11, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: anyio-4.12.1, timeout-2.4.0
collected 263 items                                                                                                                                                                                

chandra_aca/tests/test_aca_image.py .....................                                                                                                                                    [  7%]
chandra_aca/tests/test_all.py ........................                                                                                                                                       [ 17%]
chandra_aca/tests/test_attitude.py .............................................................                                                                                             [ 40%]
chandra_aca/tests/test_dark_model.py ............                                                                                                                                            [ 44%]
chandra_aca/tests/test_dark_subtract.py .........                                                                                                                                            [ 48%]
chandra_aca/tests/test_drift.py ..............................................                                                                                                               [ 65%]
chandra_aca/tests/test_manvr_mon_images.py .......                                                                                                                                           [ 68%]
chandra_aca/tests/test_maude_decom.py .........................                                                                                                                              [ 77%]
chandra_aca/tests/test_planets.py .................                                                                                                                                          [ 84%]
chandra_aca/tests/test_psf.py ...                                                                                                                                                            [ 85%]
chandra_aca/tests/test_residuals.py .....                                                                                                                                                    [ 87%]
chandra_aca/tests/test_star_probs.py .................................                                                                                                                       [100%]

================================================================================== 263 passed in 74.06s (0:01:14) ==================================================================================
(ska3-latest) flame:chandra_aca jean$ git rev-parse HEAD
981d4612bd094315b7866591a97af099e895b5ca

Functional tests

No functional testing.

@taldcroft taldcroft requested a review from jeanconn March 12, 2026 14:07
@jeanconn
Copy link
Copy Markdown
Contributor

Yeah, I've looked at this a few times and this probably fixes it. I just refrained from getting in a fix because I found I couldn't reproduce the error - it looks like we only get the error when we are in comm. But if we don't care about proving that the fix works for the high background error, we can go down this path.

@taldcroft
Copy link
Copy Markdown
Member Author

But if we don't care about proving that the fix works for the high background error, we can go down this path.

Let's do that. The proof will be when we stop getting those error alert emails.

Copy link
Copy Markdown
Contributor

@jeanconn jeanconn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion for or against this. I'm not sure if it is a great idea to try to return the unfiltered/unsmoothed value if yin is too short, or if we're better off with the error here.

@taldcroft
Copy link
Copy Markdown
Member Author

Closing since the real driver for this has been implemented elsewhere. If need be we can revisit this.

@taldcroft taldcroft closed this Mar 12, 2026
@taldcroft taldcroft deleted the fix-kernel-volume-warnings branch March 12, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants