Skip to content

Static frequency mask estimation from visibility data#381

Merged
ljgray merged 12 commits into
mainfrom
ljg/rfi-vis-mask-update
Feb 12, 2026
Merged

Static frequency mask estimation from visibility data#381
ljgray merged 12 commits into
mainfrom
ljg/rfi-vis-mask-update

Conversation

@ljgray

@ljgray ljgray commented Oct 4, 2025

Copy link
Copy Markdown
Contributor

The overall goal of this PR is to implement a task to come up with a static RFI mask from visibility data directly, without having to rely on a hard-coded set of bad channels.

This produces a flagging metric by reducing over baselines using the existing ReduceChisq code as follows:

  • The chi2 is computed over cross-pol baselines only, and excludes baselines < 5 meters.
  • Weighting is done by dividing by baseline redundancy (effectively up-weighting baselines with lower redundancy), which highlights faint frequency bands which are only otherwise visible in the high-delay chisq metric.

Flagging is then done using a simple 1D MAD metric over the 1D median in time of the chisq metric. Only nighttime data is included in the median (the hook is implemented in a ch_pipeline PR). It uses an updated version of the arPLS baseline-fitting code, which is better able to handle the lack of hard-coded static mask.

This PR does the following:

  • Updated the scale_invariant_rank operation to be a bit faster
  • Adds options to the MaskBaselines task to mask polarisation and to optionally combine the mask using an and operation
  • Extends the ReduceBase task with a hook for custom weights
  • Updates the penalized_least_squares implementation to support different weighting schemes
  • Replaces the old narrowband vis flagging task with this new static RFI mask task

This code all works and is tested. This method does not appear to overflag, although there is absolutely room for improvement.

@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 4 times, most recently from 3f35410 to 9902473 Compare December 12, 2025 23:18
@ljgray ljgray marked this pull request as ready for review December 13, 2025 01:12
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 4 times, most recently from 62834c6 to c8d3091 Compare December 15, 2025 23:45
@ljgray ljgray requested a review from ssiegelx December 17, 2025 23:39
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch from c670310 to e2c0267 Compare January 21, 2026 18:47
@ljgray ljgray requested a review from sjforeman January 27, 2026 23:18
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 2 times, most recently from cd84d81 to fc60885 Compare February 2, 2026 19:48
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch 2 times, most recently from 701a343 to f2796ea Compare February 3, 2026 22:10
@ljgray

ljgray commented Feb 5, 2026

Copy link
Copy Markdown
Contributor Author

@sjforeman

@ljgray ljgray requested a review from tristpinsm February 10, 2026 22:52
@ljgray ljgray force-pushed the ljg/rfi-vis-mask-update branch from f2796ea to b25f71d Compare February 12, 2026 22:15
@ljgray ljgray merged commit fe056d6 into main Feb 12, 2026
4 checks passed
@ljgray ljgray deleted the ljg/rfi-vis-mask-update branch February 12, 2026 22:25
@tristpinsm

Copy link
Copy Markdown
Contributor

sorry, was hoping to take a look this pm

@ljgray

ljgray commented Feb 12, 2026

Copy link
Copy Markdown
Contributor Author

sorry, was hoping to take a look this pm

No worries. It won't break anything, so it's really just a review of what's being implemented in this task, which I'd ideally like to improve on in the future anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants