feat: add avg classifier probability to unit_descriptor table

[abuzarmahmood]

When the waveform classifier is used, the average classifier probability of each sorted unit's waveforms is now stored in the unit_descriptor table. This gives a per-unit quality signal based on how confidently the classifier identified the waveforms as spikes.\n\n## Changes\n\n- Add avg_classifier_prob (Float32) column to sorted_unit_metadata and unit_descriptor table definitions\n- save_unit accepts an optional avg_classifier_prob parameter (defaults to NaN when classifier is not used)\n- Auto-processing path: auto_process_electrode now returns subcluster_prob; the caller computes np.mean(subcluster_prob[i]) for each accepted unit\n- Manual processing path: loads clf_prob.npy/clf_pred.npy for the electrode when available, computes avg prob over the selected spike indices\n- Unit summary plots (blech_units_plot.py) display the classifier probability when present\n\nCloses #760

[abuzarmahmood]

Solution Summary

This PR adds avg_classifier_prob to the unit_descriptor table when the waveform classifier is being used, as requested in #760.

How it works

Data flow: During spike sorting, the classifier produces a probability for each waveform indicating confidence that it's a real spike (stored in clf_prob.npy). When a unit is saved (either via auto or manual post-processing), we now compute the mean of these probabilities across all waveforms assigned to that unit and store it as avg_classifier_prob.

Files changed

1. utils/blech_post_process_utils.py

   • Added avg_classifier_prob = tables.Float32Col() to both sorted_unit_metadata and unit_descriptor PyTables table definitions
   • save_unit() now accepts avg_classifier_prob (defaults to NaN when classifier isn't used)
   • auto_process_electrode() now returns subcluster_prob alongside waveforms/times/fin_bool

2. blech_post_process.py

   • Auto path: Unpacks subcluster_prob from auto_process_electrode results, computes np.mean(subcluster_prob[i]) per unit, passes to save_unit
   • Manual path: After loading electrode data, checks for clf_prob.npy/clf_pred.npy. If present, loads them and computes avg prob using full_spike_inds (tracked through split/single/merge branches). If absent, passes NaN.

3. blech_units_plot.py

   • Unit summary plot title now includes Clf Prob: X.XX when the value is present and not NaN

Backward compatibility

• The new column defaults to NaN (via PyTables Float32Col() default), so existing HDF5 files that are re-processed will get the column populated
• blech_units_plot.py uses a try/except to gracefully handle older files without the column
• When the classifier is not used, the value is stored as NaN and not displayed in plots

[abuzarmahmood]

@copilot is clf prob loaded anywhere else? make sure this is not redundant code. If it is, try to simplify / refactor to avoid redundancy

[Copilot]

@abuzarmahmood I've opened a new pull request, #768, to work on those changes. Once the pull request is ready, I'll request review from you.

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • pytest_tests

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #761 at branch `feature/760-avg-classifier-prob-in-unit-descriptor`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}