UNRAVEL v1.0.0-beta.12 — WHS Rat Atlas Support, Easythresh Integration & Utility Upgrades

UNRAVEL Release: WHS Rat Atlas + Utility & Documentation Upgrades

Tag: v1.0.0-beta.12  Author: Daniel Rijsketic
Since: v1.0.0-beta.11

This release introduces expanded atlas support (WHS SD rat v4.01), new image processing utilities, enhanced FSL easythresh integration, and improved documentation throughout the UNRAVEL guides and CLI tools. It also refines data type handling, error reporting, and plotting flexibility across modules.

---

New Atlases and Resources

• Added WHS SD rat atlas v4.01
  • Includes labels, info, and hierarchical regional summary CSVs for sunurst plots and 3D visualizations.
  • Added color LUT for FSLeyes visualization

---

New & Updated Tools

Image and Voxel Tools

• filter_objects_by_size — filters small connected objects from binary images; supports optional minimum voxel threshold.
• mirror — implemented axis-0 shift logic; raises informative NotImplementedError for other axes.
• ez_thr.py — new script for thresholding FSL statistical maps using easythresh; includes improved argument parsing and mask handling.
• reorient_nii — refactored data-type inference for clarity and reliability.
• spatial_averaging.py — clarified input types, output behavior, and default .tif handling.
• extract_resolution — improved OME-TIFF parsing and error messaging.

Region & Cluster Statistics

• Added ylabel option for plot customization in mean_IF_summary.
• reverse_clusters docstring expanded with parameter/return descriptions.
• Ongoing updates to cstats (preparing for extended statistical test options).
• New output directory options in points_to_atlas.

Registration & Resampling

• Clarified documentation for initial alignment checks and padding.
• Fixed minor argument typo in save_3D_img reference image handling.

---

ABCA / MERFISH Enhancements

• Updated guides (guide.md, guide_mms.md) with more explicit instructions and examples for ABCA workflows.
• Strengthened error handling and documentation consistency across MERFISH modules.

---

Documentation & Developer Notes

• Improved general guide formatting, argument clarity, and Next Steps sections.
• Added API doc for filter_objects_by_size.
• Added TODOs for upcoming refactors (e.g., renaming mean_IF → mean_intensity, testing script in reg.py).
• Updated dependency versions in pyproject.toml.
• Numerous small text clarifications and docstring improvements across modules.

---

Internal & Maintenance

• Added newline and license formatting fix.
• Minor corrections in legacy scripts (_legacy/gubra_to_CCF50.py).
• Consolidated imports and style in utility scripts.

---

Summary

This update strengthens cross-atlas support and pipeline reliability, preparing UNRAVEL for multi-species comparative analysis.
Users can now:

• Leverage the WHS SD rat atlas for region mapping and visualization.
• Apply advanced image filtering, easythresh-based cluster thresholding, and improved resampling workflows.
• Follow updated documentation for ABCA MERFISH processing.

---

v1.0.0-beta.11

🚀 UNRAVEL Release: MapMySections + Expanded ABC Atlas Support

This release delivers a major upgrade to UNRAVEL, focused on full support for the Allen Institute MapMySections Challenge, along with broader improvements to Allen Brain Cell Atlas (ABCA) integration, documentation, CLI tools, and internal module organization.

---

🧭 New Documentation

📘 MapMySections Analysis Guide

• End-to-end guide for:
  • Downloading and organizing Genetic Tools Atlas (GTA) data
  • Segmenting STPT brains with pretrained Ilastik models
  • Registering brains to CCFv3 and warping to MERFISH space
  • Filtering MERFISH cells using segmentations
  • Visualizing and quantifying cell types across brains and regions
• Includes flowcharts, best practices, and detailed CLI usage

---

🛠️ Major Features and Tools

🧪 MapMySections CLI Tools (new)

• mms_seg_summary: quantify voxel counts per segmentation label
• mms_soma_ratio: estimate oligodendrocyte enrichment in anterior commissure
• mms_cell_type_proportions, mms_cell_type_proportions_concat: whole-brain or region-specific cell type proportion summaries
• seg_ilastik: wrapper for running Ilastik segmentation in batch mode

🧬 ABCA / MERFISH Integration Enhancements

• abca_merfish_filter: neuron-only filtering, region-based extraction
• abca_sunburst, abca_sunburst_expression: hierarchical and gene expression sunburst plots
• abca_mean_expression_color_scale, abca_percent_expression_color_scale: expression-based LUTs
• abca_merfish_join_gene: map gene expression across filtered cells
• RNAseq_join_expression_data: join gene expression with scRNA-seq metadata
• Human/Mice support handled via --species flag across tools

---

🗜️ Zarr Image Utilities (GTA)

• gta_download: download GTA Zarr images at specific resolution levels
• gta_auto_crop, gta_bbox_crop: automated 3D bounding box cropping
• gta_org_samples: organize TIFF directories into sample folders
• zarr_compress: recursive compression/decompression of .zarr to .zarr.tar.gz

---

🔀 Registration + Warping Tools

• warp_to_atlas: warp segmentations to CCFv3 atlas space
• warp_ccf30_to_merfish: convert 30 µm segmentations to MERFISH space
• reg_prep, reg, reg_check: preprocessing and registration with template
• affine_initializer_check: detect initial alignment issues during registration
• mirror, img_math: image mirroring and math utilities

---

🧩 Tabular + Utility Tools

• print_columns, print_unique_values: explore CSV column contents
• edit_columns: rename or reorder CSV columns
• filter_data_by_column: flexible row filtering with include/exclude modes
• csv_concat_with_source: add source_file column when combining CSVs

---

📦 Package & Internal Refactors

• Refactored module structure
  • Moved abca and gta scripts to unravel/allen_institute/
  • Centralized image I/O and tabular utils
  • Unified function naming, argparse logic, and CLI descriptions
• Updated pyproject.toml
  • Aliases added for new CLI tools
  • Optional [abca] dependencies (e.g., boto3, abc_atlas_access) declared
• Improved .gitignore for Sphinx and log files

---

📚 Documentation Improvements

• Added:
  • merfish_slice and utils module docs
  • Instructions for PyPI install: pip install unravel
• Improved:
  • Index and landing page formatting
  • Output path and argument help across scripts
  • Clarified Next Steps and visualization tips in guides

---

🧪 Summary

This release empowers users to:

• Process and register large-scale 3D STPT images
• Segment major cell types (somata, astrocytes, endothelial)
• Analyze filtered MERFISH cells with ABCA metadata
• Quantify and visualize gene expression across the brain

Try the new MapMySections workflow now or explore the expanded ABCA functionality!

v1.0.0-beta.10

🔖 UNRAVEL v1.0.0-beta.10 – ABCA Tool Drafts, Cluster Stats, and I/O Enhancements

Changes since v1.0.0-beta.9

---

🧬 Draft Tools for Allen Brain Cell Atlas (ABCA)

This release introduces an extensive set of draft-stage scripts for analyzing spatial transcriptomics and scRNA-seq data from the Allen Brain Cell Atlas (ABCA). These tools are available for testing and refinement prior to their integration into the core UNRAVEL CLI.

🧠 Spatial Transcriptomics (MERFISH)

• Voxel-wise expression mapping:

  • merfish_cells_to_nii.py: Convert MERFISH cells into 3D expression images
  • merfish_expression_to_nii.py: Generate MERFISH gene expression maps in CCF space
  • merfish_resample.py: Resample MERFISH coordinates or expression data
  • merfish.py: General-purpose MERFISH plotting and utility functions for spatial transcriptomics analysis

• Expression analysis and visualization:

  • merfish_cluster.py: Assess MERFISH expression in anatomical clusters
  • merfish_dot_plot.py, merfish_dot_plot_legend.py: Create annotated dot plots
  • merfish_heatmap.py: Visualize gene expression across regions or slices
  • merfish_filter.py, merfish_filter_by_mask.py, merfish_filter_expression.py: Filter cells by expression, mask, or region
  • merfish_imputed_genes.py: List or extract imputed MERFISH genes

• Region-level correlation & stats:

  • correlation_parallel.py, correlation_permutation.py, correlation_plot.py, correlation_comparison.py: Correlate MERFISH expression with voxel-wise maps and test significance
  • correlation_parallel_to_nii.py: Convert parallelized correlations into image format

• Metadata integration:

  • merfish_join_expression_data.py: Join expression matrices with metadata for analysis

---

🧬 scRNA-seq (10xv3 RNA-seq)

• RNAseq_expression.py: Extract scRNA-seq gene expression from ABCA datasets
• RNAseq_expression_in_cell_types.py: Analyze expression across cell types and subclasses
• RNAseq_expression_in_mice.py, RNAseq_expression_in_mice_heatmap.py: Visualize expression across mouse brain regions
• RNAseq_expression_in_mouse_MSNs.py: Explore MSN expression profiles
• RNAseq_expression_in_neurons_astro_ug.py, RNAseq_expression_in_mice_neurons_astro_ug.py: Filter and analyze major neural/non-neural cell classes
• RNAseq_dot_plot.py: Generate dot plots of gene expression
• RNAseq_filter.py: Filter RNA-seq datasets by metadata or expression thresholds

---

🌞 ABCA Visualization Utilities

• ABCA_sunburst.py: Generate CSVs for sunburst plots of cell type proportions at all ontology levels
• ABCA_sunburst_expression.py: Overlay mean expression values on sunburst plots
• ABCA_sunburst_filter.py, ABCA_sunburst_filter_by_proportion.py: Filter sunburst data by expression or prevalence
• ABCA_mean_expression_color_scale.py, ABCA_percent_expression_color_scale.py: Plot color scales for mean or percent expression

---

🧰 Additional Support Tools

• CCF30_to_MERFISH.py, CCF30_to_MERFISH30.py: Warp atlas or expression images to MERFISH-CCF space
• ABCA_cache_cli.py: Download Allen Brain Cell Atlas data for offline analysis

🧪 These scripts are located in _other/drafts/ABCA/ and will be integrated in future releases. The above list includes key examples, but is not exhaustive.

---

📊 Cluster & Statistical Enhancements

• summary_config.py: Auto-generate cluster_summary.ini for a given cluster image
• clusters.py: Create reverse cluster index images for downstream summary stats
• fstat_sig_vx_mask.py: Generate a binary mask from thresholded F-statistic maps
• correlation_permutation.py: Permutation test for region-level correlation significance
• Improved support for multiple thresholds, mask application, and verbose cluster stats logging

---

🧰 Image I/O, Formats & Resampling

• New utilities:
  • vstats_info.py, variance.py: Extract image stats and variance maps from NIfTI series
  • csv_to_nii.py, nii_to_csv.py: Convert between tabular data and images
• Enhanced support for:
  • TIFF series and 3D stacks
  • CZI axis reorientation
  • Anisotropic and block-based resampling with accurate header updates
• Improved handling of reference atlases, label IDs, and affine preservation

---

🚀 CLI & Usability Improvements

• Consistent argument naming (--input, --output, --mask)
• Added command aliases and unravel_commands index
• Verbose flag support for debugging and sample tracking
• Improved error messages, file checks, and output directory creation

---

📚 Documentation & Guide Updates

• New guide sections on:
  • MERFISH and ABCA workflows (upcoming)
  • Cluster validation flowchart and usage examples
  • TIFF/CZI preprocessing and Ilastik integration
• Refined installation instructions (favoring venv over pyenv)
• Updated examples and help text across commands

---

🛠️ Bug Fixes & Stability

• Fixed issues with:
  • Integer dtype checks for cluster images
  • FSLeyes LUT compatibility on macOS
  • UnboundLocalError in data loading functions
  • Label filtering and CSV parsing edge cases
• Added warnings, fallback logic, and regression tests for common pipeline steps

---

For installation and usage, see: UNRAVEL GitHub Guide

🧪 Feedback on ABCA draft tools is welcome as we prepare them for full integration in v1.0.0!

v1.0.0-beta.9

v1.0.0-beta.9 - Enhancements to Argparse Formatting and Usability

This release introduces significant improvements to argparse help formatting, leveraging the Rich library for enhanced readability and usability in terminal output.

New script help_formatter.py (replaces argparse_utils.py):

• Rich-Enhanced Argparse Help: Implemented RichArgumentParser and CustomHelpAction to integrate richly formatted help messages, including styled docstrings and argument lists.
• SuppressMetavar Class: Updated documentation on nargs configurations to avoid issue with argparse help wrapping to ensure consistent display across different terminal sizes.
• format_argparse_help Function: Applies custom styles to argparse help text, highlighting flags, defaults, and sections.
• format_docstring_for_terminal Function: Formats module docstrings with Rich's styled output for enhanced terminal display, improving readability by highlighting sections, command names, and key elements.

Improvements:

• Refactored get_samples(): Refactored for better flexibility and clearer output, now consolidated with the -e and -d arguments for directory processing.
• New: utils_get_samples Command: For testing --pattern and --dirs args in commands that perform batch processing
• Simplified Sphinx Documentation: Addressed Sphinx warnings and improved compatibility by replacing <asterisk> with * in help_formatter (except when quotes are used)
• Refactored Scripts: Various scripts, such as copy_specific_slices, filter_csv.py, and seg_ilastik, have been refactored for improved functionality and clarity.

Fixes:

• Sphinx Warnings: Resolved multiple warnings during documentation builds, ensuring a cleaner and more reliable doc generation process.

v1.0.0-beta.8

Summary of Changes

New Features:

• New Scripts:
  • filter_csv.py: Allows filtering of a CSV file by a specified column based on a list of values (useful for filtering out data from one hemisphere).
  • gubra_to_CCF_50um.py and gubra_to_CCF.py: For warping from Gubra space to CCF space.
  • points_compressor.py: Compresses coordinate points per region and can output a regional summary.
  • points_to_atlas.py: Warps a point coordinate image to atlas space.
  • resample_points.py: esampling points and saving them as NIfTI images.
  • points_to_img.py: Converts point data into images for visualization.
  • img_to_points.py: Converts images back into point data for further analysis.

Updates and Improvements:

• Script and Function Updates:

  • Converted generators to lists to handle scenarios where no files match, and the script needs to iterate over the list of matched files.
  • Added an option to move files instead of copying them.
  • Refactored functions to work with a DataFrame instead of ndarray for point data.
  • Corrected handling of single float values for current_res and target_res in the resample_and_convert_points function.
  • Fixed logic for saving output and determining whether to use the -o option.
  • Improved the code for updating the affine matrix to avoid affecting the origin and refactored resample_nii() to reduce parameters.

• Documentation:

  • Slicer 3D installation link and functions of dependencies are now mentioned.
  • Updated the installation guide and terminal cheat sheet, adding new video links for beginners.
  • Fixed multiple Sphinx warnings by updating docstrings and removing problematic parameters like *args and **kwargs.
  • Added __doc__ to epilog and improved the clarity of argument descriptions.

• Configuration and Dependency Management:

  • Fixed the default path for the -fri option.
  • Pinned Python version to 3.11 in GitHub Actions due to compatibility issues with distutils in Python 3.12.
  • Updated command names starting with cluster_ to cstats_ for consistency with vstats and rstats commands.

• Bug Fixes:

  • Fixed a warning related to plotting bar graphs.
  • Resolved issues when using small terminal windows with nargs='+' and action=SM.
  • Corrected typos and addressed minor bugs in the handling of command options and argument processing.

These changes introduce new functionality, improve existing scripts, and address bugs.

v1.0.0-beta.7

Release Update Summary

Recent Changes

• Command Renaming:

  • Commands starting with cluster_ have been renamed to cstats_ to ensure consistency with existing vstats and rstats commands.

• Guide Updates:

  • Added a terminal cheat sheet and new video links aimed at beginners.
  • Simplified argument structure to enhance usability.
  • Update the installation guide

• CSV and Logic Updates:

  • Renamed CSV files, updated help documentation, revised arguments, and adjusted logic.
  • Removed specific CSV files: region_ID_side_name_abbr.csv and sunburst_IDPath_Abbrv_wo_root.csv.

• Template and Defaults:

  • Updated information on the iDISCO/LSFM template.
  • Set the defaults to use CCFv3 (30 µm) with the iDISCO/LSFM template. The template has been warped to CCF space with refined alignment.

These changes enhance consistency, simplify usage, and improve documentation, aligning with updates to templates and default settings.

v1.0.0-beta.6

UNRAVEL v1.0.0-beta6 Release Notes

New Features

1. Resampling and Warping:

   • Added resample.py for resampling .nii.gz images.
   • Introduced warp_forward.py to forward warp a moving image (e.g., warping an atlas space image to unpadded tissue space [does not scale to full res like warp_to_native])

2. Support for CCFv3 2020:

   • Added CSV for the 2020 version of CCFv3.
   • Set some defaults for CCFv3 2020 (remaining defaults will be updated in next release).
   • Implemented features to save cell centroids and calculate label densities.

3. Image and Label Processing:

   • New :labels_to_masks.py for converting ilastik segmentation TIF series to masks.
   • New: mean_IF.py and mean_IF_summary.py for plotting data for each cluster.
   • Updated prism.py to work with data from mean_IF.py.

Improvements

1. Documentation and Help:

   • Updated installation guide with improved instructions and added sections on updating UNRAVEL.
   • Added links to the PyPI page for heifetslab-unravel.
   • Improved help messages and error handling across scripts.
   • Addressed Sphinx warnings for better documentation generation.

2. Code Refactoring and Optimization:

   • Refactored code to use decorators for printing commands and times.
   • Enhanced logging and verbosity options.
   • Updated logic for handling image registration and segmentation.
   • Removed redundant dependencies and improved package configurations.

3. Bug Fixes:

   • Fixed issues with file existence checks and path handling.
   • Corrected typos and formatting errors in documentation.
   • Pinned numpy version at 1.24.4 to avoid warnings in Windows.
   • Addressed specific bugs related to argument handling and default values.

4. Miscellaneous:

   • Added sunburst_RGBs.csv customization.
   • Improved error handling for various scripts and functions.
   • Updated arguments and default patterns for input images.

Installation and Configuration

• Added build and twine as optional dependencies.
• Provided an argument for specifying the path to the Ilastik executable, ensuring compatibility across different operating systems.
• Updated installation notes to reflect the latest changes and dependencies.

This release includes significant improvements in functionality, performance, and documentation, making it easier to use and more robust for processing LSFM data.

v1.0.0-beta

What's Changed

Major Refactor and Improvements

• The entire UNRAVEL codebase has been refactored to Python, enhancing readability and maintainability (e.g., reuse of functions).
• Several improvements have been made to streamline the processing pipeline, making it more efficient and user-friendly.

Initial Version and Publication

• The initial version of the code, which was a mixture of Python, Bash, Fiji macros, and XLSX templates, has been significantly upgraded.
• For an in-depth understanding of the initial version, refer to our publication in Neuropsychopharmacology.

New Documentation

• Please checkout our new documentation for the latest guides, etc.

v0.1.2-feature-branch

Release Log: UNRAVEL v0.1.2-feature-branch (Feature Branch Update)

Release Date: August 2024

Overview:

This release is a major update to the UNRAVEL v0.1.0 codebase, incorporating both significant improvements to the original version (v0.1.0) and early drafts of code that were foundational for the v1.0.0-beta release on the main branch. The code in this branch represents a bridge between the initial version described in Rijsketic et al., 2023 and the ongoing development efforts aimed at achieving the full functionality envisioned for v1.0.0-beta. This update focuses on enhancing the speed and accuracy of the registration pipeline, expanding the functionality of statistical analysis, and adding new tools for image processing, cluster validation, and data visualization. Additionally, this release includes numerous bug fixes and optimizations designed to streamline the workflow for voxel-wise analysis of c-Fos immunofluorescence and related tasks.

Major Updates:

• Enhanced Registration Pipeline:

  • Streamlined registration scripts (czi_to_tif3.sh, prep_reg.sh, brain_mask.sh) for faster processing and improved accuracy.
  • Added support for parallel processing and optimized memory usage in image loading and saving functions.
  • Integration of ANTsPy tools for affine initialization and non-linear transformations, enabling more robust alignment to the LSFM atlas.

• Statistical Analysis Enhancements:

  • Introduction of prep_vxw_stats.py for voxel-wise statistical analysis with improved resampling and reorientation functions.
  • New scripts for performing 2×2 ANOVA and other statistical tests with enhanced false discovery rate (FDR) correction methods.
  • Added functionality for calculating effect sizes, performing Tukey's post hoc tests, and generating summary statistics across multiple q-value thresholds.

• Cluster Validation Tools:

  • A comprehensive suite of scripts (validate_clusters.py, validate_clusters_summary_pooled_ttest.py, valid_clusters_2x2_ANOVA_draft.py) for validating voxel-wise clusters using post hoc comparisons, cell density quantification, and statistical summaries.
  • Tools for handling bilateral data, aggregating cluster validation results, and generating publication-ready tables and visualizations.

• Image Processing and Data Handling:

  • New utilities for converting image formats (czi_to_nii.py, h5_to_tifs.py, tif_to_tifs.py) and handling large datasets more efficiently.
  • Updated unravel_utils.py with functions for handling metadata, managing output directories, and processing batch data.
  • Advanced tools for manipulating image volumes, including z-scoring, background subtraction, masking, and warping to atlas space.

• Improved User Experience:

  • Refined help guides and improved command-line interface (CLI) with custom argparse formatting for better readability.
  • Enhanced progress tracking with rich library integration and more detailed error handling.

Bug Fixes:

• Resolved issues related to image orientation, file path handling, and parallel processing in various scripts.
• Fixed inconsistencies in metadata extraction and file naming conventions to ensure compatibility with downstream analysis.
• Addressed bugs in statistical scripts related to argument parsing, file aggregation, and result formatting.

Known Issues:

• Ongoing testing is required for certain new features, particularly in the native_clusters2.py and related validation scripts.
• Some scripts remain in draft form and may require further refinement before use in production environments (please see v1.0.0-beta releases on the main branch for our latest code).

Other changes

• Added initial installation guide (#1)
• Bug fixes, added ABAseg_3dc.sh for single rater regional counts (#2)
• Upgraded miracl_conv_convertTIFFtoNII.py to python3 (#3)
• Updated README.md

This release marks a significant milestone in the development of UNRAVEL, incorporating numerous enhancements that improve both the functionality and user experience. The code in this branch has been utilized in the preparation of several scientific manuscripts, underscoring its robustness and applicability to real-world neuroimaging analysis.

Full Changelog: v0.1.1...v0.1.2-feature

Initial release (w/ README markdown)

We developed UNRAVEL (UN-biased high-Resolution Analysis and Validation of Ensembles using Light sheet images) as a command line tool for voxel-wise analysis of c-Fos immunofluorescence (IF) across the cleared mouse brains and validation of identified clusters of significant voxels using c-Fos+ cell density quantification at cellular resolution.

Please refer to guides regarding necessary dependencies, paths that need to be updated, conventions for file/folder organization and detailed info on running scripts. Key scripts include: find_clusters.sh, glm.sh, and validate_clusters2.sh. Other scripts can be run in a modular fashion (e.g., overview -> prep_tifs.sh -> 488_to_nii.sh -> reg.sh -> rb.sh -> z_brain_template_mask.sh -> glm.sh -> validate_clusters2.sh). Help guides can be found at the beginning of each script or by running: <script>.sh help

If you are unfamiliar with the command line interface, please review Unix tutorials here: https://andysbrainbook.readthedocs.io/en/latest/index.html