#1 [Counterfactual image analysis] Enhancement toward project completion

[HugoHakem]

Analysis of Counterfactual images

Quality of counterfactual images needs to be assessed.
Why? to build trust. Because trusting the StarGANv2 can generate good counterfactual images that looks real is key if we want to base our biological interpretation on the difference between counterfactual and real.

Right now, it is achieved using the performance of the trained classifier: if the classifier trained on real example can classify correctly counterfactual images, then they are of decent quality.

Problem:

• It is biased toward the classifier capacity to generalize correctly. Maybe the classifier is too conservative (overfit)
• If the classifier incorrectly classify the template image, then there is little chance the counterfactual made out of the template is relevant of a new perturbation.

Solution:

1. FID metric: compute embedding with an unrelated model (inception model) and measure how distribution differs from the counterfactual set of images vs the real set of images for each class by assuming a gaussian distribution of the embedding.
2. LPIPS metric: to judge the perceptual similarity between two images. It is meant to check if images generated by the StarGANv2 are different enough from each other.
3. Cell profiler: check distribution of embedding in counterfactual vs real (maybe some statistical test with the aim of showing mean or distribution are the same.

Current problems:

• It relies on inception model and Alex net model (for FID and LPIPs respectively) which only take 3 channel as an input.
• Needs for a cell profiler python package ideally.

Solution for implementation:

• Use a custom embedding model: this could be the classifier itself (just the convolutional pass), or cell-profiler.
  Good: total flexibility
  Bad: FID is meant to be a standard metrics, so less comparability with a custom model. Plus require more work.

• Aggregate the channel into RGB (by using a weighting of each channel based on the decided color of each channel)
  Good: easy to implement
  Bad: might not render specific failing mode of specific channel because of averaging channel together. (I imagine this can be neglected)

Important Note
When computing FID, images are upsample or downsample so that it fit inception model input size: 3 * 299 * 299. With our small image (128 * 128) this might be a problem because we are heavily increasing image size. Since this concerns both set of images real and counterfactual, this might not be a problem but still important to have in mind.

Check List:

• Aggregate 5 channels over RGB using the same function as for plotting function. Just update the FID and LPIPS metric.