Patch-based Image Classification

[Jordan-Pierce]

TL;Dr

Add the ability to do patch-based image classification to images on tator (for GL, potentially MDBC and other projects). Users, on a single-image basis, could get percent (%) composition estimates for benthic substrates.

To be able to ask the questions:

• "is this image heterogenous?"
• "what stuff might be in this image?"
• "what ratios of different stuff might be in this image?"
• "what is the majorities / minorities of stuff in this image?"

Why patch-based image classification?

• Easy to label data (points annotations), and / or re-use existing annotations (bounding boxes, polygons)

Why not Semantic Segmentation?

• Requires labels for every single pixel (very consuming)

Patch-based image classification is essentially a (significant) subset of semantic segmentation (and in a dumb way).

General Plan

Training

• Pull grid data down from tator, use as the basis for training data
  • Maybe extract a patch (size tbd) centered on the grid, using the assigned label
  • Consider looking at other annotation types that might be available and useful
  • Bboxes and polygons can be used with patches by simply cropping and resizing
• Train a simple image classification model (YOLO) in Coralnet-Toolbox

Production

• Create class(es) generating patches from an input image (user's current image)
  • Inputs will include the media name / id, frame number, number of patches, patch size, sample (uniform, random, or existing grid from tator)
  • Logic for returning useful information from prediction class back to tator to be displayed to the user (categories found, %s, confidence per, etc.)
• Dockize it

Action Items

• Mark to provide sample of existing annotated / ground-truth'ed grid data from GL project already on tator, instructions on how to self-pull
• Jordan to use this^ information to pull bulk data from tator, create a YOLO-formatted image classification dataset, train a model
• ...

[Jordan-Pierce]

@marktaipan-cvisionai please modify / edit as needed