Pull Request for documented pipeline and demo files in tator_tools

[pkurtznoaa]

mdbc_cluster_demo.ipynb is the code I used for generating images and runs for the presentation. The mdbc_cluster_pipeline.py file contains the organized class for running instances, though it doesn't output and save results of embeddings to csv.

[gemini-code-assist]

Summary of Changes

Hello @pkurtznoaa, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a suite of new tools aimed at streamlining the process of preparing object detection datasets from Tator, performing advanced object similarity searches using deep learning embeddings, and providing a robust, reusable pipeline for these tasks. The changes enhance our capabilities for automated object analysis and data visualization within large media collections.

Highlights

• New Jupyter Notebook for Tator Localization Data Export: I've added tator_tools/localization_downloader.ipynb, a new Jupyter notebook designed to connect to the Tator platform, download localization data (bounding boxes) for a specified media ID, extract corresponding frames from video, and convert this data into a YOLO-compatible dataset format. It includes helper functions for frame extraction and YOLO data preparation, making it easier to prepare custom datasets for object detection model training.
• New Jupyter Notebook for Object Clustering Demo: I've introduced tator_tools/mdbc_cluster_demo.ipynb, a comprehensive Jupyter notebook that demonstrates a workflow for object detection, feature embedding using CLIP, and similarity-based clustering. This notebook showcases how to load YOLO and CLIP models, extract embeddings from detected objects in images or video frames, perform cosine similarity searches, and visualize results, including integration with FiftyOne for interactive data exploration and UMAP visualization of embeddings.
• New Reusable Object Clustering Pipeline: I've implemented tator_tools/mdbc_cluster_pipeline.py, a new Python class MDBCClusterPipeline. This class encapsulates the core logic for object detection (YOLO) and embedding (CLIP) extraction, and similarity comparison. It provides a structured, reusable component for identifying similar objects across a collection of images or video frames, and includes a command-line interface for easy execution.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The code changes introduce two new files: localization_downloader.ipynb and mdbc_cluster_demo.ipynb, along with modifications to mdbc_cluster_pipeline.py. The notebook files implement object detection and similarity search using YOLO and CLIP models. The pipeline file organizes the class for running instances. There are several issues identified, including a NameError due to an uninitialized variable, an early return in a function, and redundant model instantiations.

[gemini-code-assist]

The traceback shows a NameError: name 'k' is not defined. It seems like the variable k was intended to keep track of successful downloads but was commented out. This will cause the program to crash if an exception is encountered during the download process. It should be initialized before the try block.

    media = api.get_media(MEDIA_ID)
    out_path = f"./data/Images/{media.name}"
    print(f"Downloading {media.name}...")
    k = 0 # Initialize k before the try block
    for progress in tator.util.download_media(api, media, out_path):

[gemini-code-assist]

The extract_objects_with_embeddings function only returns the first detected object in a frame due to the return objects statement being inside the for loop. This prevents the function from processing all detected objects in the frame. To fix this, the return statement should be moved outside the loop.

            pil_image = Image.fromarray(cv2.cvtColor(cropped, cv2.COLOR_BGR2RGB))
            emb = get_clip_embeddings(pil_image)
            objects.append((emb, (x1, y1, x2, y2)))
        return objects
    

[gemini-code-assist]

The YOLO model is instantiated inside the detect_objects function. This is inefficient because the model is reloaded every time the function is called. It's better to instantiate the model once outside the function and pass it as an argument to avoid redundant loading.

def detect_objects(image, model):
    """
    Detect objects in an image using YOLOv8 from ultralytics.
    
    Args:
        image (numpy.ndarray): Input image in BGR format.
        embedding: CLIP embedding (not used in this example, but available to integrate if needed).
        
    Returns:
        list: A list of dictionaries with keys 'label' and 'bbox'. The bbox is a tuple (x_min, y_min, x_max, y_max).
    """
    # Load a pre-trained YOLOv8 model (ensure you have ultralytics installed)
    # model = YOLO("yolov8n.pt")  # or use another weight file/model as needed
    # model = SAM("sam2.1_b.pt")
    # file_path = "./Benthic-Mapping-highlight/benthic_mapping/model.pt"
    # model = YOLO(file_path) # Instantiate your model
    # Run the model on the image; the model accepts BGR images if using cv2 images
    results = model(image)
    detections = []

[gemini-code-assist]

The YOLO model is instantiated again, even though it was instantiated in cell 6. This is redundant and should be removed to avoid unnecessary reloads of the model.

file_path = "./Benthic-Mapping-highlight/benthic_mapping/model.pt"
#yolo_model = YOLO(file_path) # Instantiate your model