Question about getting really small bugs

[quitmeyer]

So our group is interested in the really tiny bugs (even if they are kinda small crappy observations)

Before we had been using a yolo11 - obb imgsize 1600 model that we trained on 4500 source images (for 38 hours on my old laptop)

and that's been working quite good for us!

I was just trying out the flatbug this evening to get a feel for it, and it was really quick and simple to use! and it actually seems to run a decent amount faster than our model! so that's cool!

it NEARLY gets everything on our images, but it tends to leave out some small bugs
like these:

do you have any suggestions on how i might tweak it to include tiny bugs?

or alternatively, are there easy ways to maybe train it on our data to make it more sensitive to those tiny bugs?

otherwise it's going good!

(Side question: if we don't need the masks, is it possible to speed things up by telling it to not compute those masks, or not?)

[qgeissmann]

Hej @quitmeyer , thanks for giving it a go!. Can you send your original image and the command you used?
I think there could be several way around:

• artificially upscale the image using the -s option
• use the larger model (tends to work better, but slower)
• add some of your data in the training set (we can help with that and include the data in the official "sub-datasets")
  looking forward!

[quitmeyer]

I didn't know you could choose different models, that's cool!
emailed you the file!

Here's my code i just lifted from your notebook

#from flat_bug import download_from_repository

image = "doubleParina_2025_02_06__13_42_06_HDR0.jpg"
#download_from_repository(image)

# Import the main model class
from flat_bug.predictor import Predictor

# Optionally import the set_log_level function to control the verbosity of flat-bug module
# from flat_bug import set_log_level
# set_log_level("DEBUG") # default is "INFO"

# Load the model
model = Predictor(device="cuda:0", dtype="float16")

# Predict
prediction = model(image)

for k, v in prediction.json_data.items():
    v = str(v)
    if len(v) > 100:
        v = v[:100] + "..."
    print(f"{k}: {v}")


from matplotlib import pyplot as plt

# Plot all the predictions
plt.figure(figsize=(20, 10))
plt.imshow(prediction.plot())
#plt.gca().axis("off")
plt.title("Predictions")




plt.show()

# Plot the cropped images
n_display = min(64, len(prediction))
crops, masks, confidences = prediction.crops[:n_display], prediction.crop_masks[:n_display], prediction.confs[:n_display]

n_crops = len(crops)
ncol = 8
nrow = n_crops // ncol
if n_crops % ncol:
    nrow += 1

fig, axs = plt.subplots(nrow, ncol, figsize=(2.5 * ncol, 2.5 * nrow))
axs = axs.flatten() if n_crops > 1 else [axs]
for ax, crop, mask, conf in zip(axs, crops, masks, confidences):
    crop = crop.permute(1, 2, 0).cpu()
    mask = 1 - mask.squeeze(0).cpu().float()
    w, h = mask.shape
    if w < h:
        mask = mask.transpose(1, 0)
        crop = crop.transpose(1, 0) 
    ax.imshow(crop)
    #ax.imshow(mask, cmap="gray", vmin=0, vmax=1, alpha=mask * 0.5) 
    ax.set_title(f"Confidence: {conf * 100:.2f}%")
    ax.axis("off")
plt.suptitle("Cropped images")
plt.show()

[quitmeyer]

in the predictor initiation, would we just add "model="flat_bug_L.pt" or something?

also @WellReadPanda from our team will be creating an open source dataset of some ground truth stuff from our deployments (we already have 4500 "source images" labelled, and aim to do a lot more) and we would be happy to publish and share it with yall! We don't actually know good practices for publishing the dataset yet (publish in zenodo?) so any guidance will be great and we are happy to help contribute!

[qgeissmann]

Fab, yes, if you have segmentation data, it would be super valuable @quitmeyer! If not, we can help annotate. We probably do not need 4500 (500 max would be good). We can email/meet just for that if you want.
Regarding your question, I tried it with the large model, but the tiny insects are still missed.

You can cheat and upscale images to increase the detection probability of these tiny ones and use e.g., -s 2

[quitmeyer]

our data is a bunch of oriented bounding boxes in yolo format with data. we don't have like masked images like you all do. We can organize and get some stuff to yall in a while!

btw when collecting data like this to train on, im guessing you don't want empty backgrounds (or maybe you do?). But then is it better to have images with LOTS of bugs in one image, or just a couple, or a mix?

also i ran it with this change to try upscaling, but with 2x scale it didn't seem to get the lil buggies. with 4x scale my lil comp couldn't do it. trying now with 3x and it's hilariously slow hahahah
#prediction = model(image)
prediction=model.pyramid_predictions(image, scale_before=2, single_scale=False)

[quitmeyer]

3x unfortunately didnt get them bugs either! so yeah maybe training will be the only way to go?

[asgersvenning]

Hi Andrew,

Thanks for trying flatbug out, looks pretty good!

I'll try to answer your questions one by one :)

1. do you have any suggestions on how i might tweak it to include tiny bugs?
   Yes, as you've already tried increasing the scale_before argument, my best suggestion would probably be to change the confidence threshold hyperparameter SCORE_THRESHOLD with:

   model.set_hyperparameters(SCORE_THRESHOLD=0.01)

   will definitely increase false-positives, but very small insects especially when they are blurry are often given a low confidence by our models for a number of reasons, might improve in the future with more data.
2. if we don't need the masks, is it possible to speed things up by telling it to not compute those masks, or not?
   Yes, definitely this is possible! It doesn't really change much on the inference side of things, since we've spent quite a bit of time optimizing that, but there is a significant overhead in drawing polygons for the overview image and masking the crops. You can disable these like this:

   pred = model(image)
   # First save JSON and crops (without masking), but no overview
   pred.save(output_directory="my_output_dir", overview=False, mask_crops=False)
   # Manually save overview, disable masks/polygons
   pred.plot(masks=False, outpath="my_output_dir/my_overview.jpg")

   which for the random image with ~100 insects I just tested it on reduced from 800ms to 100ms to save the results.
   In general saving the crops can often be much slower than everything else combined, especially when there are many insects in the image. You can integrate your other models with flatbug without storing the crops to disk in between, this can also give a significant speedup, you can get the crops directly from the prediction pred.crops returns a list of uint8 tensors which are already on device in RGB + CHW format, and can be passed directly to other models (also used for the example plot under # Plot the cropped images in the example). A small benchmark could loop over crops at around 10,000 crops/s, while storing crops with masks at around 3,200 crops/s, and 310 crops/s for storing crops with masks. So if you have a classification model in Python doing something like [classification_model(crop) for crop in fb_model(image).crops] might give a significant speedup! PS: I guess we should change "crops" to "detections" ;)
3. btw when collecting data like this to train on, im guessing you don't want empty backgrounds (or maybe you do?). But then is it better to have images with LOTS of bugs in one image, or just a couple, or a mix?
   I think covering the spectrum of insect densities is probably optimal, but weighted significantly towards dense images --- also more fun! Background is especially useful if they have something that might be confused for insects, such as the holes in the poles in your image.
4. are there easy ways to maybe train it on our data to make it more sensitive to those tiny bugs?
   Unfortunately at the moment it is not super duper easy to train it, both because it is inherently more complicated, but also because I haven't had time to make it super user-friendly ;) If you're really into the nitty gritty a good place to start is probably the code for training the models that is used for the preprint and here, but be warned very technical inside 😅.
   We also have currently hard-coded training to remove annotations for instances smaller than 32px (side-length):

   flat-bug/src/flat_bug/datasets.py

   Line 195 in 0f52efe

   _min_size : int=32 # What is the minimum size of an instance to be considered (width or height in pixels after augmentations)

   This could perhaps be changed in the future, but we had trouble ensuring that labels were consistent below that size both between labellers and even for a single person (myself included).