Tip adapter

.github/workflows/check-lint.yml

@@ -20,6 +20,7 @@ jobs:
         uses: astral-sh/setup-uv@v1
         with:
           version-file: "pyproject.toml"
+          version: latest
 
       - name: Sync Dependencies
         run: uv sync --dev

.gitignore

@@ -169,3 +169,11 @@ poetry.toml
 
 # LSP config files
 pyrightconfig.json
+
+# Datasets
+testdata2/
+dataset/
+ds/
+
+# weghts
+weights/

CHANGELOG.md

@@ -13,10 +13,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
    - Utility 'FewShotModel' [5](https://github.com/fundacaocerti/Fewpy/pull/5)
    - Utility 'Preprocessor' [5](https://github.com/fundacaocerti/Fewpy/pull/5)
    - Model 'FPTRANS' [4](https://github.com/fundacaocerti/Fewpy/pull/4)
-   - Model 'anomalyCLIP' [5](https://github.com/fundacaocerti/Fewpy/pull/5)
+   - Model 'AnomalyCLIP' [5](https://github.com/fundacaocerti/Fewpy/pull/5)
    - Model 'AirShot' [6](https://github.com/fundacaocerti/Fewpy/pull/6)
    - Model 'Qwen' [7](https://github.com/fundacaocerti/Fewpy/pull/7)
+   - Utility 'CVATAdapter' [9](https://github.com/fundacaocerti/Fewpy/pull/9)
+   - Documentation 'Fewpy/examples' [9](https://github.com/fundacaocerti/Fewpy/pull/9)
+   - Model 'TipAdapter' [20](https://github.com/fundacaocerti/Fewpy/pull/20)
 
 ### Changed
    - Utility 'FSLDataset' now supports shortest edge resizing [6](https://github.com/fundacaocerti/Fewpy/pull/6)
    - Utility 'FSLDataset' now supports bounding box normalization [7](https://github.com/fundacaocerti/Fewpy/pull/7)
+   - Model 'FPTRANS' had outputs adapted to the libraries new standard [8](https://github.com/fundacaocerti/Fewpy/pull/8)
+   - Model 'AnomalyCLIP' had outputs adapted to the libraries new standard [8](https://github.com/fundacaocerti/Fewpy/pull/8)
+   - Model 'AirShot' had outputs adapted to the libraries new standard [8](https://github.com/fundacaocerti/Fewpy/pull/8)
+   - Model 'Qwen' had outputs adapted to the libraries new standard [8](https://github.com/fundacaocerti/Fewpy/pull/8)
+   - Model 'Airshot' had its weight path search logic improved [9](https://github.com/fundacaocerti/Fewpy/pull/9)
+   - Model 'AnomalyCLIP' had its weight path search logic improved [9](https://github.com/fundacaocerti/Fewpy/pull/9)
+   - Utility 'FSLDataset' is now used directly with torch.utils.data.DataLoader [11](https://github.com/fundacaocerti/Fewpy/pull/11)
+   - Documentation 'Fewpy/examples' cvat exmple import typo fixed [11](https://github.com/fundacaocerti/Fewpy/pull/11)
+   - Model 'Airshot' now supports batched inputs (query) [11](https://github.com/fundacaocerti/Fewpy/pull/11)
+   - Model 'Qwen' now supports batched inputs (query) [11](https://github.com/fundacaocerti/Fewpy/pull/11)
+   - Model 'AnomalyCLIP' fixed prompt learner weight loading isses [11](https://github.com/fundacaocerti/Fewpy/pull/11)
+   - Utility 'FSLDataset' now implements multiple preprocessing routines to be used during inference and fine tuning [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Utility 'FewShotModel' improved to support model fine tuning [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Model 'AnomalyCLIP' predict method updated to support training mode [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Model 'Airshot' predict method updated to support training mode [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Model 'Qwen' predict method updated to support training mode [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Model 'Qwen' supports LoRA and Quantization [17](https://github.com/fundacaocerti/Fewpy/pull/17)
+   - Utility 'FSLDataset' fixed multiprocessing bug when preprocessing support set [20](https://github.com/fundacaocerti/Fewpy/pull/20)
+   - Utility 'FSLDataset' added new preprocessing method to be used with TipAdapter [20](https://github.com/fundacaocerti/Fewpy/pull/20)
+   - Utility 'FSLDataset' uses PreprocessingMethod(Enum) for preprocessing method selection [20](https://github.com/fundacaocerti/Fewpy/pull/20)
+   - Utility 'FSLDataset' fixed bounding box normalization bug [20](https://github.com/fundacaocerti/Fewpy/pull/20)
+   - Model 'FPTRANS' predict updated to support training mode [20](https://github.com/fundacaocerti/Fewpy/pull/20)

README.md

@@ -31,25 +31,27 @@ Fewpy makes it easy to load and run highly performant models on benchmarks to ex
 * **labels**: (list[dict]) Optional annotations (bboxes/classes) for the query images.
 * **img_size**: (tuple/int) Target (H, W) for resizing.
 * **max_size**: (int) Optional maximum edge length constraint for resizing.
+* **antialias**: (bool) If set True Resizing uses antialias.
+* **interpolation**: (torchvision.transforms.InterpolationMode) Interpolation mode used in the resizing operation.
 * **pixel_norm**: (tuple) Mean and Std for normalization.
-* **support_set_preprocessing_method**: (str) Strategy for support set transformation. Options include:
-    * `"standard"`: Basic transform and stacking.
-    * `"resize_annotations"`: Resizes images and mask-style `s_y`.
-    * `"normalize_annotations"`: Scales `s_y` bboxes to [0, 1].
-    * `"detection_crop"`: Crops `s_x` around `s_y` bboxes with padding.
-    * `"norm_detection_crop"`: Padded crop with pixel normalization.
-    * `"resize_labels"`: Resizes both support annotations and query labels.
-    * `"none"`: Skips preprocessing.
+* **center_crop**: (int) Size of the center crop.
+* **support_set_preprocessing_method**: (PreprocessingMethod) Strategy for support set transformation. Options include:
+    * `PreprocessingMethod.STANDARD`: Basic transform and stacking.
+    * `PreprocessingMethod.RESIZE_SUPPORT_GT`: Resizes images and mask-style `s_y`.
+    * `PreprocessingMethod.NORMALIZE_SUPPORT_GT`: Scales `s_y` bboxes to [0, 1].
+    * `PreprocessingMethod.DETECTION_CROP`: Crops `s_x` around `s_y` bboxes with padding.
+    * `PreprocessingMethod.NORM_DETECTION_CROP`: Padded crop with pixel normalization.
+    * `PreprocessingMethod.AUGMENT_SUPPORT_IMAGES`: Creates more support images through augmentation of the support set.
+    * `PreprocessingMethod.NONE`: Skips preprocessing.
 * **transform_datapoints**: (bool) If True, applies the transform pipeline to Query images and scales their associated `labels`.
 
 #### Key Logic
 1. **Transform Pipeline**: A `torchvision.transforms.Compose` pipeline is built using `Resize`, `ToTensor`, and `Normalize` based on initialization arguments.
 2. **Lazy Support Set Preprocessing**:
-   - On the first `__getitem__` call, the dataset executes a specific method determined by `support_set_preprocessing_method`.
+   - On initialization `__init__` call, the dataset executes a specific method determined by `support_set_preprocessing_method`.
    - **Stacking**: Images are stacked into a 4D tensor if shapes are uniform; otherwise, they remain a list of tensors.
    - **Detection Cropping**: Uses `_padded_crop` to zoom into support objects with a 0.5 padding ratio before resizing.
-3. **Query Label Scaling**: If `transform_datapoints` is enabled and `labels` are provided, bounding boxes for query images are automatically rescaled to match the new `img_size`.
-4. **Caching**: Once processed, `self.support_set_preproc` is set to True to bypass reprocessing for subsequent indices.
+3. **Query Label Scaling**: If `transform_datapoints` is enabled and `labels` are provided, bounding boxes and segmentation maps for query images are automatically rescaled to match the new `img_size`.
 
 #### Utility: fsl_collate
 Standard collators stack every element, which can lead to redundant memory usage in FSL because the support set is usually identical for every query in a batch. `fsl_collate` optimizes this by:
@@ -59,15 +61,15 @@ Standard collators stack every element, which can lead to redundant memory usage
 ### Usage Example
 
 ```python
-from torch.utils.data import DataLoader
+from torch.utils.data import DataLoader, PreprocessingMethod
 
 # Initialize with detection-based cropping for the support set
 ds = FSLDataset(
     x=q_imgs, 
     s_x=s_imgs, 
     s_y=s_annots, 
     img_size=(320, 320),
-    support_set_preprocessing_method="detection_crop"
+    support_set_preprocessing_method=PreprocessingMethod.DETECTION_CROP
 )
 
 # Use custom collate to prevent support set duplication in memory
@@ -103,6 +105,7 @@ model = FewShotModel(model="AirShot", config=args)
 #### 1. AnomalyCLIP
 * **Task:** Segmentation
 * **Description:** Implementation of [AnomalyCLIP](https://arxiv.org/abs/2310.18961). Optimized for zero-shot or few-shot anomaly detection and segmentation.
+
 * **Detailed Documentation:** [examples/models/anomalyclip.md](./examples/models/anomalyclip.md)
 
 #### 2. FPTRANS
@@ -120,8 +123,15 @@ model = FewShotModel(model="AirShot", config=args)
 #### 4. Qwen
 * **Task:** Detection
 * **Description:** A wrapper for [Qwen3-VL-8B-Instruct](https://arxiv.org/abs/2505.09388) that enables few-shot prompting for object detection via Vision-Language Models.
+
 * **Detailed Documentation:** [examples/models/qwen.md](./examples/models/qwen.md)
 
+#### 5. TipAdapter
+* **Task:** Classification
+* **Description:** Based on [Tip-Adapter](https://arxiv.org/abs/2207.09519). A training-free Adaption of CLIP for Few-shot Classification.
+
+* **Detailed Documentation:** [examples/models/qwen.md](./examples/models/tipadapter.md)
+
 ## CVAT
 
 Fewpy can be used with CVAT for data annotation. For that purpose inference outputs a dictionary and fewpy implements CVATAdapter. For examples of integration with cvat-cli and CVAT Community refer to [CVAT Examples](./examples/cvat/)

examples/fine_tuning/airshot.py

@@ -1,5 +1,5 @@
 from fewpy.inference import FewShotModel
-from fewpy.util.data import FSLDataset
+from fewpy.util.data import FSLDataset, PreprocessingMethod
 
 from torch.utils.data import DataLoader
 from torch.optim import AdamW
@@ -127,7 +127,7 @@ def fsl_collate(batch):
     img_size=600,
     max_size=1000,
     pixel_norm=((103.530, 116.280, 123.675), (1.0, 1.0, 1.0)),       # (mean, std)
-    support_set_preprocessing_method="detection_crop",
+    support_set_preprocessing_method=PreprocessingMethod.DETECTION_CROP,
 )
 
 dl = DataLoader(

examples/fine_tuning/anomalyclip.py

@@ -1,5 +1,5 @@
 from fewpy.inference import Preprocessor, FewShotModel
-from fewpy.util.data import FSLDataset
+from fewpy.util.data import FSLDataset, PreprocessingMethod
 from fewpy.util.loss import BinaryDiceLoss, FocalLoss
 from torch.utils.data import DataLoader
 
@@ -71,7 +71,7 @@ def collate_fn(batch):
     labels=labels,
     img_size=(IMG_SIZE, IMG_SIZE),
     pixel_norm=((0.485, 0.456, 0.0406), (0.229, 0.224, 0.225)),       # (mean, std)
-    support_set_preprocessing_method="resize_labels",
+    support_set_preprocessing_method=PreprocessingMethod.RESIZE_SUPPORT_GT,
 )
 
 dl = DataLoader(

examples/fine_tuning/fptrans.py

@@ -0,0 +1,228 @@
+from fewpy.inference import FewShotModel
+from fewpy.util.data import FSLDataset, PreprocessingMethod
+
+from pathlib import Path
+import xml.etree.ElementTree as ET
+
+import PIL
+import numpy as np
+
+import torch
+from torch.optim import AdamW
+from tqdm import tqdm
+
+
+def fptrans_collate(batch):
+
+    batch, labels, s_x, s_y = zip(*batch)
+    batch = torch.stack(batch)
+    s_x = torch.stack(s_x)
+    s_y = torch.stack(s_y)
+
+    return batch, torch.stack(labels), s_x, s_y
+
+# prepare support and query data 
+K = 1
+N = 64
+CLASSES = ["bottle",]
+IMG_SIZE = 700
+epochs = 4
+lr = 1e-5
+loss_fn = torch.nn.CrossEntropyLoss(ignore_index=255)
+
+dataset = Path("./ds").expanduser()
+annotations = dataset / "Annotations" 
+images = dataset / "JPEGImages"
+segmentation = dataset / "SegmentationClass"
+
+annotations = list(annotations.glob("*.xml"))
+# print("retrieving", len(annotations), "annotations")
+# print()
+
+indices = np.random.permutation(len(annotations))
+query_targets = []
+query_images = []
+support_images = []
+support_ground_truth = []
+sizes = []
+
+for i in indices:
+    annotation = annotations[i]
+    # print(f"looking into {annotation}")
+
+    tree = ET.parse(annotation)
+    root = tree.getroot()
+    image = images / root.find("filename").text
+    s_yi = segmentation / f"{root.find("filename").text[:-4]}.png"
+    if not s_yi.exists() or not image.exists():
+        # print(f"Segmentation {s_yi} does not exist!\n\n")
+        continue
+    first = True
+    cls = None
+
+    size = root.find("size")
+    width = int(size.find("width").text)
+    height = int(size.find("height").text)
+
+    for obj in root.findall("object"):
+        name = obj.find("name").text
+        if name not in CLASSES:
+            first = True
+            break
+
+        first = False
+        cls = name
+
+    if first == True:
+        # print("no object from class list found!")
+        continue
+
+    if len(support_images) < K:
+        mask = PIL.Image.open(s_yi).convert("L")
+        mask_tensor = torch.from_numpy(np.array(mask))
+        mask_tensor = (mask_tensor > 0).float().unsqueeze(0)
+        support_images.append(PIL.Image.open(image).convert("RGB"))
+        support_ground_truth.append(mask_tensor)
+
+    elif len(query_images) < N:
+        query_images += [PIL.Image.open(image).convert("RGB")]
+        mask = PIL.Image.open(s_yi).convert("L")
+        mask = torch.from_numpy(np.array(mask))
+        mask = (mask > 0).long().unsqueeze(0)
+        query_targets += [mask]
+        sizes.append(query_images[-1].size)
+
+    if len(support_images) >= K and len(query_images) >= N:
+        break
+
+ds = FSLDataset(
+    x=query_images,
+    s_x=support_images,
+    s_y=support_ground_truth,
+    labels=query_targets,
+    img_size=(IMG_SIZE, IMG_SIZE),
+    pixel_norm=((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),       # (mean, std)
+    support_set_preprocessing_method=PreprocessingMethod.RESIZE_SUPPORT_GT,
+)
+
+dl = torch.utils.data.DataLoader(
+    ds, 
+    batch_size=1,
+    shuffle=False,
+    collate_fn=fptrans_collate      # fptrans collate function
+)
+
+"""
+kshot: int, Number of support examples per class in the few-shot setup (e.g., 1-shot learning)
+dataset: str, The dataset being used for training or evaluation (e.g., Pascal VOC)
+backbone: str, The neural network backbone used for feature extraction (Vision Transformer Base)
+split: int, The specific dataset split used for cross-validation/testing
+checkpoint: dict | str | None, Dictionary or path containing pre-trained model weights to load
+Probs_return: bool, Whether the model should return soft probability maps instead of hard logits/masks
+drop_dim: int, The specific tensor dimension along which dropout is applied
+drop_rate: float, The dropout probability rate used to prevent overfitting
+block_size: int, The patch/block resolution for the Vision Transformer (e.g., 16x16 pixel patches)
+height: int, The input image height (often paired with img_size)
+pretrained: str, Path or identifier for the backbone's pre-trained weights (e.g., ImageNet weights)
+SAHI: bool, Flag to enable Slicing Aided Hyper Inference (used for detecting/segmenting small objects)
+bg_num: int, Number of background prototypes or background tokens used to model background context
+bsz: int, The batch size used in SAHI inference
+img_size: int, The overall spatial dimensions to resize the input image to (e.g., 700x700)
+training: bool, Boolean flag indicating whether the network is in training mode or evaluation mode
+vit_depth: int, The specific depth (number of transformer layers) extracted from the ViT backbone
+vit_stride: int, The stride used for ViT patch extraction (affects the sequence length and overlap)
+coco2pascal: bool, Flag for cross-dataset evaluation (e.g., training on COCO, evaluating on Pascal VOC)
+num_prompt: int, The number of learnable visual prompt tokens injected into the transformer
+pt_std: float, Standard deviation for the normal distribution used to initialize the prompt tokens
+"""
+
+args = {
+    "kshot": 1,
+    "dataset": "pascal",
+    "backbone": "VIT-B",
+    "checkpoint": None,
+    "Probs_return": False,
+    "drop_dim": 1,
+    "drop_rate": 0.3,
+    "block_size": 16,
+    "height": IMG_SIZE,
+    "pretrained": "",
+    "SAHI": False,
+    "bg_num": 5,
+    "bsz": 32,
+    "img_size": IMG_SIZE,
+    "training": True,
+    "vit_depth": 10,
+    "vit_stride": 23,
+    "num_prompt": 72,
+    "pair_lossW": 0.02,
+}
+
+model = FewShotModel(
+    model="FPTRANS",
+    config=args
+)
+
+model.train()
+
+for param in model.parameters():
+    param.requires_grad = False
+
+for param in model.model.purifier.parameters():
+    param.requires_grad = True
+
+parameters = [param for param in model.parameters() if param.requires_grad]
+optimizer = AdamW(parameters, lr=lr)
+
+"""
+FPTRANS.predict:
+Args:
+    batched_inputs: a list, batched outputs of :class:`DatasetMapper` .
+        Each item in the list contains the inputs for one image.
+        For now, each item in the list is a dict that contains:
+        * x: Tensor, batch of images in (B, C, H, W) format.
+        * s_x: Tensor, batch of support images in (B, K, C, H, W) format.
+        * s_y: Tensor, batch of ground truth images in (B, K, H, W) format.
+Returns:
+        list[dict]:
+            Each dict is corresponds to the output of a single input image.
+            The dict contains a string "segmentation" under the key "task" to specify the task type,
+            a "data" mask, Tensor of format (H, W) and a "postproc_data" mask, Tensor of format (H, W)
+"""
+for epoch in range(epochs):
+    loss_dict = {
+        'loss': [],
+        'prompt': [],
+        'pair': [],
+    }
+    for batch, labels, s_x, s_y in tqdm(dl):
+        optimizer.zero_grad()
+        # print(f"shapes: batch = {batch.shape}, s_x = {s_x.shape}, s_y = {s_y.shape}")
+        output = model.predict(
+            x=batch,
+            s_x=s_x,
+            s_y=s_y,
+            y=labels,
+        )
+
+        loss_pair = output["loss_pair"]
+        labels = labels.view(-1, *labels.shape[-2:])
+        loss = loss_fn(output["out"], labels)
+        loss_prompt = loss_fn(output["out_prompt"], labels)
+        total_loss = loss + loss_pair + loss_prompt
+
+        total_loss.backward()
+        optimizer.step()
+
+        loss_dict["loss"].append(loss.item())
+        loss_dict["pair"].append(loss_pair.item())
+        loss_dict["prompt"].append(loss_prompt.item())
+
+    print(f"Epoch: {epoch}/{epochs}")
+    print(f"Pairwise Loss: {sum(loss_dict["pair"]) / len(loss_dict["pair"])}")
+    print(f"Prompt Loss: {sum(loss_dict["prompt"]) / len(loss_dict["prompt"])}")
+    print(f"Output Loss: {sum(loss_dict["loss"]) / len(loss_dict["loss"])}")
+    print(f"Total Loss {sum([sum(v) for v in loss_dict.values()]) / len(loss_dict["loss"])}")
+
+print("\nFine-tuning complete!")
+torch.save(model.state_dict(), "fptrans.pth")