MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation

• Paper: MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation

🧠 Overview of MemeSense Workflow

MemeSense is a multi-stage framework for reasoning about implicitly harmful memes, leveraging diverse commonsense cues and Cognitive Shift Vectors (CSVs) for robust intervention generation.

The overall process consists of the following steps:

1. Commonsense Generation Module
   Train a commonsense generation model using a set of memes and their corresponding ground-truth commonsense annotations.

   • These commonsense labels are obtained via GPT-4o generations, followed by manual verification.
   • Refer to commonsense_generation/ for data format and training scripts.

2. Intervention Training via Cognitive Shift Vectors
   Use the curated meme–commonsense–intervention triplets to train the CSV-enhanced intervention generation model.

   • CSVs are learnable representations that capture how commonsense should influence the final response.
   • Training and inference code is available in cognitive_shift_vectors/.

3. Evaluation and Inference
   Once trained, the model can generate reasoned intervention responses for new memes — including those lacking textual cues.

   • Evaluation metrics include Semantic Similarity Score (SeSS), BERTScore, ROUGE, BLEU, and Readability.

⚙️ This modular design ensures flexibility — allowing users to integrate new memes, customize commonsense categories, or plug in alternate base models.

Install

conda create -n memesense python=3.10

conda activate memesense
pip install -r requirements.txt

# For Openflamingo, please use transformers==4.28.1 [beta]

pip install transformers==4.48.1 [tested]

# Install the lmm_icl_interface
pip install git+https://github.com/ForJadeForest/lmm_icl_interface.git
# Install the baukit
pip install git+https://github.com/davidbau/baukit.git

📌 Citation

If you use this work, please cite our paper:

@misc{adak2025memesenseadaptiveincontextframework,
      title={MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation}, 
      author={Sayantan Adak and Somnath Banerjee and Rajarshi Mandal and Avik Halder and Sayan Layek and Rima Hazra and Animesh Mukherjee},
      year={2025},
      eprint={2502.11246},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2502.11246}, 
}