ai_detection

ai_detection: For QA text dataset (question and human response), generate AI response, perform topic analysis with bertopic, classify between human responses and AI responses, and explain the model with Lime and Shap.

Project Structure

Optional feature description and simple structure
Explore the documentation »

View Demo · Report Bug · Request Feature

Content

• ai_detection
• Content
• Directory Structure
• Getting Started
  • Prerequisites
  • Installation Steps
  • Optional Features
    • Feature 1: Use your dataset
    • Feature 2: Choose Generate AI model
    • Feature 3: Choose classifier model
    • Feature 4: Use the mongodb database to save and load the dataset
• Frameworks Used
• Contributors
• Version Control
• Authors
• License
• Acknowledgements

Directory Structure

filetree 
├── uv.lock
├── pyproject.toml
├── LICENSE.txt
├── README.md
├── /images/
├── /src/
│  ├── /ai_detection/
│  │  ├── /config/
│  │  └── /data/
│  │  └── /model/
│  │  └── /fig/
│  │  └── /result/
│  │  └── /data_loader/
│  │  └── /generator/
│  │  └── /topic/
│  │  └── /classifier/
│  │  └── /explaination/
├── /tests/
└── /util/

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

Installation Steps

1. Install uv from the offical website.

2. Clone the repo

   git clone https://github.com/haowei2000/ai_detection.git

3. Make sure the python version is 3.12.7.

   python --version

   If the python is mot compatible, you can install python environment with pyenv or conda.

4. Install the dependencies with uv

   uv sync

5. Run the project

   • Generate AI response with the dataset in data.yaml and the AI model in generate.yaml

     uv run generate

   • Perform topic analysis with bertopic(topic.yaml), please run the command after generating the AI response.

     uv run topic

   • classify between human responses and AI responses with the dataset in data.yaml and the classifier model in classify.yaml, and make explaination with lime (lime.yaml) and shap (shapshap.yaml)

     uv run classify

   • Run the generate, topic, classify together

     uv run every

Optional Features

Feature 1: Use your dataset

All the data_configurations are stored in the data.yaml.

There are some ways to use your dataset:

1. Local file

   Supported file types: .csv, .json, .txt, .jsonl Open the data.yaml, add a new list item under the data_list key, and specify the path to the file or the data_name(If only specify the data_name, please make sure the {data_name}.{data.type} file in the data_folder).

   For example, if you have a csv file named human_ai_comparsion.csv in the data_folder, you can add the following configuration to the data.yaml:

   - data_type: csv
     data_name: human_ai_comparsion
     file_path: 
     question_column: question
     answer_column: answer
     max_count: 200

2. Huggingface dataset Open the data.yaml, add a new list item under the data_list key, and specify the data_type as huggingface, the data_name as the dataset name.

   If you have a huggingface dataset in https://huggingface.co/datasets/wanghw/human-ai-comparison, you can add the following configuration to the data.yaml:

     - data_type: huggingface
       data_name: human_ai_comparsion
       file_path: wanghw/human-ai-comparison
       question_column: "question"
       answer_column: "answer"
       max_count: 200

3. Optional: you can specify a filter method in [src/ai_detection/data_loader/filter.py] or just change the default filter method in [src/ai_detection/data_loader/data_loader.py]

   def default_filter(self, data: pd.DataFrame) -> pd.DataFrame:
       return data

Feature 2: Choose Generate AI model

There are 2 ways to generate AI response:

1. Use the api in openai format in generate.yaml

   The api is provided by the AI website, you can use the api by setting the api_key, api, model_name in the generate.yaml or local or remote deployment with LLaMA-Factory

2. Use the model in huggingface, you can specify the model_name in the generate.yaml

   For example, if you want to use the THUDM/glm-4-9b-chat-hf model in huggingface, you can add the following configuration to the generate.yaml:

   model_name: THUDM/glm-4-9b-chat-hf

   or download the model to local and setting the model_path

   model_path: "/root/.cache/modelscope/hub/ZhipuAI/glm-4-9b-chat-hf"

Feature 3: Choose classifier model

There are 3 optional classifier model to detect the human response and AI response:

1. Use the sklearn model.

   You can specify the model_name in the classify.yaml

   For example, if you want to use the SVM model and Tf-idf tokenizer in sklearn, you can add the following configuration to the classify.yaml:

   classifier_list:
   - model_name: "SVM"
     tokenizer_name: "TfidfVectorizer"
     split_size:
       train_size: 0.7
       test_size: 0.15
       valid_size: 0.15
     classifier_type: "sklearn"
     model_config:
       gamma: "auto"
     tokenizer_config:
       encoding: "utf-8"

2. Use the model in huggingface

   TODO: add the huggingface model code

3. Use the model in pytorch

   TODO: add the pytorch model code

Feature 4: Use the mongodb database to save and load the dataset

You can use the mongodb database to save and load the dataset, just set the mongodb connection url and the database name in the mongodb.yaml

Frameworks Used

• uv
• Huggingface

Contributors

• Haowei Wang
• Fan Wang

Version Control

This project uses Git for version control. You can check the available versions in the repository.

Authors

E-mail You can also see the list of contributors who participated in this project.

License

This project is licensed under the MIT License. See LICENSE.txt for more information.

Acknowledgements

• GitHub Emoji Cheat Sheet
• Img Shields
• Choose an Open Source License
• GitHub Pages
• Animate.css
• isort
• LLaMA-Factory
• Huggingface
• uv
• Best_README_template