🚀 Automated IT Ticket Classification (NLP - English) 🚀

Project Overview

This project aims to revolutionize how IT support tickets are managed by automating the categorization process. In environments with a high volume of calls, manual classification becomes a bottleneck, consuming valuable team time and being prone to inconsistencies.

We developed a robust solution based on Artificial Intelligence (AI) and Natural Language Processing (NLP) to efficiently and accurately classify IT tickets in Portuguese.

Objectives:

• Optimize support team time.
• Improve consistency in ticket categorization.
• Provide insights into the most frequent types of problems.
• Demonstrate the practical application of NLP techniques on real-world data.

How It Works

The classification methodology is divided into key stages:

1. Exploratory Data Analysis (EDA): The journey begins with an in-depth analysis of historical ticket data. This phase is crucial for identifying patterns, common word and phrase correlations, and understanding the nuances of the language used by users to describe their problems. This "data detective" work provides the foundation for creating the category dictionary.

2. N-gram Extraction: Based on the EDA, sequences of words (N-grams) that are strong indicators of specific categories (e.g., "laptop freezing", "internet not working", "reset password") are identified. These N-grams are mapped to their respective categories.

3. Semantic Embeddings with Sentence-Transformer: To go beyond exact N-gram matching, we use a Sentence-Transformer model (specifically trained for Brazilian Portuguese - pt-br). This model transforms entire ticket phrases into numerical "vectors" (embeddings), which capture the semantic meaning and context of the phrase.

4. Similarity Calculation and Category Suggestion: The ticket text vector is compared with the vectors of predefined categories (or reference N-grams). The category with the highest semantic similarity (calculated by metrics like cosine similarity) is suggested as the classification for the ticket.

5. Data Persistence: The categorization results (ticket ID, processed words, found N-grams, maximum similarity, and suggested category) are stored in a PostgreSQL database using SQLAlchemy for ORM management in Python.

Technologies Used

• Python: Main programming language.
• Pandas: For data manipulation and analysis (especially in EDA and ticket loading).
• Sentence-Transformers: Library for generating semantic embeddings.
• SQLAlchemy: ORM (Object-Relational Mapper) in Python for database interaction.
• PostgreSQL: Relational database management system.
• tqdm: For progress bars during large data processing.
• json and os: For configuration file management (N-gram dictionary).
• Sequelize (JavaScript): Used for database migration management (if you maintain the JS migration structure).

Environment Setup

Follow the steps below to set up and run the project on your machine:

1. Clone the Repository:

   git clone [https://github.com/ffcro3/ticket-categorization-nlp-ptbr.git](https://github.com/ffcro3/ticket-categorization-nlp-ptbr.git)
   cd ticket-categorization-nlp-ptbr

2. Create and Activate a Virtual Environment (Recommended):

   python -m venv venv
   # On Windows:
   .\venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install Python Dependencies:

   pip install -r requirements.txt

4. Database Environment Variables (.env): To securely manage your database credentials, create a file named .env in the root directory of your project.

   Add the following variables to your .env file, replacing the placeholder values with your actual PostgreSQL credentials:

   DB_USER=your_database_user
   DB_PASSWORD=your_database_password
   DB_HOST=your_database_host
   DB_PORT=5432
   DB_NAME=your_database_name
   

   These variables will be loaded automatically by the python-dotenv library in your Python scripts.

5. PostgreSQL Configuration:

   • Ensure you have a PostgreSQL server running.
   • Create a database and a user with appropriate permissions for the project.
   • Update the DATABASE_URL variable in src/db_operations.py (or wherever you defined it) with your PostgreSQL credentials:

     DATABASE_URL = "postgresql://your_user:your_password@your_host:your_port/your_database"

6. Apply Database Migrations (Sequelize - if applicable):

   • If you are using Sequelize to manage tables, ensure you have sequelize-cli configured.
   • Execute the migrations to create the tickets and tickets_categorized_tickets tables:

     npx sequelize db:migrate

   • Alternative (SQLAlchemy): If you prefer SQLAlchemy to create the tables, ensure that the Base.metadata.create_all(engine) line is present and executed in your main Python script (main.py or db_operations.py).

7. Prepare the N-gram Dictionary:

   • The N-gram dictionary (ngram_categories.json) is essential for classification. Ensure this file exists in the config/ directory.
   • You can create it manually or use a Python script to save it (as we discussed previously).

Usage

To execute the categorization process:

1. Load your ticket data: Ensure your ticket data is available (e.g., in a Pandas DataFrame dataInfo in your ngrams_data.py script).

2. Run the main script:

   python ngrams_data.py

   The script will load the N-gram dictionary, process your tickets, and insert the categorized results into the PostgreSQL database.

Contribution

Contributions are welcome! Feel free to open issues or pull requests.

License

This project is licensed under the [License Name, e.g., MIT License] - see the LICENSE.md file for details.