Commit-Time LLM-Based Vulnerability Detection System

A sophisticated automated security analysis system that detects potential vulnerabilities in code changes during the commit process using LLM-based analysis and machine learning models.

Overview

This system integrates with GitHub webhooks to automatically analyze code changes for security vulnerabilities using a combination of:

• CodeBERT Model: Pre-trained machine learning model for insecure code detection
• Google Gemini LLM: Advanced language model for detailed security analysis
• GitHub Integration: Automated commenting and notification system
• Slack Notifications: Real-time security alerts

Architecture

The system follows this workflow:

1. GitHub Webhook receives push/PR events
2. Code Changes Extraction fetches detailed commit information
3. CodeBERT Analysis performs initial ML-based security screening
4. LLM Analysis provides detailed security assessment using Google Gemini
5. Notifications posts results to GitHub and Slack

Prerequisites

• n8n workflow automation platform
• GitHub repository with webhook access
• Google Gemini API key
• Hugging Face API access
• Slack webhook URL (optional)
• GitHub Personal Access Token

Setup Instructions

1. Install n8n

# Using npm
npm install n8n -g

# Using Docker
docker run -it --rm \
  --name n8n \
  -p 5678:5678 \
  -v ~/.n8n:/home/node/.n8n \
  n8nio/n8n

2. Import the Workflow

1. Start n8n and navigate to the web interface (usually http://localhost:5678)
2. Click "Import from file" and select the workflow.json file
3. The workflow will be imported with all nodes and connections

3. Configure Credentials

GitHub API Credentials

1. Go to GitHub Settings → Developer settings → Personal access tokens
2. Generate a new token with repo and write:discussion permissions
3. In n8n, create a new "Header Auth" credential:
   • Name: GitHub API Token
   • Header Name: Authorization
   • Header Value: Bearer YOUR_GITHUB_TOKEN

Google Gemini API Credentials

1. Visit Google AI Studio
2. Create a new API key
3. In n8n, create a new "Google Gemini(PaLM) API" credential:
   • Name: Google Gemini API
   • API Key: YOUR_GEMINI_API_KEY

Hugging Face API Credentials

1. Visit Hugging Face
2. Create a new access token
3. In n8n, create a new "Header Auth" credential:
   • Name: Hugging Face API
   • Header Name: Authorization
   • Header Value: Bearer YOUR_HF_TOKEN

4. Update Node Credentials

Update the following nodes with your credentials:

1. Get Code Changes node → Use GitHub API Token credential
2. Post GitHub Comment node → Use GitHub API Token credential
3. Google Gemini Chat Model node → Use Google Gemini API credential
4. Hugging Face Security Analysis node → Use Hugging Face API credential

5. Configure Slack Webhook (Optional)

1. In your Slack workspace, create a new app
2. Enable Incoming Webhooks
3. Create a webhook URL
4. Update the Slack Notification node URL with your webhook

6. Set Up GitHub Webhook

1. Go to your GitHub repository settings
2. Navigate to Webhooks → Add webhook
3. Configure:
   • Payload URL: https://your-n8n-instance.com/webhook/github-security
   • Content type: application/json
   • Events: Select "Just the push event" or "Pull requests"
   • Secret: (optional) Add a secret for additional security

🔧 Configuration

Environment Variables

Set these environment variables in your n8n instance:

# GitHub Configuration
GITHUB_TOKEN=your_github_token
GITHUB_WEBHOOK_SECRET=your_webhook_secret

# API Keys
GEMINI_API_KEY=your_gemini_api_key
HUGGINGFACE_API_KEY=your_hf_token

# Slack Configuration (Optional)
SLACK_WEBHOOK_URL=your_slack_webhook_url

Customizing Analysis

Modify LLM Prompt

Edit the Basic LLM Chain node to customize the security analysis prompt:

You are an expert software security analyst. Your task is to analyze code changes for potential security vulnerabilities.

You are given two inputs:
1. The changed code snippet: {{ $('Process Code Diff').item.json.files[0].patch }}
2. The CodeBERT model prediction

Instructions:
- Use the code and the CodeBERT prediction together
- Assess whether the code is truly secure or insecure
- If insecure, identify potential security issues and explain why
- If secure, explain why it is safe
- Consider CodeBERT's prediction but do not blindly trust it
- Provide the result in strict JSON format

Response format:
{
  "final_verdict": "SECURE" or "INSECURE",
  "reasoning": "A clear explanation of the potential security risks or why the code is safe.",
  "recommendation": "Optional advice to fix vulnerabilities or improve security, if applicable."
}

Adjust Confidence Thresholds

Modify the Edit Fields node to change sensitivity levels for vulnerability detection.

Usage

Starting the System

1. Activate the Workflow:

   • In n8n, click the "Active" toggle on the workflow
   • The webhook will now be listening for GitHub events

2. Test the Integration:

   • Make a commit to your repository
   • Check the n8n execution log for processing details
   • Verify GitHub comments and Slack notifications

Monitoring

• n8n Dashboard: Monitor workflow executions and debug issues
• GitHub Comments: Review security analysis results on commits/PRs
• Slack Channel: Receive real-time security notifications
• Execution Logs: Detailed logs available in n8n interface

Understanding Results

Security Analysis Output

The system provides structured analysis in this format:

{
  "final_verdict": "INSECURE",
  "reasoning": "SQL injection vulnerability detected in user input handling",
  "recommendation": "Use parameterized queries or input validation"
}

Confidence Scores

• CodeBERT Model: Provides confidence scores (0-1) for secure/insecure classification
• LLM Analysis: Human-readable explanation and recommendations
• Final Verdict: Combined assessment from both models

Security Considerations

API Key Security

• Store API keys securely using n8n credentials
• Use environment variables for sensitive data
• Regularly rotate API keys

Webhook Security

• Use webhook secrets for GitHub integration
• Validate webhook payloads
• Monitor for unauthorized access

Rate Limiting

• Be aware of API rate limits for GitHub, Gemini, and Hugging Face
• Implement appropriate error handling for rate limit exceeded scenarios

Troubleshooting

Common Issues

1. Webhook Not Triggering:

   • Verify webhook URL is accessible
   • Check GitHub webhook delivery logs
   • Ensure n8n instance is running

2. API Errors:

   • Verify API keys are valid and have correct permissions
   • Check rate limits and quotas
   • Review n8n execution logs for detailed error messages

3. Missing Notifications:

   • Verify Slack webhook URL is correct
   • Check GitHub token has comment permissions
   • Review workflow execution status

Debug Mode

Enable debug logging in n8n:

n8n start --debug

Performance Optimization

Scaling Considerations

• Use n8n cloud or self-hosted with proper resources
• Consider queue management for high-volume repositories
• Implement caching for repeated code analysis

Cost Optimization

• Monitor API usage and costs
• Implement smart filtering to reduce unnecessary analysis
• Use appropriate model sizes for your use case

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For issues and questions:

• Check the n8n documentation
• Review GitHub issues
• Contact the development team

---

Note: This system is designed for educational and development purposes. Always perform manual security reviews for production code and critical systems.