FormatlyAI

A powerful web scraping and data extraction platform that uses AI to transform web content into structured data formats like CSV, Excel, and JSON.

🚀 Features

• Multi-Source Data Extraction: Extract data from URLs, text files, and web content
• AI-Powered Question Answering: Ask specific questions about content using OpenAI
• Flexible Output Formats: Export data as CSV, Excel, or JSON
• User Authentication: Secure user management with NextAuth.js
• Project Management: Organize and manage your data extraction projects
• Real-time Processing: Asynchronous job processing with status tracking
• Modern UI: Built with Next.js 15, React 19, and Tailwind CSS

🏗️ Architecture

This project consists of two main components:

Frontend (formatlyai/)

• Framework: Next.js 15 with React 19
• Styling: Tailwind CSS with Radix UI components
• Authentication: NextAuth.js
• Database: Prisma with PostgreSQL
• Forms: React Hook Form with Zod validation
• State Management: React hooks and context

Backend (formatlyaibackend/)

• Framework: FastAPI (Python)
• AI Integration: OpenAI GPT models via LangChain
• Web Scraping: HTTPX for async requests
• Data Processing: Custom cleaning and extraction methods
• Job Management: In-memory job queue with status tracking

📋 Prerequisites

• Node.js 18+ and npm
• Python 3.11+
• PostgreSQL database
• OpenAI API key

🛠️ Installation

1. Clone the Repository

git clone https://github.com/Cyro292/formatlyAi.git
cd formatlyAi

2. Frontend Setup

cd formatlyai
npm install

Create environment variables:

cp .env.example .env.local

Configure your .env.local:

# Database
DATABASE_URL="postgresql://username:password@localhost:5432/formatlyai"

# NextAuth
NEXTAUTH_URL="http://localhost:3000"
NEXTAUTH_SECRET="your-secret-key"

# Backend API
NEXT_PUBLIC_BACKEND_URL="http://localhost:8000"

# OpenAI (optional, can be set in backend)
OPENAI_API_KEY="your-openai-api-key"

3. Backend Setup

cd formatlyaibackend
pip install -r requirements.txt

Create environment variables:

cp .env.example .env

Configure your .env:

# OpenAI
OPENAI_API_KEY="your-openai-api-key"
OPENAI_MODEL="gpt-4"

# Server Configuration
HOST="0.0.0.0"
PORT=8000
DEBUG=True

# Scraping Configuration
MAX_URLS=50
GLOBAL_CONCURRENCY=5
TIMEOUT_SECS=30
POLITENESS_DELAY=1
USER_AGENT="FormatlyAI Bot 1.0"

4. Database Setup

cd formatlyai
npx prisma migrate dev
npx prisma generate

🚀 Running the Application

Start the Backend

cd formatlyaibackend
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Start the Frontend

cd formatlyai
npm run dev

The application will be available at:

• Frontend: http://localhost:3000
• Backend API: http://localhost:8000
• API Documentation: http://localhost:8000/docs

📖 Usage

1. Create a Project

1. Navigate to the Formatly page
2. Enter project name and description
3. Choose your data source (URLs, text, or files)

2. Configure Data Sources

URL Sources:

• Individual URLs: Add URLs one by one
• Bulk URLs: Paste multiple URLs separated by newlines
• File upload: Upload a text file containing URLs

3. Define Questions

Add questions to extract specific information:

• Question title (e.g., "What is the main topic?")
• Answer type (text, email, phone, number, date, etc.)
• Additional instructions (optional)

4. Choose Output Format

Select your preferred output format:

• CSV (recommended for spreadsheets)
• Excel (for complex data)
• JSON (for APIs and integrations)

5. Process and Export

Click "Start Processing" to:

1. Scrape the web content
2. Extract answers using AI
3. Generate structured data
4. Download your results

🔧 API Endpoints

Job Management

• POST /jobs/links - Create a URL processing job
• GET /jobs/{job_id} - Get job status
• GET /jobs/{job_id}/result - Get job results

Example Usage

# Create a job
curl -X POST "http://localhost:8000/jobs/links" \
  -H "Content-Type: application/json" \
  -d '{
    "raw_text": "https://example.com",
    "payload": {
      "transformQuestions": [
        {
          "title": "What is the main topic?",
          "type": "text",
          "description": "Extract the main subject"
        }
      ]
    }
  }'

# Check job status
curl "http://localhost:8000/jobs/{job_id}"

# Get results
curl "http://localhost:8000/jobs/{job_id}/result"

🏗️ Project Structure

formatlyAi/
├── formatlyai/                 # Next.js Frontend
│   ├── src/
│   │   ├── app/               # App Router pages
│   │   │   ├── formatly/      # Main application
│   │   │   ├── api/           # API routes
│   │   │   └── auth/          # Authentication
│   │   ├── components/        # Reusable components
│   │   └── lib/               # Utilities and services
│   ├── prisma/                # Database schema
│   └── package.json
├── formatlyaibackend/          # FastAPI Backend
│   ├── app/
│   │   ├── api/               # API routes
│   │   ├── core/              # Configuration
│   │   ├── models/            # Data models
│   │   ├── services/          # Business logic
│   │   └── storage/           # Data storage
│   └── requirements.txt
└── README.md

🔒 Security Features

• Environment Variables: Sensitive data stored in .env files
• Authentication: Secure user sessions with NextAuth.js
• Input Validation: Zod schemas for data validation
• Rate Limiting: Built-in request throttling
• CORS Protection: Configured for secure cross-origin requests

🧪 Development

Running Tests

# Frontend tests
cd formatlyai
npm test

# Backend tests
cd formatlyaibackend
pytest

Code Quality

# Frontend linting
cd formatlyai
npm run lint

# Backend formatting
cd formatlyaibackend
black .
isort .

🚀 Deployment

Frontend (Vercel)

1. Connect your GitHub repository to Vercel
2. Set environment variables in Vercel dashboard
3. Deploy automatically on push to main branch

Backend (Railway/Heroku)

1. Create a new project on Railway or Heroku
2. Connect your repository
3. Set environment variables
4. Deploy the formatlyaibackend directory

Database (Supabase/PlanetScale)

1. Create a PostgreSQL database
2. Update DATABASE_URL in your environment variables
3. Run migrations: npx prisma migrate deploy

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit your changes: git commit -m 'Add amazing feature'
4. Push to the branch: git push origin feature/amazing-feature
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: xxxxxx

🙏 Acknowledgments

• OpenAI for providing the AI capabilities
• Next.js team for the amazing framework
• FastAPI team for the excellent Python web framework
• All contributors and users of this project

---

Made with ❤️ by Cyro