LED File Copier

Overview

This project recursively copies files from specific folders in "LED Main - 2019" to designated department folders in a new SharePoint site. The tool handles large file transfers by moving files individually while preserving the complete folder structure.

Problem Solved

• Issue: Cannot move large folders from old SharePoint to new SharePoint via web interface due to file size limitations
• Solution: Recursively copy files one by one using Microsoft Graph API, bypassing web interface limitations
• Benefit: Maintains folder structure while handling large files and providing detailed progress tracking

Features

• ✅ Recursive file copying - Preserves complete folder structure
• ✅ Large file handling - Bypasses web interface size limitations
• ✅ Progress tracking - Resume interrupted operations
• ✅ Detailed logging - Complete operation audit trail
• ✅ Failed file reporting - Log files that cannot be moved for manual review
• ✅ Excel mapping - Configure source-to-destination folder mappings

File Structure

LED File Copier/
├── LEDFileCopier.py              # Main application
├── test_first_folder.py          # Test script for single folder
├── requirements.txt              # Python dependencies
├── Onedrive folder mapping.xlsx  # Source to destination mappings
├── .env                         # Environment variables (you create this)
└── README.md                    # This file

Generated during operation:
├── led_file_copier.log                    # Detailed operation log
├── led_copier_progress.json               # Progress tracking (resumable)
└── failed_files_microsoft_ticket.json    # Failed files report

Prerequisites

1. Python Environment

pip install -r requirements.txt

2. Azure AD App Registration

You need the same Azure AD app registration used in the original SharePoint sorter:

• Tenant ID
• Client ID
• Client Secret

3. Environment Variables

Create a .env file in the project directory:

TENANT_ID=your-tenant-id-here
CLIENT_ID=your-client-id-here  
CLIENT_SECRET=your-client-secret-here

4. SharePoint Permissions

Ensure the Azure AD app has:

• Sites.ReadWrite.All - Read/write access to SharePoint sites
• Files.ReadWrite.All - Read/write access to files

Usage

Test with First Folder

Before running the full operation, test with a single folder:

python3 test_first_folder.py

This will copy "2020 LC Spec Sheets" → "LED/Engineering" as a test.

Full Operation

python3 LEDFileCopier.py

The application will:

1. Load folder mappings from Excel file
2. Authenticate with Microsoft Graph API
3. Initialize source and destination SharePoint drives
4. Ask for confirmation before starting
5. Copy all 108 mapped folders recursively
6. Generate detailed reports

Folder Mapping

The Onedrive folder mapping.xlsx file contains 108 mappings with headers:

Current Folder (LED Main - 2019) | Destination

Example mappings (actual data rows):

2020 LC Spec Sheets              → LED/Engineering
Accounting                       → LED/Finance  
Jobs                            → LED/Jobs
B & H Lighting                  → B&H
NRG Hunters                     → NRG

Progress Tracking & Resume

Resumable Operations

The application saves progress to led_copier_progress.json. If interrupted:

• Restart the application
• It will skip already processed files
• Continue from where it left off

Progress Monitoring

During operation, you'll see:

Processing folder mapping: Accounting -> LED/Finance
Found 245 files in Accounting
Successfully copied: LED Main - 2019/Accounting/file1.xlsx -> LED/Finance/file1.xlsx
...
Completed folder copy: Accounting - Success: 243, Errors: 2

Failed Files Logging

Automatic Failed File Logging

Files that cannot be copied are automatically logged to failed_files_microsoft_ticket.json with:

• Complete file paths (source and destination)
• File sizes and metadata
• Error codes and responses
• SharePoint site information
• File categorization (Large file vs Standard file)

Using Failed File Report

When files fail to copy:

1. The application generates failed_files_microsoft_ticket.json
2. Review the file to identify problematic files
3. Use the information for manual intervention or Microsoft Support if needed
4. Large files (>100MB) are flagged for special attention

Failed File Report Structure

{
  "summary": {
    "total_failed_files": 5,
    "error_categories": {
      "Upload failed": 3,
      "Download failed": 2
    },
    "largest_files": [...],
    "instructions": "Use this report to identify files that failed to copy..."
  },
  "failed_files": [
    {
      "source_file_path": "LED Main - 2019/Accounting/large_file.zip",
      "destination_path": "LED/Finance",
      "file_size_mb": 150.5,
      "error_type": "Upload failed",
      "http_status_code": 413,
      "file_priority": "Large file"
    }
  ]
}

Output Files

Operation Logs

• led_file_copier.log - Detailed operation log with timestamps
• led_copier_progress.json - Progress tracking for resume capability
• failed_files_microsoft_ticket.json - Failed files report for manual review

Final Summary

COPY OPERATION COMPLETED
==================================================
Total files processed: 15,847
Successful copies: 15,832
Failed copies: 15
Success rate: 99.9%

⚠️  MICROSOFT TICKET REQUIRED:
Files requiring Microsoft assistance: 15
Largest failed file: 250.3 MB
Failed files report: failed_files_microsoft_ticket.json
Review failed_files_microsoft_ticket.json for files requiring manual attention.

Configuration

SharePoint Sites

Update these constants in LEDFileCopier.py if needed:

SOURCE_SITE = "ledconnection2015.sharepoint.com"
SOURCE_SITE_PATH = "/sites/Documents/LED Main - 2019"  
DEST_SITE = "ledconnection2015.sharepoint.com"
DEST_SITE_PATH = "/sites/DhilenTestProjectPage"

Rate Limiting

The application includes built-in delays:

• 0.5 seconds between individual file copies
• Progress saved every 10 files
• Automatic retry logic for temporary failures

Troubleshooting

Authentication Issues

❌ Failed to authenticate

Solution: Verify .env file contains correct Azure AD credentials

Drive Access Issues

❌ Failed to initialize SharePoint drives

Solution: Ensure Azure AD app has proper SharePoint permissions

Large File Failures

⚠️ FAILED FILE LOGGED: large_file.zip (250.1MB) - Upload failed

Solution: This is expected for very large files - review the failed files report for manual handling

Resume After Interruption

The application automatically resumes by:

1. Loading led_copier_progress.json
2. Skipping already processed files
3. Continuing from last position

Performance Expectations

Typical Performance

• Small files (<1MB): ~2-3 files per second
• Medium files (1-10MB): ~1 file per second
• Large files (>10MB): Several seconds per file
• Very large files (>100MB): May fail and require manual handling

Total Operation Time

• ~16,000 files estimated: 4-8 hours depending on file sizes
• Resume capability: Can run over multiple days if needed
• Progress tracking: Never lose work due to interruptions

Testing Workflow

1. First: Run python3 test_first_folder.py to test with one folder
2. Verify: Check that files copied correctly to LED/Engineering
3. Then: Run python3 LEDFileCopier.py for full operation
4. Monitor: Watch logs and progress files during operation

Support

• Review led_file_copier.log for detailed operation information
• Check failed_files_microsoft_ticket.json for files requiring manual attention
• Use led_copier_progress.json for resume capability status