Skip to content

STREAMING_GUIDE.md

Latest commit

 

History

History
363 lines (297 loc) · 10.4 KB

STREAMING_GUIDE.md

File metadata and controls

363 lines (297 loc) · 10.4 KB
Raw

🌊 Streaming Hugging Face Model Encoding Guide

This guide shows you how to use the streaming encoder to process Hugging Face models in real-time, which is perfect for:

  • Large models that don't fit in memory
  • Memory-constrained environments
  • Real-time processing applications
  • Progressive model analysis
  • Layer-specific analysis

🚀 Key Advantages of Streaming

Memory Efficiency

  • Process models 10x larger than available RAM
  • Constant memory usage regardless of model size
  • No need to download entire model at once

Real-time Processing

  • Immediate encoding as parameters are loaded
  • Progress tracking with live updates
  • Interruptible processing for long-running tasks

Flexible Analysis

  • Layer-specific extraction (attention, MLP, embeddings)
  • Parameter limits for focused analysis
  • Chunk-based processing for granular control

📋 Quick Start Examples

Basic Streaming

# Stream and encode a model with optimal settings
python examples/streaming_huggingface_encoder.py \
    --model distilbert-base-uncased \
    --stream \
    --max-params 50000 \
    --chunk-size 5000

# Stream a larger model with memory constraints
python examples/streaming_huggingface_encoder.py \
    --model bert-large-uncased \
    --stream \
    --max-params 100000 \
    --chunk-size 10000

Layer-Specific Analysis

# Extract only attention layers
python examples/streaming_huggingface_encoder.py \
    --model bert-base-uncased \
    --stream \
    --layers attention \
    --max-params 30000

# Extract attention and MLP layers
python examples/streaming_huggingface_encoder.py \
    --model gpt2 \
    --stream \
    --layers attention mlp \
    --max-params 40000

# Extract embeddings only
python examples/streaming_huggingface_encoder.py \
    --model roberta-base \
    --stream \
    --layers embedding \
    --max-params 20000

Chunk-Based Encoding

# Encode each chunk as separate video frame (for detailed analysis)
python examples/streaming_huggingface_encoder.py \
    --model distilbert-base-uncased \
    --stream \
    --chunk-encoding \
    --chunk-size 8000 \
    --max-params 40000

# Batch encoding (faster, single video frame)
python examples/streaming_huggingface_encoder.py \
    --model distilbert-base-uncased \
    --stream \
    --max-params 40000 \
    --chunk-size 8000

🔧 Parameter Guidelines

Chunk Size Selection

For optimal results, use these chunk sizes based on your needs:

Use Case Chunk Size Efficiency Ratio Best For
Detailed Analysis 8,000-12,000 High (>0.7) Layer-by-layer study
Balanced Processing 5,000-8,000 Good (>0.6) General model analysis
Fast Processing 3,000-5,000 Acceptable (>0.5) Quick comparisons
Memory Constrained 2,000-3,000 Lower Resource-limited environments

Parameter Limits

Choose parameter limits based on model size and analysis depth:

Model Size Recommended Limit Analysis Depth
Small (< 50M) 30,000-50,000 Comprehensive
Medium (50-200M) 20,000-40,000 Detailed
Large (200M+) 10,000-30,000 Focused
Very Large (1B+) 5,000-15,000 Selective

🎯 Real-World Usage Scenarios

Scenario 1: Large Model Analysis

# Analyze BERT-Large with memory constraints
python examples/streaming_huggingface_encoder.py \
    --model bert-large-uncased \
    --stream \
    --max-params 75000 \
    --chunk-size 7500 \
    --storage-dir bert_large_analysis

# Focus on transformer layers only
python examples/streaming_huggingface_encoder.py \
    --model bert-large-uncased \
    --stream \
    --layers attention \
    --max-params 50000 \
    --chunk-size 10000 \
    --storage-dir bert_large_attention

Scenario 2: Model Family Comparison

# Stream encode GPT family with consistent parameters
python examples/streaming_huggingface_encoder.py \
    --model gpt2 \
    --stream \
    --max-params 40000 \
    --chunk-size 8000 \
    --storage-dir gpt_family

python examples/streaming_huggingface_encoder.py \
    --model distilgpt2 \
    --stream \
    --max-params 40000 \
    --chunk-size 8000 \
    --storage-dir gpt_family

python examples/streaming_huggingface_encoder.py \
    --model microsoft/DialoGPT-medium \
    --stream \
    --max-params 40000 \
    --chunk-size 8000 \
    --storage-dir gpt_family

Scenario 3: Layer-by-Layer Analysis

# Extract different layer types separately
python examples/streaming_huggingface_encoder.py \
    --model bert-base-uncased \
    --stream \
    --layers attention \
    --max-params 30000 \
    --storage-dir bert_attention_analysis

python examples/streaming_huggingface_encoder.py \
    --model bert-base-uncased \
    --stream \
    --layers mlp \
    --max-params 30000 \
    --storage-dir bert_mlp_analysis

python examples/streaming_huggingface_encoder.py \
    --model bert-base-uncased \
    --stream \
    --layers embedding \
    --max-params 20000 \
    --storage-dir bert_embedding_analysis

📊 Monitoring and Analysis

Check Progress

# List all streaming-encoded models
python examples/streaming_huggingface_encoder.py --list-models

# Get detailed statistics
python examples/streaming_huggingface_encoder.py --statistics

Expected Output

📊 STREAMING ENCODING STATISTICS
================================================
Encoded Models: 5
Total Parameters: 185,000
Total Chunks: 23
Average Encoding Time: 3.45s
Average Chunks per Model: 4.6
Chunk Size: 8,000

🔍 Python API Usage

Basic Streaming API

from examples.streaming_huggingface_encoder import StreamingHuggingFaceEncoder

# Initialize encoder
encoder = StreamingHuggingFaceEncoder(
    video_storage_dir="my_streaming_models",
    chunk_size=8000,
    enable_progress=True
)

# Stream encode a model
result = encoder.stream_encode_model(
    "distilbert-base-uncased",
    max_total_params=40000,
    chunk_encoding=False  # Batch encoding
)

print(f"Encoded {result['parameter_count']:,} parameters in {result['encoding_time']:.2f}s")
print(f"Processing rate: {result['parameter_count'] / result['encoding_time']:.0f} params/sec")

Advanced Streaming with Layer Selection

# Stream only attention layers
result = encoder.stream_encode_model(
    "bert-base-uncased",
    target_layers=["attention"],
    max_total_params=30000,
    chunk_encoding=True  # Each chunk as separate frame
)

# Stream with custom progress tracking
for chunk, layer_info, progress in encoder.stream_model_parameters(
    "gpt2",
    target_layers=["mlp", "attention"],
    max_total_params=50000
):
    print(f"Processing {layer_info}: {progress.progress_percent:.1f}% complete")
    # Custom processing here

⚡ Performance Optimizations

Memory Usage

  • Streaming: Constant ~1-2GB memory usage regardless of model size
  • Traditional: Requires full model in memory (5-15GB for large models)
  • Improvement: 5-10x memory reduction

Processing Speed

  • Parameter extraction: 10,000-50,000 params/sec
  • Video encoding: 5,000-15,000 params/sec
  • Total throughput: 3,000-10,000 params/sec

Efficiency Tips

  1. Use optimal chunk sizes (8,000-12,000 for best efficiency ratio)
  2. Target specific layers to reduce processing time
  3. Use batch encoding for faster overall processing
  4. Set appropriate parameter limits based on analysis needs

🛠️ Troubleshooting

Common Issues

Efficiency Ratio Too Low

Error: Efficiency ratio 0.45 is below minimum 0.5

Solution: Increase chunk size to 5,000+ or total parameters to 20,000+

Out of Memory

Error: CUDA out of memory

Solution: Reduce chunk size or set CUDA_VISIBLE_DEVICES=""

Model Not Found

Error: Model not found on Hugging Face Hub

Solution: Check model name or ensure internet connectivity

Performance Issues

Slow Processing

  • Increase chunk size (up to 10,000)
  • Use batch encoding instead of chunk encoding
  • Reduce parameter limits
  • Use fewer target layers

High Memory Usage

  • Decrease chunk size (down to 2,000)
  • Set lower parameter limits
  • Process fewer layers at once

🎉 Complete Workflow Example

Here's a complete workflow for analyzing model families using streaming:

# 1. Create directory for analysis
mkdir transformer_family_analysis
cd transformer_family_analysis

# 2. Stream encode BERT family
python ../examples/streaming_huggingface_encoder.py \
    --model bert-base-uncased \
    --stream --max-params 40000 --chunk-size 8000 \
    --storage-dir bert_models

python ../examples/streaming_huggingface_encoder.py \
    --model distilbert-base-uncased \
    --stream --max-params 40000 --chunk-size 8000 \
    --storage-dir bert_models

python ../examples/streaming_huggingface_encoder.py \
    --model roberta-base \
    --stream --max-params 40000 --chunk-size 8000 \
    --storage-dir bert_models

# 3. Stream encode GPT family
python ../examples/streaming_huggingface_encoder.py \
    --model gpt2 \
    --stream --max-params 40000 --chunk-size 8000 \
    --storage-dir gpt_models

python ../examples/streaming_huggingface_encoder.py \
    --model distilgpt2 \
    --stream --max-params 40000 --chunk-size 8000 \
    --storage-dir gpt_models

# 4. Analyze results
python ../examples/streaming_huggingface_encoder.py \
    --list-models --storage-dir bert_models

python ../examples/streaming_huggingface_encoder.py \
    --statistics --storage-dir bert_models

# 5. Compare with regular encoding for similarity search
python ../examples/huggingface_video_encoder.py \
    --models bert-base-uncased gpt2 \
    --max-params 40000

python ../examples/huggingface_video_encoder.py \
    --search-similar bert-base-uncased

💡 Key Takeaways

✅ Use streaming for large models (>1GB) or memory-constrained environments
✅ Chunk size 8,000-12,000 provides the best balance of speed and efficiency
✅ Target specific layers for focused analysis and faster processing
✅ Batch encoding is faster, chunk encoding provides more granular analysis
✅ Parameter limits 20,000-50,000 work well for most analyses

The streaming encoder opens up massive models for analysis that would otherwise be impossible due to memory constraints! 🚀