Skip to content

LOCAL_TESTING.md

Latest commit

 

History

History
557 lines (396 loc) · 9.87 KB

LOCAL_TESTING.md

File metadata and controls

557 lines (396 loc) · 9.87 KB

🧪 Local Testing Guide for FlowLLM

Before You Start - Security First! 🔒

IMPORTANT: This is a PUBLIC repository. Read SECURITY.md first!

✅ Ensure .env files are in .gitignore
✅ Never commit API keys
✅ Use environment variables only

Quick Start - Test in 5 Minutes

Step 1: Verify Prerequisites

# Check Node.js version (need 18+)
node --version

# Check npm version (need 9+)
npm --version

If you don't have them, install from nodejs.org

Step 2: Clone and Install

# Navigate to project
cd /Users/n0r0bhn/Documents/flowllm

# Install dependencies
npm install

This will install all dependencies for all packages in the monorepo.

Step 3: Build the Project

# Build all packages
npm run build

This compiles TypeScript to JavaScript. You should see output like:

@flowllm/core: Build success
@flowllm/providers: Build success
@flowllm/mcp: Build success
flowllm: Build success

Step 4: Run Tests

# Run all tests
npm test

All tests should pass. If any fail, check the error messages.

Step 5: Set Up API Keys (for examples)

# Navigate to examples
cd examples

# Copy the environment template
cp .env.example .env

# Edit .env and add your REAL API keys
# Use nano, vim, or any text editor
nano .env

Add your keys:

OPENAI_API_KEY=your-openai-api-key-here
ANTHROPIC_API_KEY=sk-ant-YOUR-REAL-KEY-HERE
GOOGLE_API_KEY=YOUR-GOOGLE-KEY-HERE

Save and exit (Ctrl+O, Enter, Ctrl+X for nano)

Step 6: Test an Example

# Still in examples directory
npm run basic

You should see the agent respond to a query!

Detailed Testing Steps

1. Test Core Package

cd packages/core

# Run tests
npm test

# Run tests with coverage
npm test -- --coverage

# Watch mode (auto-rerun on changes)
npm run test:watch

What to check:

  • ✅ All tests pass
  • ✅ Memory management works
  • ✅ Tool registry functions correctly
  • ✅ No TypeScript errors

2. Test Providers Package

cd packages/providers

# Type check
npm run typecheck

# Build
npm run build

What to check:

  • ✅ TypeScript compiles without errors
  • ✅ All provider exports work
  • ✅ No import errors

3. Test MCP Package

cd packages/mcp

# Type check
npm run typecheck

# Build
npm run build

What to check:

  • ✅ TypeScript compiles
  • ✅ MCP SDK imports work
  • ✅ No dependency errors

4. Test Main Package

cd packages/flowllm

# Type check
npm run typecheck

# Build
npm run build

What to check:

  • ✅ All re-exports work
  • ✅ High-level APIs are accessible
  • ✅ No circular dependencies

Testing Examples

Test Each Example Individually

cd examples

# Test basic agent
npm run basic

# Test custom tools
npm run tools

# Test streaming
npm run stream

# Test conversations
npm run conversation

# Test multi-provider (needs all API keys)
npm run providers

# Test MCP integration (needs MCP server)
npm run mcp

Expected Output Examples

basic-agent.ts:

🤖 Agent initialized!

📝 Response: A REST API (Representational State Transfer API)...
💰 Cost: $0.000150
🔢 Tokens: 245
⏱️  Time: 1234ms

streaming.ts:

🤖 Streaming agent initialized!

📖 Generating a story...

Once upon a time, there was a robot named Pixel...
[text streams in real-time]

✅ Streaming complete!

Testing Without API Keys

You can test the framework without API keys using mock providers:

Create a Test File

// test-without-keys.ts
import { defineAgent, defineTool } from '../packages/flowllm/src';
import type { LLMProvider, Message, LLMConfig, LLMResponse, StreamChunk } from '../packages/core/src';

// Mock provider - NO API KEY NEEDED
class MockProvider implements LLMProvider {
  name = 'mock';
  
  async chat(messages: Message[], config: LLMConfig): Promise<LLMResponse> {
    return {
      content: 'Mock response from the agent',
      role: 'assistant',
      finishReason: 'stop',
      usage: { promptTokens: 10, completionTokens: 20, totalTokens: 30 },
      model: 'mock-model',
    };
  }
  
  async *stream(messages: Message[], config: LLMConfig): AsyncIterable<StreamChunk> {
    const words = ['Hello', ' from', ' mock', ' provider'];
    for (const word of words) {
      yield { delta: { content: word } };
      await new Promise(r => setTimeout(r, 100));
    }
  }
  
  countTokens(text: string): number {
    return Math.ceil(text.length / 4);
  }
  
  getMaxTokens(model: string): number {
    return 4096;
  }
  
  getCostPerToken(model: string) {
    return { prompt: 0, completion: 0 };
  }
}

// Test it!
const agent = defineAgent({
  provider: new MockProvider(),
  systemPrompt: 'You are a test agent.',
});

const response = await agent.execute('Test message');
console.log('✅ Mock test passed:', response.content);

Run it:

npx tsx test-without-keys.ts

Verify Security

Check .gitignore

# This should return nothing (no .env files tracked)
git status | grep ".env"

# This should show .env is ignored
git check-ignore .env examples/.env

Scan for Secrets

# Check if any API keys are in tracked files
git grep -i "sk-proj"  # Should find nothing
git grep -i "sk-ant"   # Should find nothing
git grep -i "AIza"     # Should find nothing

# Check staged files before commit
git diff --staged | grep -i "api"

Create Pre-commit Hook

# Create the hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
echo "🔍 Running security checks..."

# Check for API keys
if git diff --cached --name-only | xargs grep -h "sk-[Pp]roj-\|sk-[Aa]nt-\|AIza" 2>/dev/null; then
    echo "❌ ERROR: Found API key in staged files!"
    exit 1
fi

# Check for .env files
if git diff --cached --name-only | grep "^\.env$" 2>/dev/null; then
    echo "❌ ERROR: .env file is staged!"
    exit 1
fi

echo "✅ Security checks passed"
exit 0
EOF

# Make it executable
chmod +x .git/hooks/pre-commit

echo "✅ Pre-commit hook installed!"

Common Issues & Solutions

Issue: "npm: command not found"

Solution:

# Install Node.js and npm from nodejs.org
# Or use a version manager like nvm:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
nvm install 18
nvm use 18

Issue: "Cannot find module"

Solution:

# Clean and reinstall
rm -rf node_modules package-lock.json
npm install
npm run build

Issue: "TypeScript errors"

Solution:

# Check TypeScript version
npm list typescript

# Rebuild everything
npm run clean
npm run build

Issue: "API key not found"

Solution:

# Make sure .env exists
ls -la examples/.env

# Check .env content (safely)
grep "OPENAI_API_KEY" examples/.env | head -c 30

# Restart your terminal to reload environment

Issue: "Tests failing"

Solution:

# Run with verbose output
npm test -- --reporter=verbose

# Run specific test file
npm test -- memory.test.ts

# Clear cache
npm test -- --clearCache

Development Workflow

Making Changes

# 1. Create a feature branch
git checkout -b feature/my-feature

# 2. Make your changes
# Edit files...

# 3. Run tests
npm test

# 4. Build
npm run build

# 5. Test examples
cd examples && npm run basic

# 6. Check for errors
npm run typecheck
npm run lint

# 7. Commit (pre-commit hook will run)
git add .
git commit -m "feat: add my feature"

# 8. Push
git push origin feature/my-feature

Before Pushing to GitHub

CRITICAL CHECKLIST:

# 1. Run all tests
npm test

# 2. Build everything
npm run build

# 3. Check for secrets (IMPORTANT!)
git grep -i "sk-proj"
git grep -i "sk-ant"
git grep -i "AIza"

# 4. Verify .env is ignored
git status | grep ".env"

# 5. Review all changes
git diff origin/main

# 6. Double-check staged files
git diff --staged

# Only push if ALL checks pass!
git push origin main

Testing Checklist

Use this before committing:

  • All packages build successfully (npm run build)
  • All tests pass (npm test)
  • TypeScript has no errors (npm run typecheck)
  • Examples work with real API keys
  • No .env files in git status
  • No API keys in code (git grep -i "sk-")
  • Pre-commit hook is installed
  • .gitignore includes .env*
  • Documentation is updated
  • CHANGELOG.md is updated (if needed)

Performance Testing

Measure Response Times

// performance-test.ts
import { defineAgent, openai } from 'flowllm';

const agent = defineAgent({
  provider: openai('gpt-4o-mini'), // Use cheaper model
});

const start = Date.now();
const response = await agent.execute('Say hello');
const duration = Date.now() - start;

console.log(`Response time: ${duration}ms`);
console.log(`Tokens: ${response.usage.totalTokens}`);
console.log(`Cost: $${response.cost.totalCost.toFixed(6)}`);

Load Testing

// load-test.ts
import { defineAgent, openai } from 'flowllm';

const agent = defineAgent({
  provider: openai('gpt-4o-mini'),
});

const promises = Array.from({ length: 10 }, (_, i) =>
  agent.execute(`Test message ${i}`)
);

const start = Date.now();
const results = await Promise.all(promises);
const duration = Date.now() - start;

console.log(`10 requests completed in ${duration}ms`);
console.log(`Average: ${duration / 10}ms per request`);

Next Steps

Once local testing is complete:

  1. ✅ Read SECURITY.md thoroughly
  2. ✅ Set up pre-commit hooks
  3. ✅ Test all examples locally
  4. ✅ Verify no secrets in git
  5. ✅ Create a .github/workflows for CI/CD
  6. ✅ Push to GitHub
  7. ✅ Set up GitHub Secrets for CI/CD

Need Help?

Happy Testing! Remember: Security First! 🔒