fix: pass num_workers from ccproxy.yaml to litellm CLI#12
Open
npinto wants to merge 124 commits into
Open
Conversation
- Replace argparse with Tyro for type-safe CLI - Use dataclasses for configuration (ProxyConfig, GlobalConfig) - Implement decorator-based subcommand API to avoid 'command:' prefix - Maintain full feature parity with legacy CLI - Add ccproxy-legacy entry point for backwards compatibility - Add type stubs for tyro to ensure mypy compatibility - Update dependencies to include tyro>=0.7.0
- Removed ConfigProvider dependency injection pattern - Updated all usages to use get_config() singleton directly - Fixed thread safety with double-check locking pattern - Fixed all ruff and mypy pre-commit failures - Updated type stubs for external libraries - All tests passing with 92.96% coverage
- Remove Start, Stop, Status commands and CCProxyManager class - Add new LiteLLM command that wraps litellm with config file - Update tests to match new CLI structure - Fix httpx type stub (TimeoutError -> TimeoutException) - Maintain 92.43% test coverage The ccproxy CLI now provides: - `ccproxy litellm [args]` - Run LiteLLM with ccproxy configuration - `ccproxy install` - Install configuration files - `ccproxy run <command>` - Run commands with proxy environment
- Tyro converts PascalCase class names to kebab-case commands - LiteLLM was becoming lite-llm instead of litellm - Changed to Litellm which properly converts to litellm command
- Add detach parameter to Litellm dataclass with -d alias - Implement background process execution with PID file tracking - Save PID to config_dir/litellm.lock - Redirect stdout/stderr to config_dir/litellm.log (non-appending) - Check for existing running process before starting new one - Clean up stale PID files automatically - Add comprehensive tests for detach functionality - All tests passing with 92.08% coverage
- Add Stop dataclass for the stop command - Implement stop_litellm function that: - Checks for litellm.lock PID file - Attempts graceful shutdown with SIGTERM - Falls back to SIGKILL if needed after 0.5s - Cleans up stale PID files - Provides clear user feedback - Add comprehensive tests for all stop scenarios - All tests passing with 92.55% coverage
- Add `ccproxy logs` command to view LiteLLM log file - Support -f/--follow option for tail -f functionality - Support -n/--lines option to control number of lines shown - Use system PAGER for viewing logs (defaults to less) - Add comprehensive tests for all log viewing scenarios - Add rich type stubs for mypy compatibility
- Add defensive type checking to handle both float timestamps and timedelta objects - Fix TypeError when LiteLLM passes timedelta objects instead of float timestamps - Apply fix to async_log_success_event, async_log_failure_event, and async_log_stream_event - Add proper mypy type ignores for operator overloading on union types - Resolves runtime error: type datetime.timedelta doesn't define __round__ method
- Set CCPROXY_CONFIG_DIR environment variable in CLI before starting LiteLLM - Update config loading to check environment variable first, then fallback - Change fallback path from current directory to ~/.ccproxy - Fixes MatchModelRule not matching claude-3-5-haiku-20241022 model name - Resolves issue where proxy_server.config_path is None in LiteLLM runtime
- Add OAuth token forwarding logic in CCProxyHandler.async_pre_call_hook() - Only forward tokens when User-Agent contains 'claude-cli' - Only apply to Anthropic models (anthropic/* or claude*) - Extract OAuth token from secret_fields.raw_headers.authorization - Forward via provider_specific_header.extra_headers.authorization - Add comprehensive edge case handling with null checks - Update all tests to match actual implementation structure - Add tests for edge cases and missing data scenarios This enables Claude Code OAuth tokens (sk-ant-oat01-*) to be properly forwarded to Anthropic's API when using the LiteLLM proxy.
- Add ShellIntegration subcommand to CLI - Auto-detect shell type (bash/zsh) or allow explicit specification - Generate shell scripts that check if LiteLLM proxy is running - Dynamically create/remove 'claude' alias based on proxy status - Support automatic installation to shell config files - Use precmd_functions for zsh and PROMPT_COMMAND for bash - Check proxy status via PID file (litellm.lock) - Add comprehensive tests for shell integration - Update README with shell integration documentation This allows users to automatically have 'claude' aliased to 'ccproxy run claude' whenever the LiteLLM proxy server is running, and removes the alias when stopped.
- Remove YAML loading fallback, use litellm.proxy.proxy_server.llm_router directly - Remove _load_models_from_yaml and _get_fallback_model methods - Simplify fallback to use 'default' labeled model - Move time calculation logic to utils module (calculate_duration_ms) - Remove error message redacting as litellm handles it - Update all tests to mock proxy_server instead of YAML loading - Add test helpers for consistent proxy_server mocking - Fix subprocess.run env parameter in CLI tests - Add rich library type stubs for mypy - Maintain 93% test coverage with 194 passing tests
…nsistency - Remove dead code: ccproxy_get_model function was only used in tests - Refactor tests to use CCProxyHandler.async_pre_call_hook directly - Fix label inconsistency in test_handler_logging.py (large_context → token_count) - Maintain test coverage above 90% (achieved 92.36%) This simplifies the codebase by removing unnecessary backward compatibility and makes tests more direct by using the actual handler methods.
- Modified _determine_routed_model to properly use router's get_model_for_label - Removed hardcoded 'claude-3-5-sonnet-20241022' fallback - Changed fallback to 'unknown' when no model is specified - Added test for behavior when no 'default' model is configured - Fixed test import to include RuleConfig BREAKING CHANGE: When no 'default' label is configured and no rules match, the handler now preserves the original model instead of using a hardcoded fallback
OAuth tokens from Claude CLI are now forwarded based on whether the final routed model is going to the Anthropic provider, not based on the original request. This ensures OAuth tokens are properly forwarded when any model gets routed to Anthropic. - Changed handler.py to check routed_model instead of original_model - Updated test_oauth_forwarding_with_routed_model to verify forwarding - Added test_no_oauth_forwarding_when_routed_to_non_anthropic test
- Create hooks.py module with separate processing hooks: - classify_hook: Handles request classification - rewrite_model_hook: Routes to appropriate model based on label - forward_oauth_hook: Handles OAuth token forwarding for Claude CLI - Remove monolithic _determine_routed_model function - Simplify CCProxyHandler to use hook pipeline pattern - Improve OAuth forwarding to check actual API destination - Update metadata field names for clarity: - ccproxy_original_model → ccproxy_alias_model - ccproxy_routed_model → ccproxy_litellm_model - Add comprehensive tests for new OAuth forwarding logic BREAKING CHANGE: Metadata field names have changed. Update any code that relies on ccproxy_original_model or ccproxy_routed_model fields.
…ivate - Delete unused singleton.py module - Make clear_rules() private by renaming to _clear_rules() - Remove redundant reset_rules() method - Update tests to use private methods appropriately - Remove types.py reference from CLAUDE.md - Fix test mock specifications
- Remove OAuth tokens from logs to prevent credential exposure - Replace hard assertions with safe defaults and error logging - Add exception handling for hook execution to prevent request failures - Update input validation to handle edge cases gracefully Security improvements: - OAuth tokens no longer logged, only auth presence indicated - Failed hooks now logged but don't crash entire request - Invalid inputs handled with warnings and default values BREAKING CHANGE: Hook failures no longer raise exceptions, they log errors and continue processing. This may change error handling behavior for custom hooks that expect exceptions to propagate.
fix: address security, performance, and accuracy issues - Performance: add debug check for rich console output to reduce latency - Security: fix OAuth domain validation to prevent subdomain attacks - Accuracy: implement tiktoken-based token counting for better precision - Tests: update tests to work with realistic token counting behavior - Fix: improve exception handling and add tiktoken type stubs Fixes handler.py:94-127, hooks.py:74-83, rules.py:62-72 EOF )
- Add comprehensive documentation for config precedence order - Clarify priority: ENV > proxy dir > ~/.ccproxy (fallback) - Add structured logging for config source discovery - Improve error handling and debugging information Fixes config.py:166-205 unpredictable behavior issue
- Update test data to use realistic text patterns for accurate token counting - Replace simple repeated characters with varied sentences for proper tokenization - Add comprehensive tests for calculate_duration_ms utility function - Remove obsolete test_env.py and empty test_handler_temp.py files - Fix string formatting style in utils.py error message
- Add capture_headers hook for logging HTTP headers with sensitive redaction - Add SENSITIVE_PATTERNS for authorization, x-api-key, cookie redaction - Update forward_oauth to use new multi-provider OAuth system
- Replace deprecated 'credentials' with 'oat_sources' - Add capture_headers hook - Add example for extended OAuth config with user_agent
- Add hook parameter support via dict format in ccproxy.yaml
- Hooks can now specify params: { headers: [...] } for hook-specific configuration
- capture_headers hook now accepts optional headers filter parameter
- ccproxy status command now displays hooks table with parameters
- ccproxy status command now displays model deployments table from LiteLLM config
- Model aliases (e.g., 'default') resolve to target model's API base for display
Eliminates the need to manually type '--' when running commands with flags: Before: ccproxy run -- claude -p foo After: ccproxy run claude -p foo The entry_point now intercepts argv and automatically inserts '--' after 'run' to prevent tyro from parsing command arguments as ccproxy flags. Backwards compatible - explicit '--' is still supported.
Remove tests that asserted old Anthropic-only OAuth forwarding behavior. The new implementation uses get_llm_provider for multi-provider support. Changes: - Remove TestCredentialsLoading (obsolete 'credentials' field) - Remove tests asserting OAuth NOT forwarded for non-Anthropic - Update credential fallback tests to use oat_sources API - Fix CLI tests to handle full litellm executable path - Fix test_multiple_providers to use passthrough mode
Add 22 tests covering: - Basic header capture with and without filtering - Case-insensitive header filtering - Sensitive header redaction (authorization, x-api-key, cookie) - Long header value truncation - HTTP method and path extraction - Raw headers from secret_fields merging - Edge cases (empty headers, missing metadata, etc.) Coverage: hooks.py 75% → 97%, total 78.65% → 81.55%
LiteLLM doesn't preserve custom metadata from async_pre_call_hook to logging callbacks. Implemented thread-safe global store to pass trace_metadata between callbacks, then update LangFuse traces directly via SDK in async_log_success_event.
Claude Code embeds session info in the metadata.user_id field with format:
user_{hash}_account_{uuid}_session_{uuid}
The extract_session_id hook parses this and sets:
- metadata["session_id"] for LangFuse session grouping
- trace_metadata["claude_user_hash"] and ["claude_account_id"]
Detect user's custom ccproxy.py files and skip auto-generation to avoid overwriting. Files containing "Auto-generated handler file" marker are safe to overwrite; all others are preserved with a warning panel. - Check for auto-generated marker before overwriting - Display rich warning panel with instructions - Suggest removing file for auto-generation or setting handler config - Add comprehensive tests for all scenarios
Enables external tools to detect ccproxy context by comparing ANTHROPIC_BASE_URL against the proxy URL, preventing infinite recursion in wrapper scripts.
Health checks are tagged with "litellm-internal-health-check" in metadata. When ccproxy's hooks rewrite model names, health checks validate the wrong model, causing failures. Now we detect health check requests early and return unmodified data, allowing LiteLLM to validate actual configured models.
Add critical note that the project name is `ccproxy` (lowercase), not "CCProxy". PascalCase is reserved for class names only.
Add ML artifacts and Prisma ignores from feat/logging and feat/mitm.
- Replace deprecated `credentials` with `oat_sources` (multi-provider OAuth) - Add Hooks section documenting all 6 built-in hooks - Update dev workflow to use `uv tool install --editable` - Document custom handler file preservation - Add `ccproxy status --json` to CLI docs
- Add Request Lifecycle mermaid diagram to Development section in README - Document hook parameters support in CLAUDE.md and docs/configuration.md - Rename Development Setup → Development with Local Setup subsection
Add add_beta_headers hook that adds required anthropic-beta headers for Claude Code impersonation, enabling Claude Max OAuth tokens to work through the proxy. Headers added: - oauth-2025-04-20 - claude-code-20250219 - interleaved-thinking-2025-05-14 - fine-grained-tool-streaming-2025-05-14
Released 2026-02-05. Claude Code 2.1.37+ requests this model.
feat: add claude-opus-4-6 to model config
The `litellm.num_workers` setting in `ccproxy.yaml` was silently ignored. `start_litellm()` built the litellm command without reading this value, so the proxy always started with litellm's default worker count regardless of what was configured. Read `num_workers` from the `litellm:` section of `ccproxy.yaml` and pass it as `--num_workers` to the litellm subprocess when set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
The
litellm.num_workerssetting inccproxy.yamlis silently ignored.start_litellm()builds the litellm command aslitellm --config <path>without readingnum_workersfrom the config, so the proxy always starts with litellm's default worker count.Template suggests it works:
Actual command (before fix):
Expected command (after fix):
Fix
Read
num_workersfrom thelitellm:section ofccproxy.yamland append--num_workers Nto the litellm subprocess command when configured. When omitted, litellm uses its own default (CPU count or 4).9 lines in
src/ccproxy/cli.py.Test
One unit test following the existing
@patch("subprocess.run")pattern intest_cli.py: write accproxy.yamlwithnum_workers: 8, callstart_litellm(), assert--num_workers 8appears in the subprocess command.Fails on current HEAD, passes with fix. Zero regressions on existing tests.