Allow custom model pricing without modifying source code

[elronbandel]

Closes #106

Summary

Users can now configure custom model pricing without modifying source code by creating a ~/.exgentic/pricing.json file. This solves the problem where every new model required a code change to the name_map in exgentic/utils/cost.py.

The implementation automatically loads the pricing file on first cost lookup and registers all models with LiteLLM's register_model() API. This works seamlessly for both CLI and Python API usage.

Changes

• Added _load_custom_pricing() function in src/exgentic/utils/cost.py that:
  • Reads ~/.exgentic/pricing.json if it exists
  • Validates the JSON structure
  • Registers each model with litellm.register_model()
  • Handles errors gracefully with appropriate logging
  • Uses a global flag to ensure loading happens only once
• Updated error message to mention the new ~/.exgentic/pricing.json config file as the first option
• Added comprehensive tests in tests/utils/test_cost_custom_pricing.py covering:
  • Valid pricing file loading
  • Invalid JSON handling
  • Invalid structure handling
  • Single-load guarantee
  • Integration with litellm_tokens_cost()
  • Error message verification
• Updated documentation in docs/custom-models.md:
  • Added "Option 1: Config file" section with examples
  • Kept existing "Option 2: Python API" section
  • Removed outdated note about CLI limitations

Testing

All tests pass:

• 8 new tests for custom pricing functionality
• All existing cost tests continue to pass
• Pre-commit hooks pass (ruff, codespell, SPDX headers, etc.)

Example Usage

Create ~/.exgentic/pricing.json:

{
  "my-custom-model": {
    "input_cost_per_token": 0.0001,
    "output_cost_per_token": 0.0002,
    "litellm_provider": "openai"
  }
}

Then use the model normally:

exgentic evaluate --benchmark gsm8k --agent tool_calling --model my-custom-model

Cost tracking will work automatically without any code changes.

[elronbandel]

Solid implementation that solves issue #106. The config file approach works for both CLI and Python API. Tests are comprehensive, error handling is appropriate, and there are no breaking changes. Ready to merge.