diff --git a/app/tests/README.md b/app/tests/README.md new file mode 100644 index 0000000..3f458ec --- /dev/null +++ b/app/tests/README.md @@ -0,0 +1,224 @@ +# Testing & Quality Assurance Documentation + +This document outlines the testing strategy, tools and workflows used to ensure the reliability and quality of Perry's codebase. + +Our testing architecture relies heavily on **Pytest** for both unit and integration tests, alongside strict pre-commit hooks and automated CI/CD pipelines. + +--- + +## Testing stack + +We use a Python testing stack defined in `app/requirements.txt`: + +* **[`pytest`](https://docs.pytest.org/)**: The core testing framework. +* **[`pytest-asyncio`](https://pytest-asyncio.readthedocs.io/)**: For testing asynchronous code (`async`/`await`). +* **[`pytest-mock`](https://pytest-mock.readthedocs.io/)**: For mocking dependencies and isolating tests (using `monkeypatch`). +* **[`pytest-cov`](https://pytest-cov.readthedocs.io/)**: To measure code coverage. +* **[`pytest-dotenv`](https://pypi.org/project/pytest-dotenv/)**: To automatically load `.env` variables during tests. +* **[`pytest-testdox`](https://pypi.org/project/pytest-testdox/)**: To generate human-readable terminal output for test results. + +Additionally, **code quality** is enforced by **[`Ruff`](https://docs.astral.sh/ruff/)** (linter/formatter) and **[`Safety`](https://github.com/pyupio/safety)** (dependency vulnerability scanner). + +--- + +## Directory structure + +The `tests` directory is designed to mirror the structure of the `src` folder. This makes it easy to locate the tests for any given module. + +
+ Test Folder Structure +

Structure of the app/tests/ directory

+
+ +* **`agents/`**: Tests for the base agent and specialized agents (CRM, Finance, Calendar, Chat, etc.). +* **`controllers/odoo/`**: Tests for data cleaning, LLM routing (`llm_service`) and context handling. +* **`guardrails/`**: Tests for output validation and security limits. +* **`mcp_server/`**: Client-side tests for FastMCP tools integration. +* **`models/`**: Pydantic schema validation tests. +* **`routes/`**: API endpoint tests (FastAPI). +* **`conftest.py`**: Global test configuration and shared fixtures. + +--- + +## Configuration and Fixtures + +### `pytest.ini` +Our Pytest configuration automatically handles async tests and loads environment variables to ensure local testing doesn't break: +```ini +[pytest] +asyncio_mode = auto +testpaths = tests +python_files = test_*.py +pythonpath = . +env_override_existing_values = 1 +env_files = ../.env +``` + +### Global fixtures (`conftest.py`) +To avoid repetitive code, we use global fixtures that can be injected into any test: +* `client`: Provides an `AsyncClient` to simulate FastAPI requests without running the server. +* `agent_state`: Returns a fresh `AgentState` memory block, ensuring isolation between tests. +* `sample_message`: Provides a boilerplate `Message` object. +* `clean_factory` (autouse): This is a critical automated fixture that **clears the AgentFactory instances dictionary before every test**. This prevents state leakage where an agent initialized in Test A affects the outcome of Test B. + +--- + +## Running the tests + +We use a `Makefile` to simplify the execution of test commands from the terminal. + +> **Important:** All `make` commands must be executed from within the `ingest-api/app` directory. + +1. **Run all tests (with readable output):** + ```bash + make test + ``` + *Executes: `pytest --testdox tests/`* + + ![Make Test Output](./img/make-test.png) + *Example output of a successful `make test` run* + +2. **Run tests with Coverage Report:** + ```bash + make test-cov + ``` + *Executes: `pytest --cov=src --testdox tests/`* + + ![Make Test Coverage Output](./img/make-test-cov.png) + *Terminal output displaying the test coverage summary* + +--- + +## Pre-commit hooks + +Before any code is committed, we enforce quality checks using `.pre-commit-config.yaml`. The pipeline ensures that broken code never reaches the repository. + +The hooks run in this order: +1. **Ruff Check**: Lints the code for syntax and style errors. +2. **Ruff Format**: Automatically formats the Python code. +3. **Safety Scan**: Checks for known vulnerabilities in the installed dependencies. +4. **Run Tests**: Executes the full Pytest suite. The commit will be aborted if any test fails. + +--- + +## Continuous Integration (GitHub Actions) + +We use GitHub Actions to run our test suite automatically on the cloud whenever code is pushed to or merged into the `main` branch. This guarantees that integrations don't break existing features. + +![GitHub Actions Workflow](./img/github-actions.png) +*Working test workflow execution in GitHub Actions* + +The workflow (`.github/workflows/test.yml`) sets up Python 3.10, installs the requirements and runs Pytest while injecting necessary secrets: + +```yaml +name: Run tests + +on: + push: + branches: ["main", "master"] + pull_request: + branches: ["main", "master"] + +jobs: + test: + runs-on: ubuntu-latest + defaults: + run: + working-directory: app + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.10" + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + + - name: Run tests + env: + LLM_PROVIDER: groq + GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }} + run: | + PYTHONPATH=. pytest +``` + +### How to add a new environment variable + +If you build a new feature or agent that requires a new API key or configuration variable (e.g., `ODOO_API_KEY`), you must update the CI pipeline so the cloud tests don't fail. + +Follow these two steps: + +#### 1. Add the Secret to the GitHub Repository +You need to store the actual value safely in GitHub so it isn't exposed in your code. +1. Go to the repository on GitHub. +2. Click on the **Settings** tab. +3. In the left sidebar, scroll down to the **Security and quality** section. +4. Click on **Secrets and variables** and then select **Actions**. +5. Click the green **New repository secret** button. +6. Set the **Name** (e.g., `ODOO_API_KEY`) and paste the actual key into the **Secret** field. Click **Add secret**. + +#### 2. Update the Workflow file +Now, tell the GitHub Action to pull that secret and inject it into the testing environment. +Open `.github/workflows/test.yml` and add the new variable under the `env` block of the "Run tests" step: + +```yaml + - name: Run tests + env: + LLM_PROVIDER: groq + GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }} + ODOO_API_KEY: ${{ secrets.ODOO_API_KEY }} # Inject the new secret here + run: | + PYTHONPATH=. pytest +``` + +--- + +## Guide: How to add tests for a new file + +If you are creating a new feature (e.g., `src/agents/inventory_agent.py`), you must ensure it is covered by tests. Follow these steps: + +### Step 1: Create the test file +Navigate to the corresponding folder inside `app/tests/` (e.g., `app/tests/agents/`) and create a new file. **The file name must start with `test_`** so Pytest can discover it: +`test_inventory_agent.py` + +### Step 2: Import dependencies and fixtures +Import the module you want to test and any necessary Pytest tools: +```python +import pytest +from src.agents.inventory_agent import InventoryAgent + +# Fixtures from conftest.py (agent_state, sample_message) are automatically available +# if you pass them as arguments to the test functions +``` + +### Step 3: Write the tests +* Start the function name with `test_`. +* If testing an `async` function, use the `@pytest.mark.asyncio` decorator. +* Use `monkeypatch` to mock environment variables or external API calls (e.g., LLM generation) to ensure the test runs quickly and locally. + +**Example:** +```python +@pytest.mark.asyncio +async def test_inventory_agent_logic(agent_state, sample_message, monkeypatch): + """Test that the inventory agent responds correctly to stock queries.""" + + # 1. Arrange (Setup mocks and instances) + agent = InventoryAgent(state=agent_state) + + # 2. Act (Execute the logic) + agent.receive(sample_message) + + # 3. Assert (Verify the outcome) + assert len(agent.state.memory) == 1 + assert agent.state.memory[0].content == "Perry, show me the inventory" +``` + +### Step 4: Run and verify +Run `make test` or `make test-cov` locally to ensure the new tests pass. + +Once you try to commit, the pre-commit hook will also validate the work automatically. diff --git a/app/tests/img/github-actions.png b/app/tests/img/github-actions.png new file mode 100644 index 0000000..ae68a5f Binary files /dev/null and b/app/tests/img/github-actions.png differ diff --git a/app/tests/img/make-test-cov.png b/app/tests/img/make-test-cov.png new file mode 100644 index 0000000..04504c4 Binary files /dev/null and b/app/tests/img/make-test-cov.png differ diff --git a/app/tests/img/make-test.png b/app/tests/img/make-test.png new file mode 100644 index 0000000..77cfefd Binary files /dev/null and b/app/tests/img/make-test.png differ diff --git a/app/tests/img/tests_structure.png b/app/tests/img/tests_structure.png new file mode 100644 index 0000000..a88e6aa Binary files /dev/null and b/app/tests/img/tests_structure.png differ