Docs: add testing and CI documentation

app/tests/README.md

@@ -0,0 +1,224 @@
+# Testing & Quality Assurance Documentation
+
+This document outlines the testing strategy, tools and workflows used to ensure the reliability and quality of Perry's codebase. 
+
+Our testing architecture relies heavily on **Pytest** for both unit and integration tests, alongside strict pre-commit hooks and automated CI/CD pipelines.
+
+---
+
+## Testing stack
+
+We use a Python testing stack defined in `app/requirements.txt`:
+
+*   **[`pytest`](https://docs.pytest.org/)**: The core testing framework.
+*   **[`pytest-asyncio`](https://pytest-asyncio.readthedocs.io/)**: For testing asynchronous code (`async`/`await`).
+*   **[`pytest-mock`](https://pytest-mock.readthedocs.io/)**: For mocking dependencies and isolating tests (using `monkeypatch`).
+*   **[`pytest-cov`](https://pytest-cov.readthedocs.io/)**: To measure code coverage.
+*   **[`pytest-dotenv`](https://pypi.org/project/pytest-dotenv/)**: To automatically load `.env` variables during tests.
+*   **[`pytest-testdox`](https://pypi.org/project/pytest-testdox/)**: To generate human-readable terminal output for test results.
+
+Additionally, **code quality** is enforced by **[`Ruff`](https://docs.astral.sh/ruff/)** (linter/formatter) and **[`Safety`](https://github.com/pyupio/safety)** (dependency vulnerability scanner).
+
+---
+
+## Directory structure
+
+The `tests` directory is designed to mirror the structure of the `src` folder. This makes it easy to locate the tests for any given module.
+
+<div align="center">
+  <img src="./img/tests_structure.png" alt="Test Folder Structure">
+  <p><em>Structure of the <code>app/tests/</code> directory</em></p>
+</div>
+
+*   **`agents/`**: Tests for the base agent and specialized agents (CRM, Finance, Calendar, Chat, etc.).
+*   **`controllers/odoo/`**: Tests for data cleaning, LLM routing (`llm_service`) and context handling.
+*   **`guardrails/`**: Tests for output validation and security limits.
+*   **`mcp_server/`**: Client-side tests for FastMCP tools integration.
+*   **`models/`**: Pydantic schema validation tests.
+*   **`routes/`**: API endpoint tests (FastAPI).
+*   **`conftest.py`**: Global test configuration and shared fixtures.
+
+---
+
+## Configuration and Fixtures
+
+### `pytest.ini`
+Our Pytest configuration automatically handles async tests and loads environment variables to ensure local testing doesn't break:
+```ini
+[pytest]
+asyncio_mode = auto
+testpaths = tests
+python_files = test_*.py
+pythonpath = .
+env_override_existing_values = 1
+env_files = ../.env
+```
+
+### Global fixtures (`conftest.py`)
+To avoid repetitive code, we use global fixtures that can be injected into any test:
+*   `client`: Provides an `AsyncClient` to simulate FastAPI requests without running the server.
+*   `agent_state`: Returns a fresh `AgentState` memory block, ensuring isolation between tests.
+*   `sample_message`: Provides a boilerplate `Message` object.
+*   `clean_factory` (autouse): This is a critical automated fixture that **clears the AgentFactory instances dictionary before every test**. This prevents state leakage where an agent initialized in Test A affects the outcome of Test B.
+
+---
+
+## Running the tests
+
+We use a `Makefile` to simplify the execution of test commands from the terminal.
+
+> **Important:** All `make` commands must be executed from within the `ingest-api/app` directory.
+
+1.  **Run all tests (with readable output):**
+    ```bash
+    make test
+    ```
+    *Executes: `pytest --testdox tests/`*
+
+    ![Make Test Output](./img/make-test.png)
+    *Example output of a successful `make test` run*
+
+2.  **Run tests with Coverage Report:**
+    ```bash
+    make test-cov
+    ```
+    *Executes: `pytest --cov=src --testdox tests/`*
+
+    ![Make Test Coverage Output](./img/make-test-cov.png)
+    *Terminal output displaying the test coverage summary*
+
+---
+
+## Pre-commit hooks
+
+Before any code is committed, we enforce quality checks using `.pre-commit-config.yaml`. The pipeline ensures that broken code never reaches the repository.
+
+The hooks run in this order:
+1.  **Ruff Check**: Lints the code for syntax and style errors.
+2.  **Ruff Format**: Automatically formats the Python code.
+3.  **Safety Scan**: Checks for known vulnerabilities in the installed dependencies.
+4.  **Run Tests**: Executes the full Pytest suite. The commit will be aborted if any test fails.
+
+---
+
+## Continuous Integration (GitHub Actions)
+
+We use GitHub Actions to run our test suite automatically on the cloud whenever code is pushed to or merged into the `main` branch. This guarantees that integrations don't break existing features.
+
+![GitHub Actions Workflow](./img/github-actions.png)
+*Working test workflow execution in GitHub Actions*
+
+The workflow (`.github/workflows/test.yml`) sets up Python 3.10, installs the requirements and runs Pytest while injecting necessary secrets:
+
+```yaml
+name: Run tests
+
+on:
+  push:
+    branches: ["main", "master"]
+  pull_request:
+    branches: ["main", "master"]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: app
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Run tests
+        env:
+          LLM_PROVIDER: groq
+          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+        run: |
+          PYTHONPATH=. pytest
+```
+
+### How to add a new environment variable
+
+If you build a new feature or agent that requires a new API key or configuration variable (e.g., `ODOO_API_KEY`), you must update the CI pipeline so the cloud tests don't fail. 
+
+Follow these two steps:
+
+#### 1. Add the Secret to the GitHub Repository
+You need to store the actual value safely in GitHub so it isn't exposed in your code.
+1. Go to the repository on GitHub.
+2. Click on the **Settings** tab.
+3. In the left sidebar, scroll down to the **Security and quality** section.
+4. Click on **Secrets and variables** and then select **Actions**.
+5. Click the green **New repository secret** button.
+6. Set the **Name** (e.g., `ODOO_API_KEY`) and paste the actual key into the **Secret** field. Click **Add secret**.
+
+#### 2. Update the Workflow file
+Now, tell the GitHub Action to pull that secret and inject it into the testing environment. 
+Open `.github/workflows/test.yml` and add the new variable under the `env` block of the "Run tests" step:
+
+```yaml
+      - name: Run tests
+        env:
+          LLM_PROVIDER: groq
+          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+          ODOO_API_KEY: ${{ secrets.ODOO_API_KEY }}    # Inject the new secret here
+        run: |
+          PYTHONPATH=. pytest
+```
+
+---
+
+## Guide: How to add tests for a new file
+
+If you are creating a new feature (e.g., `src/agents/inventory_agent.py`), you must ensure it is covered by tests. Follow these steps:
+
+### Step 1: Create the test file
+Navigate to the corresponding folder inside `app/tests/` (e.g., `app/tests/agents/`) and create a new file. **The file name must start with `test_`** so Pytest can discover it:
+`test_inventory_agent.py`
+
+### Step 2: Import dependencies and fixtures
+Import the module you want to test and any necessary Pytest tools:
+```python
+import pytest
+from src.agents.inventory_agent import InventoryAgent
+
+# Fixtures from conftest.py (agent_state, sample_message) are automatically available 
+# if you pass them as arguments to the test functions
+```
+
+### Step 3: Write the tests
+*   Start the function name with `test_`.
+*   If testing an `async` function, use the `@pytest.mark.asyncio` decorator.
+*   Use `monkeypatch` to mock environment variables or external API calls (e.g., LLM generation) to ensure the test runs quickly and locally.
+
+**Example:**
+```python
+@pytest.mark.asyncio
+async def test_inventory_agent_logic(agent_state, sample_message, monkeypatch):
+    """Test that the inventory agent responds correctly to stock queries."""
+
+    # 1. Arrange (Setup mocks and instances)
+    agent = InventoryAgent(state=agent_state)
+
+    # 2. Act (Execute the logic)
+    agent.receive(sample_message)
+
+    # 3. Assert (Verify the outcome)
+    assert len(agent.state.memory) == 1
+    assert agent.state.memory[0].content == "Perry, show me the inventory"
+```
+
+### Step 4: Run and verify
+Run `make test` or `make test-cov` locally to ensure the new tests pass. 
+
+Once you try to commit, the pre-commit hook will also validate the work automatically.