Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
9004c81
feat: defining new data structure for Discovery + Extraction of Tenders
BrunoCarusoc May 24, 2026
08ffcdb
feat: added support for transforming HTML Content into Markdown
BrunoCarusoc May 26, 2026
048e108
feat: updated pyproject.toml and new dependencies
BrunoCarusoc May 26, 2026
025e64e
feat: updated portals configuration to do discovery and extraction of…
BrunoCarusoc May 26, 2026
944019f
refactor: added discovery models to models.py
BrunoCarusoc May 26, 2026
0a860f0
feat: added Tenders Overview Extractor Tool
BrunoCarusoc May 26, 2026
5f421b5
fix: small corrections
BrunoCarusoc May 26, 2026
2cf406c
feat: added browser context
BrunoCarusoc May 26, 2026
f5e3478
fix: possible missing value
BrunoCarusoc May 26, 2026
ba9822a
feat: added Single Page Loader Tool
BrunoCarusoc May 26, 2026
99743fc
feat: added new agent
BrunoCarusoc May 26, 2026
743cdd5
feat: added new tasks
BrunoCarusoc May 26, 2026
046c993
feat: added guardrail for single json object
BrunoCarusoc May 26, 2026
87499e2
feat: added new agents and tassk for discovery and extraction
BrunoCarusoc May 26, 2026
f69dc6b
fix: updated import on models.py
BrunoCarusoc May 26, 2026
f510f44
test: disabling all portal except 'TED'
BrunoCarusoc May 27, 2026
0148fb1
feat: update config for discovery and extraction agents/tasks
BrunoCarusoc May 27, 2026
311656a
fear: updated ScoutCrew to include the new agents
BrunoCarusoc May 27, 2026
36a8aa0
refactor: removed unused imports
BrunoCarusoc May 27, 2026
37afa24
fix: add missing modules
Alessandro624 May 27, 2026
bb1d9ec
feat: update scout task prompts
Alessandro624 May 27, 2026
018d05a
fix: async execution failure
Alessandro624 May 27, 2026
adcac0f
feat: update json guardrail extractors
Alessandro624 May 27, 2026
f9af5db
fix: '/n' error in utils
Alessandro624 May 27, 2026
5bea166
fix: extract array from wrapper always returns None
Alessandro624 May 27, 2026
303f91a
fix: update contract parsing in flow
Alessandro624 May 27, 2026
3df6c20
feat: updated resolution agent prompt
BrunoCarusoc May 27, 2026
e999d25
Merge branch 'feat-tender-discovery' of https://github.com/Alessandro…
BrunoCarusoc May 27, 2026
ed27f26
fix: correction on comment
BrunoCarusoc May 27, 2026
8b6bd49
refactor: cleaning task_scout.yaml
BrunoCarusoc May 27, 2026
11f78e1
refactor: cleaning agents_scout.yaml
BrunoCarusoc May 27, 2026
39f8f97
refactor: reordering of tasks definitions
BrunoCarusoc May 27, 2026
5baf9b7
feat: update resultion_task formatting, reordering of tasks
BrunoCarusoc May 27, 2026
576d2f9
feat: update task compliance prompt
Alessandro624 May 27, 2026
97857fb
fix: remove current_date in task scout
Alessandro624 May 27, 2026
0f9d445
feat: update company profile
Alessandro624 May 27, 2026
6b6b30f
fix: remove mock compliance tool
Alessandro624 May 27, 2026
d1821ca
feat: clean up scout crew
Alessandro624 May 27, 2026
1158ac0
feat: clean up crawler tools
Alessandro624 May 27, 2026
3c9ac6d
feat: better contracts parsing in flow
Alessandro624 May 27, 2026
90a2cbf
feat: better json array validation in guardrails
Alessandro624 May 27, 2026
8c3e5a8
feat: add contract loading from output
Alessandro624 May 27, 2026
092ce64
feat: update main with contract loading
Alessandro624 May 27, 2026
2772b26
fix: update implicit no go detection
Alessandro624 May 27, 2026
687a2dc
docs: update documentation
Alessandro624 May 27, 2026
6120060
test: add new tests
Alessandro624 May 27, 2026
4c54632
update flow logging
Alessandro624 May 27, 2026
aa5a893
feat: update pyptoject.toml with new scripts
Alessandro624 May 28, 2026
c8b2805
feat: add html report generation
Alessandro624 May 28, 2026
795ff55
docs: update documentation with new report generation command
Alessandro624 May 28, 2026
34a7d24
test: add report generation unit tests
Alessandro624 May 28, 2026
6a9b66a
fix: update guardrails and utils
Alessandro624 May 28, 2026
b47540b
feat: update flow with specific typing
Alessandro624 May 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Quick links:
| [Characters](docs/guides/characters.md) | Agent personas, roles, behavioral traits |
| [Flow State Machine](docs/reference/flow_state_machine.md) | Every transition mapped with routing logic |
| [Knowledge System](docs/reference/knowledge_system.md) | StringKnowledgeSource, chunking, RAG pipeline |
| [CLI Reference](docs/reference/cli_reference.md) | `bandai`, `bandai --dry-run`, `run_with_trigger`, CrewAI utilities |
| [CLI Reference](docs/reference/cli_reference.md) | `bandai`, `bandai --mode report`, `bandai --dry-run`, CrewAI utilities |
| [Testing](docs/testing.md) | Test strategy, layers, commands, mocking, fixtures |

---
Expand Down Expand Up @@ -163,12 +163,13 @@ bandai --mode scout
# Propose for a known contract (skip scouting)
bandai --mode propose --contract GD-2026-00123

# Generate stakeholder HTML report from output/
bandai --mode report

# Validate config without LLM calls
bandai --dry-run
```

The same runtime is also exposed as `run_with_trigger` for webhook or API-driven execution.

### Tests

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ How the system is built. Start here for the structural overview.
| [main_pipeline.md](architecture/main_pipeline.md) | Flow state machine, execution modes (`full`/`scout`/`propose`), entry points, human interaction points, output files, logging |
| [crews.md](architecture/crews.md) | ScoutCrew, ComplianceCrew, ProposalCrew - agents, task chains, build signatures, shared config |
| [models.md](architecture/models.md) | All 11 Pydantic models: knowledge models, scouting models, compliance models, proposal models |
| [tools.md](architecture/tools.md) | 4 custom CrewAI tools: TenderCrawlerTool, ContractDetailTool, ComplianceCheckerTool, ProposalWriterTool |
| [tools.md](architecture/tools.md) | Runtime CrewAI tools: Playwright discovery/detail loaders and ProposalWriterTool |
| [configuration.md](architecture/configuration.md) | Environment variables, LLM providers, portal YAML, startup validation, NO-GO keywords |

## Guides
Expand Down
20 changes: 11 additions & 9 deletions docs/architecture/crews.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,27 @@ BandAI has three crews, they all use `memory=get_memory()` for cross-session lea
**File:** `crews/scout_crew.py`
**Config:** `config/agents_scout.yaml`, `config/tasks_scout.yaml`

Discovers and deduplicates Italian public tenders across multiple procurement portals. The number of crawler agents is dynamic - one per portal defined in `config/portals.yaml`.
Discovers and deduplicates Italian public tenders across configured procurement portals. For each portal, Scout builds a discovery agent and an extraction agent.

### Agents

| Agent | LLM | Tools | Notes |
| ------------------------ | ------- | -------------------------------- | ------------------------------ |
| Crawler Agent (×N) | fast | TenderCrawlerTool, ContractDetailTool | One per portal, async |
| Resolution Agent | main | ContractDetailTool | `inject_date=True` |
| Discovery Agent (×N) | fast | TendersOverviewExtractorTool | One per portal, async |
| Extraction Agent (×N) | fast | SinglePageLoaderTool | One per portal, reads discovery context |
| Resolution Agent | main | - | `inject_date=True` |
| Preference Filter | main | - | `human_input=True` |

### Task Chain

```text
crawl_task (×N, async) -> resolution_task -> preference_filter_task
discovery_task (×N, async) -> extraction_task (×N) -> resolution_task -> preference_filter_task
```

1. **crawl_task** - Each crawler searches one portal for tenders matching Italian ICT/cloud CPV codes. Returns `RawContract` JSON arrays.
2. **resolution_task** - Deduplicates across portals using Contract ID matching and +/-5% value tolerance. Applies weighted consensus polling using portal reliability scores. Returns `ResolvedContract` JSON array ranked by `value_eur` descending. Guardrail: output must be a raw JSON array starting with `[`.
3. **preference_filter_task** - Filters and re-ranks based on user preferences. Adds `fit_reason` to each entry. Guardrail: strips markdown fences, validates JSON array format.
1. **discovery_task** - Uses Playwright to collect listing-page Markdown and returns `TenderOverview` JSON arrays.
2. **extraction_task** - Loads tender detail pages and returns `TenderInfo` JSON arrays.
3. **resolution_task** - Maps `TenderInfo` records to `ResolvedContract`, deduplicates, and applies portal reliability weights. Guardrail rejects null required fields and placeholder values.
4. **preference_filter_task** - Filters and re-ranks based on user preferences. Adds `fit_reason` to each entry.

### Build Signature

Expand All @@ -50,8 +52,8 @@ Runs a structured advocate/auditor debate to produce a bid verdict for a single

| Agent | LLM | Tools | Notes |
|--------------------|-------|-----------------------|--------------------------------|
| Advocate | main | ComplianceCheckerTool | Optimistic bid manager |
| Auditor | main | ComplianceCheckerTool | Former ANAC inspector |
| Advocate | main | - | Optimistic bid manager |
| Auditor | main | - | Former ANAC inspector |
| Compliance Officer | main | - | `reasoning=True` for synthesis |

### Task Chain (Initial)
Expand Down
6 changes: 5 additions & 1 deletion docs/architecture/main_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,17 @@ BandAI runs a three-phase procurement pipeline orchestrated by a CrewAI Flow. Th
| `full` | Scout -> Compliance -> Proposal | Preferences + conditional review | Complete end-to-end run |
| `scout` | Scout only | Preferences | Quick opportunity discovery |
| `propose` | Compliance -> Proposal | Conditional review | Single known contract, skip scouting |
| `report` | Report generation only | None | Stakeholder review of existing outputs |

Mode is set via `--mode` flag or by calling `BandAIState(mode=...)`.

## Entry Point

`main.py` is a thin wrapper. It validates configuration (provider, API key, knowledge file, portals), parses CLI args, prepares the initial `BandAIState`, then delegates to `BandAIFlow.kickoff(inputs=state.model_dump())`.

For `--mode propose`, the wrapper injects a stub contract into the state before kickoff so the flow can start directly from compliance without running scouting.
For `--mode propose`, the wrapper loads the selected contract from `output/01_scout_results.json` before kickoff so the flow can start directly from compliance without running scouting.

For `--mode report`, the wrapper skips CrewAI entirely and generates `output/report.html` from the JSON artifacts already present in `output/`.

```text
main.py -> _startup_validation() -> BandAIState(...) -> BandAIFlow().kickoff(inputs=state.model_dump())
Expand Down Expand Up @@ -101,6 +104,7 @@ All artifacts land in `output/`:
| `02_compliance_{nn}_{id}.json` | Individual compliance verdicts |
| `02_no_go_review_required.json` | Consolidated NO-GO contracts |
| `03_proposal_{nn}_{id}.json` | Final proposal per contract |
| `report.html` | Stakeholder-ready HTML summary |

## Logging

Expand Down
82 changes: 20 additions & 62 deletions docs/architecture/tools.md
Original file line number Diff line number Diff line change
@@ -1,63 +1,34 @@
# Tools

Four custom CrewAI tools, all in `tools/crawler_tools.py`. All extend `BaseTool` and use Pydantic input schemas.
Runtime CrewAI tools live in `tools/crawler_tools.py`. All extend `BaseTool` and use Pydantic input schemas.

## TenderCrawlerTool
## TendersOverviewExtractorTool

Crawls a procurement portal for matching tenders.
Navigates a configured tender listing portal with Playwright and returns list-page content as Markdown for the Discovery Agent to parse into `TenderOverview` objects.

**Input:** `CrawlerInput`
**Input:** `TendersOverviewExtractorInput`

```text
portal_name: str - Human-readable portal name
base_url: str - Portal entry URL
keywords: list[str] - Search terms (accepts string or JSON array)
max_results: int - Limit, default 10
portal_name: str
tenders_count: int
```

**Output:** JSON array of tender notices with keys: `portal`, `url`, `title`, `contract_id`, `contracting_authority`, `deadline`, `value_euros`, `raw_text`.

**Status:** Stub. Returns mock data with randomized values. The real implementation needs `httpx` + `BeautifulSoup` for portal-specific scraping. The mock is sufficient for end-to-end pipeline testing.

**Used by:** Crawler Agent (Scout crew, one instance per portal).
**Used by:** Discovery Agent in `ScoutCrew`.

---

## ContractDetailTool
## SinglePageLoaderTool

Fetches full metadata for a single tender by Contract ID.
Loads a single tender detail page with Playwright and returns cleaned Markdown for the Extraction Agent to parse into `TenderInfo` objects.

**Input:** `ContractLookupInput`
**Input:** `SinglePageLoaderInput`

```text
contract_id: str - Unique tender identifier
portal_url: str - Source portal URL
portal_name: str
url: str
```

**Output:** JSON with `contract_id`, `source_url`, `completeness_score`, and boolean flags for `has_technical_spec`, `has_admin_clauses`, `has_award_criteria`.

**Status:** Stub. The real implementation should hit the ANAC open data API at `https://dati.anticorruzione.it/opendata/`.

**Used by:** Crawler Agent and Resolution Agent (Scout crew).

---

## ComplianceCheckerTool

Cross-references company profile against tender requirements.

**Input:** `ComplianceInput`

```text
tender_raw_text: str - Full tender text
company_profile_json: str - Company profile as JSON string
```

**Output:** JSON with three lists: `met`, `missing`, `uncertain`.

**Status:** Stub. Returns hardcoded example analysis. The real implementation needs NLP extraction to map tender requirements to company certifications, turnover thresholds, and past contract history.

**Used by:** Advocate and Auditor agents (Compliance crew).
**Used by:** Extraction Agent in `ScoutCrew`.

---

Expand All @@ -68,33 +39,20 @@ Writes a Markdown proposal document to disk.
**Input:** `DocumentGeneratorInput`

```text
title: str - Document title
sections: dict[str, str] - Section name -> Markdown content
output_path: str - File path, default "output/proposal_output.md"
title: str
sections: dict[str, str]
output_path: str
```

**Output:** Confirmation string with the file path, or an error message.

**Behavior:** Creates parent directories automatically (`mkdir(parents=True)`). Overwrites existing files. Writes UTF-8 encoded Markdown.

**Status:** Functional. This is the only tool with a real implementation.

**Used by:** Proposal Architect (Proposal crew).

---

## Input Schema Details

All schemas use Pydantic `Field` descriptions for LLM-readable documentation:
**Behavior:** Creates parent directories automatically and writes UTF-8 Markdown.

- `CrawlerInput` - `keywords` field has a `@field_validator` that accepts either a JSON string or a raw string, converting single values to `["value"]`.
- `ContractLookupInput` - Straightforward key-value lookup.
- `DocumentGeneratorInput` - Default output path is `output/proposal_output.md`.
- `ComplianceInput` - Expects the full company profile as a serialized JSON string, which the agent constructs from knowledge.
**Used by:** Proposal Architect in `ProposalCrew`.

## Adding a New Tool

1. Define a Pydantic input schema in `tools/crawler_tools.py`.
2. Create a class extending `BaseTool` with `name`, `description`, `args_schema`, and `_run()`.
3. Import and assign to the relevant agent via the `tools=[]` parameter.
4. Add tests in `tests/test_config.py` or a new `tests/test_tools.py`.
3. Assign it to the relevant agent only when it provides real runtime behavior.
4. Add focused tests for the schema and behavior.
8 changes: 4 additions & 4 deletions docs/guides/customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ The Scout crew automatically creates a Crawler Agent for each portal. No code ch

**Reliability guidelines:**

- 0.901.00 - Official national/EU sources (ANAC, TED)
- 0.750.89 - Regional procurement platforms (MePA, Sardegna CAT)
- 0.500.74 - Municipal or niche portals
- 0.90-1.00 - Official national/EU sources (ANAC, TED)
- 0.75-0.89 - Regional procurement platforms (MePA, Sardegna CAT)
- 0.50-0.74 - Municipal or niche portals

The Resolution Agent uses reliability as a weighting factor in consensus polling. A portal with 0.70 contributes less to the canonical record than one with 1.00.

Expand Down Expand Up @@ -142,7 +142,7 @@ class RAGSearchTool(BaseTool):
```python
agent = Agent(
...,
tools=[RAGSearchTool(), ContractDetailTool()],
tools=[RAGSearchTool()],
)
```

Expand Down
12 changes: 11 additions & 1 deletion docs/guides/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,15 @@ Discovers tenders and prints results. No compliance or proposal generation.
bandai --mode propose --contract GD-2026-00123
```

Skips scouting. Creates a stub contract with the given ID and runs compliance + proposal phases.
Skips scouting. Loads the contract with the given ID from `output/01_scout_results.json` and runs compliance + proposal phases.

### Report mode

```bash
bandai --mode report
```

Generates `output/report.html` from the current output JSON files. Use it after `full`, `scout`, or `propose` runs to review results in a browser.

### Dry run

Expand All @@ -102,6 +110,7 @@ output/
├── 02_compliance_01_GD-2026-00123.json
├── 02_no_go_review_required.json
├── 03_proposal_01_GD-2026-00123.json
├── report.html
└── proposal_output.md
```

Expand All @@ -122,6 +131,7 @@ uv run pytest_unit
| `bandai` | Run full pipeline |
| `bandai --mode scout` | Scout-only mode |
| `bandai --mode propose --contract ID` | Propose mode for known contract |
| `bandai --mode report` | Generate stakeholder HTML report |
| `bandai --dry-run` | Validate config without LLM calls |
| `crewai test` | Run CrewAI evaluation (2 iterations) |
| `crewai train -n 5` | Train crew with 5 iterations |
Expand Down
24 changes: 18 additions & 6 deletions docs/reference/cli_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,25 @@ bandai --mode scout

### `bandai --mode propose --contract <ID>`

Propose mode for a known contract. Skips scouting, creates a stub contract entry, and runs Compliance → Proposal.
Propose mode for a known contract. Skips scouting, loads the contract from `output/01_scout_results.json`, and runs Compliance → Proposal.

```bash
bandai --mode propose --contract GD-2026-00123
```

`--mode propose` requires `--contract`. Without it, exits with an error.
`--mode propose` requires `--contract`. Without it, or if the contract ID is not present in the previous scout output, exits with an error.

### `bandai --mode report`

Generates a stakeholder-ready HTML report from the JSON files already present in `output/`.

```bash
bandai --mode report
# or
bandai_report
```

Writes `output/report.html`. This mode does not call LLMs and does not require provider/API validation.

### `bandai --dry-run`

Expand All @@ -51,13 +63,13 @@ Exits 0 on success, 1 on any validation failure.

The following commands are declared in `pyproject.toml` and point to the same runtime entry points used by the CLI:

### `run_crew`
### `kickoff`

Alias of `bandai`.
CrewAI Flow-compatible alias for `bandai`.

### `run_with_trigger`
### `plot`

Calls the same runtime as `bandai`. Use this when the pipeline is launched by an external trigger such as a webhook.
CrewAI Flow-compatible script used by `crewai flow plot`.

### `pytest_unit`

Expand Down
Loading
Loading