Needs some kind of browser/fetch mechanism?

[njbrake]

We need some way to access data like that found on the internet (think mechanics asking for pricing info about parts etc)

How does openclaw do it an how can we make it secure....

Big question

[pankaj-bind]

Built a quick prototype for this. Screenshot attached -- contractor asks for a main breaker panel, agent searches the web and returns results with links.

How it works:

• New web_search tool using [ddgs](https://pypi.org/project/ddgs/) (DuckDuckGo, no API key needed)
• Config: WEB_SEARCH_ENABLED, WEB_SEARCH_MAX_RESULTS, WEB_SEARCH_TIMEOUT_SECONDS
• Tested with Gemini 2.5 Pro on Vertex AI

Limitations:

• No fetch_url tool yet (can't read a specific webpage if contractor shares a link)

Questions:

1. Is fetch_url part of this issue or separate?
2. Any preferred search backend?

@njbrake

[njbrake]

Ooooo thank you for the initial exploration! I think for this one it's really important that we take our time to make sure we get this right: the implementation of itself may not be as difficult as figuring out exactly how we should've wanted to work so that it's secure.

I think this is one piece where Clawbolt can really shine: Santa is being built and expected to be a run by someone working at a very specific trade, odds are that they're generally only going to be looking things up from a few sites. Generally, I think for assistance, it would be impractical to have a gradual whitelisting process, but for clawbolt maybe this is actually ok.

Here's the general flow I'm thinking of:

1. During initial on boarding, they asked what website they normally visit. I think we're expecting that they're going to be dictating it so we would need to translate natural language to actual URLs and have them confirm.
2. Those websites will form the initial whitelist for Web requests.
3. If during usage, they ask for something where it appears that we need to search or access somewhere that's not waitlisted then we need to ask them if they want that new site that we find to be added to the waitlist or if we just want it to be a one time look up.
4. For all web requests, we run the results through a guardrail model from our any-guardrail library to act as another safeguard.

Since we're trying to build something to help out people that may generally be less knowledgeable about tech and AI in general I want to make sure that we make this as safe as possible and make it easy for them to use while also protecting them

[njbrake]

Note: This research summary was generated by Claude Opus via Claude Code, through a conversation between Claude and @njbrake.

Research Summary

Understanding

The agent needs a way to access external web data (parts pricing, supplier catalogs, permit requirements) to answer contractor questions beyond its training data. The key insight from @njbrake's comment is that Clawbolt's contractor-specific vertical makes a per-contractor whitelist approach practical here, unlike a general-purpose assistant where it wouldn't be. The implementation needs to be safe for users who may not be deeply technical.

Relevant Code

• backend/app/agent/tools/registry.py -- Tool factory/registration system. Supports core vs specialist tool distinction, dependency injection via ToolContext, auto-discovery of *_tools modules.
• backend/app/agent/tools/base.py -- Tool and ToolResult dataclasses. All tools require a Pydantic params_model for input validation. Error kinds: VALIDATION, SERVICE, NOT_FOUND, PERMISSION, INTERNAL.
• backend/app/agent/core.py lines 532-770 -- Agent loop. Runs up to 10 tool rounds, validates params via Pydantic, handles specialist activation dynamically.
• backend/app/agent/onboarding.py -- Onboarding flow where trade, location, business hours, etc. are collected. This is where the initial website whitelist conversation would live.
• backend/app/agent/tools/profile_tools.py -- Profile update tools. The whitelist could be stored as a profile field or a related model.
• backend/app/config.py -- Existing llm_provider/llm_model pattern shows how to keep search providers config-swappable. Also has http_timeout_seconds (30s) and max_media_size_bytes (20MB).
• backend/app/models.py -- Contractor model. Would need a new allowed_domains field or a related ContractorWebWhitelist table.

Analysis

Current state: The agent has zero web access. 12 tools across 6 categories, none of which can fetch external data. No MCP integration. No any-guardrail integration yet (planned in architecture docs but not implemented).

How OpenClaw handles it: Three tiers of web access (search, fetch, full browser), all with security concerns. Their security model uses tool deny/allow lists, SSRF validation, and Docker sandbox isolation. Despite all this, they've had multiple CVEs in 2026 and a known prompt injection gap where CSS-hidden text in fetched pages bypasses sanitization (openclaw/openclaw#8027). Their model is "trust the operator" -- our model should be "protect the contractor."

What any-guardrail brings: Mozilla AI's own guardrail library (same stack as any-llm which we already use). Provides a unified interface for content safety: prompt injection detection, toxic content filtering, jailbreak detection. Supports multiple guardrail backends (Deepset, Llama Guard, ShieldGemma, etc.) swappable via config. Running web results through any-guardrail before injecting them into the LLM context would catch prompt injection payloads embedded in web pages.

Suggested Approach

Following @njbrake's proposed flow, with implementation details:

1. Onboarding: collect initial whitelist

During onboarding (after trade/location), ask the contractor what websites they normally use for their work. Since they're likely dictating via voice/Telegram, the agent needs to:

• Accept natural language ("I usually check Home Depot and Ferguson for parts")
• Resolve to actual domains (homedepot.com, ferguson.com)
• Present the resolved list and ask for confirmation before saving
• Store as a per-contractor whitelist (new model or profile field)

2. web_search tool (specialist, whitelist-aware)

• Provider-agnostic search interface, following the llm_provider/llm_model config pattern. OSS users bring their own API key, premium manages it for them.
• Search results filtered/annotated by whitelist status: results from whitelisted domains are fetched directly, others are flagged.
• Pydantic validation on query length and result count.
• Per-contractor daily limit.

3. web_fetch tool (specialist, whitelist-gated)

• Fetches full page content only from whitelisted domains by default.
• If a non-whitelisted URL is needed (contractor shares a link, or search result points to an unfamiliar site), the agent asks: "I found info on partstown.com but it's not on your approved sites list. Want me to add it permanently, or just look it up this one time?"
• HTTPS-only, SSRF protection (block private/internal IPs), response size cap, timeout.
• Content extraction (HTML to readable text) with sanitization.

4. Guardrail pipeline on all web content

Before injecting any web-sourced content into the LLM context:

• Run through any-guardrail to detect prompt injection payloads, toxic content, and other risks.
• If the guardrail flags content, discard it and tell the contractor the page couldn't be safely read.
• This is the defense-in-depth layer beyond the whitelist.

5. OSS vs Premium split

• OSS: All tools, config fields for search provider/API key, web_search_enabled flag (off by default). Self-hosters configure everything themselves. The whitelist is stored per-contractor in the DB.
• Premium: Manages API key and billing. Daily quota tied to subscription tier via existing quota pipeline. Could pre-populate common whitelist entries based on trade during onboarding.

Open Questions

• Should the whitelist be domain-level (homedepot.com) or path-level (homedepot.com/p/electrical/*)? Domain-level is simpler and probably sufficient.
• Is fetch_url part of this issue or should it be a separate issue? (echoing the prototype comment's question)
• For the guardrail step, should we block on any-guardrail integration (not yet implemented) or ship search-only first and add guardrails as a fast-follow?

---

🔍 Automated research by Claude Code

[pankaj-bind]

Thanks for the detailed vision. The whitelist-per-contractor approach is a great fit for this vertical since contractors typically rely on the same handful of supplier sites repeatedly.

A couple of observations from building the prototype:

• DuckDuckGo scraping (ddgs) worked for the demo but it's not stable enough for production. A proper API (Brave, SearXNG self-hosted) would be better. Designing a provider interface early would make swapping easy.
• Search without fetch_url is still useful on its own since the LLM can summarize snippets from search results. Could be a good incremental first step.
• The onboarding whitelist collection via voice/dictation will need careful UX. Contractors saying "Home Depot" needs to reliably resolve to homedepot.com. Might need a confirmation step.

The any-guardrail layer on fetched content is a solid call, especially given the prompt injection risks from raw web pages.

[njbrake]

I just switched over clawbolt away from the ChatCompletions API of any-llm, and over to the Messages API, which should hopefully make tool calling and web fetches a bit cleaner.

For the immediate term (next few weeks) I want to try to focus on ensuring that all of the existing planned features of Clawbolt are working as expected: stuff like the heartbeat/checklist/filemanagement/gdrive/dropbox. Then once we're feeling good about the stability of that, web integration makes sense to pursue 🚀 .

[pankaj-bind]

Makes sense. Stability first, then new surface area.

I'll hold off on the web search PR and look for ways to help with the existing features in the meantime. Let me know if any of the heartbeat/checklist/file management work needs hands.