3.5. Add LLM chat over all certificates

[MaryBak]

Description
Add LLM chat over all certificates or a user-defined subset. This should either be based on some RAG solution or an LLM with custom tools to search and access certificates. @J08nY has worked on this and there is a prototype.

Additional context
CCAT project

[J08nY]

Let me give a summary of what I have tried, what worked and what did not.

Current chat feature

Currently, we are using the chat.ai.e-infra.cz backend provided by MetaCentrum (see the docs). This gives us a few open-weight models that they are running. So far our usage is quite limited so they are okay with this, though they intended this to be for personal use only (by researchers) so this may change in the future or if our usage increases.

The API they offer is the usual chat/completions API but also the OpenWebUI API. However, it seems they retired offering the OpenWebUI API as I am seeing errors from that. I am working on this with them, so lets see. We can always tun our own instance of OpenWebUI and point it at their upstream for inference, it would just be more maintenance and deployment issues.

The current feature is limited to one certificate. There are 4 options the user can choose from:

• Full context (both): Both the certification report and security target documents are stuffed (in text form) into the context as part of the system prompt.

  sec-certs/config/example.config.py

  Line 63 in 09cca5c

  WEBUI_PROMPT_CC_CERT_FULL = "You are answering questions about a Common Criteria certificate '{{ cert_name }}'.\n {% if report %}Here is the certification report of the certificate: <document>{{ report }}</document>{% endif %}\n {% if target %}Here is the security target of the certificate: <document>{{ target }}</document>{% endif %}"

  
  For most certificates, the context windows of the models are large enough that they fit.
• Full context (report): The same as above, but only the certification report is used.
• Full context (target): The same as above, but only the target is used.
• (RAG) This certificate: This uses the OpenWebUI RAG on the certificate documents. It requires that the certificate documents were previously uploaded to the OpenWebUI backend (and processed there). The model does not get the full document but the users message(query) is embedded and only similar chunks are retrieved from the document. This can support any size of the documents, even if they do not fit into context.

The prototypes

There are several ways of implementing chat over all of the certificates (of a particular framework). I have experimented with two.

RAG

Using the RAG feature provided by OpenWebUI is the simplest way of doing this. We simply upload all of the certificate documents onto the OpenWebUI backend when we get them (or their content changes). There is code for this in the web backend:

sec-certs/sec_certs_page/common/tasks/webui.py

Line 21 in 09cca5c

class KBUpdater: # pragma: no cover

Then, one can create a collection containing all of the documents from one framework (or one type of document such as the report). Then the collection can be added as an argument to the model inference call:

sec-certs/sec_certs_page/common/ai/chat.py

Lines 45 to 60 in 09cca5c

	def chat_rag(
	queries, model: str, collection: str, hashid: Optional[str] = None, about: str = "entry"
	) -> requests.Response:
	"""
	Chat with the model using RAG (Retrieval-Augmented Generation).
	:param queries: THe list of message dictionaries.
	:param model: Model name.
	:param collection: Collection name (e.g., "cc", "fips", "pp").
	:param hashid: Certificate hash ID.
	:param about: Whether to do RAG over the "entry" (certificate documents only),
	the whole "collection" (but not any particular certificate), or "both".
	For "entry" and "both", a valid hashid must be provided.
	For "collection", no hashid is needed.
	:return:
	"""

What happens on the OpenWebUI backend is well described in their docs.

Some findings:

• 👍 This scales quite well, the computation overhead for us is very small. However, I found that it had very poor performance for our use-case.
• 👎 Since we are using the upstream OpenWebUI instance directly, we can not change any of its settings to tune the RAG to our liking.
• 👎 Often, the user's question lead to very bad RAG queries (there is likely some query generation model that creates queries that get embedded but I did not fully investigate) and those led to the model receiving useless chunks of documents from many certificates.
• 👎 The model was constantly confused about the certificates, and was mixing them up, as it did not know it received the chunks from multiple certificate documents.
• 👎 The documents are not very "RAG"-able I think. Let's take a simple query for the certificate ID. Often they contain it, but without any context around it that would be semantically close to "an ID", so the chunks selected are garbage and the model doesn't get it.

Perhaps some of these drawbacks could be improved:

• if we ran our own OpenWebUI instance and were able to change and experiment with the settings
• if we improved the prompts and added additional context

Tools

Many of the models support some form of tool calling. The OpenWebUI backend also supports it and contains some tools natively, such as Web Scrape and Web Search. There is also support for custom tools, which are essentially any piece of Python code that the model gets access to (needs to be well-described for the model to be able to know when and how to use it). However, this feature is disable on the e-Infra OpenWebUI deployment as it is impossible to secure it.

I played around with using the native tools on the OpenWebUI backend with this prompt:

You are a helpful assistant for the sec-certs project and an expert in security certifications. You can answer questions about Common Criteria certificates.
In your context, you may have relevant information extracted from Common Criteria certification reports and security targets. Each source with a different ID is a different certificate, do not confuse them.
If you have a certificate's dgst, only use the sources in the context that belong to it, do not use any other sources from the context.

If you want to search for a certificate (by its name or ID) and the context you have is not sufficient, use the web scrape tool to search using the following URL:
https://sec-certs.org/cc/mergedsearch/?searchType=by-name&cat=abcdefghijklmop&type=any&status=any&sort=match&scheme=any&q=<your query here>

If you want to search for a certificate by the contents of its report or target and the context you have is not sufficient, use the web scrape tool to search using the following URL:
https://sec-certs.org/cc/mergedsearch/?searchType=fulltext&cat=abcdefghijklmop&type=any&status=any&sort=match&scheme=any&q=<your query here>

If you believe you have the certificate ID of a certificate, you can look it up by using the web scrape tool at the following URL:
https://sec-certs.org/cc/id/<the certificate ID>

This allows you to obtain the certificate's dgst, which you can use as described below to obtain the full text of the certification report and security target documents. The dgst is found in the raw data section of the /cc/id/ website, in the JSON under the "dgst" key.
If the user asks a question about the certificate that may benefit from having the full text of the certification report or security target, use web scrape to download it.

https://sec-certs.org/cc/<dgst>/report.txt
https://sec-certs.org/cc/<dgst>/target.txt

• 👍 This worked quite nicely.
• 👍 The model was able to use both search methods on the site.
• 👍 Also, it was able to download the contents of the certification report and target and inject them into the context. However, there were some drawbacks:
• 👎 The model got confused about the certificate ID and dgst quite a lot.
• 👎 The model did not use the tools properly, quite often it complained that it "CAN NOT use any tools but if it could it would do it like ".
• 👎 The native tools pass the web content through some sort of markdown converter and only inject that into the context. This meant links were mostly lost and the model had no way of navigating them. This also likely stripped some other useful data from the site that is obvious to an actual user.

An example chat is here:
https://gist.github.com/J08nY/834bfb6d56dc25deebdb442cff547fdf

The future

I think the way forward is to:

• Deploy our own OpenWebUI instance.
• Use custom tools, there is an outdated example. Maybe add LLM specific endpoints to the site that the tool could use. The site currently has no API, it is purely a frontend though with some features that are programmatically nice for access. We could go further.
• Experiment more with prompts.
• Add framework-specific documents to the context or otherwise give the model access. I mean things like the Common Criteria standard, etc.