Ben O'Mahony and Danilo Sato host weekly AI-focused webinars, covering the latest developments in AI-related technology.
During one of their webinars Ben demonstrated a software solution he had built which acted as a framework for performing regular, automated editorial reviews of a book he is currently authoring.
I wanted to create something similar to Ben's system, albeit on a smaller scale and targeting blog articles as the authored content.
This software solution uses local LLMs to help users plan and author articles.
- Implement a web application providing Plan and Author functions.
- Integrate an LLM for evaluating a plan.
- Integrate an LLM for providing plan research functionality.
- Integrate an LLM for evaluating an article.
- Integrate basic persistence.
- Provide short, single-screen user documentation.
- Deeper research capabilities for planning.
- Improve presentation of evaluation output.
- Support saving to Google Drive.
- Investigate issues related to tool use on Llama.cpp.
- Auto-evaluation based on a timeout.
No known defects.
GitHub Copilot was used to assist in the development of this software.
Note
I was originally using Llama.cpp to host local LLMs for use by this software. However, I encountered issues when implementing tool use with gpt-oss on Llama.cpp. Switching to LM Studio solved this problem. I would like to revisit Llama.cpp as the host at a later date.
A system capable of running LM Studio is required.
Details of my personal system are below.
Note
The hardware in use on my PC includes an Accelerated Processor Unit (APU) which combines CPU and GPU on a single chip. Recommendations for alternative hardware can be found here; performance will depend upon the models you choose to run (and other operational factors).
Configure LM Studio as per the documentation.
Download an appropriate model; you can use community leaderboards to help select an appropriate model.
Note
The software was designed to work with gpt-oss models. Other models may work; chosen models must support tool use.
Use the Developer tab to run your chosen model as an API server.
Note
For optimal performance, configure LM Studio to offload to your GPU.
Clone the repository.
Open in Visual Studio Code.
Build the projects.
Press F5 to run.
- Plan Page: Define article purpose, audience, core message, and goals with validation.
- Author Page: Write article content using Markdown with live preview; evaluate your article against the plan.
Navigate to the Plan page:
For each field, capture the requirements for the article you plan to author.
You can use the AI tools to help in the definition of the plan:
- Research: Your specification is sent to an LLM for analysis which uses a web search tool to find relevant internet sources to help expand your definition.
- Evaluate: Your specification is sent to an LLM which analyses the content and provides suggestions for potential improvements.
You can iterate the use of these tools, evaluating your content periodically as you specify your plan.
When your plan is ready, move to the Author page:
The Author page provides:
- Edit: Markdown editing tools for authoring your article.
- Preview: View how your article will look once published.
- Evaluate: Your article is sent to an LLM for evaluation (more below).
Author your article and then switch to the Evaluate tab:
Clicking Evaluate will submit your article to an LLM, along with your plan definition.
The LLM will review your article against your plan and provide recommendations for improvements.
You can iterate the use of this tool, evaluating your content periodically as you author your article.
Ben O'Mahony and Danilo Sato for hosting their AI webinars and highlighting the LLM use case this software implements.
This repository was created primarily for my own exploration of the technologies involved.
I have selected an appropriate license using this tool.
This software is licensed under the MIT license.
I am interested in the idea of LLM-as-a-judge and have implemented a basic evaluation pipeline which integrates this technique. If you are interested in the general use of LLMs for evaluation purposes, LLM-as-a-judge may be of interest.