Add Vale AI evaluator and CLI command; config, reporter, and tests updates by ayo6706 · Pull Request #7 · TRocket-Labs/vectorlint

ayo6706 · 2025-11-13T20:45:01Z

This PR introduces a new Vale AI evaluator that augments Vale CLI findings with LLM-generated suggestions, along with a dedicated CLI command and configuration support in vectorlint.ini.

Key Additions

🔹 New Vale AI Evaluator

Runs Vale CLI and enhances findings with AI-generated suggestions.
Extracts before/after context windows around each finding.
Supports batched suggestion generation with robust fallbacks.
Falls back gracefully to Vale’s original descriptions if AI generation fails.
Normalized severity mapping and improved error handling.

🔹 New CLI Command

vale-ai command runs Vale with AI-enhanced suggestions.
Registered in src/index.ts.

🔹 Config Enhancements

Added parsing for an [Evaluators] section and per-evaluator configuration:

[Evaluators]
enabled = [vale-ai]

[Evaluator:vale-ai]
contextWindowSize = 100

loadConfig() now includes evaluators: { enabled, valeAI }.

🔹 Reporting Improvements

Pretty-printed Vale findings with optional AI suggestions.
New summary printers:
- printValeIssueRow
- printValeFileSummary
- printValeGlobalSummary

🔹 Test Updates

Updated imports to remove .js extensions (TypeScript path imports).
Adjusted tests to match new config, evaluator, and reporter behavior.

File Summary

🆕 Added

src/cli/vale-ai-command.ts — new CLI for AI-enhanced Vale.

✏️ Modified (highlights)

config-loader.ts
Parsing evaluator sections + returning new evaluators config shape.
suggestion-generator.ts
Context formatting updates + constructor accepts provider.
types.ts
Introduced BatchSuggestionResponse.
vale-ai-evaluator.ts
Major refactor: context extraction, batching, fallbacks, caching.
vale-runner.ts
Improved exit-code handling + typo fix.
reporter.ts
New Vale issue and summary formatters.
config-schemas.ts
Extended schema for evaluator configuration.
tests/
Import cleanup + updated evaluator/CLI behavior tests.

Behavior Notes

Evaluator only activates when declared in vectorlint.ini.
Requires:
- Vale CLI installed and on $PATH.
- LLM provider env vars depending on the provider (e.g., OPENAI_API_KEY).
Graceful error paths:
- Missing Vale → helpful installation message.
- Missing context → warning, but still processes finding.
- AI generation failure → fallback to Vale description.

How to Use

Run

npm run dev -- vale-ai path/to/file.md
# or
node ./dist/index.js vale-ai path/to/file.md -v

Verify Vale Installation

vale --version

Install Vale (quick reference)

macOS: brew install vale
Windows: choco install vale
Linux: download from official releases

Reviewer Checklist

Install Vale and ensure vale --version works.
Set required LLM provider environment variables.

- Add ValeRunner class for comprehensive Vale CLI interaction - Implement methods to check Vale installation and retrieve version - Create robust error handling for various Vale execution scenarios - Extend ValeIssue type with additional metadata fields - Add Node types to tsconfig for improved type support - Implement JSON parsing and error management for Vale output - Support cross-platform Vale installation detection Enables advanced linting capabilities with Vale CLI, providing a flexible and resilient integration mechanism for text analysis and style checking.

…integration - Add new ValeAIEvaluator class to orchestrate Vale CLI execution - Implement file content caching mechanism for efficient processing - Create robust context window extraction method for Vale findings - Add error handling and logging for file reading and context extraction - Normalize Vale severity levels to standard format - Support dynamic file evaluation with optional file path input - Prepare infrastructure for future AI suggestion generation

…gestions - Add new SuggestionGenerator class for batch AI suggestion generation - Implement structured output schema for batch suggestion responses - Create method to generate context-aware suggestions for Vale findings - Add robust error handling for LLM suggestion generation - Include fallback to Vale's original descriptions on LLM failure - Implement private method to build comprehensive batch prompts - Add detailed type definitions and interfaces for suggestion generation Enhances Vale AI evaluation by providing intelligent, context-aware suggestions for writing improvements using LLM providers.

…gestion generation - Add EvaluatorRegistry with factory pattern for managing evaluators - Enhance ValeAIEvaluator to support AI-powered suggestion generation - Implement batch suggestion generation with robust error handling - Add context window extraction and caching for improved AI context - Introduce SuggestionGenerator integration with LLM providers - Improve error handling and logging for Vale CLI and AI suggestion processes - Refactor Vale AI evaluation workflow to support more flexible configuration

- Extend CONFIG_SCHEMA to include optional evaluators configuration - Add support for specifying enabled evaluators - Introduce Vale AI specific configuration with context window size - Provide default context window size of 100 for Vale AI evaluator - Enhance configuration flexibility for advanced evaluation settings

…tation and error handling - Refactor ValeAIEvaluator class with detailed JSDoc comments - Add comprehensive documentation for `evaluate()` method with usage examples - Improve error handling and logging for file content caching - Enhance context window extraction with robust edge case handling - Add more descriptive error messages and fallback mechanisms - Improve method-level documentation for `extractContextWindow()` and `normalizeSeverity()` - Update default description handling to use `Message` if `Description` is empty - Improve code readability and maintainability with detailed inline comments

… support - Add .vale.ini configuration file with default Vale settings - Enhance config-loader.ts to parse evaluator-specific configuration sections - Implement parsing for Vale AI context window size configuration - Add support for enabled evaluators configuration in vectorlint.ini - Update SuggestionGenerator with improved LLM prompt and documentation - Refactor config loading to support more flexible configuration options - Improve error handling and configuration parsing robustness This change introduces more comprehensive configuration support for VectorLint, specifically focusing on Vale and AI evaluator configurations.

…chema support - Add comprehensive Vale suggestion generator prompt template - Refactor SuggestionGenerator to support dynamic prompt loading - Implement initial schema and criteria evaluation structure - Prepare for advanced AI-driven Vale suggestion generation - Improve type definitions and modular architecture for suggestion generation This change introduces a robust framework for generating context-aware Vale suggestions using AI, with a focus on flexible prompt management and structured evaluation criteria.

…gestion generation - Add EvaluatorRegistry with factory pattern for managing evaluators - Enhance ValeAIEvaluator to support AI-powered suggestion generation - Implement batch suggestion generation with robust error handling - Add context window extraction and caching for improved AI context - Introduce SuggestionGenerator integration with LLM providers - Improve error handling and logging for Vale CLI and AI suggestion processes - Refactor Vale AI evaluation workflow to support more flexible configuration

- Extend CONFIG_SCHEMA to include optional evaluators configuration - Add support for specifying enabled evaluators - Introduce Vale AI specific configuration with context window size - Provide default context window size of 100 for Vale AI evaluator - Enhance configuration flexibility for advanced evaluation settings

…tation and error handling - Refactor ValeAIEvaluator class with detailed JSDoc comments - Add comprehensive documentation for `evaluate()` method with usage examples - Improve error handling and logging for file content caching - Enhance context window extraction with robust edge case handling - Add more descriptive error messages and fallback mechanisms - Improve method-level documentation for `extractContextWindow()` and `normalizeSeverity()` - Update default description handling to use `Message` if `Description` is empty - Improve code readability and maintainability with detailed inline comments

… support - Add .vale.ini configuration file with default Vale settings - Enhance config-loader.ts to parse evaluator-specific configuration sections - Implement parsing for Vale AI context window size configuration - Add support for enabled evaluators configuration in vectorlint.ini - Update SuggestionGenerator with improved LLM prompt and documentation - Refactor config loading to support more flexible configuration options - Improve error handling and configuration parsing robustness This change introduces more comprehensive configuration support for VectorLint, specifically focusing on Vale and AI evaluator configurations.

- Create new vale-ai-command.ts module to encapsulate Vale AI CLI logic - Move vale-ai command registration from index.ts to dedicated handler - Implement comprehensive error handling for CLI options, environment variables, and Vale installation - Add verbose logging support with Vale version display - Display formatted results with severity icons and contextual information - Move Vale CLI types to schemas/vale-responses.ts for better organization - Refactor index.ts to register commands via dedicated functions instead of inline definitions - Improve command isolation to prevent option conflicts between different commands

- Remove trailing whitespace in suggestion-generator.ts - Normalize import paths by removing .js extensions in vale-ai-evaluator.ts - Remove verbose JSDoc comments from vale-runner.ts private methods - Remove unnecessary comments from index.ts CLI setup - Normalize import paths by removing .js extensions across all test files - Ensure consistent code formatting and import style throughout codebase

- Remove unnecessary comment about context window extraction - Fix inconsistent whitespace and indentation in extractContextWindow method - Add newline at end of vale-ai-evaluator.ts file - Add newline at end of vale-runner.ts file - Improve code consistency and readability across Vale AI evaluator module

…ions - Separate context display into distinct "before" and "after" parts instead of reconstructing full match - Remove context preview reconstruction that could misalign with Vale's span information - Add clarifying comments explaining why match reconstruction was removed - Provide more granular context information for better AI suggestion generation

- Remove unused SuggestionGenerator import from vale-ai-command.ts - Replace all process.exit() calls with throw new Error() for proper error handling - Change successful completion from process.exit(0) to early return statement - Improve type safety for context window parsing with explicit type checking - Simplify error handling in ValeAIEvaluator by removing unnecessary try-catch wrapper - Improve error message handling by safely extracting error messages from Error objects - Remove exit code logic for Vale findings as they are suggestions, not fatal errors - Enable proper error propagation through the CLI command chain instead of abrupt process termination

- Revert error throwing to process.exit(1) in vale-ai-command with eslint disable comment - Enhance createValeConfigError to parse JSON error responses from Vale stderr - Extract error message text from JSON parsed stderr when available - Fall back to raw stderr if JSON parsing fails - Improve error message clarity by using extracted text instead of raw output

- Move EvaluatorsConfig interface definition from config-loader to vale-ai/types - Import EvaluatorsConfig type in config-loader for type consistency - Replace inline type definitions with centralized EvaluatorsConfig type - Improve type reusability across evaluator modules

oshorefueled · 2025-11-18T09:47:44Z

src/cli/vale-ai-command.ts

+    .command('vale-ai [files...]')
+    .description('Run Vale with AI-enhanced suggestions')
+    .option('-v, --verbose', 'Enable verbose logging')
+    .option('--context-window <size>', 'Number of characters to extract before/after each issue for AI context', '100')


This config needs a clearer name, it could be confused with context window of the LLM.

oshorefueled · 2025-11-18T09:50:05Z

src/cli/vale-ai-command.ts

+import { DefaultRequestBuilder } from '../providers/request-builder';
+import { loadDirective } from '../prompts/directive-loader';
+import { parseCliOptions, parseEnvironment } from '../boundaries/index';
+import { handleUnknownError } from '../errors/index';


We should be able to import this without index in the path.

oshorefueled · 2025-11-18T09:51:04Z

src/cli/vale-ai-command.ts

+      const contextWindowSize = parseInt(contextWindowStr, 10);
+      if (isNaN(contextWindowSize) || contextWindowSize < 0) {
+        console.error('Error: context-window must be a non-negative number');
+        throw new Error('Invalid context-window value');


context-window not a clear name.

oshorefueled · 2025-11-18T09:51:30Z

src/cli/vale-ai-command.ts

+      const valeRunner = new ValeRunner();
+      if (!valeRunner.isInstalled()) {
+        console.error('Error: Vale is not installed or not in PATH.');
+        console.error('Install Vale: https://vale.sh/docs/vale-cli/installation/');


make this a constant.

oshorefueled · 2025-11-18T09:52:32Z

src/cli/vale-ai-command.ts

+      try {
+        if (cliOptions.verbose) {
+          console.log(`[vale-ai] Running Vale on ${files.length > 0 ? files.join(', ') : 'all files'}`);
+        }
+
+        // Run evaluation
+        const result = await evaluator.evaluate(files.length > 0 ? files : undefined);
+
+        // Display results
+        if (result.findings.length === 0) {
+          console.log('✓ No issues found by Vale.');
+          return;
+        }
+
+        console.log(`Found ${result.findings.length} issue(s):\n`);
+
+        for (const finding of result.findings) {
+          const severityIcon = finding.severity === 'error' ? '✖' : 
+                              finding.severity === 'warning' ? '⚠' : 'ℹ';
+
+          console.log(`${severityIcon} ${finding.file}:${finding.line}:${finding.column}`);
+          console.log(`  Rule: ${finding.rule}`);
+          console.log(`  Match: "${finding.match}"`);
+          console.log(`  Suggestion: ${finding.suggestion}`);
+
+          if (cliOptions.verbose && finding.context) {
+            const contextPreview = `${finding.context.before}${finding.match}${finding.context.after}`;
+            console.log(`  Context: "${contextPreview.substring(0, 100)}${contextPreview.length > 100 ? '...' : ''}"`);
+          }
+
+          console.log('');
+        }


This function can be broken up to make it easier to read.

oshorefueled · 2025-11-18T09:53:54Z

src/evaluators/vale-ai/vale-ai-evaluator.ts

+   * @param severity - Vale severity string (case-insensitive)
+   * @returns Normalized severity: 'error', 'warning', or 'suggestion'


No need for JsDocs params here. Comment before it is sufficient.

oshorefueled · 2025-11-18T09:56:07Z

src/cli/vale-ai-command.ts

+export function registerValeAICommand(program: Command): void {
+  program
+    .command('vale-ai [files...]')
+    .description('Run Vale with AI-enhanced suggestions')
+    .option('-v, --verbose', 'Enable verbose logging')
+    .option('--context-window <size>', 'Number of characters to extract before/after each issue for AI context', '100')


Since it's now an evaluator system, and every evaluator would need to register its command, this should be made a method on the evaluator class, so every evaluator defines its Command registration method.

We could do that on a separate PR to keep things clean.

oshorefueled · 2025-11-18T10:04:26Z

@ayo6706 Let's make the terminal output look similar to the vale output. Rather than create a new layout, let's mirror the existing Vale output by simply adding one more column to the report, which would be "suggestion".

It keeps it familiar, making the feature look like an extension to Vale, rather than a whole new thing.

- Rename `--context-window` option to `--context-chars` for clarity - Rename internal `contextWindowSize` variable to `contextCharsSize` for consistency - Extract error messages to constants `VALE_NOT_INSTALLED_ERROR` and `VALE_INSTALLATION_URL` - Simplify import paths by removing `/index` from boundary and error imports - Remove redundant JSDoc comment from `normalizeSeverity` method - Improve error message formatting with template literals

…functions - Extract CLI options parsing and validation into `parseAndValidateCliOptions()` - Extract environment variable parsing and validation into `parseAndValidateEnvironment()` - Extract context characters size parsing into `parseContextCharsSize()` - Extract Vale runner creation and validation into `createAndValidateValeRunner()` - Extract LLM provider creation into `createLLMProvider()` - Extract results display logic into `displayResults()` - Extract main evaluation workflow into `executeValeAIEvaluation()` - Add `ValeFinding` type import for better type safety - Add `CliOptions` and `EnvConfig` type imports - Improve code readability and maintainability by reducing action handler complexity - Enable easier testing of individual command components

… output - Group findings by file and display in tabular format matching Vale's output style - Extract result display logic into dedicated reporter functions (printValeIssueRow, printValeFileSummary, printValeGlobalSummary) - Remove verbose parameter from displayResults function as formatting is now consistent - Adjust column widths in printValeIssueRow (location: 8, severity: 8, message: 40) for better alignment - Simplify message truncation and remove complex word-wrapping logic for cleaner output - Improve suggestion formatting with consistent indentation and line wrapping at 80 characters - Add file-level and global summary statistics to match Vale's reporting format - Remove blank lines between individual issues for more compact output

…dths - Remove multi-line word wrapping logic for AI suggestions - Change suggestion display from separate indented lines to single inline column - Adjust message column width from 40 to 35 characters - Add new rule column with 15 character width for better table alignment - Update suggestion fallback behavior to use empty strings instead of Vale descriptions - Simplify suggestion rendering to single-line truncation at 80 characters - Remove unnecessary indent and arrow formatting for suggestions - Update JSDoc comment to reflect new suggestion column in output - Improve consistency across suggestion-generator and vale-ai-evaluator error handling

- Remove file-level error, warning, and suggestion count calculations - Remove printValeFileSummary call that duplicates global summary information - Eliminate unnecessary blank line output after file findings - Simplify displayResults function by consolidating summary logic to global level only

- Change suggestion severity color from blue to cyan for better consistency - Always apply color formatting to summary counts regardless of value - Replace commas with "and" conjunction in file and global summaries - Make Params field nullable in VALE_ACTION_SCHEMA to handle optional parameters - Add blank line before global summary output for improved readability - Ensures consistent visual hierarchy and better readability of Vale findings output

ayo6706 added 21 commits November 12, 2025 19:32

Merge branch 'main' into ft/vale-ai

69f22d7

Resolve merge conflicts with remote ft/vale-ai branch

48a19fa

Fix wrong merge changes

1eb4392

eslint fix

f1d926f

ayo6706 changed the title ~~Ft/vale ai~~ Add Vale AI evaluator and CLI command; config, reporter, and tests updates Nov 17, 2025

ayo6706 added 2 commits November 18, 2025 10:14

oshorefueled requested changes Nov 18, 2025

View reviewed changes

ayo6706 added 4 commits November 18, 2025 12:09

ayo6706 added 3 commits November 18, 2025 17:09

eslint fix

f272e39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Vale AI evaluator and CLI command; config, reporter, and tests updates#7

Add Vale AI evaluator and CLI command; config, reporter, and tests updates#7
ayo6706 wants to merge 30 commits intomainfrom
ft/vale-ai

ayo6706 commented Nov 13, 2025 •

edited

Loading

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled Nov 18, 2025

Uh oh!

oshorefueled commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

		* @param severity - Vale severity string (case-insensitive)
		* @returns Normalized severity: 'error', 'warning', or 'suggestion'

Conversation

ayo6706 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Additions

🔹 New Vale AI Evaluator

🔹 New CLI Command

🔹 Config Enhancements

🔹 Reporting Improvements

🔹 Test Updates

File Summary

🆕 Added

✏️ Modified (highlights)

Behavior Notes

How to Use

Run

Verify Vale Installation

Install Vale (quick reference)

Reviewer Checklist

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

oshorefueled commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

ayo6706 commented Nov 13, 2025 •

edited

Loading