Remove severity levels and add progress reporting

.lintus.yml

@@ -21,12 +21,17 @@ rules:
     criteria:
       "true": At least one raised message is vague, such as "invalid" or "failed", with no detail.
       "false": Every raised message names the problem, or no errors are raised.
-    severity: warning
 
   classes_are_documented:
     description: Each class and module should open with a comment describing its responsibility.
-    question: Does every top-level class or module in this file have a comment describing what it is for?
-    offense_when: false
-    severity: warning
+    question: Is there a class or module defined in this file whose definition is not preceded by a comment describing what it is for?
+    criteria:
+      "true": >-
+        A class or module with a body of its own is introduced without a descriptive comment on the
+        lines right above its `class` or `module` line.
+      "false": >-
+        Every class or module that carries behaviour has such a comment, or the file defines none.
+        Ignore namespace wrappers whose body only nests other definitions, classes reopened only to
+        nest another definition, and one-line subclasses such as `class Error < StandardError; end`.
     exclude:
       - "exe/*"

CHANGELOG.md

@@ -1,5 +1,12 @@
 ## [Unreleased]
 
+- A progress counter on stderr while files are being checked.
+- The JSON output lists every file with the probability each rule gave it, not only offenses.
+- The starter config's documentation rule asks about the offense and scopes it with criteria,
+  so namespace wrappers no longer trip it.
+- Removed rule `severity` and the `--fail-on` flag: an offense is an offense. A config that still
+  sets `severity` is rejected with a message naming the key.
+
 ## [0.1.0] - 2026-09-21
 
 - Initial release.

README.md

@@ -19,7 +19,7 @@ rules:
 
 ```
 $ lintus
-app/jobs/retry_job.rb: [no_sleep_in_jobs] Background jobs must never block on sleep. (error, noul 0.94)
+app/jobs/retry_job.rb: [no_sleep_in_jobs] Background jobs must never block on sleep. (noul 0.94)
 
 42 files inspected, 1 offense detected
 ```
@@ -81,13 +81,11 @@ rules:
       - "app/**/*.rb"
     exclude:
       - "app/models/legacy/**"
-    severity: error
 
   service_objects_are_documented:
     description: Service objects carry a comment explaining what they do.
     question: Does every class in this file have a comment describing its responsibility?
     offense_when: false
-    severity: warning
     paths:
       - "app/services/**/*.rb"
 ```
@@ -102,7 +100,6 @@ rules:
 | `threshold`    | no       | Probability above which the answer counts as `true`. Defaults to 0.5. Raise it for rules where a false positive is costly. |
 | `paths`        | no       | Globs the rule applies to. Defaults to the top-level `paths`; with neither, every file. |
 | `exclude`      | no       | Globs the rule never applies to, on top of the top-level `exclude`. |
-| `severity`     | no       | `error` (default) or `warning`. Only errors fail the run, unless `--fail-on` says otherwise. |
 | `offense_when` | no       | `true` (default) or `false`. Set to `false` for rules phrased positively, such as "Does every class have a comment?". |
 
 Rule ids are snake_case. They become the question identifiers in the Jev request and the
@@ -121,6 +118,22 @@ model. So prefer several narrow questions over one broad one: "Does this file ca
 and "Does this file rescue `Exception`?" as two rules beat "Does this file do anything a job
 should not?".
 
+**Ask about the offense, not about compliance.** "Does every class have a comment?" turns
+false on a single edge case, and there is always one: the namespace wrapper, the reopened
+class, the one-line error subclass. "Is there a class without a comment?" is the same
+question, but its `criteria` can now spell out which classes count. Keep `offense_when: false`
+for questions that are genuinely easier to phrase positively.
+
+**Name the unit and list what to ignore.** The model reads a question literally. When Lintus
+first linted itself with "does every class or module have a comment?", every file was flagged
+with a probability around 0.15: each one opens with an undocumented `module Lintus`. The fix
+was not the threshold but the criteria, which now exclude wrappers whose body only nests other
+definitions.
+
+**Read the probabilities before touching the threshold.** Scores clustered far from 0.5 mean
+the model is sure of its reading of the question. If that reading is not yours, reword. Move
+the threshold only when the scores of clean and offending files overlap around it.
+
 Lintus sends the model the file's path and its full content. A question can therefore refer to
 the file name ("Is this a controller?") as well as the code.
 
@@ -140,11 +153,15 @@ Only files that at least one rule applies to are sent. Deleted files are never s
 
 `--format text` is the default. `--format github` prints GitHub Actions workflow commands, so
 each offense becomes an annotation on the file in the pull request; it is the default when
-`GITHUB_ACTIONS` is set. `--format json` is for other tools.
+`GITHUB_ACTIONS` is set. `--format json` is for other tools, and it also lists the probability
+every rule gave every file under `files`, offense or not, which is what you want when tuning
+a rule's wording or threshold.
+
+Exit status is `0` when clean, `1` when there are offenses, and `2` when a request failed or
+the invocation was wrong.
 
-Exit status is `0` when clean, `1` when there are offenses at or above `--fail-on`
-(`error` by default, or `warning`, or `never`), and `2` when a request failed or the
-invocation was wrong.
+While a run is in flight, a counter on stderr shows how many files are done. Off a terminal
+(CI logs, pipes) it is a single line announcing the run, so captured output stays clean.
 
 ## GitHub Actions
 
@@ -204,7 +221,6 @@ Either way `JEV_API_KEY` must be in the environment of the shell running the com
 -c, --config PATH      Config file to use instead of searching for one
 -j, --jobs N           Concurrent requests to the Jev API (default 4)
     --api-key KEY      Jev API key (default: $JEV_API_KEY)
-    --fail-on LEVEL    error (default), warning, or never
 -l, --list             Show what would be checked without calling the API
 ```
 

action.yml

@@ -17,7 +17,7 @@ inputs:
     required: false
     default: ""
   args:
-    description: Extra arguments for the lintus command, for example "--fail-on warning" or "--jobs 8".
+    description: Extra arguments for the lintus command, for example "--jobs 8".
     required: false
     default: ""
   version:

lib/lintus/cli.rb

@@ -1,3 +1,3 @@
 # frozen_string_literal: true
 
 require "optparse"
@@ -15,7 +15,7 @@
     # `command` is what to do; `selection` is which files, with `ref` for --diff
     # and `files` for paths given on the command line.
     Options = Struct.new(
-      :command, :selection, :ref, :files, :config, :format, :jobs, :api_key, :list, :fail_on,
+      :command, :selection, :ref, :files, :config, :format, :jobs, :api_key, :list,
       keyword_init: true
     )
 
@@ -50,9 +50,9 @@
       return list(tasks) if options.list
 
       configure_api_key!(options)
-      report = runner.run(tasks)
+      report = with_progress(tasks.size) { |progress| runner.run(tasks) { |done| progress.update(done) } }
       Formatter.for(options.format).new(stdout).render(report)
-      report.exit_status(fail_on: options.fail_on)
+      report.exit_status
     end
 
     def parse(argv)
@@ -79,7 +79,7 @@
     def default_options
       Options.new(
         command: :lint, selection: :all, format: @env["GITHUB_ACTIONS"] == "true" ? "github" : "text",
-        jobs: 4, api_key: @env["JEV_API_KEY"], list: false, fail_on: "error"
+        jobs: 4, api_key: @env["JEV_API_KEY"], list: false
       )
     end
 
@@ -109,9 +109,6 @@
           options.jobs = jobs
         end
         parser.on("--api-key KEY", "Jev API key (default: $JEV_API_KEY)") { |key| options.api_key = key }
-        parser.on("--fail-on LEVEL", %w[error warning never], "Exit non-zero on: error (default), warning, never") do |level|
-          options.fail_on = level
-        end
         parser.on("-l", "--list", "List the files and rules that would be checked, without calling the API") do
           options.list = true
         end
@@ -138,6 +135,14 @@
       end
     end
 
+    def with_progress(total)
+      progress = Progress.new(stderr, total)
+      progress.start
+      yield progress
+    ensure
+      progress.finish
+    end
+
     def configure_api_key!(options)
       raise Error, "no API key: set JEV_API_KEY or pass --api-key" if options.api_key.nil? || options.api_key.empty?
 

lib/lintus/formatter/github.rb

@@ -1,3 +1,3 @@
 # frozen_string_literal: true
 
 module Lintus
@@ -8,7 +8,7 @@
       private
 
       def offense_line(offense)
-        command(offense.severity, offense.message, file: offense.path, title: "lintus: #{offense.rule.id}")
+        command("error", offense.message, file: offense.path, title: "lintus: #{offense.rule.id}")
       end
 
       def skipped_line(skipped)

lib/lintus/formatter/json.rb

@@ -1,3 +1,3 @@
 # frozen_string_literal: true
 
 require "json"
@@ -11,16 +11,22 @@
           summary: {
             files_inspected: report.checked.size,
             offenses: report.offenses.size,
-            errors: report.errors.size,
-            warnings: report.warnings.size,
             skipped: report.skipped.size,
             failures: report.failures.size
           },
           offenses: report.sorted_offenses.map(&:to_h),
+          files: report.checked.sort_by(&:path).map { |checked| file_entry(checked) },
           skipped: report.skipped.map { |skipped| { path: skipped.path, reason: skipped.reason } },
           failures: report.failures.map { |failure| { path: failure.path, error: failure.error.message } }
         )
       end
+
+      private
+
+      # Every rule asked about the file and the probability it got, offense or not.
+      def file_entry(checked)
+        { path: checked.path, answers: checked.answers.transform_values { |noul| noul.round(3) } }
+      end
     end
   end
 end

lib/lintus/formatter/text.rb

@@ -1,3 +1,3 @@
 # frozen_string_literal: true
 
 module Lintus
@@ -7,7 +7,7 @@
       private
 
       def offense_line(offense)
-        "#{offense.path}: [#{offense.rule.id}] #{offense.message} (#{offense.severity}, noul #{offense.noul.round(2)})"
+        "#{offense.path}: [#{offense.rule.id}] #{offense.message} (noul #{offense.noul.round(2)})"
       end
 
       def skipped_line(skipped) = "#{skipped.path}: skipped, #{skipped.reason}"

lib/lintus/offense.rb

@@ -1,14 +1,12 @@
 # frozen_string_literal: true
 
 module Lintus
   # A rule the model flagged on a file, with the probability it gave.
   Offense = Struct.new(:path, :rule, :noul, keyword_init: true) do
-    def severity = rule.severity
-    def error? = rule.error?
     def message = rule.description
 
     def to_h
-      { path: path, rule: rule.id, severity: severity, message: message, noul: noul.round(3) }
+      { path: path, rule: rule.id, message: message, noul: noul.round(3) }
     end
   end
 end

lib/lintus/progress.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module Lintus
+  # Tells the person waiting that something is happening. On a terminal it is a
+  # counter updated in place and erased when done; elsewhere (CI logs, pipes) it
+  # is a single line announcing the run, so nothing garbles captured output.
+  class Progress
+    def initialize(io, total)
+      @io = io
+      @total = total
+      @tty = io.respond_to?(:tty?) && io.tty?
+      @width = 0
+    end
+
+    def start
+      return if @total.zero?
+
+      if @tty
+        draw(0)
+      else
+        @io.puts("Checking #{@total} #{@total == 1 ? "file" : "files"}...")
+      end
+    end
+
+    def update(done)
+      draw(done) if @tty
+    end
+
+    def finish
+      return unless @tty && @width.positive?
+
+      @io.print "\r#{" " * @width}\r"
+      @io.flush
+    end
+
+    private
+
+    def draw(done)
+      line = "Checking #{done}/#{@total} files"
+      @width = [@width, line.size].max
+      @io.print "\r#{line.ljust(@width)}"
+      @io.flush
+    end
+  end
+end

lib/lintus/report.rb

@@ -3,6 +3,7 @@
 module Lintus
   # Everything a run produced: what was checked, what was flagged, what was skipped, what broke.
   class Report
+    Checked = Struct.new(:path, :answers, keyword_init: true)
     Skipped = Struct.new(:path, :reason, keyword_init: true)
     Failure = Struct.new(:path, :error, keyword_init: true)
 
@@ -16,9 +17,11 @@ def initialize
       @mutex = Mutex.new
     end
 
-    def record_checked(path, offenses)
+    # `answers` maps every rule asked about the file to the probability it got,
+    # offense or not, so a run can be read for confidence and not just verdicts.
+    def record_checked(path, offenses, answers = {})
       synchronize do
-        checked << path
+        checked << Checked.new(path: path, answers: answers)
         self.offenses.concat(offenses)
       end
     end
@@ -31,21 +34,13 @@ def record_failure(path, error)
       synchronize { failures << Failure.new(path: path, error: error) }
     end
 
-    def errors = offenses.select(&:error?)
-    def warnings = offenses.reject(&:error?)
-
     def sorted_offenses = offenses.sort_by { |offense| [offense.path, offense.rule.id] }
 
-    # 2 when a file could not be checked, 1 when offenses reach `fail_on`, else 0.
-    def exit_status(fail_on: "error")
+    # 2 when a file could not be checked, 1 when there are offenses, else 0.
+    def exit_status
       return 2 if failures.any?
 
-      failing = case fail_on.to_s
-                when "warning" then offenses
-                when "never" then []
-                else errors
-                end
-      failing.any? ? 1 : 0
+      offenses.any? ? 1 : 0
     end
 
     private

lib/lintus/rule.rb

@@ -8,11 +8,10 @@ module Lintus
   # `offense_when` (true by default, so questions are phrased to describe the
   # offense: "Does this file call sleep?").
   class Rule
-    KEYS = %w[question description criteria threshold paths exclude severity offense_when].freeze
-    SEVERITIES = %w[error warning].freeze
+    KEYS = %w[question description criteria threshold paths exclude offense_when].freeze
     ID_FORMAT = /\A[a-z][a-z0-9_]*\z/
 
-    attr_reader :id, :description, :question, :criteria, :threshold, :paths, :exclude, :severity, :offense_when
+    attr_reader :id, :description, :question, :criteria, :threshold, :paths, :exclude, :offense_when
 
     def initialize(id, attrs, default_paths: [], default_exclude: [])
       @id = validate_id(id)
@@ -24,7 +23,6 @@ def initialize(id, attrs, default_paths: [], default_exclude: [])
       @threshold = build_threshold(attrs["threshold"])
       @paths = Schema.string_list(attrs.fetch("paths", default_paths))
       @exclude = default_exclude + Schema.string_list(attrs["exclude"])
-      @severity = build_severity(attrs.fetch("severity", "error"))
       @offense_when = build_offense_when(attrs.fetch("offense_when", true))
     end
 
@@ -41,8 +39,6 @@ def add_to(query)
 
     def offense?(answer) = answer.result == offense_when
 
-    def error? = severity == "error"
-
     private
 
     def validate_id(id)
@@ -86,16 +82,6 @@ def build_threshold(threshold)
       threshold.to_f
     end
 
-    def build_severity(severity)
-      severity = severity.to_s
-      unless SEVERITIES.include?(severity)
-        raise ConfigError,
-              "rule #{id}: `severity` must be one of #{SEVERITIES.join(", ")}"
-      end
-
-      severity
-    end
-
     def build_offense_when(value)
       raise ConfigError, "rule #{id}: `offense_when` must be true or false" unless [true, false].include?(value)