Skip to content

Exclude prose/data from language stats; documentation exclusion; programming-only dominant language#12

Merged
nielsbosma merged 1 commit into
mainfrom
fix/language-aggregation
Jun 23, 2026
Merged

Exclude prose/data from language stats; documentation exclusion; programming-only dominant language#12
nielsbosma merged 1 commit into
mainfrom
fix/language-aggregation

Conversation

@nielsbosma

Copy link
Copy Markdown
Contributor

Implements the three language-statistics accuracy suggestions for github-linguist parity. All three parts are done (none deferred).

Part 1 — Exclude prose + data from language statistics

LanguageAggregator.Aggregate now skips LanguageType.Prose and LanguageType.Data entirely (rows and denominator), so percent is the share of programming + markup bytes only. XML-doc updated. (Ref: language-aggregator-exclude-prose-data.md)

Part 2 — Documentation/example exclusion

  • New embedded data file src/Ivy.StackAnalyzer/data/documentation.yml (seeded from linguist documentation.yml: examples/samples/demos/docs/documentation/changelog/README/license path patterns).
  • New DataStore.LoadDocumentation (resource suffix .data.documentation.yml) mirroring LoadVendor, exposed as DataStore.DocumentationPatterns, with user-directory overlay support.
  • New ScannedFile.IsDocumentation, set in FileSystemScanner the same way as IsVendored but WITHOUT pruning the directory from the walk (example/doc dirs still surface as [auxiliary] components; ComponentDetector still reads the unfiltered list).
  • Aggregate guard extended with || f.File.IsDocumentation.
    (Ref: language-examples-overcount.md)

Part 3 — Programming-only dominant language

Added LanguageAggregator.DominantProgrammingLanguage, restricting the dominant/headline language pick to type == Programming (linguist parity), so a data (JSON) or prose (Text) language can never out-rank the real code language. After Part 1 the per-component Languages list (consumed for the hash language slot) is already free of data/prose; this helper makes the rule explicit and also excludes markup. (Ref: dominant-language-includes-data-prose.md)

Tests

  • dotnet test: 97 passed / 0 failed (95 baseline + 2 new aggregator tests).
  • Unit tests updated in LanguageAggregatorTests.cs: rewrote Aggregate_excludes_* to assert prose/data exclusion + re-percent; added Aggregate_excludes_documentation_files and DominantProgrammingLanguage_ignores_markup_data_and_prose.
  • Snapshots updated (all 4): go-service (drops Go Module data row), numpy-style-lib (drops TOML), django-spa (drops JSON/Markdown/Pip Requirements), next-dotnet-monorepo (drops XML/JSON/YAML/Markdown). Percentages re-computed over programming+markup; docs-only auxiliary subtrees now show languages: [].

Note: no test fixture contains an example/docs subtree of real code, so Part 2's path exclusion is covered by the dedicated unit test rather than a snapshot.

@nielsbosma
nielsbosma merged commit 7f0033a into main Jun 23, 2026
1 check passed
@nielsbosma
nielsbosma deleted the fix/language-aggregation branch June 23, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant