diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4b3476d..df36748 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -85,6 +85,17 @@ Checks if a file contains a specific string. Useful for tools configured inside "pyproject.toml" = ["[tool.ruff]"] ``` +### yaml_resources (high confidence) + +Matches Kubernetes-style YAML resources by API group and, optionally, kind. +Both values must occur in the same YAML document. + +```toml +[[detect.yaml_resources]] +api_groups = ["argoproj.io"] +kinds = ["Application", "AppProject", "ApplicationSet"] +``` + ### key_exists (medium confidence) Checks if a dot-separated key path exists in a JSON or TOML file. Useful for checking if a tool has configuration in a structured file. diff --git a/README.md b/README.md index 2a4ba60..a648ca5 100644 --- a/README.md +++ b/README.md @@ -292,7 +292,7 @@ Language ecosystems and development tools across multiple categories. **Languages:** Ada, C, C#, C++, COBOL, Clojure, Common Lisp, Crystal, D, Dart, Deno, Elixir, Elm, Emacs Lisp, Erlang, F#, Fortran, GDScript, Gleam, Go, Groovy, Haskell, Haxe, Java, JavaScript, Julia, Kotlin, Lua, Mojo, Nim, Nix, OCaml, Objective-C, Odin, PHP, Perl, Prolog, Python, R, Racket, Roc, Ruby, Rust, Scala, Scheme, Solidity, Swift, Tcl, TypeScript, V, VHDL, Verilog, Zig. -**Package Managers:** Alire, Bun, Bundler, Cabal, Cargo, CocoaPods, Composer, Conan, Conda, DUB, Deno Modules, Flit, Gleam Packages, Go Modules, Gradle, Hatch, Haxelib, Maven, Mix, Nix Flakes, NuGet, PDM, Pipenv, Pkg, Poetry, Quicklisp, Shards, Swift Package Manager, Yarn, cpanm, dotnet CLI, elm, fpm, npm, opam, pip, pnpm, pub, rebar3, renv, sbt, setuptools, uv, vcpkg. +**Package Managers:** Alire, Bun, Bundler, Cabal, Cargo, CocoaPods, Composer, Conan, Conda, DUB, Deno Modules, Flit, Gleam Packages, Go Modules, Gradle, Hatch, Haxelib, Jsonnet Bundler, Maven, Mint, Mix, Nix Flakes, NuGet, PDM, Pipenv, Pkg, Poetry, Quicklisp, Shards, Stack, Swift Package Manager, Yarn, cpanm, dotnet CLI, elm, fpm, npm, opam, pip, pnpm, pub, rebar3, renv, sbt, setuptools, uv, vcpkg. **Test:** ASV, AVA, Alcotest, Artillery, Bats, BenchmarkTools.jl, Bruno, Catch2, Cucumber, Cypress, EUnit, ExUnit, Gatling, Ginkgo, Google Test, Hspec, Hurl, JUnit, Jasmine, Jest, Kotest, Lighthouse CI, Locust, MSW, Minitest, Mocha, Newman, PHPUnit, Pest, Playwright, REST Client, RSpec, ScalaTest, Selenium, Testify, Testing Library, Vitest, XCTest, axe-core, benchmark-ips, cargo test, clojure.test, criterion, crystal spec, dart test, deno test, dotnet test, gleam test, go test, hyperfine, k6, kotlin.test, nf-test, pytest, pytest-benchmark, tape, testament, testthat, tinytest, tox, vdiffr, zig test. @@ -304,7 +304,7 @@ Language ecosystems and development tools across multiple categories. **Docs:** Docsify, Documenter.jl, Docusaurus, Dokka, Doxygen, ExDoc, Hugo, Javadoc, Jekyll, MkDocs, MultiQC, MyST-Parser, Nextra, Quarto, R Markdown, Read the Docs, Redoc, Sphinx, Starlight, Storybook, Swagger UI, TypeDoc, VitePress, Yard, Zensical, cargo doc, dart doc, deno doc, knitr, mdBook, phpDocumentor, pkgdown, pkgsite, roxygen2. -**Build:** Actix Web, AdonisJS, Angular, Astro, Autotools, Axum, CMake, Django, Dune, Echo, Electron, Eleventy, Ember.js, Express, FastAPI, Fastify, Fiber, Flask, Flutter, Foundry, Gatsby, Gin, GoReleaser, Hardhat, Hono, Invoke, Just, Koa, Laravel, Less, Mage, Make, Meson, NestJS, Next.js, Nextflow, Nuxt, Parcel, Phoenix, PostCSS, Qwik, Rails, Rake, React Native, Remix, Rocket, Rollup, Rspack, SWC, Sass, Sinatra, Snakemake, Spin, Spring Boot, Svelte, SvelteKit, Symfony, Task, Tauri, Vite, Vue, Webpack, cibuildwheel, cross, esbuild, nf-core, targets, tsup. +**Build:** Actix Web, AdonisJS, Angular, Astro, Autotools, Axum, Babashka, BitBake, CMake, Django, Dune, Echo, Electron, Eleventy, Ember.js, Express, FastAPI, Fastify, Fiber, Flask, Flutter, Foundry, Gatsby, Gin, GoReleaser, Hardhat, Hono, Invoke, Just, Koa, Laravel, Less, Mage, Make, Meson, Meteor, NestJS, Next.js, Nextflow, Nuxt, Parcel, Phoenix, PostCSS, Qwik, Rails, Rake, React Native, Remix, Rocket, Rollup, Rspack, SWC, Sass, Sinatra, Snakemake, Spin, Spring Boot, Svelte, SvelteKit, Symfony, Task, Tauri, Vite, Vue, Webpack, cibuildwheel, cross, esbuild, nf-core, targets, tsup. **Native Ext:** Maturin, Neon, Rustler, meson-python, mkmf, napi-rs, node-gyp, phpize, rb-sys, setuptools Extension, setuptools-rust. @@ -320,11 +320,11 @@ Language ecosystems and development tools across multiple categories. **Container:** Cloud Native Buildpacks, Dev Container, Docker, Docker Compose, Podman. -**Infra:** AWS CDK, Ansible, CloudFormation, DVC, Dockstore, Helm, Kubernetes, Kustomize, Packer, Pulumi, Serverless Framework, Terraform, Vagrant. +**Infra:** AWS CDK, Ansible, Argo CD, Chef, CloudFormation, DVC, Dockstore, Flux, Helm, Helmfile, Kubernetes, Kustomize, OpenTofu, Packer, Pulumi, Puppet, Serverless Framework, Terraform, Vagrant. -**Monorepo:** Bazel, Cargo workspaces, Go workspace, Lerna, Moon, Nx, Pants, Rush, Turborepo, Yarn workspaces, pnpm workspaces. +**Monorepo:** Bazel, Buck, Cargo workspaces, Go workspace, Lerna, Moon, Nx, Pants, Rush, Turborepo, Yarn workspaces, pnpm workspaces. -**Environment:** Flipper, JetBrains IDE, Jupyter, LaunchDarkly, Mise, Pixi, Unleash, VS Code, Volta, asdf, direnv, dotenv, pyenv. +**Environment:** Devbox, Flipper, JetBrains IDE, Jupyter, LaunchDarkly, Mise, Pixi, Unleash, VS Code, Volta, asdf, direnv, dotenv, pyenv. **i18n:** Crowdin, Fluent, FormatJS, Lingui, Rails i18n, Transifex, gettext, i18next, vue-i18n. diff --git a/cmd/brief/main_test.go b/cmd/brief/main_test.go index cd438cc..a59566a 100644 --- a/cmd/brief/main_test.go +++ b/cmd/brief/main_test.go @@ -16,6 +16,7 @@ const scanHelperRootEnv = "BRIEF_SCAN_HELPER_ROOT" const diffHelperEnv = "BRIEF_DIFF_HELPER" const submoduleHelperRootEnv = "BRIEF_SUBMODULE_HELPER_ROOT" const submoduleDiffHelperEnv = "BRIEF_SUBMODULE_DIFF_HELPER" +const yamlResourceDiffHelperEnv = "BRIEF_YAML_RESOURCE_DIFF_HELPER" func TestScanDefaultsBoundRecursiveDetection(t *testing.T) { if root := os.Getenv(scanHelperRootEnv); root != "" { @@ -79,6 +80,39 @@ func TestDiffAppliesScanOverrides(t *testing.T) { } } +func TestDiffReportsYAMLResourceWithUppercaseExtension(t *testing.T) { + if os.Getenv(yamlResourceDiffHelperEnv) != "" { + cmdDiff([]string{"-json", "HEAD"}) + os.Exit(0) + } + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not installed") + } + + dir := t.TempDir() + initGitScanFixture(t, dir) + writeScanFixture(t, dir, "README.md", "# Example\n") + runGitFixture(t, dir, "add", "README.md") + runGitFixture(t, dir, "commit", "-q", "-m", "initial") + writeScanFixture(t, dir, "manifest.YAML", "apiVersion: argoproj.io/v1alpha1\nkind: Application\n") + + cmd := exec.Command(os.Args[0], "-test.run=^TestDiffReportsYAMLResourceWithUppercaseExtension$") + cmd.Dir = dir + cmd.Env = append(os.Environ(), yamlResourceDiffHelperEnv+"=1") + out, err := cmd.Output() + if err != nil { + t.Fatalf("diff command failed: %v", err) + } + + var report brief.Report + if err := json.Unmarshal(out, &report); err != nil { + t.Fatalf("parsing diff output: %v\n%s", err, out) + } + if !reportHasTool(&report, "infrastructure", "Argo CD") { + t.Fatalf("tools = %+v, want Argo CD from changed manifest.YAML", report.Tools) + } +} + func TestScanIncludeSubmodulesFlag(t *testing.T) { if root := os.Getenv(submoduleHelperRootEnv); root != "" { cmdScan([]string{"-json", "-include-submodules", root}) diff --git a/detect/detect.go b/detect/detect.go index 28999f6..8ea1ab5 100644 --- a/detect/detect.go +++ b/detect/detect.go @@ -36,6 +36,8 @@ const ( DefaultScanLimit = 10000 // DefaultLineCountTimeout bounds external line counters. DefaultLineCountTimeout = 2 * time.Second + // contentGlobReadLimit bounds content inspected from each glob-matched file. + contentGlobReadLimit = 1 << 20 microsPerMS = 1000.0 scanReadBatchSize = 128 @@ -606,6 +608,13 @@ func (e *Engine) matchTool(tool *kb.ToolDef) brief.Confidence { } } + for _, resource := range tool.Detect.YAMLResources { + if e.hasYAMLResource(resource) { + best = brief.ConfidenceHigh + break + } + } + if len(tool.Detect.Dependencies) > 0 || len(tool.Detect.DevDependencies) > 0 { if e.hasDependency(tool) { best = brief.ConfidenceHigh @@ -868,6 +877,12 @@ func (e *Engine) loadFileExts() { // that point outside the root to prevent file disclosure attacks. // It opens the file via O_NOFOLLOW to avoid TOCTOU races between stat and read. func (e *Engine) safeReadFile(file string) ([]byte, error) { + return e.safeReadFileLimit(file, 0) +} + +// safeReadFileLimit applies safeReadFile's path checks and reads at most limit +// bytes. A non-positive limit reads the complete file. +func (e *Engine) safeReadFileLimit(file string, limit int64) ([]byte, error) { path := filepath.Join(e.Root, file) info, err := os.Lstat(path) if err != nil { @@ -889,8 +904,14 @@ func (e *Engine) safeReadFile(file string) ([]byte, error) { if !targetInfo.Mode().IsRegular() { return nil, fmt.Errorf("path is not a regular file: %s", file) } - // Safe symlink within root: read the resolved target directly. - return os.ReadFile(target) + // Safe symlink within root: open the resolved target without following + // a symlink swapped into place after the checks above. + f, err := openNoFollow(target) + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + return readFileLimit(f, limit) } if !info.Mode().IsRegular() { return nil, fmt.Errorf("path is not a regular file: %s", file) @@ -902,7 +923,14 @@ func (e *Engine) safeReadFile(file string) ([]byte, error) { return nil, err } defer func() { _ = f.Close() }() - return io.ReadAll(f) + return readFileLimit(f, limit) +} + +func readFileLimit(r io.Reader, limit int64) ([]byte, error) { + if limit > 0 { + r = io.LimitReader(r, limit) + } + return io.ReadAll(r) } // contains checks if an exact file or any regular file matching a glob contains @@ -936,7 +964,7 @@ func (e *Engine) globContains(pattern string, contentPatterns []string) bool { if !e.matchesProjectPattern(pattern, rel) { continue } - data, err := e.safeReadFile(rel) + data, err := e.safeReadFileLimit(rel, contentGlobReadLimit) if err == nil && containsAny(string(data), contentPatterns) { return true } @@ -953,6 +981,41 @@ func containsAny(content string, patterns []string) bool { return false } +type yamlResource struct { + APIVersion string `yaml:"apiVersion"` + Kind string `yaml:"kind"` +} + +func (e *Engine) hasYAMLResource(signal kb.YAMLResourceInfo) bool { + e.loadProjectFiles() + for _, rel := range e.projectFiles { + ext := strings.ToLower(filepath.Ext(rel)) + if ext != ".yaml" && ext != ".yml" { + continue + } + + data, err := e.safeReadFileLimit(rel, contentGlobReadLimit) + if err != nil { + continue + } + decoder := yaml.NewDecoder(bytes.NewReader(data)) + for { + var resource yamlResource + if err := decoder.Decode(&resource); err != nil { + break + } + group, _, found := strings.Cut(resource.APIVersion, "/") + if !found || !slices.Contains(signal.APIGroups, group) { + continue + } + if len(signal.Kinds) == 0 || slices.Contains(signal.Kinds, resource.Kind) { + return true + } + } + } + return false +} + // matchPathPattern matches slash-separated paths and treats ** as zero or more // complete path segments. func matchPathPattern(pattern, name string) bool { diff --git a/detect/detect_test.go b/detect/detect_test.go index d53e755..07e9215 100644 --- a/detect/detect_test.go +++ b/detect/detect_test.go @@ -2695,6 +2695,15 @@ func TestFileContainsGlob(t *testing.T) { t.Error("exact file_contains path should retain its existing behavior") } }) + + t.Run("bounded read", func(t *testing.T) { + dir := t.TempDir() + writeProjectFile(t, dir, "deploy/large.yaml", strings.Repeat("x", contentGlobReadLimit)+marker) + + if New(loadKB(t), dir).contains("**/*.yaml", []string{marker}) { + t.Error("file_contains glob should not inspect content beyond its read limit") + } + }) } func TestDirectoryGlobPattern(t *testing.T) { diff --git a/detect/development_tools_test.go b/detect/development_tools_test.go new file mode 100644 index 0000000..e5a4d63 --- /dev/null +++ b/detect/development_tools_test.go @@ -0,0 +1,295 @@ +package detect + +import ( + "slices" + "testing" + + "github.com/git-pkgs/brief" +) + +func TestMissingDevelopmentToolSignals(t *testing.T) { + tests := []struct { + name string + files map[string]string + category string + tool string + packageManager bool + notTool string + }{ + { + name: "Babashka project", + files: map[string]string{"bb.edn": "{:tasks {test {:task (println \"test\")}}}\n"}, + category: "build", + tool: "Babashka", + }, + { + name: "nested OpenTofu configuration", + files: map[string]string{"infra/main.tofu": "terraform { required_version = \">= 1.8\" }\n"}, + category: "infrastructure", + tool: "OpenTofu", + }, + { + name: "Stack project", + files: map[string]string{"stack.yaml": "resolver: lts-23.18\npackages: [.]\n"}, + tool: "Stack", + packageManager: true, + }, + { + name: "Chef Ruby metadata", + files: map[string]string{"metadata.rb": "name 'example'\nversion '1.0.0'\n"}, + category: "infrastructure", + tool: "Chef", + }, + { + name: "Chef JSON metadata", + files: map[string]string{"metadata.json": "{\"name\":\"example\",\"version\":\"1.0.0\"}\n"}, + category: "infrastructure", + tool: "Chef", + }, + { + name: "Chef Berksfile", + files: map[string]string{"Berksfile": "source 'https://supermarket.chef.io'\nmetadata\n"}, + category: "infrastructure", + tool: "Chef", + }, + { + name: "BitBake recipe", + files: map[string]string{"recipes/example/example_1.0.bb": "SUMMARY = \"Example recipe\"\n"}, + category: "build", + tool: "BitBake", + }, + { + name: "BitBake recipe append", + files: map[string]string{"recipes/example/example_1.0.bbappend": "FILESEXTRAPATHS:prepend := \"${THISDIR}/files:\"\n"}, + category: "build", + tool: "BitBake", + }, + { + name: "Buck build file", + files: map[string]string{"app/BUCK": "cxx_binary(name = \"app\", srcs = [\"main.cc\"])\n"}, + category: "monorepo", + tool: "Buck", + }, + { + name: "Buck metadata file", + files: map[string]string{"prelude/METADATA.bzl": "METADATA = {}\n"}, + category: "monorepo", + tool: "Buck", + }, + { + name: "Devbox config", + files: map[string]string{"devbox.json": "{\"packages\": [\"ripgrep@latest\"]}\n"}, + category: "environment", + tool: "Devbox", + }, + { + name: "Devbox lockfile", + files: map[string]string{"devbox.lock": "{\"lockfile_version\": \"1\"}\n"}, + category: "environment", + tool: "Devbox", + }, + { + name: "Jsonnet Bundler manifest", + files: map[string]string{"jsonnetfile.json": "{\"version\": 1, \"dependencies\": []}\n"}, + tool: "Jsonnet Bundler", + packageManager: true, + }, + { + name: "Jsonnet Bundler lockfile", + files: map[string]string{"jsonnetfile.lock.json": "{\"version\": 1, \"dependencies\": []}\n"}, + tool: "Jsonnet Bundler", + packageManager: true, + }, + { + name: "Puppet control repository", + files: map[string]string{"Puppetfile": "forge 'https://forge.puppet.com'\nmod 'puppetlabs/stdlib'\n"}, + category: "infrastructure", + tool: "Puppet", + }, + { + name: "Mint package list", + files: map[string]string{"Mintfile": "realm/SwiftLint@0.57.0\n"}, + tool: "Mint", + packageManager: true, + }, + { + name: "Meteor package definition", + files: map[string]string{"package.js": "Package.describe({ name: 'example:package', version: '1.0.0' });\n"}, + category: "build", + tool: "Meteor", + }, + { + name: "Meteor versions file", + files: map[string]string{".meteor/versions": "meteor-base@1.5.2\n"}, + category: "build", + tool: "Meteor", + }, + { + name: "Meteor JSON versions file", + files: map[string]string{"versions.json": "{\"meteor\": \"3.0.0\"}\n"}, + category: "build", + tool: "Meteor", + }, + { + name: "Helmfile config", + files: map[string]string{"helmfile.yaml": "releases: []\n"}, + category: "infrastructure", + tool: "Helmfile", + }, + { + name: "templated Helmfile config", + files: map[string]string{"helmfile.yaml.gotmpl": "releases: []\n"}, + category: "infrastructure", + tool: "Helmfile", + }, + { + name: "Helmfile directory config", + files: map[string]string{"helmfile.d/production.yaml": "releases: []\n"}, + category: "infrastructure", + tool: "Helmfile", + }, + { + name: "templated Helmfile directory config", + files: map[string]string{"helmfile.d/production.yaml.gotmpl": "releases: []\n"}, + category: "infrastructure", + tool: "Helmfile", + }, + { + name: "Argo CD YAML manifest", + files: map[string]string{"deploy/argocd.yaml": "apiVersion: argoproj.io/v1alpha1\nkind: Application\n"}, + category: "infrastructure", + tool: "Argo CD", + notTool: "Flux", + }, + { + name: "Argo CD YML manifest", + files: map[string]string{"deploy/argocd.yml": "apiVersion: argoproj.io/v1alpha1\nkind: AppProject\n"}, + category: "infrastructure", + tool: "Argo CD", + notTool: "Flux", + }, + { + name: "Argo CD ApplicationSet manifest", + files: map[string]string{"deploy/applicationset.yaml": "apiVersion: argoproj.io/v1alpha1\nkind: ApplicationSet\n"}, + category: "infrastructure", + tool: "Argo CD", + notTool: "Flux", + }, + { + name: "Flux YAML manifest", + files: map[string]string{"clusters/flux.yaml": "apiVersion: source.toolkit.fluxcd.io/v1\nkind: GitRepository\n"}, + category: "infrastructure", + tool: "Flux", + notTool: "Argo CD", + }, + { + name: "Flux YML manifest", + files: map[string]string{"clusters/flux.yml": "apiVersion: kustomize.toolkit.fluxcd.io/v1\nkind: Kustomization\n"}, + category: "infrastructure", + tool: "Flux", + notTool: "Argo CD", + }, + } + + knowledgeBase := loadKB(t) + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + dir := t.TempDir() + for path, content := range test.files { + writeProjectFile(t, dir, path, content) + } + + report, err := New(knowledgeBase, dir).Run() + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if test.packageManager { + if !slices.ContainsFunc(report.PackageManagers, func(d brief.Detection) bool { + return d.Name == test.tool + }) { + t.Errorf("expected %s package manager, got %v", test.tool, packageManagerNames(report)) + } + return + } + assertToolDetected(t, report, test.category, test.tool) + if test.notTool != "" { + assertToolNotDetected(t, report, test.category, test.notTool) + } + }) + } +} + +func TestMissingDevelopmentToolFixtures(t *testing.T) { + tests := []struct { + fixture string + category string + tool string + packageManager bool + }{ + {fixture: "babashka-project", category: "build", tool: "Babashka"}, + {fixture: "opentofu-project", category: "infrastructure", tool: "OpenTofu"}, + {fixture: "stack-project", tool: "Stack", packageManager: true}, + {fixture: "chef-project", category: "infrastructure", tool: "Chef"}, + {fixture: "bitbake-project", category: "build", tool: "BitBake"}, + {fixture: "buck-project", category: "monorepo", tool: "Buck"}, + {fixture: "devbox-project", category: "environment", tool: "Devbox"}, + {fixture: "jsonnet-bundler-project", tool: "Jsonnet Bundler", packageManager: true}, + {fixture: "puppet-project", category: "infrastructure", tool: "Puppet"}, + {fixture: "mint-project", tool: "Mint", packageManager: true}, + {fixture: "meteor-project", category: "build", tool: "Meteor"}, + {fixture: "helmfile-project", category: "infrastructure", tool: "Helmfile"}, + {fixture: "argocd-project", category: "infrastructure", tool: "Argo CD"}, + {fixture: "flux-project", category: "infrastructure", tool: "Flux"}, + } + + for _, test := range tests { + t.Run(test.tool, func(t *testing.T) { + report := runOn(t, "../testdata/"+test.fixture) + if test.packageManager { + if !slices.ContainsFunc(report.PackageManagers, func(d brief.Detection) bool { + return d.Name == test.tool + }) { + t.Errorf("expected %s package manager, got %v", test.tool, packageManagerNames(report)) + } + return + } + assertToolDetected(t, report, test.category, test.tool) + }) + } +} + +func TestGitOpsContentDetectionIgnoresGenericKubernetesYAML(t *testing.T) { + tests := []struct { + name string + content string + }{ + { + name: "generic Kubernetes manifest", + content: "apiVersion: apps/v1\nkind: Deployment\n", + }, + { + name: "Argo Workflows manifest", + content: "apiVersion: argoproj.io/v1alpha1\nkind: Workflow\n", + }, + { + name: "KubeVela application manifest", + content: "apiVersion: core.oam.dev/v1beta1\nkind: Application\n", + }, + { + name: "markers split across YAML documents", + content: "apiVersion: argoproj.io/v1alpha1\nkind: Workflow\n" + + "---\napiVersion: core.oam.dev/v1beta1\nkind: Application\n", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + dir := t.TempDir() + writeProjectFile(t, dir, "deploy/manifest.yaml", test.content) + + report := runOn(t, dir) + assertToolNotDetected(t, report, "infrastructure", "Argo CD") + assertToolNotDetected(t, report, "infrastructure", "Flux") + }) + } +} diff --git a/detect/filter.go b/detect/filter.go index c250d6e..402c4a0 100644 --- a/detect/filter.go +++ b/detect/filter.go @@ -27,7 +27,7 @@ func newFilterContext(knowledgeBase *kb.KnowledgeBase, changedFiles []string) *f changedExts := make(map[string]bool) for _, f := range changedFiles { if ext := filepath.Ext(f); ext != "" { - changedExts[ext] = true + changedExts[strings.ToLower(ext)] = true } } @@ -291,7 +291,8 @@ func matchesConfigFiles(tool *kb.ToolDef, changed map[string]bool) bool { func matchesDetectionPatterns(tool *kb.ToolDef, changed map[string]bool, changedExts map[string]bool) bool { return matchesPathPatterns(tool.Detect.Files, changed, changedExts) || - matchesPathPatterns(tool.Detect.ExcludeFiles, changed, changedExts) + matchesPathPatterns(tool.Detect.ExcludeFiles, changed, changedExts) || + (len(tool.Detect.YAMLResources) > 0 && (changedExts[".yaml"] || changedExts[".yml"])) } func matchesPathPatterns(patterns []string, changed map[string]bool, changedExts map[string]bool) bool { diff --git a/detect/filter_test.go b/detect/filter_test.go index 8079369..bf173f1 100644 --- a/detect/filter_test.go +++ b/detect/filter_test.go @@ -196,6 +196,21 @@ func TestToolMatchesChangedFiles_FileContainsGlob(t *testing.T) { } } +func TestToolMatchesChangedFiles_YAMLResource(t *testing.T) { + tool := &kb.ToolDef{ + Detect: kb.DetectInfo{ + YAMLResources: []kb.YAMLResourceInfo{{APIGroups: []string{"example.io"}}}, + }, + } + + if !toolMatchesChangedFiles(tool, map[string]bool{"deploy/example.yaml": true}, map[string]bool{".yaml": true}) { + t.Error("expected YAML resource tool to match changed YAML file") + } + if toolMatchesChangedFiles(tool, map[string]bool{"deploy/example.json": true}, map[string]bool{".json": true}) { + t.Error("expected YAML resource tool not to match changed non-YAML file") + } +} + func TestToolMatchesChangedFiles_ExcludeFileContains(t *testing.T) { tool := &kb.ToolDef{ Detect: kb.DetectInfo{ diff --git a/kb/kb.go b/kb/kb.go index 6fa7bcd..18b51a4 100644 --- a/kb/kb.go +++ b/kb/kb.go @@ -45,9 +45,17 @@ type DetectInfo struct { FileContains map[string][]string `toml:"file_contains"` ExcludeFileContains map[string][]string `toml:"exclude_file_contains"` KeyExists map[string][]string `toml:"key_exists"` + YAMLResources []YAMLResourceInfo `toml:"yaml_resources"` Ecosystems []string `toml:"ecosystems"` } +// YAMLResourceInfo identifies Kubernetes-style YAML resources by API group and +// optional kinds. Both fields are matched within the same YAML document. +type YAMLResourceInfo struct { + APIGroups []string `toml:"api_groups"` + Kinds []string `toml:"kinds"` +} + // CommandInfo holds the commands associated with a tool. type CommandInfo struct { Run string `toml:"run"` @@ -448,7 +456,7 @@ func (base *KnowledgeBase) Validate() error { func validateToolPaths(tool *ToolDef) error { total := len(tool.Detect.Files) + len(tool.Detect.ExcludeFiles) + len(tool.Detect.FileContains) + len(tool.Detect.ExcludeFileContains) + - len(tool.Detect.KeyExists) + len(tool.Config.Files) + len(tool.Detect.KeyExists) + len(tool.Detect.YAMLResources) + len(tool.Config.Files) if tool.Config.Lockfile != "" { total++ } @@ -496,6 +504,25 @@ func validateToolPaths(tool *ToolDef) error { return err } } + return validateYAMLResources(tool.Source, tool.Detect.YAMLResources) +} + +func validateYAMLResources(source string, resources []YAMLResourceInfo) error { + for i, resource := range resources { + if len(resource.APIGroups) == 0 { + return fmt.Errorf("%s: detect.yaml_resources[%d].api_groups must not be empty", source, i) + } + for _, group := range resource.APIGroups { + if strings.TrimSpace(group) == "" { + return fmt.Errorf("%s: detect.yaml_resources[%d].api_groups must not contain empty values", source, i) + } + } + for _, kind := range resource.Kinds { + if strings.TrimSpace(kind) == "" { + return fmt.Errorf("%s: detect.yaml_resources[%d].kinds must not contain empty values", source, i) + } + } + } return nil } diff --git a/kb/kb_test.go b/kb/kb_test.go index 84937e6..96ee5e2 100644 --- a/kb/kb_test.go +++ b/kb/kb_test.go @@ -195,6 +195,66 @@ func TestValidateRejectsBroadContentGlob(t *testing.T) { } } +func TestValidateRejectsExtensionContentGlob(t *testing.T) { + base := &kb.KnowledgeBase{Tools: []*kb.ToolDef{{ + Source: "knowledge/example.toml", + Detect: kb.DetectInfo{ + FileContains: map[string][]string{"**/*.yaml": {"marker"}}, + }, + }}} + if err := base.Validate(); err == nil { + t.Fatal("expected extension content glob validation error") + } +} + +func TestValidateYAMLResources(t *testing.T) { + t.Run("valid", func(t *testing.T) { + base := &kb.KnowledgeBase{Tools: []*kb.ToolDef{{ + Source: "knowledge/example.toml", + Detect: kb.DetectInfo{YAMLResources: []kb.YAMLResourceInfo{{ + APIGroups: []string{"example.io"}, + Kinds: []string{"Example"}, + }}}, + }}} + if err := base.Validate(); err != nil { + t.Fatalf("expected valid YAML resource signal: %v", err) + } + }) + + tests := []struct { + name string + resource kb.YAMLResourceInfo + }{ + { + name: "missing API group", + resource: kb.YAMLResourceInfo{Kinds: []string{"Example"}}, + }, + { + name: "blank API group", + resource: kb.YAMLResourceInfo{APIGroups: []string{" "}}, + }, + { + name: "blank kind", + resource: kb.YAMLResourceInfo{ + APIGroups: []string{"example.io"}, + Kinds: []string{" "}, + }, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + base := &kb.KnowledgeBase{Tools: []*kb.ToolDef{{ + Source: "knowledge/example.toml", + Detect: kb.DetectInfo{YAMLResources: []kb.YAMLResourceInfo{test.resource}}, + }}} + if err := base.Validate(); err == nil { + t.Fatal("expected YAML resource validation error") + } + }) + } +} + func TestValidateRejectsContentDirectory(t *testing.T) { base := &kb.KnowledgeBase{Tools: []*kb.ToolDef{{ Source: "knowledge/example.toml", diff --git a/knowledge/_shared/argocd.toml b/knowledge/_shared/argocd.toml new file mode 100644 index 0000000..735910e --- /dev/null +++ b/knowledge/_shared/argocd.toml @@ -0,0 +1,18 @@ +[tool] +name = "Argo CD" +category = "infrastructure" +homepage = "https://argo-cd.readthedocs.io" +docs = "https://argo-cd.readthedocs.io/en/stable/" +repo = "https://github.com/argoproj/argo-cd" +description = "Declarative GitOps continuous delivery for Kubernetes" + +[[detect.yaml_resources]] +api_groups = ["argoproj.io"] +kinds = ["Application", "AppProject", "ApplicationSet"] + +[commands] +run = "argocd app sync" + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/_shared/bitbake.toml b/knowledge/_shared/bitbake.toml new file mode 100644 index 0000000..a24d7ec --- /dev/null +++ b/knowledge/_shared/bitbake.toml @@ -0,0 +1,20 @@ +[tool] +name = "BitBake" +category = "build" +homepage = "https://www.openembedded.org/wiki/BitBake" +docs = "https://docs.yoctoproject.org/bitbake/" +repo = "https://github.com/openembedded/bitbake" +description = "Task executor and build engine for embedded Linux" + +[detect] +files = ["*.bb", "**/*.bb", "*.bbappend", "**/*.bbappend"] + +[commands] +run = "bitbake" + +[config] +files = ["*.bb", "**/*.bb", "*.bbappend", "**/*.bbappend"] + +[taxonomy] +role = ["build-tool"] +function = ["automation"] diff --git a/knowledge/_shared/buck.toml b/knowledge/_shared/buck.toml new file mode 100644 index 0000000..e6da161 --- /dev/null +++ b/knowledge/_shared/buck.toml @@ -0,0 +1,20 @@ +[tool] +name = "Buck" +category = "monorepo" +homepage = "https://buck2.build" +docs = "https://buck2.build/docs/" +repo = "https://github.com/facebook/buck2" +description = "Large-scale build system for monorepos" + +[detect] +files = ["BUCK", "**/BUCK", "METADATA.bzl", "**/METADATA.bzl"] + +[commands] +run = "buck2 build //..." +alternatives = ["buck2 test //...", "buck build //..."] + +[config] +files = [".buckconfig", "BUCK", "**/BUCK", "METADATA.bzl", "**/METADATA.bzl"] + +[taxonomy] +role = ["build-tool", "orchestrator"] diff --git a/knowledge/_shared/chef.toml b/knowledge/_shared/chef.toml new file mode 100644 index 0000000..67b6276 --- /dev/null +++ b/knowledge/_shared/chef.toml @@ -0,0 +1,22 @@ +[tool] +name = "Chef" +category = "infrastructure" +homepage = "https://www.chef.io" +docs = "https://docs.chef.io" +repo = "https://github.com/chef/chef" +description = "Infrastructure automation and configuration management" + +[detect] +files = ["metadata.rb", "metadata.json", "Berksfile"] + +[commands] +run = "chef-client" +alternatives = ["berks install", "chef exec"] + +[config] +files = ["metadata.rb", "metadata.json", "Berksfile", "Berksfile.lock"] +lockfile = "Berksfile.lock" + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/_shared/devbox.toml b/knowledge/_shared/devbox.toml new file mode 100644 index 0000000..181d2db --- /dev/null +++ b/knowledge/_shared/devbox.toml @@ -0,0 +1,22 @@ +[tool] +name = "Devbox" +category = "environment" +homepage = "https://www.jetify.com/devbox" +docs = "https://www.jetify.com/docs/devbox" +repo = "https://github.com/jetify-com/devbox" +description = "Portable development environment manager powered by Nix" + +[detect] +files = ["devbox.json", "devbox.lock"] + +[commands] +run = "devbox shell" +alternatives = ["devbox install", "devbox run"] + +[config] +files = ["devbox.json", "devbox.lock"] +lockfile = "devbox.lock" + +[taxonomy] +role = ["cli-tool"] +function = ["runtime-management"] diff --git a/knowledge/_shared/flux.toml b/knowledge/_shared/flux.toml new file mode 100644 index 0000000..432707a --- /dev/null +++ b/knowledge/_shared/flux.toml @@ -0,0 +1,23 @@ +[tool] +name = "Flux" +category = "infrastructure" +homepage = "https://fluxcd.io" +docs = "https://fluxcd.io/flux/" +repo = "https://github.com/fluxcd/flux2" +description = "GitOps continuous delivery toolkit for Kubernetes" + +[[detect.yaml_resources]] +api_groups = [ + "helm.toolkit.fluxcd.io", + "image.toolkit.fluxcd.io", + "kustomize.toolkit.fluxcd.io", + "notification.toolkit.fluxcd.io", + "source.toolkit.fluxcd.io", +] + +[commands] +run = "flux reconcile kustomization" + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/_shared/helmfile.toml b/knowledge/_shared/helmfile.toml new file mode 100644 index 0000000..d023720 --- /dev/null +++ b/knowledge/_shared/helmfile.toml @@ -0,0 +1,21 @@ +[tool] +name = "Helmfile" +category = "infrastructure" +homepage = "https://helmfile.readthedocs.io" +docs = "https://helmfile.readthedocs.io" +repo = "https://github.com/helmfile/helmfile" +description = "Declarative deployment manager for Helm charts" + +[detect] +files = ["helmfile.yaml", "helmfile.yaml.gotmpl", "helmfile.d/*.yaml", "helmfile.d/*.yaml.gotmpl"] + +[commands] +run = "helmfile sync" +alternatives = ["helmfile apply", "helmfile diff"] + +[config] +files = ["helmfile.yaml", "helmfile.yaml.gotmpl", "helmfile.d/*.yaml", "helmfile.d/*.yaml.gotmpl"] + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/_shared/jsonnet-bundler.toml b/knowledge/_shared/jsonnet-bundler.toml new file mode 100644 index 0000000..e4f6830 --- /dev/null +++ b/knowledge/_shared/jsonnet-bundler.toml @@ -0,0 +1,21 @@ +[tool] +name = "Jsonnet Bundler" +category = "package_manager" +homepage = "https://github.com/jsonnet-bundler/jsonnet-bundler" +docs = "https://github.com/jsonnet-bundler/jsonnet-bundler#readme" +repo = "https://github.com/jsonnet-bundler/jsonnet-bundler" +description = "Package manager for Jsonnet dependencies" + +[detect] +files = ["jsonnetfile.json", "jsonnetfile.lock.json"] + +[commands] +run = "jb install" +alternatives = ["jb update"] + +[config] +files = ["jsonnetfile.json", "jsonnetfile.lock.json"] +lockfile = "jsonnetfile.lock.json" + +[taxonomy] +role = ["package-manager"] diff --git a/knowledge/_shared/opentofu.toml b/knowledge/_shared/opentofu.toml new file mode 100644 index 0000000..6722983 --- /dev/null +++ b/knowledge/_shared/opentofu.toml @@ -0,0 +1,22 @@ +[tool] +name = "OpenTofu" +category = "infrastructure" +homepage = "https://opentofu.org" +docs = "https://opentofu.org/docs/" +repo = "https://github.com/opentofu/opentofu" +description = "Open source infrastructure as code provisioning tool" + +[detect] +files = ["*.tofu", "**/*.tofu"] + +[commands] +run = "tofu plan" +alternatives = ["tofu apply", "tofu init"] + +[config] +files = ["*.tofu", "**/*.tofu", ".terraform.lock.hcl"] +lockfile = ".terraform.lock.hcl" + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/_shared/puppet.toml b/knowledge/_shared/puppet.toml new file mode 100644 index 0000000..bebaa40 --- /dev/null +++ b/knowledge/_shared/puppet.toml @@ -0,0 +1,21 @@ +[tool] +name = "Puppet" +category = "infrastructure" +homepage = "https://www.puppet.com" +docs = "https://help.puppet.com/core/" +repo = "https://github.com/puppetlabs/puppet" +description = "Infrastructure automation and configuration management" + +[detect] +files = ["Puppetfile"] + +[commands] +run = "puppet apply" +alternatives = ["r10k puppetfile install"] + +[config] +files = ["Puppetfile"] + +[taxonomy] +layer = ["infrastructure"] +function = ["deployment"] diff --git a/knowledge/clojure/babashka.toml b/knowledge/clojure/babashka.toml new file mode 100644 index 0000000..22e6258 --- /dev/null +++ b/knowledge/clojure/babashka.toml @@ -0,0 +1,21 @@ +[tool] +name = "Babashka" +category = "build" +homepage = "https://babashka.org" +docs = "https://book.babashka.org" +repo = "https://github.com/babashka/babashka" +description = "Clojure scripting runtime and task runner" + +[detect] +files = ["bb.edn"] + +[commands] +run = "bb tasks" +alternatives = ["bb"] + +[config] +files = ["bb.edn"] + +[taxonomy] +role = ["build-tool"] +function = ["automation"] diff --git a/knowledge/haskell/stack.toml b/knowledge/haskell/stack.toml new file mode 100644 index 0000000..9a9c9e4 --- /dev/null +++ b/knowledge/haskell/stack.toml @@ -0,0 +1,21 @@ +[tool] +name = "Stack" +category = "package_manager" +homepage = "https://www.haskellstack.org" +docs = "https://docs.haskellstack.org" +repo = "https://github.com/commercialhaskell/stack" +description = "Haskell build tool and package manager" + +[detect] +files = ["stack.yaml"] + +[commands] +run = "stack build" +alternatives = ["stack test", "stack install"] + +[config] +files = ["stack.yaml", "stack.yaml.lock"] +lockfile = "stack.yaml.lock" + +[taxonomy] +role = ["package-manager"] diff --git a/knowledge/node/meteor.toml b/knowledge/node/meteor.toml new file mode 100644 index 0000000..b685f1d --- /dev/null +++ b/knowledge/node/meteor.toml @@ -0,0 +1,22 @@ +[tool] +name = "Meteor" +category = "build" +homepage = "https://www.meteor.com" +docs = "https://docs.meteor.com" +repo = "https://github.com/meteor/meteor" +description = "Full-stack JavaScript application platform" + +[detect] +files = ["package.js", ".meteor/versions", "versions.json"] + +[commands] +run = "meteor run" +alternatives = ["meteor build", "meteor test"] + +[config] +files = ["package.js", ".meteor/versions", "versions.json"] + +[taxonomy] +role = ["framework"] +layer = ["full-stack", "frontend", "backend"] +domain = ["web-development"] diff --git a/knowledge/swift/mint.toml b/knowledge/swift/mint.toml new file mode 100644 index 0000000..a71d648 --- /dev/null +++ b/knowledge/swift/mint.toml @@ -0,0 +1,20 @@ +[tool] +name = "Mint" +category = "package_manager" +homepage = "https://github.com/yonaskolb/Mint" +docs = "https://github.com/yonaskolb/Mint#readme" +repo = "https://github.com/yonaskolb/Mint" +description = "Package manager for executable Swift packages" + +[detect] +files = ["Mintfile"] + +[commands] +run = "mint bootstrap" +alternatives = ["mint run"] + +[config] +files = ["Mintfile"] + +[taxonomy] +role = ["package-manager"] diff --git a/testdata/argocd-project/application.yaml b/testdata/argocd-project/application.yaml new file mode 100644 index 0000000..9eed0ad --- /dev/null +++ b/testdata/argocd-project/application.yaml @@ -0,0 +1,2 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application diff --git a/testdata/babashka-project/bb.edn b/testdata/babashka-project/bb.edn new file mode 100644 index 0000000..846cbc5 --- /dev/null +++ b/testdata/babashka-project/bb.edn @@ -0,0 +1,2 @@ +{:tasks + {test {:task (println "test")}}} diff --git a/testdata/bitbake-project/example_1.0.bb b/testdata/bitbake-project/example_1.0.bb new file mode 100644 index 0000000..6e040a4 --- /dev/null +++ b/testdata/bitbake-project/example_1.0.bb @@ -0,0 +1 @@ +SUMMARY = "Example recipe" diff --git a/testdata/bitbake-project/example_1.0.bbappend b/testdata/bitbake-project/example_1.0.bbappend new file mode 100644 index 0000000..8802adb --- /dev/null +++ b/testdata/bitbake-project/example_1.0.bbappend @@ -0,0 +1 @@ +FILESEXTRAPATHS:prepend := "${THISDIR}/files:" diff --git a/testdata/buck-project/BUCK b/testdata/buck-project/BUCK new file mode 100644 index 0000000..ac7cbe2 --- /dev/null +++ b/testdata/buck-project/BUCK @@ -0,0 +1 @@ +cxx_binary(name = "app", srcs = ["main.cc"]) diff --git a/testdata/buck-project/METADATA.bzl b/testdata/buck-project/METADATA.bzl new file mode 100644 index 0000000..4247d0d --- /dev/null +++ b/testdata/buck-project/METADATA.bzl @@ -0,0 +1 @@ +METADATA = {} diff --git a/testdata/chef-project/Berksfile b/testdata/chef-project/Berksfile new file mode 100644 index 0000000..e0cac03 --- /dev/null +++ b/testdata/chef-project/Berksfile @@ -0,0 +1,2 @@ +source "https://supermarket.chef.io" +metadata diff --git a/testdata/chef-project/metadata.json b/testdata/chef-project/metadata.json new file mode 100644 index 0000000..efb3d67 --- /dev/null +++ b/testdata/chef-project/metadata.json @@ -0,0 +1 @@ +{"name":"example","version":"1.0.0"} diff --git a/testdata/chef-project/metadata.rb b/testdata/chef-project/metadata.rb new file mode 100644 index 0000000..c4d18f7 --- /dev/null +++ b/testdata/chef-project/metadata.rb @@ -0,0 +1,2 @@ +name "example" +version "1.0.0" diff --git a/testdata/devbox-project/devbox.json b/testdata/devbox-project/devbox.json new file mode 100644 index 0000000..6639909 --- /dev/null +++ b/testdata/devbox-project/devbox.json @@ -0,0 +1 @@ +{"packages":["ripgrep@latest"]} diff --git a/testdata/devbox-project/devbox.lock b/testdata/devbox-project/devbox.lock new file mode 100644 index 0000000..256492e --- /dev/null +++ b/testdata/devbox-project/devbox.lock @@ -0,0 +1 @@ +{"lockfile_version":"1"} diff --git a/testdata/flux-project/gotk-sync.yaml b/testdata/flux-project/gotk-sync.yaml new file mode 100644 index 0000000..6d1816f --- /dev/null +++ b/testdata/flux-project/gotk-sync.yaml @@ -0,0 +1,2 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization diff --git a/testdata/helmfile-project/helmfile.yaml b/testdata/helmfile-project/helmfile.yaml new file mode 100644 index 0000000..dd3b7f0 --- /dev/null +++ b/testdata/helmfile-project/helmfile.yaml @@ -0,0 +1 @@ +releases: [] diff --git a/testdata/jsonnet-bundler-project/jsonnetfile.json b/testdata/jsonnet-bundler-project/jsonnetfile.json new file mode 100644 index 0000000..6262de4 --- /dev/null +++ b/testdata/jsonnet-bundler-project/jsonnetfile.json @@ -0,0 +1 @@ +{"version":1,"dependencies":[]} diff --git a/testdata/jsonnet-bundler-project/jsonnetfile.lock.json b/testdata/jsonnet-bundler-project/jsonnetfile.lock.json new file mode 100644 index 0000000..6262de4 --- /dev/null +++ b/testdata/jsonnet-bundler-project/jsonnetfile.lock.json @@ -0,0 +1 @@ +{"version":1,"dependencies":[]} diff --git a/testdata/meteor-project/.meteor/versions b/testdata/meteor-project/.meteor/versions new file mode 100644 index 0000000..41b05cf --- /dev/null +++ b/testdata/meteor-project/.meteor/versions @@ -0,0 +1 @@ +meteor-base@1.5.2 diff --git a/testdata/meteor-project/package.js b/testdata/meteor-project/package.js new file mode 100644 index 0000000..c5a39fd --- /dev/null +++ b/testdata/meteor-project/package.js @@ -0,0 +1 @@ +Package.describe({ name: "example:package", version: "1.0.0" }); diff --git a/testdata/meteor-project/versions.json b/testdata/meteor-project/versions.json new file mode 100644 index 0000000..cea35a5 --- /dev/null +++ b/testdata/meteor-project/versions.json @@ -0,0 +1 @@ +{"meteor":"3.0.0"} diff --git a/testdata/mint-project/Mintfile b/testdata/mint-project/Mintfile new file mode 100644 index 0000000..3fdf476 --- /dev/null +++ b/testdata/mint-project/Mintfile @@ -0,0 +1 @@ +realm/SwiftLint@0.57.0 diff --git a/testdata/opentofu-project/main.tofu b/testdata/opentofu-project/main.tofu new file mode 100644 index 0000000..102e4ab --- /dev/null +++ b/testdata/opentofu-project/main.tofu @@ -0,0 +1,3 @@ +terraform { + required_version = ">= 1.8" +} diff --git a/testdata/puppet-project/Puppetfile b/testdata/puppet-project/Puppetfile new file mode 100644 index 0000000..51a89b2 --- /dev/null +++ b/testdata/puppet-project/Puppetfile @@ -0,0 +1,2 @@ +forge "https://forge.puppet.com" +mod "puppetlabs/stdlib" diff --git a/testdata/stack-project/stack.yaml b/testdata/stack-project/stack.yaml new file mode 100644 index 0000000..c103f70 --- /dev/null +++ b/testdata/stack-project/stack.yaml @@ -0,0 +1,2 @@ +resolver: lts-23.18 +packages: [.]