From c2d6e037d8da346a92c3ac5932510ee767e53e1b Mon Sep 17 00:00:00 2001 From: Ben Fairless Date: Tue, 18 Aug 2026 08:59:45 +0800 Subject: [PATCH 1/3] Add AGENTS.md for AI coding agents Records the things about this gem that are easy to get wrong: that ~/.morph holds a live API key and must never end up in output or a fixture, that the upload includes every file not under a dot directory, that the server streams newline-delimited JSON rather than plain text, and that create_tar deliberately hands back an open rewound tempfile. Also notes that bare rake fails here because the Rakefile has no default task, that SimpleCov enforces a 90% floor so the suite can fail with every example passing, that .rubocop_todo.yml is an accepted backlog rather than something to regenerate, and that scraper.rb in the root is a leftover sample scraper rather than gem code. CLAUDE.md and .github/copilot-instructions.md are pointers, so the guidance lives in one place regardless of which tool reads it. Org-wide workflow and disclosure rules are referenced rather than restated, with the gh commands to fetch them. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ben Fairless --- .github/copilot-instructions.md | 3 + AGENTS.md | 98 +++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + 3 files changed, 102 insertions(+) create mode 100644 .github/copilot-instructions.md create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..ec7ce75 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,3 @@ +# Copilot instructions + +See [AGENTS.md](../AGENTS.md) for repository guidance. It applies here too. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5ab2575 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,98 @@ +# AGENTS.md + +This file provides guidance to AI coding agents (Claude Code, GitHub Copilot, +and others) when working with code in this repository. `CLAUDE.md` and +`.github/copilot-instructions.md` point here so the guidance lives in one place. + +## What this gem is + +`morph-cli` gives the `morph` command, which tars up the scraper in the current +directory, uploads it to a morph.io server, runs it there and streams the +output back to the terminal. The scraper never runs locally, which is the whole +point: developers get the morph.io scraper environment without installing it. + +The audience is developers, so error messages and documentation here can assume +command-line familiarity. + +## Layout + +- `lib/morph-cli.rb` is the substance: finding the files, building the tar, + posting it, and decoding the streamed response. +- `lib/morph-cli/cli.rb` is a thin Thor wrapper handling options, the API key + prompt and turning Faraday exceptions into messages worth reading. +- `bin/morph` runs `execute` when given no arguments, so bare `morph` and + `morph execute` are the same thing. + +## Things that will catch you out + +- **`~/.morph` holds a live API key.** It is YAML with separate + `:development` and `:production` sections, written with mode 0600 and read + back with `YAML.safe_load_file(..., permitted_classes: [Symbol])`. Never echo + its contents, paste them into an issue, or use a real key in a spec or + fixture. Use an obvious placeholder. +- **The upload includes every file in the directory** that isn't under a + directory starting with `.`, so `.git` is skipped but anything else the + person happens to have sitting there is not, including `data.sqlite`. Bear + that in mind before changing `all_paths`. +- **The server streams newline-delimited JSON, not plain text.** Each line has + a `stream` (`stdout`, `internalout` or `stderr`) and `text`, and `log` raises + on any other stream value. Partial chunks are buffered on the newline, so + changes there need to keep handling a chunk that splits a line. +- `create_tar` deliberately returns an open, rewound tempfile: it closes the + tar writer to flush the trailer but leaves the underlying handle usable for + the upload. The comments in that method say so; keep them true if you touch + it. +- The default request timeout is 600 seconds, overridable per environment with + `:timeout` in the config. `--dev` switches the whole thing to + `http://127.0.0.1:3000` for people working on morph.io itself. +- `scraper.rb` in the repository root is a leftover sample morph.io scraper + for a NSW council. It is tracked, but it is not gem code, nothing in `lib/` + loads it, and it is neither linted nor tested. It is there so you can run + `morph` against this directory by hand. Don't treat it as an example of the + house style. + +## Commands + + bundle exec rspec + bundle exec rubocop + bundle exec bundler-audit check --update + bundle exec rake build + +Those are the four CI jobs; the test job runs on Ruby 3.2, 3.3 and 3.4 and the +rest on 3.2. `.ruby-version` pins 3.2.2 locally. + +**`bundle exec rake` on its own fails here** with "Don't know how to build task +'default'". The Rakefile is nothing but `require "bundler/gem_tasks"`, so there +is no default task and no `rake spec`. Run `rspec` directly. + +`spec/spec_helper.rb` sets SimpleCov `minimum_coverage 90`, so the suite fails +on a coverage regression even when every example passes. It also sets +`disable_monkey_patching!`, so use `RSpec.describe` and the `expect` syntax, +and it loads `webmock/rspec` so specs don't reach the network. + +`.rubocop.yml` inherits `.rubocop_todo.yml`, which is an existing backlog of +accepted offences. Fix them or leave them, but don't regenerate that file +wholesale as part of an unrelated change. + +`Gemfile.lock` is gitignored and untracked. `bundle install` changes it +locally, which is expected. Never force-add it. + +## Releasing + +Don't run `bundle exec rake release`. It exists only because of +`bundler/gem_tasks` and would tag and publish from your machine. Releases are +automated: a version bump in `lib/morph-cli/version.rb` merged to `main` +publishes the gem through RubyGems trusted publishing in the `rubygems` +environment. Follow the README's "Releasing a new version" section. + +## Org-level guidance + +Workflow, branch naming, commit sign-off, AI disclosure and review conventions +are org-wide and deliberately not restated here. Fetch them when you need them: + + gh api repos/openaustralia/.github/contents/.github/CONTRIBUTING.md -H "Accept: application/vnd.github.raw" + gh api repos/openaustralia/.github/contents/AGENTS.md -H "Accept: application/vnd.github.raw" + +The README's Contributing section points people at the same guide. This +repository has no overrides of it. If one is ever agreed, record it here with +the reason, so the difference reads as a decision rather than drift. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md From e7671e29b791923ae0a822ac8b372adcab0d3a81 Mon Sep 17 00:00:00 2001 From: Ben Fairless Date: Tue, 18 Aug 2026 09:07:51 +0800 Subject: [PATCH 2/3] Don't assume the GitHub CLI is installed The org-level guidance section told agents to fetch CONTRIBUTING.md and AGENTS.md with "gh api", which only works where the GitHub CLI happens to be present. A Copilot session, a fresh container or an outside contributor may have neither it nor an authenticated token, and the guidance is public, so it doesn't need one. Links to both files plus their raw URLs instead, so any HTTP client will do. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ben Fairless --- AGENTS.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5ab2575..38a1f75 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,10 +88,18 @@ environment. Follow the README's "Releasing a new version" section. ## Org-level guidance Workflow, branch naming, commit sign-off, AI disclosure and review conventions -are org-wide and deliberately not restated here. Fetch them when you need them: +are org-wide and deliberately not restated here, so there is one copy to keep +current rather than a stale paraphrase in every repository. They live in +`openaustralia/.github`: - gh api repos/openaustralia/.github/contents/.github/CONTRIBUTING.md -H "Accept: application/vnd.github.raw" - gh api repos/openaustralia/.github/contents/AGENTS.md -H "Accept: application/vnd.github.raw" +- [CONTRIBUTING.md](https://github.com/openaustralia/.github/blob/main/.github/CONTRIBUTING.md) +- [AGENTS.md](https://github.com/openaustralia/.github/blob/main/AGENTS.md) + +To read them without leaving the terminal, fetch the raw files with any HTTP +client. Don't assume a particular tool is installed, the GitHub CLI included: + + https://raw.githubusercontent.com/openaustralia/.github/main/.github/CONTRIBUTING.md + https://raw.githubusercontent.com/openaustralia/.github/main/AGENTS.md The README's Contributing section points people at the same guide. This repository has no overrides of it. If one is ever agreed, record it here with From e68be38dd28d4dc180b361baca05c3e0bd56cb05 Mon Sep 17 00:00:00 2001 From: Ben Fairless Date: Tue, 18 Aug 2026 09:14:05 +0800 Subject: [PATCH 3/3] Be exact about which files the upload skips all_paths prunes directories whose name starts with a dot, but a dot-file isn't a directory, so it gets packed. Verified by calling all_paths against a temporary tree: a top-level .env comes back in the list, while a file inside .hidden_dir does not. The previous wording was true but easy to read as "hidden files are skipped", which is the wrong way round on the one case that has consequences. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ben Fairless --- AGENTS.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 38a1f75..a474e89 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,10 +30,13 @@ command-line familiarity. back with `YAML.safe_load_file(..., permitted_classes: [Symbol])`. Never echo its contents, paste them into an issue, or use a real key in a spec or fixture. Use an obvious placeholder. -- **The upload includes every file in the directory** that isn't under a - directory starting with `.`, so `.git` is skipped but anything else the - person happens to have sitting there is not, including `data.sqlite`. Bear - that in mind before changing `all_paths`. +- **The upload includes almost everything in the directory, dot-files + included.** `all_paths` prunes directories whose name starts with `.`, so + `.git` and `.bundle` are skipped, but a dot-*file* is not a directory and so + gets packed: a top-level `.env` is uploaded, as is `data.sqlite` and anything + else the person happens to have sitting there. Keep that distinction in mind + before changing `all_paths`, and don't describe the behaviour as "skips + hidden files", because it doesn't. - **The server streams newline-delimited JSON, not plain text.** Each line has a `stream` (`stdout`, `internalout` or `stderr`) and `text`, and `log` raises on any other stream value. Partial chunks are buffered on the newline, so