diff --git a/CHANGELOG.md b/CHANGELOG.md index c8c6669..c4c65f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -229,6 +229,18 @@ server without it, `send_early_hints` is a no-op and nothing changes. Off with `config.importmap.early_hints = false`. +### Changed + +- **Everything this gem knows about esm.run lives in `Importmap::EsmRun`.** + The provider name, the bundle URL shapes, the rewrite that turns a bundle's + `/npm/dep@ver/+esm` imports into bare specifiers and the jsDelivr version + lookup were nine things in `Importmap::Packager`, which had grown to the + 800-line ceiling with nowhere to put the next addition. Behaviour is + unchanged and no documented setting moved: `Importmap::Packager.esm_run_resolver` + still reads and writes the resolver, now on the new class. Only the `:nodoc:` + constants `Packager::ESM_RUN_*` are gone, as `Importmap::EsmRun::PROVIDER`, + `::CDN`, `::URL_REGEXP` and `::IMPORT_REGEXP`. + ### Fixed - **A CDN that fails mid-crawl leaves the pin alone.** Vendoring a graph makes diff --git a/CLAUDE.md b/CLAUDE.md index 316113d..fc159a3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,6 +88,7 @@ HttpRetries lib/importmap/http_retries.rb bounded retries ModuleInspector lib/importmap/module_inspector.rb whether a download can be served as one file (fork-only file) PackageGraph lib/importmap/package_graph.rb crawls a chunked package's siblings, rewrites them (fork-only file) VendoredGraph lib/importmap/vendored_graph.rb the graph directory and the pin_all_from line (fork-only file) +EsmRun lib/importmap/esm_run.rb the esm.run provider: URLs, import rewrite, versions (fork-only file) Installer lib/install/, lib/tasks/importmap_tasks.rake rails importmap:install ``` @@ -112,7 +113,7 @@ Two paths, kept apart: the **request path** (engine → Map → helpers, no I/O | Gemspec | `importmap-rails.gemspec` (deleted here) | `importmap-plus.gemspec` | | Entry point | `lib/importmap-rails.rb` (kept — still the real entry) | `lib/importmap-plus.rb` requires it | | Release | `bin/release` pushed from a laptop with an API key | `bin/release` → GitHub Release → trusted publishing (`release.yml`) | -| Fork-only files | — | `minifier.rb`, `http_retries.rb`, `module_inspector.rb`, `package_graph.rb`, `vendored_graph.rb`, `provider_chain.rb`, `integrity.rb`, `CHANGELOG.md`, `release.yml`, `deploy-docs.yml`, `docs-ci.yml`, `docs/` | +| Fork-only files | — | `minifier.rb`, `http_retries.rb`, `module_inspector.rb`, `package_graph.rb`, `vendored_graph.rb`, `provider_chain.rb`, `integrity.rb`, `esm_run.rb`, `CHANGELOG.md`, `release.yml`, `deploy-docs.yml`, `docs-ci.yml`, `docs/` | Upstream files this fork has modified heavily, which WILL conflict on sync: `commands.rb`, `packager.rb`, `npm.rb`, `README.md`, `ci.yml`, `test/commands_test.rb`, `test/packager_test.rb`. Per-file resolution rules: `.claude/rules/upstream-sync.md`. diff --git a/lib/importmap/esm_run.rb b/lib/importmap/esm_run.rb new file mode 100644 index 0000000..17573ca --- /dev/null +++ b/lib/importmap/esm_run.rb @@ -0,0 +1,118 @@ +require "uri" +require "json" + +# Everything this gem knows about jsDelivr's bundling endpoint +# (https://www.jsdelivr.com/esm): one minified ESM file per package, with its +# dependencies referenced as /npm/dep@ver/+esm. +# +# It is the one provider whose URLs a CDN host can't identify — an esm.run +# bundle and a plain jsDelivr file are both served from cdn.jsdelivr.net — and +# the one whose downloads have to be rewritten before they can be vendored, so +# both facts live here rather than in Importmap::Packager, which asks this +# class the two questions it has: is this that provider, and is this one of its +# URLs. +# +# Nothing here writes a file or reads config/importmap.rb. The version lookup +# is the one request, and it goes out through the Packager so a jsDelivr that +# rate-limits or resets gets the same bounded retries as every other request +# this gem makes, and fails as the same Importmap::Packager::HTTPError. +class Importmap::EsmRun + PROVIDER = "esm.run".freeze # :nodoc: + CDN = "https://cdn.jsdelivr.net/npm/".freeze # :nodoc: + URL_REGEXP = %r{\Ahttps://cdn\.jsdelivr\.net/npm/.+/\+esm\z}.freeze # :nodoc: + # A bundle's own imports: `from"/npm/dep@1.2.3/+esm"`, `import"…"`, + # `import("…")`, `export … from"…"`. Anchored on the keyword so an ordinary + # string that happens to look like a bundle URL is left alone. What it does + # not do is parse JavaScript, so the same text inside a string or a comment + # would still be rewritten — a jsDelivr bundle is esbuild output whose only + # surviving comment is the banner, and a root-relative /npm/ URL is + # meaningless anywhere but in one of its own imports. + IMPORT_REGEXP = + %r{((?:\bfrom|\bimport)\s*\(?\s*)(["'])/npm/((?:@[^/"'@]+/)?[^/"'@]+)@([^/"']+)((?:/[^"']*?)?)/\+esm\2}.freeze # :nodoc: + + # The jsDelivr data API versions are resolved through. Also readable and + # writable as Importmap::Packager.esm_run_resolver, which is where an app + # that points this at a mirror has always set it. + singleton_class.attr_accessor :resolver + self.resolver = URI("https://data.jsdelivr.com/v1/packages/npm/") + + class << self + def provider?(provider) + provider.to_s == PROVIDER + end + + def url?(url) + url.to_s.match?(URL_REGEXP) + end + + def url_for(name, version, subpath = nil) + "#{CDN}#{name}@#{version}#{subpath}/+esm" + end + + # Turns import "/npm/dep@1.2.3/+esm" into import "dep" so the bundle + # resolves through the import map, and lists what it needs pinned as + # [ [ package, url ], … ]. + def rewrite_imports(source) + dependencies = {} + versions = Hash.new { |hash, key| hash[key] = [] } + + rewritten = source.gsub(IMPORT_REGEXP) do + keyword, quote, name, version, subpath = $1, $2, $3, $4, $5.to_s + key = "#{name}#{subpath}" + dependencies[key] ||= url_for(name, version, subpath) + versions[key] << version unless versions[key].include?(version) + "#{keyword}#{quote}#{name}#{subpath}#{quote}" + end + + warn_about_conflicting_versions(versions) + + [ rewritten, dependencies.to_a ] + end + + private + # An import map maps a bare specifier to one file, so a bundle that + # imports the same package at two versions can only get the first one it + # asked for. Say so rather than pick silently. + def warn_about_conflicting_versions(versions) + versions.each do |key, seen| + next if seen.one? + + warn %(#{key} is imported at #{seen.join(", ")} by this bundle; pinning @#{seen.first}, an import map holds one version) + end + end + end + + def initialize(packager) + @packager = packager + end + + # The import map Importmap::Packager#import answers with, built from the + # version jsDelivr resolves each spec to; nil when it hasn't got one of them, + # because a bundle URL for a version nobody published is a 404 at download. + def imports(specs) + imports = Array(specs).to_h do |spec| + name, requested, subpath = spec.to_s.match(Importmap::Packager::PACKAGE_SPEC_REGEXP)&.captures + raise Importmap::Packager::Error, "Can't parse package spec #{spec.inspect}" unless name + + version = resolve_version(name, requested) + return nil unless version + + [ "#{name}#{subpath}", self.class.url_for(name, version, subpath) ] + end + + { imports: imports } + end + + private + def resolve_version(name, requested) + uri = self.class.resolver.dup + uri.path += "#{name}/resolved" + uri.query = "specifier=#{URI.encode_www_form_component(requested)}" if requested + + body = @packager.fetch_remote(uri, allow_missing: true, description: "resolving #{uri}") + + body && JSON.parse(body)["version"] + rescue JSON::ParserError + raise Importmap::Packager::HTTPError, "Unexpected response from #{uri}" + end +end diff --git a/lib/importmap/module_inspector.rb b/lib/importmap/module_inspector.rb index c3075da..2db1e74 100644 --- a/lib/importmap/module_inspector.rb +++ b/lib/importmap/module_inspector.rb @@ -6,7 +6,7 @@ # itself — a sibling module, a worker script, a wasm binary, its own directory # via import.meta.url — resolves to a 404 in the browser. # -# Like Packager::ESM_RUN_IMPORT_REGEXP this reads the source with regexes +# Like Importmap::EsmRun::IMPORT_REGEXP this reads the source with regexes # rather than parsing JavaScript, so the same text inside a string still counts. # It is deliberately the cautious direction: a false positive keeps a working # remote pin, and `pin --vendor` is the escape hatch. Every judgement call here diff --git a/lib/importmap/package_graph.rb b/lib/importmap/package_graph.rb index 32a741a..86747a9 100644 --- a/lib/importmap/package_graph.rb +++ b/lib/importmap/package_graph.rb @@ -36,7 +36,7 @@ class Importmap::PackageGraph # Importmap::ModuleInspector::RELATIVE_IMPORT_REGEXP with the specifier # captured, so the same forms it counts are the ones rewritten here. Like - # that one and Packager::ESM_RUN_IMPORT_REGEXP it doesn't parse JavaScript, + # that one and Importmap::EsmRun::IMPORT_REGEXP it doesn't parse JavaScript, # so a data string that spells out an import statement is rewritten inside # the string too, and a form it can't read — a magic comment between the # keyword and the specifier, an unterminated literal — isn't rewritten at diff --git a/lib/importmap/packager.rb b/lib/importmap/packager.rb index 9ce6642..06cf23c 100644 --- a/lib/importmap/packager.rb +++ b/lib/importmap/packager.rb @@ -7,6 +7,7 @@ require "importmap/vendored_graph" require "importmap/http_retries" require "importmap/integrity" +require "importmap/esm_run" class Importmap::Packager include Importmap::HttpRetries @@ -32,20 +33,6 @@ class Importmap::Packager "esm.sh" => "esm.sh" }.freeze # :nodoc: - # jsDelivr's bundling endpoint (https://www.jsdelivr.com/esm): one minified - # ESM file per package, with its dependencies referenced as /npm/dep@ver/+esm. - ESM_RUN_PROVIDER = "esm.run".freeze # :nodoc: - ESM_RUN_CDN = "https://cdn.jsdelivr.net/npm/".freeze # :nodoc: - ESM_RUN_URL_REGEXP = %r{\Ahttps://cdn\.jsdelivr\.net/npm/.+/\+esm\z}.freeze # :nodoc: - # An esm.run bundle's own imports: `from"/npm/dep@1.2.3/+esm"`, `import"…"`, - # `import("…")`, `export … from"…"`. Anchored on the keyword so an ordinary - # string that happens to look like a bundle URL is left alone. What it does - # not do is parse JavaScript, so the same text inside a string or a comment - # would still be rewritten — a jsDelivr bundle is esbuild output whose only - # surviving comment is the banner, and a root-relative /npm/ URL is - # meaningless anywhere but in one of its own imports. - ESM_RUN_IMPORT_REGEXP = - %r{((?:\bfrom|\bimport)\s*\(?\s*)(["'])/npm/((?:@[^/"'@]+/)?[^/"'@]+)@([^/"']+)((?:/[^"']*?)?)/\+esm\2}.freeze # :nodoc: # name[@version][/subpath] — a leading "@" distinguishes a scoped name # (@scope/pkg) from an unscoped name with a subpath (apexcharts/core). PACKAGE_SPEC_REGEXP = %r{\A(@[^@/]+/[^@/]+|[^@/]+)(?:@([^/]+))?(/.+)?\z}.freeze # :nodoc: @@ -111,9 +98,6 @@ def initialize(integrity: nil) singleton_class.attr_accessor :endpoint self.endpoint = URI("https://api.jspm.io/generate") - singleton_class.attr_accessor :esm_run_resolver - self.esm_run_resolver = URI("https://data.jsdelivr.com/v1/packages/npm/") - # CDNs reset connections and rate-limit bursts. Each request is tried this # many times, pausing retry_wait × attempt between tries, before it fails. # Shared with Importmap::Npm, which talks to the registry the same way. @@ -127,6 +111,13 @@ def retry_wait = Importmap::HttpRetries.wait def retry_wait=(value) Importmap::HttpRetries.wait = value end + + # Where --from esm.run resolves versions. Documented on this class since + # before Importmap::EsmRun held it, so it keeps answering here. + def esm_run_resolver = Importmap::EsmRun.resolver + def esm_run_resolver=(value) + Importmap::EsmRun.resolver = value + end end # Anything responding to #call(source) => String. Defaults to the first of @@ -159,7 +150,7 @@ def initialize(importmap_path = "config/importmap.rb", vendor_path: "vendor/java def import(*packages, env: "production", from: "jspm") @last_import_error = nil - return import_from_esm_run(packages) if esm_run?(from) + return Importmap::EsmRun.new(self).imports(packages) if esm_run?(from) response = post_json({ "install" => Array(packages), @@ -331,12 +322,12 @@ def download(package, url, minify: false, force: false, graph: true) # going to be vendored or only hashed for a remote pin. Writes nothing and # inspects nothing. Like #post_json, a failure the retry doesn't recognise # still comes out as this class's HTTPError rather than a backtrace. - def fetch_remote(url, allow_missing: false) - response = get_response(url) + def fetch_remote(url, allow_missing: false, description: "downloading #{url}") + response = get_response(url, description: description) # jspm answers some files brotli whatever the request advertises, and # Net::HTTP decodes gzip and deflate only. Not asked up front: supplying an # Accept-Encoding at all stops it decoding the gzip it does understand. - response = get_response(url, IDENTITY_ENCODING) if response.code == "200" && encoded?(response.body) + response = get_response(url, IDENTITY_ENCODING, description: description) if response.code == "200" && encoded?(response.body) if response.code == "200" # Still unreadable asked plain: say so, rather than let ModuleInspector raise. @@ -408,7 +399,7 @@ def remote_pin?(package) end def provider_for_url(url) - return ESM_RUN_PROVIDER if url.to_s.match?(ESM_RUN_URL_REGEXP) + return Importmap::EsmRun::PROVIDER if Importmap::EsmRun.url?(url) PROVIDER_HOSTS[URI(url.to_s).host] rescue URI::InvalidURIError @@ -416,7 +407,7 @@ def provider_for_url(url) end def esm_run?(provider) - provider.to_s == ESM_RUN_PROVIDER + Importmap::EsmRun.provider?(provider) end # The import-map key a package spec pins: "apexcharts@7.1.0/core" pins @@ -613,8 +604,8 @@ def parse_service_error(response) nil end - def get_response(url, headers = nil) - with_retries("downloading #{url}") { Net::HTTP.get_response(URI(url), headers) } + def get_response(url, headers = nil, description: "downloading #{url}") + with_retries(description) { Net::HTTP.get_response(URI(url), headers) } end # An encoding Net::HTTP couldn't undo is the one thing that reaches here as @@ -654,7 +645,7 @@ def remove_package_from_importmap(package) def download_package_file(package, url, minify: false, force: false, graph: true) body = fetch_remote(url) source = body.dup.force_encoding("UTF-8") - source, dependencies = rewrite_esm_run_imports(source) if url.match?(ESM_RUN_URL_REGEXP) + source, dependencies = Importmap::EsmRun.rewrite_imports(source) if Importmap::EsmRun.url?(url) @last_graph = Importmap::PackageGraph.for_download(self, package, url, source, body) if graph source = @last_graph.entry_source if @last_graph @@ -736,65 +727,6 @@ def commit_entry(package, partial) File.rename(partial, vendored_package_path(package)) end - # Turns import "/npm/dep@1.2.3/+esm" into import "dep" so the bundle - # resolves through the import map, and lists what it needs pinned. - def rewrite_esm_run_imports(source) - dependencies = {} - versions = Hash.new { |hash, key| hash[key] = [] } - - rewritten = source.gsub(ESM_RUN_IMPORT_REGEXP) do - keyword, quote, name, version, subpath = $1, $2, $3, $4, $5.to_s - key = "#{name}#{subpath}" - dependencies[key] ||= "#{ESM_RUN_CDN}#{name}@#{version}#{subpath}/+esm" - versions[key] << version unless versions[key].include?(version) - "#{keyword}#{quote}#{name}#{subpath}#{quote}" - end - - # An import map maps a bare specifier to one file, so a bundle that - # imports the same package at two versions can only get the first one - # it asked for. Say so rather than pick silently. - versions.each do |key, seen| - next if seen.one? - - warn %(#{key} is imported at #{seen.join(", ")} by this bundle; pinning @#{seen.first}, an import map holds one version) - end - - [rewritten, dependencies.to_a] - end - - def import_from_esm_run(packages) - imports = packages.to_h do |spec| - name, requested, subpath = spec.match(PACKAGE_SPEC_REGEXP)&.captures - raise Error, "Can't parse package spec #{spec.inspect}" unless name - - version = resolve_esm_run_version(name, requested) - return nil unless version - - ["#{name}#{subpath}", "#{ESM_RUN_CDN}#{name}@#{version}#{subpath}/+esm"] - end - - { imports: imports } - end - - def resolve_esm_run_version(name, requested) - uri = self.class.esm_run_resolver.dup - uri.path += "#{name}/resolved" - uri.query = "specifier=#{URI.encode_www_form_component(requested)}" if requested - - response = with_retries("resolving #{uri}") { Net::HTTP.get_response(uri) } - - case response.code - when "200" - JSON.parse(response.body)["version"] - when "404" - nil - else - handle_failure_response(response) - end - rescue JSON::ParserError - raise HTTPError, "Unexpected response from #{uri}" - end - def remove_sourcemap_comment_from(source) source.gsub(/^\/\/# sourceMappingURL=.*/, "") end diff --git a/test/esm_run_test.rb b/test/esm_run_test.rb new file mode 100644 index 0000000..bf91d78 --- /dev/null +++ b/test/esm_run_test.rb @@ -0,0 +1,105 @@ +require "test_helper" +require "importmap/esm_run" +require "importmap/packager" + +class Importmap::EsmRunTest < ActiveSupport::TestCase + test "recognises the provider by the name --from and a pin comment use" do + assert Importmap::EsmRun.provider?("esm.run") + assert Importmap::EsmRun.provider?(:"esm.run") + assert_not Importmap::EsmRun.provider?("jsdelivr") + assert_not Importmap::EsmRun.provider?(nil) + end + + test "tells a bundle URL apart from a plain jsDelivr file" do + assert Importmap::EsmRun.url?("https://cdn.jsdelivr.net/npm/md5@2.2.0/+esm") + assert Importmap::EsmRun.url?("https://cdn.jsdelivr.net/npm/@hotwired/stimulus@3.2.2/+esm") + assert_not Importmap::EsmRun.url?("https://cdn.jsdelivr.net/npm/md5@2.2.0/md5.js") + assert_not Importmap::EsmRun.url?("https://ga.jspm.io/npm:md5@2.2.0/md5.js") + end + + test "builds the bundle URL for a package, version and subpath" do + assert_equal "https://cdn.jsdelivr.net/npm/md5@2.2.0/+esm", + Importmap::EsmRun.url_for("md5", "2.2.0") + assert_equal "https://cdn.jsdelivr.net/npm/apexcharts@7.1.0/core/+esm", + Importmap::EsmRun.url_for("apexcharts", "7.1.0", "/core") + assert Importmap::EsmRun.url?(Importmap::EsmRun.url_for("@scope/pkg", "1.0.0", "/sub")) + end + + test "rewrites a bundle's own imports to bare specifiers and lists them" do + source = <<~JS + import{a}from"/npm/charenc@0.0.2/+esm";import b from '/npm/@scope/pkg@1.0.0/sub/path/+esm'; + const c = () => import("/npm/crypt@0.0.2/+esm"); + export default a + JS + + rewritten, dependencies = Importmap::EsmRun.rewrite_imports(source) + + assert_includes rewritten, %(from"charenc") + assert_includes rewritten, %(from '@scope/pkg/sub/path') + assert_includes rewritten, %(import("crypt")) + assert_no_match %r{["']/npm/}, rewritten + + assert_equal [ + [ "charenc", "https://cdn.jsdelivr.net/npm/charenc@0.0.2/+esm" ], + [ "@scope/pkg/sub/path", "https://cdn.jsdelivr.net/npm/@scope/pkg@1.0.0/sub/path/+esm" ], + [ "crypt", "https://cdn.jsdelivr.net/npm/crypt@0.0.2/+esm" ] + ], dependencies + end + + test "leaves a string that only looks like a bundle URL alone" do + source = %(import a from"/npm/charenc@0.0.2/+esm";const u="/npm/sneaky@1.0.0/+esm";export default[a,u]) + + rewritten, dependencies = Importmap::EsmRun.rewrite_imports(source) + + assert_includes rewritten, %(const u="/npm/sneaky@1.0.0/+esm") + assert_equal [ "charenc" ], dependencies.map(&:first) + end + + test "warns when a bundle imports one dependency at two versions and pins the first" do + source = %(import a from"/npm/charenc@0.0.1/+esm";import b from"/npm/charenc@0.0.2/+esm";export default[a,b]) + + dependencies = nil + _out, err = capture_io { _rewritten, dependencies = Importmap::EsmRun.rewrite_imports(source) } + + assert_match(/charenc is imported at 0\.0\.1, 0\.0\.2/, err) + assert_equal [ [ "charenc", "https://cdn.jsdelivr.net/npm/charenc@0.0.1/+esm" ] ], dependencies + end + + test "Packager.esm_run_resolver reads and writes the resolver this class holds" do + original = Importmap::EsmRun.resolver + + assert_equal original, Importmap::Packager.esm_run_resolver + + Importmap::Packager.esm_run_resolver = URI("https://mirror.example.com/v1/packages/npm/") + assert_equal URI("https://mirror.example.com/v1/packages/npm/"), Importmap::EsmRun.resolver + ensure + Importmap::EsmRun.resolver = original + end + + test "resolves a version through the resolver and answers nil for a package it hasn't got" do + packager = Importmap::Packager.new(Rails.root.join("config/importmap.rb")) + requested = [] + resolved = Class.new do + def code() "200" end + def body() { "version" => "9.9.9" }.to_json end + end.new + + Net::HTTP.stub(:get_response, ->(uri, *) { requested << uri.to_s; resolved }) do + assert_equal({ imports: { "md5" => "https://cdn.jsdelivr.net/npm/md5@9.9.9/+esm" } }, + Importmap::EsmRun.new(packager).imports([ "md5@2" ])) + end + + assert_equal [ "https://data.jsdelivr.com/v1/packages/npm/md5/resolved?specifier=2" ], requested + + missing = Class.new { def code() "404" end }.new + Net::HTTP.stub(:get_response, missing) do + assert_nil Importmap::EsmRun.new(packager).imports([ "nope-not-a-package" ]) + end + end + + test "an unparseable spec is this gem's own error, not a nil package" do + packager = Importmap::Packager.new(Rails.root.join("config/importmap.rb")) + + assert_raises(Importmap::Packager::Error) { Importmap::EsmRun.new(packager).imports([ "@scope" ]) } + end +end