Build and display the org-wide msdmd repository map - #44
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5b148dd0a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const repoData = JSON.parse(await readFile(GENERATED_REPOS, 'utf8')); | ||
| const inputs = await fetchRepositoryInputs(repoData.repositories || []); | ||
| const data = buildOrgMap(inputs); |
There was a problem hiding this comment.
When refresh:github falls back during a GitHub API outage or rate-limit event, repoData.fallback is true and the checked-in repos.last-known-good.json has no head_sha fields. Passing those repositories into this collector therefore marks every collection missing, after which the result is written as a successful map and last-known-good snapshot. The Pages workflow can consequently publish a zero-collection map during a transient upstream failure instead of preserving the previous map; detect the upstream fallback and use the organization-map fallback rather than rebuilding from headless entries.
Useful? React with 👍 / 👎.
| if (process.env.OFFLINE === '1') { | ||
| const fallback = JSON.parse(await readFile(SNAPSHOT_OUT, 'utf8')); |
There was a problem hiding this comment.
On a clean checkout, OFFLINE=1 npm run refresh:data now always fails here because this commit does not add src/_data/snapshots/org-msdmd.last-known-good.json. The new refresh:msdmd step reaches this unconditional read before generating any map, producing ENOENT and aborting the offline build; commit a seed snapshot or provide a fallback that can be constructed from checked-in data.
Useful? React with 👍 / 👎.
| if (collection.repo && collection.repo !== item.name) { | ||
| hmmm.push(`Collection declares repo=${collection.repo}; consumed repository is ${item.name}.`); |
There was a problem hiding this comment.
For generated collections that use the existing organization-qualified identity format—such as repo: "The-Interdependency/The-Interdependency.github.io" in The-Interdependency.github.io_msdmd.ts—item.name contains only the GitHub repository name. This comparison therefore reports a source-identity mismatch for a valid collection and pollutes its provenance evidence; compare against ${ORGANIZATION}/${item.name} or normalize both accepted forms first.
Useful? React with 👍 / 👎.
| } catch (error) { | ||
| const stderr = String(error?.stderr || ''); | ||
| const status = /404|not found/i.test(stderr) ? 'missing' : 'invalid'; |
There was a problem hiding this comment.
When a commit-pinned raw download fails because of DNS, timeout, TLS, or a transient raw.githubusercontent.com outage, its stderr does not contain 404, so this branch labels the repository collection invalid. The collector then publishes the degraded map and overwrites the last-known-good snapshot instead of activating fallback, potentially turning a temporary network failure into missing production evidence; reserve invalid for bytes that were successfully downloaded but could not be parsed, and propagate transport failures to the snapshot fallback.
Useful? React with 👍 / 👎.
| hmmm.push('Collection-declared source_commit does not match the repository head consumed by this website build.'); | ||
| } | ||
|
|
||
| for (const declaration of collection.declarations || []) { |
There was a problem hiding this comment.
If a syntactically valid remote collection contains a non-array declarations value, such as {}, parsing succeeds and the collection is marked ok, but this loop throws TypeError: object is not iterable. That aborts aggregation for every repository and invokes the global fallback rather than retaining this repository as explicit invalid evidence; validate declarations and gaps with Array.isArray as is already done for edges, and mark only the offending collection invalid.
Useful? React with 👍 / 👎.
| } catch { | ||
| return { sha: null, committed_at: null }; |
There was a problem hiding this comment.
If the organization repository listing succeeds but an individual commits API request transiently fails, this catch converts the error to a normal null head while leaving the overall snapshot in non-fallback mode. The refresh then skips that repository's reviewed manifest, marks its msdmd collection missing, and overwrites both last-known-good snapshots with the degraded data; distinguish a genuinely headless repository from an API failure and preserve the prior entry or fail into fallback for the latter.
Useful? React with 👍 / 👎.
1 participant
What
Makes the website the organization-scale collector and display consumer for repo-owned msdmd data.
Flow:
Repository authority does not transfer to the website.
Collector
defineMsdmdCollection(...)files without evaluating them;hmmmrather than guessing;src/_data/generated/orgMsdmd.jsonplus last-known-good snapshot when online.Display
/projects/map/;/assets/data/org-msdmd.json;Usage / limits
npm run refresh:datanow runsrefresh:githubbeforerefresh:msdmd. The repo-head lookup happens once inrefresh:github; msdmd collection bodies use commit-pinned raw GitHub, so the feature does not double the REST request count.Validation
Added
tests/org-msdmd.test.mjscovering:The repository's existing PR workflow runs the full
npm run checkplus Playwright browser/accessibility checks withGITHUB_TOKENsupplied.hmmm
The website's checked-in root
_msdmd.tscollection point predates this branch. The new source modules carry their own MODULE_BUILD/BOUNDARIES/CONTRACTS blocks, so its next canonical collection refresh will surface this collector and renderer; until then the org map will correctly expose any collection/source-head mismatch rather than silently treating the website's collection as current.