[Proposal/Architecture] Keep MUIOGO upstream-compatible by default

[SeaCelo]

This is the repo-level follow-on that matters beyond v5.5.

The v5.5 dry run is encouraging because the overlap with upstream is still fairly concentrated. But that is only helpful if we manage the repo in a way that keeps future pulls small and predictable.

I want us to treat upstream compatibility as a default design rule for MUIOGO. That does not mean we stop doing downstream work. It means we should avoid creating incompatibility when an adapter, wrapper, or additive downstream layer would do the job just as well.

What this issue should decide:

• which shared MUIO files we consider the protected overlap surface
• when it is acceptable to edit those files directly versus adding a downstream wrapper/adapter
• what we expect PRs to say when they deliberately diverge from upstream
• how we review upstream releases so drift does not pile up for months

The practical goal is simple: future upstream pulls should feel more like maintaining a structured fork and less like rediscovering a custom merge strategy every time.

Done when:

• we have an explicit overlap inventory
• we have a default rule to prefer wrappers/adapters/additive modules over invasive rewrites
• PRs touching overlap files are expected to explain whether the change stays upstream-friendly or deliberately diverges
• future upstream sync work has a lightweight playbook

Related work:

• [Task] Absorb selected MUIO v5.5 changes on origin/main #388 [Task] Absorb selected MUIO v5.5 changes on origin/main (OPEN)
• **Proposal:** This issue defines the contribution-intake framework we intend to adopt to reduce PR firehose and improve quality/signal. #255 Proposal: This issue defines the contribution-intake framework we intend to adopt to reduce PR firehose and improve quality/signal. (OPEN)
• [Task] Implement Foundational CI/CD Infrastructure #375 [Task] Implement Foundational CI/CD Infrastructure (OPEN)
• [Task] [Proposal/Architecture] Canonical CLEWS Result Extraction & Validation Layer for OG-Core Integration #348 [Task] [Proposal/Architecture] Canonical CLEWS Result Extraction & Validation Layer for OG-Core Integration (OPEN)
• Feature: Unified User Interface for OG-Core and OSeMOSYS Integration (OG-CLEWS Framework) #380 Feature: Unified User Interface for OG-Core and OSeMOSYS Integration (OG-CLEWS Framework) (OPEN)

[parthdagia05]

I’ve been working through the v5.5 absorption PRs (#397, #398, #405 for #390, #391, #392), and I ran into exactly the kind of confusion this proposal is trying to address - figuring out what should be taken directly from upstream vs. where MUIOGO should intentionally differ.

A couple of things stood out from that work:

Variables.json and Const.Class.js were pretty clear cases of overlap. The upstream v5.5 versions are effectively the source of truth, and our local copies had drifted (incorrect unit rules, missing result groups, etc.). If we had an explicit “overlap inventory” earlier, it would’ve been much easier to spot that these should just be aligned with upstream.
DataFileRoute.py was more nuanced. Some changes from v5.5 could be ported directly, but others conflicted with MUIOGO’s path safety handling. In cases like this, having a guideline like “prefer wrapping over directly modifying upstream logic” would really help contributors make consistent decisions.
For my GSoC proposal (OG-Core integration), I’ve been trying to follow that same principle by keeping everything additive - introducing new files (OGCore.Class.js, OGParams.js, OGResults.js) instead of modifying existing CLEWS components. That feels much safer and aligns well with the adapter/wrapper approach being discussed here.

[lil-aditya]

Hi @SeaCelo I wanted to share a concise summary of the v5.5 sync work I’ve completed so far and clarify how I’ve approached the remaining items.

Work completed

PR #402 — Guardrails + smoke harness (closes #389, part of #388)

• Added a maintainer playbook for upstream syncs
  (docs/dev/upstream_sync_playbook.md)
  → documents overlap hotspots and patterns to explicitly reject during sync

• Implemented a clean-base verification script
  (scripts/verify_clean_base.py) that checks:

  • unresolved git state (merge/rebase indicators under .git/*HEAD)
  • conflict markers in tracked source files
  • Python syntax via py_compile
  • runs the repo’s smoke tests

• Added a minimal stdlib-only smoke test harness (tests_smoke/)

  • uses unittest (no new dependencies)
  • validates app import and a few stable routes

• Added an optional CI workflow (Python 3.11) to run the verification/smoke checks

• Updated CONTRIBUTING.md with instructions to run:

Constraints respected:

• no dependency additions
• stdlib-only test harness
• no assumption that repo root is writable

Goal: ensure upstream pulls have a reliable “clean-base gate” so a clean merge is not mistaken for a safe merge.

---

PR #407 — Backend/runtime port (closes #390)

• Ported selected upstream v5.5 backend/runtime changes while preserving MUIOGO constraints

• Updated solver model file

• Fixed resData / resDataPath bookkeeping

• Ensured cleanup preserves viewDefinitions.json (no loss of saved views)

• Added safe backend routes:

• /readModelFile

• /readLogFile
  → path-confined, read-only, no arbitrary filesystem access

• Adopted logging improvements:

• config-driven runtime directory (outside WebAPP)

• no cwd-relative paths, no startup log deletion

MUIOGO invariants preserved:

• absolute path handling
• safe subprocess execution (no shell=True)
• env → PATH → bundled solver resolution
• path validation / upload safety

---

PR #408 — Diagnostics UI (closes #391)

• Ported upstream v5.5 diagnostics UI:

• DataFile model/log viewer flow

• standalone ModelFile page

• MathJax-based equation rendering

• scenario toggle-all control

• Connected UI strictly to backend-safe routes from PR [v5.5] Backend/runtime port: safe routes, logging, resData fixes #407

• Enforced safe rendering:

• raw model/log text via DOM text nodes (no innerHTML)

• MathJax scoped to equation view only

Adaptations kept minimal and targeted:

• on-demand fetch instead of eager preload
• SPA routing corrections
• toggle-all respects locked SC_0

---

About PR4 / #392 (results metadata + shadow-price rendering)

I also implemented the result-metadata alignment locally (including adopting upstream Variables.json and aligning pivot/unit handling), which would correspond to PR4 for #392.

However, I did not open a separate PR for this because there are already active PRs addressing the same track (e.g., #405 / #406). I wanted to avoid creating a third competing PR and adding review noise.

Happy to:

• align with whichever PR is selected for [Task] Fix result metadata and shadow-price rendering from MUIO v5.5 #392
• or contribute follow-up fixes if anything remains uncovered there

---

Next step

I also believe it may be better to:

• refine and extend the existing documentation (e.g., playbook / CONTRIBUTING)
• incorporate:
• an explicit overlap inventory (protected surface)
• guidance on when to modify overlap files vs use wrappers/adapters
• expectations for PRs that touch overlap files (explicit compatibility/divergence note)

---

Please let me know if you’d like me to:

• rebase PR [v5.5] Backend/runtime port: safe routes, logging, resData fixes #407 / [v5.5] Diagnostics UI: model/log viewer + ModelFile page + MathJax #408 after upstream merges
• help validate or extend the [Task] Fix result metadata and shadow-price rendering from MUIO v5.5 #392 work

[Anirudh77715]

I went through this issue alongside #388 and the current codebase layout, and I think the framing here is exactly right —upstream compatibility as a default design rule, not a constraint that blocks downstream work.

A few observations from the repo as it stands:

Where the overlap surface is concentrated

Looking at the codebase, the shared upstream core appears to be concentrated in API/Classes/ — specifically the Base layer (FileClass.py, Config.py, CustomThreadClass.py) and the Case layer (CaseClass.py, DataFileClass.py, OsemosysClass.py). The Routes layer and WebAPP are more likely to be downstream-only. If that mapping is roughly correct, it might be worth making the boundary explicit — for example, a UPSTREAM_OVERLAP.md manifest listing the files that upstream owns semantically, so PRs touching those files get flagged early.

Adapter pattern as the practical default

The v5.5 sync (#388) reinforces the case for this. If downstream features are built as wrappers or extension modules rather than inline edits to shared files, then pulling v5.5 (or future versions) becomes a straightforward cherry-pick rather than a conflict resolution exercise. For example, rather than adding OG-Core logic directly into DataFileClass.py, a pattern like OGCoreDataFileAdapter that extends or wraps the upstream class would keep the shared surface clean.

PR-level enforcement

The intake check (check_pr_intake.py) already validates PR structure. It would be natural to add a "touches overlap files" flag — the script could compare the PR's changed file list against the overlap manifest and require a short "upstream-compatibility note" when overlap files are modified. This connects directly to the governance work in #255.

Lightweight sync playbook

For future syncs beyond v5.5, a documented playbook could be as simple as:

1. Diff upstream tag against last synced tag
2. Filter to overlap-manifest files only
3. Flag files with downstream edits for manual review
4. Everything else cherry-picks cleanly

Happy to help with any of the concrete deliverables here — the overlap inventory or the intake-check integration — if that would be useful.

[brightyorcerf]

Reading through the 'Done when' criteria and the recent sync challenges, it seems we’re converging on a Structured Fork model. To help move this from a design rule to a repeatable process, here are some of my thoughts:

• Explicit Overlap Inventory: We could categorize files by 'Ownership' (e.g., Upstream Assets like Variables.json, Core Overlap in API/Classes/, and MUIOGO Native extensions). This clarifies for contributors where an adapter is required vs. where a direct edit is safe.

• Default Design Rule: I suggest a 'Regression-Risk' threshold. Direct edits stay reserved for bug fixes/optimizations that don't touch MUIOGO-specific invariants (like absolute path handling or security). Structural changes or environmental conflicts (like the v5.5 result metadata) trigger a Wrap/Adapter requirement.

• Upstream Disclosure in PRs: To prevent drift from piling up, we could add a simple 'Upstream Compatibility Note' to the PR template for any change touching the overlap surface. It just needs to state if the change is Aligned, Divergent, or Wrapped, and the reasoning behind it.

• Lightweight Sync Workflow: Future pulls could follow a simple Diff → Filter → Classify process, isolating Tier B (Overlap conflicts) for manual review while cherry-picking the rest.

This could serve as a starting point for the Governance/Playbook deliverables. I’m happy to help draft the initial UPSTREAM_OVERLAP.md manifest or the intake-check logic if this approach aligns with the project's long-term maintenance goals.