Debugging why model not simulated correctly

[hazhirr]

Hi,
After a bunch of adjustments in a model, I have finally been able to get the model in the link below to be processed by SDEverywhere, and graphs to be created, but the webassembly simulations simply don't match the vensim outputs, e.g. on 'Retired Residents' and 'Total Households', among others. Interestingly setting 'Min migration time' to a very large value changes the strange behavior and makes it more aligned with the vensim simulation (steady state), but still not the same (different numerical values). Any suggestions on how to debug the problem?
Here is the output of sdeverywhere

vs. the vensim results:

Here is the link to the model:
https://www.dropbox.com/scl/fi/23zybpd3g62z4udkm8lj1/SimpleCity-V05.mdl?rlkey=q37v0fmyy8gr1n51gd59ibdd7&dl=0

Thanks so much!

[chrispcampbell]

Hi @hazhirr, I'm happy to see that you're trying out SDEverywhere! Can you attach a zip file with some more files from your project? It doesn't need to be the whole thing, just the config folder and the sde.config.js file would be enough to help me investigate.

We have an internal tool for diagnosing these sorts of discrepancies, which helps one pinpoint where differences begin to occur between two runs (Vensim and SDE, or any combination). I'm in the middle of porting it over to the SDEverywhere repo (as part of the model-check tool) so that it can be used more easily by anyone. It's not quite ready yet, but your model/project will be a good test, and it will be good motivation to help me finish that up.

[hazhirr]

Thanks so much Chris! Here is the file with config folder and js file (attachment was not allowed, so dropbox link here): https://www.dropbox.com/scl/fi/t7606rdrighn0a4wtg1dv/ModelAndConfig.zip?rlkey=1wbqgclcu1c4lvk8vus832ztq&dl=0 Great to hear that you have a tool to help identify these and hopefully that can be shared with the community soon (though I am certainly not an advanced user or an informed coder). best, hazhir

…

On Thu, Oct 24, 2024 at 2:34 PM Chris Campbell ***@***.***> wrote: Hi @hazhirr <https://github.com/hazhirr>, I'm happy to see that you're trying out SDEverywhere! Can you attach a zip file with some more files from your project? It doesn't need to be the whole thing, just the config folder and the sde.config.js file would be enough to help me investigate. We have an internal tool for diagnosing these sorts of discrepancies, which helps one pinpoint where differences begin to occur between two runs (Vensim and SDE, or any combination). I'm in the middle of porting it over to the SDEverywhere repo (as part of the model-check tool) so that it can be used more easily by anyone. It's not quite ready yet, but your model/project will be a good test, and it will be good motivation to help me finish that up. — Reply to this email directly, view it on GitHub <#557 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AORHR5RPNBK7YBAYV4BP3E3Z5E4R3AVCNFSM6AAAAABQRLLENSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBUGQZTIMA> . You are receiving this because you were mentioned.Message ID: <climateinteractive/SDEverywhere/repo-discussions/557/comments/11044340@ github.com>

[chrispcampbell]

Hi @hazhirr, thanks for sharing the files, those are helpful. Prior to running your model through my (work-in-progress) tracing tool, I ran the "Model Explorer" app (the app in the generated project), and I saw a number of error messages related to ALLOCATE AVAILABLE in the browser's developer console:

(These messages are coming from the ALLOCATE AVAILABLE implementation in SDEverywhere's C runtime, and we don't currently surface these error messages in the runtime or in the generated app, which is why you can only see them in the developer console for now.)

I determined that your model is hitting two known issues (or caveats) in SDE's implementation of ALLOCATE AVAILABLE, plus one other small issue that was causing an incorrect conditional evaluation. I added temporary workarounds for the first three issues (I will file and fix these soon), and am working through a fourth issue that is likely the remaining cause of the discrepancies you are seeing.

It's unusual to have a model tickle multiple bugs in SDE at once like this; I'm sorry that you hit the jackpot here :) We have not had a lot of real world users of ALLOCATE AVAILABLE other than EPS (for which Todd implemented that function originally), so there are still some kinks to be worked out. Fortunately the tracing tool I mentioned has been helpful in narrowing down these discrepancies. I'll post again with more details soon once I sort out the rest. Thanks for your patience!

[hazhirr]

Thanks a lot Chris. I appreciate all the work you are doing on this. Under the hood, allocate available is indeed more complex than most other operations in Vensim and I am glad sdeverywhere has built it into the available functions, for my model it would be hard to implement the relevant allocations without it. Also, in case you are looking for potential enhancement opportunities, my original model was giving 3 other errors in sdeverywhere which I needed to fix with workarounds:
1- when the last input into allocate available (total resources) is a single element of a subscript, the function gave an error. That is why I had to separately define unsubscripted variables for Available P1, etc.
2- The vector elm map implementation I was using for calculating cascading flows in a subscripted stock gave an error, so I had to switch to a different implementation using subranges (for variable "Buildings"). This may be the same issue as the next item though.
3- SDeverywhere was not happy when I used sum() functions inside stock definitions, for aggregating flows coming from different subscripts, when going from a subscripted stock to one without, (i.e. Retiring taking people from Working Age Residents to Retired Residents).
All these errors will stop this earlier version of the model from being processed by sdeverywhere, and technically they all work in Vensim (but 2-3 are a bit non-standard, so I understand if you choose not to change the stricter checks in sdeverywhere).
https://www.dropbox.com/scl/fi/ouu7mtfgd3kb8io7624ha/SimpleCity-V04.mdl?rlkey=nbhvh6egts3edr254daae95c0&st=avw44qop&dl=0

Thanks!

[chrispcampbell]

Hi @hazhirr, thanks for pointing out those issues and including the link to the v04 model; that will be helpful. I will take a look at those and get issues filed once I get through the issues from your v05 model (I made more progress on those, just need to polish off the fixes and get them integrated).

To set some expectations on timing, I'm currently finalizing some new En-ROADS features, which will occupy most of my time in the next few days, but I will keep you updated as I get time for the SDE fixes.

Also, I'd be happy to set up a call at some point to show you the latest features, especially in model-check and how we use it on our team, since there have been a number of improvements since that last presentation. But probably should get through the current round of issues first since model-check will only be useful to you once we get these bugs sorted out!

[chrispcampbell]

Hi @hazhirr, just wanted to say that I haven't forgotten about this and that I'm still chipping away at the above. The latest En-ROADS release unexpectedly occupied all of my time the last couple weeks, but I've made progress on the SDE bug fixes and hope to finalize them soon. Will keep you posted.

[chrispcampbell]

Hi @hazhirr, I'm sorry that it took so long to work through these issues , but I'm happy to report that I have identified all the issues affecting your v05 model, and I have fixes in place on branches. I haven't merged the fixes to the main branch or published new packages yet because I want to run them by Todd for a review and I have a few more tests to add. But I do have a pre-release package that contains all the fixes rolled into one for you to try in the meantime (more on that below).

In the course of fixing the issues, I greatly increased our test coverage around cases involving different combinations and amounts of subscripts/dimensions and also around cases involving ALLOCATE AVAILABLE. Some of these issues have been known for a few years but seemed too daunting to fix at the time, so I'm glad your model gave me motivation to finally fix these :)

For the record, here are the relevant issues in the compile package:

• Incorrect code generated when multi-dimensional variable accessed by aliased dimensions #179
• Incorrect code generated when multiple subscripts resolve to same family #278
• ALLOCATE AVAILABLE uses the wrong priority subscript #377
• Math error for subscripts in expression position #568
• Unary negation has different precedence than in Vensim #569

The first four have been fully fixed and require no changes or workarounds in your v05 model. The last one is a case of ambiguous behavior, so for that one I suggest the following edits to your mdl:

diff --git a/SimpleCity-V05.mdl b/SimpleCity-V05.mdl
index b894606..d7a2e91 100644
--- a/SimpleCity-V05.mdl
+++ b/SimpleCity-V05.mdl
@@ -637,7 +637,7 @@ Max Feasible consumption[Edu]=
        ~               |
 
 Medu[Edu,Edu2]= INITIAL(
-       exp(-Dedu[Edu,Edu2]^2/2/Education Mixing^2))
+       exp(-(Dedu[Edu,Edu2]^2)/2/Education Mixing^2))
        ~       dmnl
        ~               |
 
@@ -2005,8 +2005,7 @@ INITIAL TIME  = 0
        ~       The initial time for the simulation.
        |
 
-SAVEPER  = 
-        TIME STEP
+SAVEPER  = 1
        ~       Year [0,?]
        ~       The frequency with which output is stored.
        |

There are two changes here:

• Add parentheses to the first term of Medu to avoid the operator precedence issue noted in Unary negation has different precedence than in Vensim #569.
• Change SAVEPER to 1. (Or you can change this to another value like 0.5 or 0.25.) This isn't strictly necessary, but it will improve performance slightly and reduce the amount of memory needed for running the model since you generally don't need to graph every data point for each TIME STEP, currently set to 0.03125.

To update your project to use the pre-release build, change one line in your package.json file:

diff --git a/package.json b/package.json
index b87e16a..d2a2d64 100644
--- a/package.json
+++ b/package.json
@@ -15,7 +15,7 @@
   "dependencies": {
     "@sdeverywhere/build": "^0.3.4",
     "@sdeverywhere/check-core": "^0.1.2",
-    "@sdeverywhere/cli": "^0.7.23",
+    "@sdeverywhere/cli": "climateinteractive/gitpkg-registry-public#sde-cli-069f3e8",
     "@sdeverywhere/plugin-check": "^0.3.5",
     "@sdeverywhere/plugin-config": "^0.2.4",
     "@sdeverywhere/plugin-vite": "^0.1.8",

Then run npm i to install, and npm run dev to start the local builder.

The graph from your example configuration now looks correct:

There are still some very slight (around 0.0001) differences in output as compared to Vensim. Most of the time SDEverywhere produces even closer results to Vensim, except in the case of ALLOCATE AVAILABLE which is noted in this caveats comment.

Please let me know if you run into any issues with the above information, or have any questions.

My next steps are:

• Add the remaining tests and get those issues reviewed, merged, and published.
• File issues for any those v04 issues that you noted earlier (some may have already been addressed by my recent fixes).
• Add more documentation to the wiki about using the model-check tool with more examples of how to write comparison and check tests.

[hazhirr]

Thanks so much Chris! Really appreciate the updates and looking forward to trying them on my model.

[chrispcampbell]

Hi @hazhirr, I published new versions of the @sdeverywhere packages that include the fixes noted above, so you no longer need to use that special pre-release version.

The easiest way to upgrade all @sdeverywhere packages in your project is to open a terminal window, cd to your project directory, then type the following:

npx npm-check-updates -ws --root -u --install=always -f "@sdeverywhere/*"

That's the more automatic way of upgrading to the latest versions of the SDE packages. The more manual alternative is to edit the package.json files to use the latest versions, then type npm install to install them.

To follow up on the 3 issues you noted above that affected your v04 model:

1- when the last input into allocate available (total resources) is a single element of a subscript, the function gave an error. That is why I had to separately define unsubscripted variables for Available P1, etc.

I confirmed that this issue is resolved by my fixes for #377 which were included in the latest published @sdeverywhere/compile package. Therefore, you can change your v05 model formulation back to the way it was in v04, for example, you can use Available P[xjtN] and can remove those Available PN variables.

BTW, I noticed that there was a typo in your v05 in Available P3, it was using Available P[xjt2] instead of xjt3, so I guess that's a good reason to go back to the Available P approach :)

2- The vector elm map implementation I was using for calculating cascading flows in a subscripted stock gave an error, so I had to switch to a different implementation using subranges (for variable "Buildings"). This may be the same issue as the next item though.

I see. The VECTOR ELM MAP implementation looks a bit fragile in the case of more than one subscript, and it looks like we may have a bug that needs more investigating (I started looking but it's daunting).

For the case you have in v04 (or even your different implementation in v05), since there aren't too many subscript combinations involved, I wonder if it would be easier / clearer to define separate equations for the inner part, like:

Mapped Building Maturing[BuildingTypes,BA1] = Building Completion[BuildingTypes] ~~|
Mapped Building Maturing[BuildingTypes,BA2] = Building Maturing[BuildingTypes, BA1] ~~|
...

But that's just a hand-wavy suggestion, not sure if it's helpful.

3- SDeverywhere was not happy when I used sum() functions inside stock definitions, for aggregating flows coming from different subscripts, when going from a subscripted stock to one without, (i.e. Retiring taking people from Working Age Residents to Retired Residents).

For this one I wasn't able to find the exact place in the v04 model that you're referring to. If you have time, could you point me to the specific variable names that are involved (or show me a minimal reproducible example)?

[hazhirr]

Dear @chrispcampbell,
Thanks so much for all the updates and fixes, they make a big difference and I am looking forward to trying SDE again.
Here is a link to v04 model which shows the equation that did not work (e.g. for 'Retired' stock), with sum() inside the stock variable equation.
https://www.dropbox.com/scl/fi/ouu7mtfgd3kb8io7624ha/SimpleCity-V04.mdl?rlkey=nbhvh6egts3edr254daae95c0&dl=0

Best,
Hazhir