arm64 Darwin behaviour

[dthierry]

As Kevin has reported sometimes, k_aug does not behave as expected in other systems.
This requires a bit more testing, particularly with Pyomo.

[kwmcbride]

For clarification, this issue does not appear on a machine running Linux Mint (the latest version at this time). I will be testing this on a Windows machine in the next couple of days to see if that works too.

At the moment, when I try to use k_aug, all problems fail with a similar message as the following (MacOS 11.2.3, M1):

ReactionModel: Generating base model (self._model)
# TemplateBuilder: Preparing model for p_estimator
# TemplateBuilder: Preparing model for None
WARNING: More finite elements were found in ContinuousSet 'alltime' than the
    number of finite elements specified in apply. The larger number of finite
    elements will be used.
meas_scaling 0.05
W[K_AUG]...	[K_AUG_ASL]No n_rhs declared.
W[K_AUG]...	[K_AUG_ASL]Target log10mu:= -11.
W[K_AUG]...	[K_AUG_ASL]This problem has no degrees of freedom
Pass the option square_override for the desired behaviour
W[K_AUG_ASL]...	[K_AUG_ASL]No ipopt_zL_in suffix declared, setting zL = 0.
W[K_AUG_ASL]...	[K_AUG_ASL]No ipopt_zU_in suffix declared, setting zU = 0.
W[K_AUG]...	[K_AUG_ASL]No f_timestamp suffix declared, Fallback to default writing mode.
I[K_AUG]...	[K_AUG_ASL] Filename for dot_sens dsdp_in_.in
W[K_AUG]...	[K_AUG_ASL]dsdp for linear C(x) + I*p = 0 override.
[K_AUG] 0.1.0, Part of the IDAES PSE framework
Please visit https://idaes.org/
: dsdp_mode=
dsdp_mode=
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]Attempting to make i_pert->d_c > 0.
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
E[K_AUG]...	[MA57_FACTOR]Failure, pivot tol is at it maximum.
W[K_AUG]...	[MA57_FACTOR]Inexact solution is now activated[Warning: results might not be good].
I[K_AUG]...	[K_AUG_ASL]File read successfully.
I[K_AUG]...	[K_AUG_ASL]Number of Right hand sides: 0
I[K_AUG]...	[K_AUG_ASL]Number of variables       : 10810
I[K_AUG]...	[K_AUG_ASL]Number of constraints     : 10810
I[K_AUG]...	[K_AUG_ASL]Number of valid n_dof     : 0
I[K_AUG]...	[ADJUST_MU]	Warning no relevant info from the problem can predict logmu
I[K_AUG]...	[FIND_INEQUALITIES]summary: eq: 10810, leq: 0, geq: 0 
I[K_AUG]...	[K_AUG_ASL]Nonzeroes in the sparse Jacobian 48610
I[K_AUG]...	[GET_HESS_ASL]Objective found
I[K_AUG]...	[GET_HESS_ASL]Nonzeroes in the sparse hessian 6905
I[K_AUG]...	[GET_HESS_ASL]Minimization problem detected
I[K_AUG]...	[GET_HESS_ASL]Current objective 121840159476.264603
I[K_AUG]...	[GET_HESS_ASL]Missing nz in the Hessian of the Lag: 9305
I[K_AUG]...	[K_AUG_ASL]Barrier term added.
I[K_AUG]...	[K_AUG_ASL]MC19 scaling...
I[K_AUG]...	[ASSM_RHS_DCDP]According to the suffixes declared len p is 4 
I[MA57]...	[]***
 Runtime parameters:
   Objective type: METIS_OBJTYPE_VOL
   Coarsening type: Unknown!
   Initial partitioning type: METIS_IPTYPE_GROW
   Refinement type: Unknown!
   Perform a 2-hop matching: Yes
   Number of balancing constraints: 1
   Number of refinement iterations: 227025648
   Random number seed: 304157472
   Number of separators: 1
   Compress graph prior to ordering: Yes
   Detect & order connected components separately: Yes
   Prunning factor for high degree vertices: 21488882.000000
   Allowed maximum load imbalance: 1.004 

Input Error: Incorrect objective type.
 nbrpool statistics
        nbrpoolsize:            0   nbrpoolcpos:            0
    nbrpoolreallocs:            0

I[MA57]...	[ma57bd_]ma57ad_: Status Ok.
I[MA57]...	[MA57_FACTOR]
I[K_AUG]...	[MA57_FACTOR]n_neig = 0
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000001
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000032
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000422
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.002943
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.012635
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.037685
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.085532
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.158160
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.250797
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.354399
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.459324
I[MA57]...	[Factorize Success.]***
I[MA57]...	[MA57_RATIO]
I[K_AUG]...	[MA57_SOLVE]: Ratio of norm of scaled residuals (reported); NaN 
I[K_AUG]...	[MA57_SOLVE]Accuracy at an acceptable level.

E[MA57]...	[Inertia check Failure!!!.]ERROR: Solver (asl) returned non-zero return code (255)
ERROR: See the solver log above for diagnostic information.
Traceback (most recent call last):

  File "/Users/kevinmcbride/Development/kipet_units/kipet_examples/examples/example_8/Ex_8_estimability.py", line 74, in <module>
    listparams = r1.e_analyzer.rank_params_yao(meas_scaling=meas_uncertainty,

  File "/Users/kevinmcbride/Development/kipet_units/kipet/kipet/estimability_tools/estimability_analysis.py", line 279, in rank_params_yao
    dsdp, idx_to_param = self.get_sensitivities_for_params(tee=True, sigmasq=sigmas)

  File "/Users/kevinmcbride/Development/kipet_units/kipet/kipet/estimability_tools/estimability_analysis.py", line 199, in get_sensitivities_for_params
    k_aug.solve(m, tee=True)

  File "/Users/kevinmcbride/Development/kipet_units/kipet/.venv/lib/python3.8/site-packages/pyomo/opt/base/solvers.py", line 601, in solve
    raise ApplicationError(

ApplicationError: Solver (asl) did not exit normally

[kwmcbride]

Another issue with this system is that the kaug_hess folder does not appear at all. This works fine on my current Linux machine. I will be testing this on windows too.

[dthierry]

It seems your problem has a (close to) singular Jacobian, so the back solve fails.
It can happen because of several reasons, e.g. if the initial point is not the same.
I've found that using the ThirdParty-Metis from coin-or has some weird behaviour in the arm64 Macs. So it has to be deactivated somehow. In other words, this could be a) numerical issue or b) Mac M1 thing.

If you generate the .nl file of this particular run, I can analyse the numerical part of it, because we can run it, in say linux, and see if it fails as well. The Metis thing might be more difficult to understand for now.

As for the folder, I've tried the print_kkt option on Mac 11.4 M1 and it does generate the GJH folder with the matrices does it not generate any folder at all in your system? E.g. try running k_aug somenlfile.nl print_kkt in the terminal and see if the folder is generated.

[kwmcbride]

It's probably Metis on the M1 then. I can run these problems without issue on the linux machine.

I'll try to generate the nl file and get it to you.

I can generate the GJH folder by using k_aug in the terminal.

[kwmcbride]

I found out that if the option "deb_kkt" is used at the same time as "print_kkt", the GJH folder is not created. This solves part of the problem.

[kwmcbride]

Okay, at least the reduced hessian parameter selection part works on MacOS now after making adjustments due to the different structure of the output info.

The others still produce the error shown above. I emailed you a copy of the nl file.

[kwmcbride]

Another point I should make is that k_aug is returning confidence intervals that are basically zero width (mac os). This may be related to Metis.

Sipopt is working though and can be used in the meantime.

[kwmcbride]

The problem in the last post is fixed. Kipet depended heavily on the old debug files and their absence lead to a chain of problems. I converted my reduced hessian methods to account for this.

[dthierry]

I've tried the nl file in Mac and I basically get the same situation as you've reported. I've also tried it on Linux and so forth and I get the same failure.
I guess the starting point is different in linux, in other words the local solution from the optimization phase is different. It would make sense if k_aug fails in every platform, otherwise the the local solutions from Ipopt are platform dependent, which is weird but totally possible.
I also noticed that the objective value is very large in comparison with the rest of the problem, so perhaps having different scaling within kipet might help.

[kwmcbride]

That is interesting. I'll have to run those examples again then.

I have been rewriting a bunch of reduced hessian code to make Kipet work with k_aug. I have finally tackled the covariance issue when using multiple datasets. I now run into this issue:

Solver solution file: '/var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpswqc0zw2.pyomo.sol'
Solver problem files: ('/var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpswqc0zw2.pyomo.nl',)
        0.41 seconds required for solver
        0.00 seconds required to read logfile 
        0.03 seconds required to read solution file
        0.09 seconds required for postsolve

W[K_AUG]...	[K_AUG_ASL]No n_rhs declared.
W[K_AUG]...	[K_AUG_ASL]Target log10mu:= -11.
[K_AUG] 0.1.0, Part of the IDAES PSE framework
Please visit https://idaes.org/
: print_kkt=
print_kkt=
bad line 71855 of /var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpgf4xwy7o.pyomo.nl: d28724
I[K_AUG]...	[K_AUG_ASL]File read successfully.
I[K_AUG]...	[K_AUG_ASL]Number of Right hand sides: 0
I[K_AUG]...	[K_AUG_ASL]Number of variables       : 14366
I[K_AUG]...	[K_AUG_ASL]Number of constraints     : 14364
I[K_AUG]...	[K_AUG_ASL]Number of valid n_dof     : 2
ERROR: Solver (asl) returned non-zero return code (1)
ERROR: See the solver log above for diagnostic information.

I assume this is a modeling error on my part and not related to the error above. Can you confirm this? I don't know what is meant by the d28724 being considered a bad line.

[dthierry]

I think something is wrong when the nl file is being written. Like passing something that is not a float/integer in your constraint expressions and so forth.
I bet that if you check your printed model you'd find a problematic expression. Or you could just open the nl file and go to the line 71855 and see if there's something there that points you to the model itself.

[kwmcbride]

You can close this now. I found out what the problem was and have no issues now. The error above is caused when using more than one collocation point. Other than that, the estimability code is working again.