Add OptiMind code and document it on README

.gitignore

@@ -25,3 +25,4 @@ build/*
 *.swp
 *.swo
 
+**/.DS_Store

README.md

@@ -30,6 +30,18 @@ The [milp-evolve](milp-evolve/) folder includes the following material to open s
 }
 ```
 
+### OptiMind: Teaching LLMs to Think Like Optimization Experts
+The [optimind](optimind/) folder includes the following material to open source the paper, evaluation code, and data:
+
+```latex
+@article{chen2025optimind,
+  title={OptiMind: Teaching LLMs to Think Like Optimization Experts},
+  author={Zhang, Xinzhi and Chen, Zeyi and Zope, Humishka and Barbalho, Hugo and Mellou, Konstantina and Molinaro, Marco and Kulkarni, Janardhan and Menache, Ishai and Li, Sirui},
+  journal={arXiv preprint arXiv:2509.22979},
+  year={2025}
+}
+```
+
 
 ## Responsible AI Considerations  
 

optimind/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Microsoft
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

optimind/README.md

@@ -0,0 +1,96 @@
+# OptiMind: Teaching LLMs to Think Like Optimization Experts
+
+[![arXiv](https://img.shields.io/badge/arXiv-2509.22979-b31b1b.svg)](https://arxiv.org/abs/2509.22979)
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-OptiMind--SFT-blue)](https://huggingface.co/microsoft/OptiMind-SFT)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://mit-license.org/)
+
+This repository contains the official evaluation code and cleaned datasets for the paper **"OptiMind: Teaching LLMs to Think Like Optimization Experts"**.
+
+OptiMind is a comprehensive framework designed to enhance Large Language Models (LLMs) for formulating Mixed-Integer Linear Programming (MILP) problems. By systematically integrating optimization domain expertise into both training (via data cleaning) and inference (via class-specific error hints), OptiMind significantly improves formulation accuracy over base models and other open-source baselines.
+
+## Overview
+
+The repository provides:
+1.  **Evaluation Scripts**: The exact code used in our paper to evaluate models using single-turn generation, majority voting, and multi-turn self-correction with solver feedback.
+2.  **Cleaned Benchmarks**: Expert-verified versions of IndustryOR, Mamo-Complex, and OptMATH, where we fixed missing data, ambiguities, and incorrect ground truths.
+3.  **Expert Hints**: The library of error-analysis hints used to guide the model during inference.
+
+## Usage
+
+### 1. Install Enviornment
+We recommend using `uv` for fast package management. Please follow the steps below to set up the environment.
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+uv venv .sglang --python 3.12 --seed
+source .sglang/bin/activate
+uv pip install "sglang" --prerelease=allow
+uv pip install pandas numpy matplotlib json_repair nest_asyncio gurobipy hf_transfer azure-identity pyarrow termcolor
+```
+
+**Note:** Make sure you have a valid Gurobi license to execute the solver-based evaluation and self-correction loops.
+
+### 2. Model
+Our fine-tuned model is available on Hugging Face: [microsoft/OptiMind-SFT](https://huggingface.co/microsoft/OptiMind-SFT).
+
+### 3. Running Evaluations
+First, clone the repository and navigate to the project directory:
+
+```bash
+git clone [https://github.com/microsoft/OptiGuide.git](https://github.com/microsoft/OptiGuide.git)
+cd OptiMind
+```
+
+We provide shell scripts to automate the evaluation pipeline, which includes running experiments across multiple random seeds, temperatures, and majority-voting configurations. Before running the evaluation scripts, you must update the file paths in `submit_eval.sh` and `submit_eval_gptoss.sh` to match your local environment.
+
+For `gpt-oss-20b` and its fine-tuned variants (including [our fine-tuned model](https://huggingface.co/microsoft/OptiMind-SFT)), use `submit_eval_gptoss.sh`:
+```
+bash submit_eval_gptoss.sh
+```
+For other open-source models (e.g., Qwen, Llama), use `submit_eval.sh` that excludes the  `--gpt-oss` and `--reasoning` flags.
+```
+bash submit_eval.sh
+```
+The evaluation result will be automatically saved in `eval_results/<benchmark_name>/<model_name_and_run_configurations>`. To aggregate the mean and stand deviation across the runs, run
+```
+python calculate_avg_results.py <path_to_the_run>
+```
+
+
+
+
+
+## Data and Benchmarks
+We provide our rigorously cleaned versions of three challenging benchmarks in the `/data` folder. These files are provided in CSV format with question and answer columns.
+
+
+- `data/optimind_cleaned_classified_industryor.csv`: Cleaned and classified IndustryOR benchmark.
+- `data/optimind_cleaned_classified_mamo_complex.csv`: Cleaned and classified Mamo-Complex benchmark.
+- `data/optimind_cleaned_classified_optmath.csv`: Cleaned and classified OptMATH benchmark.
+- `data/hints.csv`: The dictionary mapping problem classes to specific error summaries and hints.
+
+### Dataset Cleaning Comparisons
+
+In the `data/comparison/` folder, we provide detailed HTML summary tables that transparently document every modification made to the original benchmarks. These files allow you to align the original instances with our cleaned versions and verify the specific fixes (e.g., addressing missing parameters, ambiguity, or wrong ground truths).
+
+- `data/comparison/industryOR_original_vs_ours.html`: Comparison table aligning the original [IndustryOR](https://huggingface.co/datasets/CardinalOperations/IndustryOR) instances with our cleaned versions.
+- `data/comparison/OptMATH_original_vs_ours.html`: Comparison table aligning the [OptMATH](https://github.com/optsuite/OptMATH/blob/main/benchmark/OptMATH_Bench.json) instances with our cleaned versions.
+- `data/comparison/compare_SIRL_Ours.html`: A comparison between our cleaned IndustryOR set and the cleaned version from [SIRL](https://github.com/Cardinal-Operations/SIRL/blob/main/test_data/IndustryOR_fixedV2.json) (identifying residual issues in the latter).
+
+Each row in these HTML tables contains the Problem Index, Original Problem, Original Answer, Updated Problem, Updated Answer, and a "How did we fix it" description.
+
+
+## License 
+This project is licensed under the [MIT Licence](https://mit-license.org/).
+
+## Citation
+If you find our work, code, or datasets useful, please cite our paper:
+```
+@article{chen2025optimind,
+  title={OptiMind: Teaching LLMs to Think Like Optimization Experts},
+  author={Chen, Zeyi and Zhang, Xinzhi and Zope, Humishka and Barbalho, Hugo and Mellou, Konstantina and Molinaro, Marco and Kulkarni, Janardhan and Menache, Ishai and Li, Sirui},
+  journal={arXiv preprint arXiv:2509.22979},
+  year={2025}
+}
+```

optimind/SECURITY.md

@@ -0,0 +1,14 @@
+<!-- BEGIN MICROSOFT SECURITY.MD V1.0.0 BLOCK -->
+
+## Security
+
+Microsoft takes the security of our software products and services seriously, which
+includes all source code repositories in our GitHub organizations.
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+For security reporting information, locations, contact information, and policies,
+please review the latest guidance for Microsoft repositories at
+[https://aka.ms/SECURITY.md](https://aka.ms/SECURITY.md).
+
+<!-- END MICROSOFT SECURITY.MD BLOCK -->

optimind/calculate_avg_results.py

@@ -0,0 +1,93 @@
+from pathlib import Path
+import json
+import re
+import sys
+import math
+
+def average_turn_accuracies(root_dir: str, turns=None, use_new: bool = False):
+    """
+    Compute average accuracies and std dev for turn_0..turn_4 over subdirectories named 'xxx_j'.
+    Returns (averages_dict, stdevs_dict, counts_dict).
+
+    If use_new is True, read from stats_new.json instead of stats.json.
+    """
+    if turns is None:
+        turns = [f"turn_{i}" for i in range(5)]  # turn_0 .. turn_4
+
+    subdir_pat = re.compile(r".+_\d+$")
+    sums = {t: 0.0 for t in turns}
+    sumsq = {t: 0.0 for t in turns}
+    counts = {t: 0 for t in turns}
+
+    stats_filename = "stats_new.json" if use_new else "stats.json"
+
+    for d in Path(root_dir).iterdir():
+        if not d.is_dir() or not subdir_pat.match(d.name):
+            continue
+        stats_path = d / stats_filename
+        if not stats_path.exists():
+            continue
+        try:
+            data = json.loads(stats_path.read_text())
+        except Exception:
+            continue
+        acc = data.get("accuracy_per_turn", {})
+        for t in turns:
+            v = acc.get(t)
+            if isinstance(v, (int, float)):
+                v = float(v)
+                sums[t] += v
+                sumsq[t] += v * v
+                counts[t] += 1
+
+    averages = {t: (sums[t] / counts[t] if counts[t] else None) for t in turns}
+    stdevs = {}
+    for t in turns:
+        n = counts[t]
+        if n > 1:
+            # sample standard deviation (ddof=1)
+            var = (sumsq[t] - (sums[t] ** 2) / n) / (n - 1)
+            # guard tiny negative from FP error
+            stdevs[t] = math.sqrt(var) if var > 0 else 0.0
+        elif n == 1:
+            stdevs[t] = 0.0
+        else:
+            stdevs[t] = None
+
+    return averages, stdevs, counts
+
+def main():
+    prog = Path(sys.argv[0]).name
+    args = sys.argv[1:]
+
+    if not args:
+        print(f"Usage: {prog} [--new] <root_dir>")
+        sys.exit(1)
+
+    use_new = False
+    if "--new" in args:
+        use_new = True
+        args.remove("--new")
+
+    if not args:
+        print(f"Usage: {prog} [--new] <root_dir>")
+        sys.exit(1)
+
+    root_dir = args[0]
+    averages, stdevs, counts = average_turn_accuracies(root_dir, use_new=use_new)
+
+    # Print sorted by turn index
+    for t in sorted(averages, key=lambda k: int(k.split('_')[1])):
+        avg = averages[t]
+        std = stdevs[t]
+        n = counts[t]
+
+        if avg is None:
+            print(f"{t}: None (std=None) (n={n})")
+        else:
+            avg *= 100.0
+            std_str = "None" if std is None else f"{std * 100.0:.6f}"
+            print(f"{t}: {avg:.6f} ± {std_str} (n={n})")
+
+if __name__ == "__main__":
+    main()