Kinetic radiation fit parameters Python sub-repository

.gitignore

@@ -68,3 +68,5 @@ gyrokinetic/data/adas/*.npy
 # Adas data files
 data/adas/*.npy
 install-deps/*.npy
+gyrokinetic/data/adas/__pycache__/
+gyrokinetic/data/adas/upt_read_radiation.py

gyrokinetic/data/adas/README.md

@@ -6,4 +6,8 @@ ADAS data is not included in gkylzero and must be downloaded by user.
 To download ADAS data and convert to format accessible by gkylzero,
 run machines/mkdeps.<machine_name>.sh with flag --download-adas=yes
 
-Radiation fits are produced with [code maintained by J. Roeltgen](https://github.com/jRoeltgen/radiation_operator.git), requiring Matlab.
+When running a simulation, gkeyll reads radiation data from `radiation_fit_parameters.txt`. The radiation fits were originally are produced with [code maintained by J. Roeltgen](https://github.com/jRoeltgen/radiation_operator.git), requiring Matlab. The radiation fits from this implementation are in `radiation_fit_parameters_legacy.txt`
+
+Now, we ported the matlab implemenation to python and the fits can be produced natively. The sub-repository to produce and adjust fits is present in the folder `kin-rad-py`, as well as intructions on how to use it. The current `radiation_fit_parameters.txt` file is produced with this new implementation. If fits are adjusted, remember to have the new `radiation_fit_parameters.txt` in this location, then compile gkeyll. This will copy the data to `${PREFIX:=$HOME/gkylsoft}/gkeyll/share/adas/radiation_fit_parameters.txt` and this effectevely the file use at runtime. 
+
+

gyrokinetic/data/adas/kin-rad-py/.gitignore

@@ -0,0 +1,21 @@
+# =========================================================================
+# kin-rad-py local gitignore
+# =========================================================================
+
+# ignore all local data folders completely
+raw_data/*.dat
+formatted_data/*.txt
+fits_data/fit-db_test*/
+fits_data/fit-db_prod_rs_server_1/plots/
+fits_data/fit-db_prod_rs_server_1/old_plots/
+
+# ignore python execution artifacts
+__pycache__
+__pycache__/
+*.pyc
+
+# ignore any ad-hoc script experiments
+# update-plots.py
+upt_read_radiation.py
+# get_densities.py # might be useful 
+

gyrokinetic/data/adas/kin-rad-py/README.md

@@ -0,0 +1,93 @@
+# kin-rad-gkeyll
+**automated kinetic radiation parameter fitting for gkeyll.**
+
+This repository houses the complete data ingestion and non-linear optimization pipeline for calculating plasma radiation parameters according to the [original repository](https://github.com/jRoeltgen/radiation_operator) written on matlab following the methodology outlined by [J. Roeltgen et al. NF 65 2025](https://iopscience.iop.org/article/10.1088/1741-4326/adff28): "A kinetic line-driven radiation operator and its application to Gyrokinetics". It takes raw bremsstrahlung and line radiation data from openadas, fits it to the analytical equations from the mentioned paper, and compiles the results into a lightweight database ready for the gkeyll kinetic simulation framework.
+
+---
+
+### Filesystem architecture
+
+The repository is separated data into ingestion, batch processing, testing and fine-tuning environments:
+
+```text
+kin-rad-gkeyll/
+├── raw_data/                  # raw openadas .dat files (git-ignored)
+├── formatted_data/            # roeltgen-formatted 4-column .txt files (git-ignored)
+├── fits_data/                 # the master output directory
+│   ├── roeltgen_data/         # legacy database from matlab implementation 
+│   ├── fit-db_<ID>/           # output folder for a specific batch run
+│   │   ├── master-fit_db_<ID>.txt   # csv database of all fits and metadata
+│   │   ├── gkeyll-db_fit_<ID>.txt   # the compiled gkeyll-ready text file
+│   │   └── plots/             # dual log-log/linear-log pngs of every fit
+│   └── individual_fits/       # directory for manually tuned fits (TBD)
+├── src/                       # core python modules
+│   ├── compile_gk_db.py       # produces gkeyll-ready .txt from a database id 
+│   ├── error_analysis.py      # handles zone-based fractional error physics
+│   ├── optimizer_core.py      # handles the scipy/matlab integration wrapper
+│   ├── data_parser.py         # gets a function Li(ne, Te) from adas data
+│   ├── fetch_adas_plt.py      # helper script to fetch raw ADAS plt files from openadas
+│   ├── format_adas.py         # formats raw .dat adas to formatted .txt
+│   ├── get_densities.py       # helper scrip to extract ne points use in original roeltgen fits
+│   ├── opt.py                 # helper scripts to run fmincon matlab (no longer needed)
+│   ├── optimize.m             # helper scripts to run fmincon matlab (no longer needed)
+│   └── update-plots.py        # re-produce the plots for a given database id 
+├── test/                      
+│   └── check_fit.py           # checks fit according to ID (TBD)
+├── download_data.py           # downloads and formats raw ADAS data 
+├── fit_batch.py               # fits a batch of radiation parameters for given ID 
+│                                and elements/charge/densities
+├── fit_single.py              # fine-tune a single fit given an ID
+├── swap_fits.py               # tool to overwrite batch fits with manual ones (TBD)
+├── .gitignore
+├── requirements.txt
+├── environment.yml
+└── README.md
+```
+
+### Usage
+
+Producing fit parameters for some elements is a straightforward two-step process: fetch the data, then run the batch optimizer.
+
+#### Data ingestion (`download_data.py`)
+Automatically fetches raw OpenAdas radiation data and translates it into our strict formatted `.txt` grids.
+
+```bash
+# download and format all supported elements
+python download_data.py
+# or just target a specific element
+python download_data.py --elements He
+```
+
+#### Batch optimization 
+fit_batch.py it sweeps through the $N_e$ grids, fits the radiation curves to roeltgen's analytical models, and dynamically saves the best parameters and plots to in the fits_data/ directory.
+
+```bash
+# fit all available elements and assign a unique dataset id saved in fits_data/fit-db_<ID>/
+python fit_batch.py --id run_1 --elements all --min-te 1.5
+```
+
+```bash
+# add elements to an existing database 
+python fit_batch.py --id run_1 --elements Al, He 
+```
+> *note: in case you want to overwrite the fits of an existing database, check the flag `--overwrite`*
+
+> *note: the current database included in this repository `prod_rs_server_1` is the current radiation parameters used in gkeyll.*
+
+#### Compile gkeyll database and how to use it at runtime
+After the batch optimization is done, you can compile the results into a gkeyll-ready text file with:
+```bash
+python src/compile_gk_db.py --id prod_rs_server_1
+```
+> *note: This is already done during `fit_batch.py`, but is just a helper script to re generate the database if needed. IMPORTANT: to actually use the generate data in gkeyll, remember to copy the generated `gkeyll-db_fit_<ID>.txt` file into `gkeyll/gyrokinetic/data/adas/` with the specific name: `radiation_fit_parameters.txt` and re-compile gkeyll. This will copy that file to `${PREFIX:=$HOME/gkylsoft}/gkeyll/share/adas/radiation_fit_parameters.txt` and this will be the file used at runtime.*
+
+#### Manual tuning  (TBD)
+
+If ever the automated fits need to be fine-tuned, the next step of this repository is to include a manual fitting environment in which a specific fit can be dialed "by hand" and store in `fits_data/individual_fits/` for later inserting into a database (e.g. `prod_rs_server_1`). The scripts for this environment are still in development, but the general idea is to have a set of tools that allow you to:
+
+- `fit_single.py`: manually dial in weights and $V_0$ bounds for a highly specific $N_e$ / $T_e$ slice.
+- `swap_fits.py`: replace a bad batch fit with your handcrafted fit inside the master database.
+- `test/check_fit.py`: visual plots for any fit ID directly from the terminal.
+
+> *note: created and designed by jaime caballero for my phd at DIFFER and the Gkeyll team to mimic original https://github.com/jRoeltgen/radiation_operator/tree/main by J. Roeltgen; the scripts in this repository were created with the help of ai.*
+

gyrokinetic/data/adas/kin-rad-py/download_data.py

@@ -0,0 +1,74 @@
+# created and designed by jaime caballero for my phd at differ and the gkeyll team
+# to mimic original https://github.com/jRoeltgen/radiation_operator/tree/main by J. Roeltgen
+# code written with some help of ai 
+#
+# download_data.py
+# this script is the entry point for the adas radiation data ingestion pipeline.
+# it reaches out to openadas, grabs the raw bremsstrahlung and line radiation 
+# files (.dat), and translates them into the strict 4-column layout defined 
+# by roeltgen. this gives us the clean L_z(T_e, n_e) to do non-linear fitting.
+
+
+import os
+import sys
+import glob
+import argparse
+
+# point python to the src directory so we can import our engine scripts
+sys.path.append(os.path.join(os.path.dirname(__file__), 'src'))
+
+# import the tools we built previously
+from fetch_adas_plt import fetch_plt_file, ADAS_PLT_FILES
+from format_adas import format_adas_to_roeltgen
+
+def main():
+    parser = argparse.ArgumentParser(description="fetch and format adas radiation data.")
+    parser.add_argument("--elements", type=str, default="all", 
+                        help="element symbol (e.g., H, He, Li) or 'all' to ingest everything.")
+
+    args = parser.parse_args()
+
+    # define our pipeline directories
+    raw_dir = "raw_data"
+    fmt_dir = "formatted_data"
+
+    print("=========================================")
+    print("--- ADAS DATA INGESTION PIPELINE ---")
+    print("=========================================\n")
+
+    # figure out which elements we are processing
+    if args.elements.lower() == "all":
+        elements_to_process = list(ADAS_PLT_FILES.keys())
+    else:
+        element_cap = args.elements.capitalize()
+        if element_cap not in ADAS_PLT_FILES:
+            print(f"[error] element '{args.elements}' is not supported.")
+            print(f"supported elements: {list(ADAS_PLT_FILES.keys())}")
+            sys.exit(1)
+        elements_to_process = [element_cap]
+
+    # phase 1: fetch the raw data from openadas
+    print(f"\n--- phase 1: fetching raw data ---")
+    downloaded_files = []
+    for el in elements_to_process:
+        # fetch_plt_file handles checking if it already exists to save bandwidth
+        filepath = fetch_plt_file(el, data_dir=raw_dir)
+        if filepath:
+            downloaded_files.append(filepath)
+
+    # phase 2: format the raw data for the optimizer
+    print(f"\n--- phase 2: formatting data to roeltgen standard ---")
+    if not downloaded_files:
+        print("no files were downloaded or found. aborting format phase.")
+        sys.exit(1)
+
+    for raw_file in downloaded_files:
+        # format_adas_to_roeltgen parses the adf11 grid and outputs the T_e, n_e, and L_z arrays
+        format_adas_to_roeltgen(raw_file, output_dir=fmt_dir)
+
+    print("\n=========================================")
+    print("ingestion complete. ready for batch fitting.")
+    print("=========================================")
+
+if __name__ == "__main__":
+    main()