Feat/energy consumption

.gitignore

@@ -13,4 +13,6 @@ models
 wandb/*
 .cache
 *.gradio*
-*.egg*
+*.egg*
+*.sbatch
+

NOTES.md

@@ -0,0 +1,55 @@
+ The estimator's FLOP-based model overestimates for larger models because it doesn't account for real-world hardware efficiency gains (optimized kernels, memory hierarchy effects, batching).
+
+Larger models emit more CO₂, but not linearly — going from 1B to 14B parameters (~14× more params) only increases emissions ~3-5×, probably because runtime doesn't scale linearly with parameter count for inference/evaluation workloads.
+
+GPU utilization increases with model size — small models (1-2B) use only 25–35% of the GPU, while large models (8–14B) push utilization to 70–88%.
+
+LLaMA 3.2 is the most efficient (1B: 6.1g CO₂, 216s)
+
+Qwen3 is mid-range (1.7B: 11.9g, 454s)
+
+and Gemma 3 is the least efficient (~1B: 11.0g, 458s)
+
+
+Comparison con le stesse famiglie anche magari tra gemma2, gemma3
+In Context Learning
+
+Con il CPU load mode fallback utilizza P=TDP×(0.1+0.9×(cpu_load/100) 
+3 ) con TDP = 450w
+
+cpu_load = psutil.cpu_percent(interval=0.5, percpu=False)
+
+the cpu_load reflects the entire node's CPU usage — including other users' jobs if the node isn't exclusively allocated. That's why your force_cpu_power=450 matters: even if the node shows 80% load, some of that may not be yours.
+
+Provare a runnare di nuovo con tracking process
+
+Configurazioni per il train.
+
+
+
+
+
+
+
+
+
+### 1. Dataset Selection and Preprocessing
+To isolate model architecture as the primary variable for energy consumption, your dataset pipeline must be perfectly rigid.
+
+* **Select a Single Baseline Dataset:** As discussed, use a single dataset like **MedQA (USMLE)** or **MedMCQA** to ensure consistent reasoning complexity and average sequence length across all models.
+* **Standardize the Token Count (Crucial):** Do not train for "3 epochs." Because different models use different tokenizers (e.g., Llama's tokenizer splits words differently than Gemma's), 1 epoch of MedQA might be 10 million tokens for Model A but 11 million for Model B. Set your training loop to stop after exactly $X$ million tokens have been processed. If possible we can have this stop_criteria as a parameter in the config
+* **Unified Prompt Template:** Wrap every training example in the exact same instruction format (e.g., `Instruction: [Question] \n Options: [A,B,C,D] \n Answer: [Target]`). 
+
+
+### 3. Model Configuration (PEFT / QLoRA)
+Full-parameter fine-tuning is likely too resource-intensive and noisy for this comparison. You should use Parameter-Efficient Fine-Tuning (PEFT).
+
+* **Standardize LoRA Parameters:** Apply the exact same adapter size to all models. For example, fix your LoRA Rank ($r=16$) and Alpha ($\alpha=32$). This ensures you are testing how efficiently the base model integrates new knowledge, rather than testing different adapter sizes.
+* **Maximize Batch Size:** To measure true efficiency, you must saturate the GPU. Find the maximum batch size that fits into the GPU memory for *each* model. A model running at 40% memory utilization will yield artificially poor energy efficiency metrics.
+
+### 5. Evaluation and Comparison
+Once the models are fine-tuned, you need to map their energy cost against their new capabilities.
+
+* **Standardize Inference:** When testing the models on the test split to get their accuracy scores, use strict, fixed hyperparameters (e.g., `temperature = 0.1`, `top_p = 0.95`).
+* **Plot the Pareto Frontier:** Create a scatter plot with **Total Fine-Tuning Energy (Wh)** on the X-axis and **Benchmark Accuracy (%)** on the Y-axis. The models that sit on the top-left edge of this plot represent the optimal trade-off between energy efficiency and learned performance. Add this as a separate function called energy_accuracy_plot in a folder called plots or similar. Add it to evaluation.
+

README.md

@@ -110,6 +110,126 @@ include_path: config_files/evaluator_configs/tasks
 - Running via SLURM:
   - Configure venv and config paths in the slurm script ``scripts/autollm_eval.sbatch``. 
   - Run the script: ``sbatch scripts/autollm_eval.sbatch``.
+
+#### Step 3. Refresh Pareto frontier panels (post-hoc)
+
+Each eval run logs its own energy (`emissions/*`) and accuracy (`eval/*`)
+metrics to wandb, **but not** the `pareto/<label>/*` fields used by the
+Pareto-frontier workspace. Pareto optimality is a **cross-run** property: a
+run can only know its rank relative to every other run in the project, so
+the frontier must be computed once per batch — not during a single run.
+
+After a sweep or batch of eval jobs has finished, refresh the frontier flags
+and workspace panels by running the standalone backfill script. This is a
+lightweight wandb-API call (no GPU, no dataset loading) — run it directly on
+the cluster login/head node, inside the same venv used for eval. **Do not
+submit it via `sbatch`.**
+
+```shell
+# On the cluster login node:
+source $VENV_PATH/bin/activate
+source $ENV_VARIABLES_PATH       # exports WANDB_API_KEY
+
+python scripts/wandb_pareto_plot.py \
+    --entity <wandb-entity> \
+    --project <wandb-project>
+```
+
+The script is idempotent — re-running it after new eval jobs simply
+recomputes the frontier and overwrites the `pareto/<label>/*` summary fields
+on every run. Useful flags:
+
+- ``--dry-run``: report per-label frontier sizes without writing anything.
+- ``--skip-panel``: only backfill summary fields, skip the workspace upsert.
+- ``--skip-backfill``: only (re)create the workspace view.
+- ``--tag <tag>``: restrict to runs carrying a given tag (default: ``energy-profiling``).
+
+##### Auto-refresh option (opt-in)
+
+If you want each energy-eval run to update the workspace itself — without
+having to remember to launch `scripts/wandb_pareto_plot.py` after every batch
+— add an ``auto_pareto`` block to your evaluator YAML. When the eval job
+finishes and the energy metrics have been flushed to wandb, the same Slurm
+job calls `refresh_pareto_workspace(...)` in-process, on the same venv,
+using the same ``WANDB_API_KEY`` already exported for energy logging.
+
+The hook lives in `auto_llm/evaluator/run.py` (right after
+`WandbEnergyLogger.flush()`). It only fires when ``energy_profiling: true``
+**and** ``auto_pareto.enabled: true`` are both set — so existing configs that
+omit the block keep their previous behavior (no auto-refresh).
+
+```yaml
+# config_files/evaluator_configs/.../my-eval.yaml
+model: hf
+tasks: medmcqa,medqa_4options,...
+model_args: pretrained=google/gemma-2-2b-it
+wandb_args: project=open-medical-llm-energy,name=my-run
+energy_profiling: true
+
+auto_pareto:
+  enabled: true
+  entity: llm4kmu                       # required if WANDB_ENTITY is unset
+  project: open-medical-llm-energy      # defaults to wandb_args.project
+  # Everything below is optional — defaults match scripts/wandb_pareto_plot.py
+  energy_key: emissions/actual_energy_consumed_kWh
+  score_scale: 100.0
+  tag: energy-profiling
+  workspace_name: Pareto Frontier
+  skip_preset: false                    # set true once preset is registered
+  skip_panel: false
+  skip_backfill: false
+  dry_run: false
+```
+
+What the hook does, step by step:
+
+1. After ``wandb_logger.flush()`` finishes the energy run, reads the
+   ``auto_pareto`` block.
+2. Resolves the wandb entity (``auto_pareto.entity`` →
+   ``wandb_args.entity`` → ``WANDB_ENTITY`` env var). Aborts the refresh
+   (with a warning, not an error) if none is set.
+3. Calls ``refresh_pareto_workspace(...)`` from
+   ``auto_llm.evaluator.plots.wandb_pareto_plot`` — the same library
+   function the manual CLI script wraps, so behavior is identical.
+4. The function fetches every run in the target project via the wandb
+   API, recomputes the frontier for all 13 labels (1 overall + 3 groups
+   + 9 tasks), writes ``pareto/<label>/*`` keys back into each run's
+   summary, then upserts the saved workspace view.
+5. On success, logs the workspace URL. On any failure (network glitch,
+   wandb 5xx, missing entity, …), logs a single warning of the form
+   ``auto_pareto refresh failed (best-effort, eval job will succeed): …``
+   and the eval job still exits 0.
+
+Trade-offs to be aware of:
+
+- **Best-effort only.** A refresh failure never fails the eval job. If
+  reliability matters for a specific batch, leave ``enabled: false`` and
+  run the manual command from the login node at the end.
+- **Concurrent jobs race.** When two energy-eval Slurm jobs finish at
+  the same time and both attempt to refresh, both write
+  ``pareto/<label>/*`` into every run's summary. Last writer wins;
+  flags converge as more jobs finish. This is acceptable because the
+  Pareto frontier is a deterministic function of the data already in
+  wandb — every refresh that runs after the last ``wandb.finish()``
+  produces the same answer.
+- **Cost.** The refresh fetches every run in the project via the wandb
+  API (no GPU, no dataset loading). For projects with hundreds of runs
+  this adds a few seconds per eval job. On large sweeps, prefer
+  ``enabled: false`` plus one manual run at the end.
+- **Workspace only.** This refresh updates the live ``Pareto Frontier``
+  workspace via ``wandb_workspaces.workspaces``. It does **not**
+  publish a frozen Report — for that, run the manual command and call
+  the ``wandb_workspaces.reports.v2`` API yourself.
+
+When to leave ``auto_pareto`` disabled (the default):
+
+- Local debugging — no need to overwrite the project workspace from
+  every smoke test.
+- Configs not running with ``energy_profiling: true`` — the hook is
+  only reached when the energy block executes.
+- Massive sweeps where you only care about the final frontier — let
+  the sweep complete and run the manual command once on the login
+  node.
 </details>
 
 # Contact Us

auto_llm/builder/task_data_builder/med_mcqa_data_builder.py

@@ -0,0 +1,59 @@
+from datasets import DatasetDict, load_dataset, Dataset
+
+from auto_llm.builder.task_data_builder.task_data_builder import TaskDataBuilder
+from auto_llm.dto.builder_config import (
+    DatasetSplit,
+    TaskDatasetFeatures,
+)
+
+
+class MedmcqaDataBuilder(TaskDataBuilder):
+    """Build the MedMCQA dataset into ``input_text`` / ``output_text`` format.
+
+    Source: ``openlifescienceai/medmcqa`` on HuggingFace.
+
+    Raw columns: ``question``, ``opa``/``opb``/``opc``/``opd`` (four options),
+    ``cop`` (0-indexed correct option: 0=A, 1=B, 2=C, 3=D), ``choice_type``
+    (``"single"`` or ``"multi"``).
+
+    """
+
+    OPTION_MAP = {0: "A", 1: "B", 2: "C", 3: "D"}
+
+    def build(self) -> DatasetDict:
+        ds_dict = load_dataset("openlifescienceai/medmcqa", trust_remote_code=True)
+
+        return DatasetDict(
+            {
+                DatasetSplit.TRAIN: self._process_split(ds_dict["train"]),
+                DatasetSplit.VALIDATION: self._process_split(ds_dict["validation"]),
+                DatasetSplit.TEST: self._process_split(ds_dict["test"]),
+            }
+        )
+
+    def _process_split(self, ds: Dataset) -> Dataset:
+        samples = []
+        for item in ds:
+            if item["cop"] not in self.OPTION_MAP:
+                continue
+
+            question = item["question"]
+            input_text = (
+                f"Question: {question}\n"
+                f"Options:\n"
+                f"A. {item['opa']}\n"
+                f"B. {item['opb']}\n"
+                f"C. {item['opc']}\n"
+                f"D. {item['opd']}"
+            )
+
+            output_text = self.OPTION_MAP[item["cop"]]
+
+            samples.append(
+                {
+                    TaskDatasetFeatures.INPUT_TEXT: input_text,
+                    TaskDatasetFeatures.OUTPUT_TEXT: output_text,
+                }
+            )
+
+        return Dataset.from_list(samples)