folderpath Inconsistent behaviour - "input" needs to be inserted in the path to the worldprice file

[paulrougieux]

The two error messages explained below are contradictory:

1. Either folderpath defines the "Path to directory with Input/Output folder.", then it should be at the parent of the input directory and the first error shouldn't happen. See how to reproduce this error with the sample python code below.
2. Or folderpath defines the input directory level, then it seems strange that the second error happens, asking for something in input / input.

Current behaviour

TiMBA/cli/cli.py specifies that folderpath should be the "Path to directory with Input/Output folder":

@click.option('-FP', '--folderpath', 'folderpath', required=False, type=click.Path(
     file_okay=False, writable=True, path_type=Path), help="Path to directory with Input/Output folder.")

Defining it at the wrong level returns DataManager.py returns this error

folderpath = "/home/paul/eu_cbm/TiMBA_Data/"
# FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/eu_cbm/TiMBA_Data/02_Additional_Information/worldprice_world500.xlsx'

Changing it to one level below DataManager.py returns this next error

folderpath = "/home/paul/eu_cbm/TiMBA_Data/input"
# FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/eu_cbm/TiMBA_Data/input/input/01_Input_Files'

Reproduce error 1

This is the error that appears when defining folderpath to a parent directory that contains both the input and output directories, in the example below: folderpath = "/home/paul/eu_cbm/TiMBA_Data/" .

"""
Run this example:

    conda activate timbaenv
    cd /tmp
    ipython -i run_timba.py

"""

from pathlib import Path
import datetime as dt
import importlib
from TiMBA.main_runner.main_runner import main
from TiMBA.data_management.ParameterCollector import ParameterCollector

folderpath = "/home/paul/eu_cbm/TiMBA_Data/"

user_input_cli = {
    "year": 2020,
    # Periods lengths are defined in the ExogChange sheet, first column.
    "max_period": 10,
    "product_price": "shadow_PP",
    "world_price": "shadow_WP",
    "transportation_factor": 1.0,
    "material_balance": "C_specific_MB",
    "global_material_balance": False,
    "serialization": False,
    "constants": [False, False, False],
    "dynamization_activated": True,
    "cleaned_opt_quantity": False,
    "capped_prices": False,
    "verbose_optimization_logger": True,
    "verbose_calculation_logger": False,
    "addInfo": True,
}

Parameters = ParameterCollector(user_input=user_input_cli, folderpath=folderpath)
world = "scenario_input_SSP2v3_2020_2050_new.xlsx"
current_dt = dt.datetime.now().strftime("%Y%m%dT%H-%M-%S")
PACKAGEDIR = Path("/home/paul/eu_cbm/TiMBA/TiMBA")

main(
    UserIO=Parameters,
    world_version=world,
    time_stamp=current_dt,
    package_dir=PACKAGEDIR,
    sc_name=world[: len(world) - 5],
)

Will return the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/eu_cbm/TiMBA_Data/02_Additional_Information/worldprice_world500.xlsx'

In [1]: # Path("/home/paul/eu_cbm/TiMBA_Data/02_Additional_Information/worldprice_world500.xlsx").exists()

In [2]: Path("/home/paul/eu_cbm/TiMBA_Data/02_Additional_Information/worldprice_world500.xlsx").exists()
Out[2]: False

In [3]: Path("/home/paul/eu_cbm/TiMBA_Data/input/02_Additional_Information/worldprice_world500.xlsx").exists()
Out[3]: True

Proposal

folderpath should define the parent directory of both the input and output directory.

[franziska-schier]

Feat. help CLI command -> Folderpath in fact is the path to the "parent" directory were both the input and the output folder for data are located. That is what "Path to directory with Input/Output folder" actually mean.
The input folder then contains sub-folder where, e.g., input data or additional information are stored / from where they needs to be called. The model will not run if the parent directory isn't called correctly because necessary data to run simulations will not be found (e.g., ... input/input/... does not exist).

Feat. # Proposal: the folderpath already define the parent directory of both the input and output direrectory.

[paulrougieux]

Feat. # Proposal: the folderpath already define the parent directory of both the input and output directory.

If folderpath defines the parent directory of both the in put and output directories, then that error should be fixed:

folderpath = "/home/paul/eu_cbm/TiMBA_Data/"
# FileNotFoundError: [Errno 2] No such file or directory: '/home/paul/eu_cbm/TiMBA_Data/02_Additional_Information/worldprice_world500.xlsx'

I updated the issue above with the code to reproduce the error.

"input" needs to be inserted in the path to the worldprice file.

[paulrougieux]

Updating TiMBA to the latest commits on the main branch from 6 days ago a1efeb8, the python script to reproduce the error above now returns a different error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /tmp/run_timba.py:37
     16 folderpath = "/home/paul/eu_cbm/TiMBA_Data/"
     18 user_input_cli = {
     19     "year": 2020,
     20     # Periods lengths are defined in the ExogChange sheet, first column.
   (...)     34     "addInfo": True,
     35 }
---> 37 Parameters = ParameterCollector(user_input=user_input_cli, folderpath=folderpath)
     38 world = "scenario_input_SSP2v3_2020_2050_new.xlsx"
     39 current_dt = dt.datetime.now().strftime("%Y%m%dT%H-%M-%S")

File ~/rp/eu_cbm/TiMBA/TiMBA/data_management/ParameterCollector.py:38, in ParameterCollector.__init__(self, user_input, folderpath)
     36 self._addInfo = user_input[ParamNames.addInfo.value]
     37 self._folderpath = folderpath
---> 38 self._activate_add_on_cmodule = user_input[ParamNames.activate_cmodule.value]
     39 self._sc_num = user_input[ParamNames.sc_num.value]
     40 self._read_in_pkl = user_input[ParamNames.read_in_pkl.value]

KeyError: 'activate_add_on_cmodule'

I will check later next week. Any advice on how to properly run TiMBA direcly from python using TiMBA.main_runner or an alternative would be welcomed.