[LATTE-dev] Cannot handle some LATTEv1-generated metadata, e.g., data/all_targets_S0<n>_v1.txt

[orionlee]

When I tried to run LATTE for TIC 65263849 from sector 2, I got the KeyError: 'Camera' during parsing data/all_targets_S002_v1.txt

Phase folded plot... done.
Traceback (most recent call last):
  File "C:\pkg\_winNonPortables\Anaconda3\envs\LATTE-dev\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\pkg\_winNonPortables\Anaconda3\envs\LATTE-dev\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\dev\_juypter\LATTE\LATTE\__main__.py", line 542, in <module>
    utils.interact_LATTE(tic, indir, syspath, sectors_all, sectors, ra, dec, args)  # the argument of whether to shos the images or not
  File "C:\dev\_juypter\LATTE\LATTE\LATTEutils.py", line 690, in interact_LATTE
    brew.brew_LATTE(tic, indir, syspath, transit_list, simple, BLS, model, save, DV, sectors, sectors_all, alltime, allflux, allflux_err, all_md, alltimebinned, allfluxbinned, allx1, allx2, ally1, ally2, alltime12, allfbkg, start_sec, end_sec, in_sec, tessmag, teff, srad, ra, dec, args)
  File "C:\dev\_juypter\LATTE\LATTE\LATTEbrew.py", line 316, in brew_LATTE
    ticids, distance, target_ra, target_dec = utils.nn_ticids(indir, transit_sec, tic)
  File "C:\dev\_juypter\LATTE\LATTE\LATTEutils.py", line 2595, in nn_ticids
    tic_list = pd.read_csv("{}/data/all_targets_S{}_v1.txt".format(indir,download_sector)).sort_values(['Camera', 'RA', 'Dec']).reset_index()
  File "C:\pkg\_winNonPortables\Anaconda3\envs\LATTE-dev\lib\site-packages\pandas\core\frame.py", line 5273, in sort_values
    keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
  File "C:\pkg\_winNonPortables\Anaconda3\envs\LATTE-dev\lib\site-packages\pandas\core\frame.py", line 5273, in <listcomp>
    keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
  File "C:\pkg\_winNonPortables\Anaconda3\envs\LATTE-dev\lib\site-packages\pandas\core\generic.py", line 1560, in _get_label_or_level_values
    raise KeyError(key)
KeyError: 'Camera'

Inspecting the data/all_targets_S002_v1.txt file in question, it looks like the following (not a csv):

#
# Full target list for orbit 11, sector 2
# Created Fri Aug 24 12:54:36 EDT 2018 
# Concatenation of GI astero exo bright DDT ppas lists
#
# TICID	      Camera   CCD	Tmag	   RA		  Dec�
  2733208	1	3	 8.03	356.0455	-13.1907
  2733409	1	3	 7.55	356.1195	-14.4202
...

The file was created back when I was using LATTE v1.0.0

Comparing the file against recent target files generated from LATTE-dev, it seems that the format was changed to comma-separated.

TICID,Camera,CCD,Tmag,RA,Dec,sec
12621262,1,3,10.06,335.4806,-17.7932,28
12649449,1,3,10.07,335.5289,-16.8613,28

The code as-is is equipped to handle the comma-separated one, but not the old tab-separated format with comment

[orionlee]

The format of all_targets_...txt is changed.
Workaround for users who have been using LATTE v1 is to make it re-generate all required metadata

• Assume your current latte data directory is ~/data_LATTE, the metadata is in ~/data_LATTE/data
• Make LATTE use a new data directory and redownload metadata, e.g., ~/data_LATTE_new
  • python -m LATTE --new-path
  • python -m LATTE --new-data
• The updated metadata is then in ~/data_LATTE_new/data
• You can either use the new data directory. Or if you want to use the old directory, replace the data with the new one, then re-specify LATTE's data directory with python -m LATTE --new-path