todo.org

#+TITLE: Demeter/Athena/Artemis To Do List
#+AUTHOR: Bruce Ravel
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="stylesheets/orgstyle.css" />
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="stylesheets/orgtocstyle.css" />
#+OPTIONS: toc:2

* top priorities
** small things
  - [ ] Dispersive algorithm, include in Athena
  - [X] S02 display issue
  - [ ] clone data sets and path lists in Artemis
  - [X] Artemis history improvments
  - [X] xkcd style plots, see
    http://tex.stackexchange.com/questions/74878/create-xkcd-style-diagram-in-tex
    and http://rfonseca.github.io/xkcd-gnuplot/ and
    http://antiyawn.com/uploads/humorsans.html
  - [ ] plot key for neg part of envelope is incorrect

** big things
  - [ ] From Joel Ullom: In our data, the error bar in each energy bin
    of the EXAFS spectrum is known from Poisson statistics since the
    photons are (laboriously) counted one by one. This creates the
    opportunity to propagate these errors through the Artemis fitting
    process.  Potentially, the resulting error bars on the fit
    parameters are more meaningful.  This software capability would be
    interesting to explore. /This is a Larch chore/
  - [X] From Joel Ullom: In the rebinning functionality of Athena, all
    rebinnings reduced the amplitude of the white line feature in our
    ferrioxalate data.  You explained well why rebinning is seldom
    useful when using a conventional EXAFS energy grid.  However, we
    only reproduce the conventional energy grid through rebinning so I
    think rebinning is still relevant for us.  It would be nice to be
    able to rebin without reducing spectral detail in the XANES
    region. -- What is the width of the box?
  - [X] 5 and 6 legged paths
  - [ ] PCA improvements
  - [ ] feff8.5L
  - [X] persistance of analysis methods in Athena
        This can be implemented in the same way I implemented XDI
        persistance -- for each group, serialize a blessed scalar and
        write that to the record

* larch chores

+ [ ] /l/ versions of the A,A,&H need path wrangling on Windows that includes PATH info for larch
+ [X] XDI headers and saving current with larch
+ [X] finsh Athena templates, including recently added ones (e.g. MEE templates, pixel)
+ [ ] start work on Artemis-related templates
+ [ ] kaiser vs. kaiser-bessel
+ [ ] use of stddev array in LCF is slightly incorrect, needs to be sensitive to the sapce in which the merge was made.
+ [ ] +command line in command buffer in A&A+ & capture feedback from server
+ [X] testing of Larch + XDI & of Larch - XDI
+ [ ] empirical standard generation in Athena

** Larch v. Ifeffit problem areas

   + Xray/Absorption/CL.pm -- this needs to be refactored
   + t/ tests need to be tested with Larch
   + in particular, need a t/001_larch.t test (with an appropriate SKIP in case of no server)
   + "nergy": XDAC vs. Ifeffit.  Larch will see "energy".
      * Perhaps a larch or ifeffit configuration attribute
      * ack-ing for '\bnergy\b' yields:
        : examples/recipes/MultiChannel/mc2prj
        : 54:  #my @energy    = $mc->get_array('nergy');
	:
        : lib/Demeter/Plugins/10BMMultiChannel.pm
	: 53:    my @energy = $mc->get_array('nergy');
	:
	: lib/Demeter/Plugins/X23A2MED.pm
	: 324:becomes "nergy".  Go figure.
	:
	: lib/Demeter/Plugins/x23a2med.demeter_conf
	: 10:default=nergy
   + Need to add Larch copyright statement throughout
        : ##             Larch is copyright (c) 2013 Matt Newville, Tom Trainer
        : ##                                 http://xraypy.github.io/xraylarch/


* perl modules to evaluate

 + Path:Tiny to replace File::Spec
 + Test::Class and Test::Class::Moose

* things that need doing
  
** Moose issues

** wx issues
  - [X] why does Wx::FileDialog with ~wxFD_OVERWRITE_PROMPT~ style not work correctly?
        see https://bugzilla.gnome.org/show_bug.cgi?id=631908 and 
        https://bugs.launchpad.net/ubuntu/+source/gtk+2.0/+bug/558674 for more info
        Turns out this is a gtk problem -- I can make the same thing happen with Firefox!
        *This is now deprecated.*

** installation issues [2/2]
  - [X] Relocation needs testing on Windows, need to make sure that
        the executables can all be found,
  - [X] also need to be able to find all of Ifeffit CL and CAPS files.

** Demeter bugs, shortcomings, missing features  [5/21] [23%]
  - [ ] in =Demeter::Data::IO::save_many=, why doesn't updating in n specifying "background" not work to bring data up to date?  (fft, i.e. doing autobk, does work)
  - [ ] tie multiple groups together (e.g. MED data with reference)
  - [ ] config param updated AND that param is currently showing in the Prefs tool -- need to update display in Prefs tool
  - [ ] add a record to an existing Athena project file
  - [ ] clnorm + Julia and Jingen's MBACK replacement
  - [ ] Data object should note that datatype is xanes and not compute fbkg since the bkg array does not exist
  - [ ] cannot specify title for stddev and variance plots
  - [ ] Demeter command backend needs lots of polishing
  - [ ] 3rd derivative
  - [ ] SDK: better documentation for freeze/thaw//serialize/deserialize
  - [ ] error check deserialization for all objects, verify that it is a yaml or whatnot, return 0 if not
  - [ ] OpenBabel interface, tied into Feff
  - [ ] MRU lists and unicode, unicode-y file paths fail the -e test at Demeter::MRU line 77
  - [ ] Feff8 integration
  - [ ] Use PDL in pathfinder
  - [ ] background groups for AXAFS
  - [X] Add tests to Path ranking framework (Karine Provost's idea, implemented in Demeter)
  - [X] serialize VPath, put VPath into dpj and fpj files
  - [X] set E0 to peak of white line
  - [X] need to disable indicators during the Rk plot.  Indicators (as implemented) are not consistent with multiplot
  - [X] Plucking from gnuplot really sucks



** Athena

*** needs testing [/]

*** Tools [22/25] [88%]
   1. [X] main
   2. [X] calibrate
   3. [X] align
   4. [X] deglitch
   5. [X] /smooth with Savitsky-Golay filter/ [[file:notes/sgolay.m][Matlab routine for SG filter]]; +smooth with boxcar average [[http://comments.gmane.org/gmane.comp.lang.perl.pdl.general/6333][See this]]+
   6. [X] truncate
   7. [X] rebin
   8. [X] convolution and noise
   9. [ ] /deconvolution/
   10. [X] self absorption
   11. [X] copy series
   12. [X] data summation

   13. [ ] dispersive
   14. [X] multi-electron removal

   15. [X] LCF
   16. [X] /PCA/ [[http://mailman.jach.hawaii.edu/pipermail/perldl/2006-August/000588.html][PCA from PDL mail archives]]
   17. [X] Peak fit
   18. [X] log ratio
   19. [X] difference

   20. [ ] explain FT
   21. [X] disk watcher
   22. [X] XDI
   23. [X] journal
   24. [X] plugins
   25. [X] preference 

**** Requires PDL
    1. smoothing by S-G (done with Larch)
    2. deconvolution
    3. +PCA+
    4. +boxcar average+

*** needs doing [5/15] [33%]
   - [ ] beter error handling in plugins -- specifically need to do a
         better job of trapping errors in the ~fix~ method.  Perhaps
         ~fix~ should always be called within an ~eval~?
   - [X] report to a text dialog on values of a particular parameter for all/marked groups
   - [ ] implement megamerge in Athena
   - [X] links to documentation on each tool
   - [X] *Main window* [3/3] [100%]
     + [X] +SpinCtrl for use with Rbkg+, event handler triggered twice on first use (found work-around)
     + [X] +SpinCtrl for use with bkg_spl1/2 (I find this to be a much worse idea)+ (not implemented)
     + [X] push bkg_e0 /algorithm/ onto marked/all groups and recalculate
   - [ ] *Convolve/noise tool*
     + add noise to chi data
   - [ ] *Difference tool* [0/1]
     + [ ] difference in k
   - [-] *LCF tool* [1/5]
     + [X] persistance of LCF results -- /use serialized blessed scalar, like XDI/
     + [ ] satisfying both inclusive and unity can be trouble for
           poorly normalized data and/or poor standard selection
           (see email from Jack Hitch)
     + [ ] compute R-factor in k or R using fit from energy.  (see
           mail from Scott 7 January 2012)
     + [ ] boundary on E0 shift.  See email from Dean Hesterburg from
           30 May, 2014.  /This is a Larch chore./
     + [ ] plot in norm(E) for this fit in chi(k) or plot in chi(k)
           for this fit in norm(E)
   - [-] *PCA tool* [3/11]
     + [ ] test setection against analysis space.  do the right thing
       with xanes data and chi(k) analysis or chi(k) data and energy
       analysis
     + [X] persistance -- what's the value of persistance for PCA?
     + [ ] manual entry of component coefficients (??)
     + [ ] cluster analysis, select two components and plot the
           associations of each data with those two in a scatter plot.
     + [ ] worry about "def group.tt" line being too long in tt template
     + [ ] plotting in pgplot
     + [ ] error analysis, RET, SPOIL, etc, determination of number
           of components, validity of TT, etc
     + [ ] specialized file headers for reconstruction and tt save files
     + [ ] pluck buttons for analysis range don't work
     + [X] save eigenvectors to a file
     + [X] if the marked groups are changed after performing the PCA
           but before reconstructing data, the plot will fail with
           and exception.  need to save the list of data that went
           into the PCA and check against that rather than the
           current list of marked groups
   - [-] *Peakfit tool* [8/12]
     + [X] persistance -- /use serialized blessed scalar, like XDI/
     + [ ] verify that fit is sensible (ndata/ninfo vs nparam)
     + [ ] broken using XES
     + [ ] explicitly state ninfo and epsilon
     + [ ] for Larch, bounds on parameters
     + [X] implement pseudovoight
     + [X] verify that all centroids are within (or near) fit range
     + [X] behavior when changing data
     + [X] pluck fit ranges
     + [X] +ndata is full data range rather than fit range with Fityk backend+ (igoring this)
     + [X] +explicitly set title of plot+ (i can live with this)
     + [X] sequence over marked groups
   - [ ] *Data watcher*
   - [-] *MEE tool* [2/3]
     + [X] arctan removal
     + [ ] other algorithms from the literature
     + [X] good guesses for parameters -- at least a pluck button for shift
   - [ ] *Plot menu*
     + zoom/unzoom/cursor for pgplot
   - [X] *metadata tool*
   - [X] *Data summer*
     + push_values method needs to update menus to reflect changes in group list

** raw data and plugins
   - Need to expand the filetype system by examining data from *all* the XAS beamlines in the world.  Yes ... all of them.
   - solicit help from the facility representatives

** Windows issues [4/9] [44%]
  - [X] parameter group context menus don't get posted (see
	http://www.nntp.perl.org/group/perl.wxperl.users/2011/03/msg7929.html)
  - [ ] the frickin' Gnuplot error logs seem to remain open and locked
	on Windows when a crash happens
  - [ ] status bar does not get color for wait or error messages
	/this may be unfixable, see/
	http://www.nntp.perl.org/group/perl.wxperl.users/2011/04/msg7943.html
  - [ ] The atoms panel background color is too light.  Which window
	needs its BackgroundColour set to wxNullColour?
  - [ ] clampbox does not get enabled/disabled explicitly, is this
	another aspect of StaticBox that is different on Windows? (see
	link above)
  - [ ] Need to test that paths with (parens|commas|quotes) get
	followed correctly in all situations
  - [X] Initial initialization of gnuplot and feff executable
        locations in the situation where the package has been moved or
        reinstalled such that and old demeter.ini still exists
  - [X] relocation of Strawberry leaves Ifeffit unable to find phase
	shift and CL tables (use an ENV variable?)
  - [X] The Strawberry+Demeter package does not coexist at all with
        ActivePerl.  Best solution is to generate ppd for Demeter
        armed with all dependencies.  Would need to compile wrapper
        and somehow get gnuplot on the machine. *Solved by explicitly
        calling Strawberry in the .bat files*


** Artemis
*** Artemis bugs and missing features [1/8]  [12%]
   - [X] More work on history, see Anthony Ardizzi's issue
   - [ ] Reorganize lists, move individual items up and down, move
	 blocks up and down, Path list, +Plot list+
   - [ ] do SSPaths get serialized and deserialized with the pointers
         to the feff calculation set correctly and no additional
         folders being created in stash (as was the case for FSPath)?
   - [ ] implementing derivative of phase plotting would require
         proper handling of this signal in the bkg, residual, and
         running R-factor plots.  Also probably want to disallow it
         for R123 plots.
   - [ ] better way of moving an empirical standard from Athena to
         Artemis -- should be able to import it directly from the
         athena project file.
   - [ ] Display of S02 glyph on some Windows is broken, breaking S02 context menu
   - [ ] Eric says: "if u use the automatic choice of parameters for
         the paths, in combination with a few quick first shell models
         and an imported feff.inp the whole thing runs in trouble. I
         don't really expect name collisions, but it seems to loose
         track between all the feff runs."  /This is a bit unclear.../
   - [ ] Path drag and drop [0/3]
     - [ ] DND of an SSPath does not work correctly
     - [ ] DND of FPath also broken
     - [ ] DND of selection (currently only one at a time)

**** Project [2/4]
   - [X] VPaths to/from project file
   - [X] SSPaths not saved/restored properly?
   - [ ] Indicators to/from project file
   - [ ] GDS objects get created with same groupname over and over as
	 they are used in fits in the history

**** Advanced fitting
***** MFC [0/1]
   - [ ] Balance interstitial energies for MFC fits
***** MDS & Fit Sequence [0/4]
   - [ ] Import mutiple data sets from an Athena project file
   - [ ] feffit.inp import: needs testing; MDS that is not merely MKW
   - [ ] Clone data sets such that the path list gets replicated efficiently (i.e. for MDS fits)
   - [ ] Clone all/marked paths between data sets


*** Histograms [12/14] [85%]
   - [X] Triangle object
     - yields a DS path and a TS path
     - by R and theta
     - +by a trio of Cartesian coordinates+
   - [ ] triangle histogram
   - [-] *Error checking* in Artemis, e.g. check that there is at least one bin in the supplied range(s)
	 - [X] SS
	 - [ ] ncl
	 - [ ] thru

** Hephaestus
  More standards!!

** Other object types [0/3]
  - [ ] Structural Units
      + Extension of VPath
      + Store GDS, feff, and path objects in a zip file.
      + On import, mark GDS parameters as merge if in conflict
  - [ ] MSPaths
      + Much like SSPath, make an arbitrary n-legged path
  - [ ] Nearly collinear paths, ThreeBody
      + Extension of VPath
      + need to worry about load order in Demeter.pm
      + Define a three body configuration, generate its 4-legged path and a sequence of three-legged paths along with a mixing parameter.
      + It will take a single set of path parameters that are pushed onto the generated Path objects, except for the amplitude, which will be computed from the mixing parameter.
      + This is a single object for the user to interact with which expands into 2 or 3 3-legged paths and a single 4-legged path
      
      
* Atoms and Feff


** CIF [0/3] [0%]
  - [ ] CIF issue: CIF file with "_eof" token at end of file, as in [[file:notes/H16PW12O46.cif][this cif file]]
  - [ ] Handle CIF import problems more gracefully (See Shelly's other email from 17 June 2011)
  - [ ] CIF errors are not handled gracefully (e.g. multiple occupancy)

** Atoms [7/10] [70%]
  - [ ] Parsing an atoms.inp file for the space group is fragile.  It
    currently grabs the 13 characters following the equals sign and
    white space.  13 is the longest standard HM symbol with a modifier
    -- an example is 'I 41/a m d :1' (see
    http://www.crystallography.net/cod/cod/cif/7/20/60/7206075.cif as
    an example).  However, if additional spaces are present or if
    another keyword follows in fewer than 13 characters, the parser
    /will/ do the wrong thing.
  - [X] Setting of Ba2Co9O14 (Chanapa Kongmark), also some problem with ZnFe2O4
  - [X] should be a way to insert tabulated shift vector, if it exists
  - [X] handle ": setting" notation: =:R=, =:H=, =:1=, =:2= [[http://research.cems.umn.edu/morse/code/svn/pscf/doc/man/sec.php?section%3DAppendix:%2520Space%2520Groups&][(for example)]]
    + =:1= and =:2= seem to have to do with whether the shift vector should be used, see [[file:examples/atoms/ZnFe2O4.cif][ZnFe2O4.cif]] and use the shift vector
    + =:R=, =:H= specify rhombohedral or tri/hexagonal setting
  - [X] Lepidocrocite issue (amam gets changed to cmcm)
  - [ ] 2 sites at the same position with occupancies <1.  see file above for an example
  - [ ] George Sterbinsky's [[http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2011-May/009936.html][mailing list post]] that turned out to be about 
	atoms' sphere and rhomboid in a non-orthogonal group
  - [ ] very confusing error message when core is not a tag
  - [X] Shell tags in the feff.inp ATOMS list
  - [X] Rhombic groups seem not be handled properly.  This example fails to generate a subshell of 3 atoms at ~1.9A
         : title name:     Fe2O3  hematite
         : space  R -3 c
         : a    = 5.0380	b    = 5.0380	c    = 13.7720
         : rmax = 6.00	core = Fe1
         : atom
         :   Fe     0.00000   0.00000   0.35530  Fe1
         :   O      0.30590   0.00000   0.25000  O1
  - [X] This input data fails
         : title formula:  LaCoO3
         : title refer1:  PRB V. 66 P. 094408 (2002)
         : title notes:   T = 300 K
         : space  r -3 c
         : a = 5.44864       c = 13.1035
         : rmax = 6.00       core = Co1
         : atom
         :   Co     0.00000   0.00000   0.00000  Co1
         :   La     0.00000   0.00000   0.25000  La1
         :   O      0.55032   0.00000   0.25000  O1


** Feff

*** Feff8 is unsupported except as an Atoms output type

https://github.com/xraypy/feff85exafs

*** Polarization and ellipticity

*** Five and six legged paths
 1. [[file:examples/h12213.cif][This CIF file]] is an example of a structure that has five and six
    legged nearly collinear scattering paths at around 4 Angstroms.  It
    has metal atoms bridged by cyanide (CN), like Prussian Blue (see for
    instance [[https://doi.org/10.1021/ic50177a008]])

 2. Another example: 1-4 dibromo benzene has Br scatterers across a
    benzene ring which can be seen at about 6 Angstroms, along with
    3s, 4s, 5s, and 6s:
        :        C-C
        :       /   \
        : Br - C     C - Br
        :       \   /
        :        C-C

 3. In fact, any benzene ligand has important 5s and 6s going around
    the ring.  There are various Abs-C-C 3s, 4s, 5s, and 6s.
        :         C-C
        :        /   \
        : Abs - C     C 
        :        \   /
        :         C-C

 4. Metals out to 8 Angstroms require 5s and 6s



* Ifeffit

In file ~src/lib/iff_show.f~, change line 431 from 

:     character*(*) s, t , messg*256

to

:     character*(*) s, t , messg*1024

+Also need to specify locations of CL data and phase shift tables via an ENV variable+

* Weird stuff I'd prefer not to implement unless demanded
 1. xfit output (only used by women who glow and men who plunder)
 2. csv and text report (excel *is* implemented)
 3. point finder (this was Shelly's request)
 4. session defaults (did anyone but me actually use these?)
 5. set to standard (i.e. the one that is marked) -- confusing and
    little used
 6. tie relative energy value to changes in E0 (this was something
    Jeremy requested originally)
 7. +set e0 by algorithm for all and marked -- also confusing and little used+ (implemented)
 8. +plot margin lines for deglitching, deglitch many points (this was something that was most useful for a timing problem at 10ID that no longer exists)+
    (implemented)
 9. preprocessing truncation and deglitching (truncation might be
    worth implementing)