FY 22 Q2 Software Support: Products and Missions (01/31/2022)

[AustinSanders]

Overview:

This is an initial list of issues to be categorized and prioritized for a support sprint starting 01/31/22. This is a 2 week sprint, and it is set to conclude on 02/13/22.

Prioritized tickets:

High Priority

Modify tgocassisrdrgen to include browse keyword information #4760
Jigsaw "Unable to set coefficients vector" issue with 6.0.0 #4686.
Findfeatures - "Can't invert empty matrix" Errors #4639 Important, but potentially too large to fit into a support sprint. May require some digging to assess scope. Potential external PR coming in.
State of ISIS Test Data #4744
Opening cubes outside of Process objects is error prone #3805 Converted apps are no longer processing attributes. This should be addressed.
USGSCSM: Issues with some LRO NAC datasets #357 Should be closeable on merge of PR 358.

Medium Priority

Error using spiceinit with Dawn mission images #4684 - Work-blocking issue that may be a quick fix. Potentially similar fix as PR #4670
CAM2MAP Only Map-Projects First Layer in a CUB for Some Cameras (at least MARCI) #4635 Progress has been made, unclear how much additional time is required for a solution.
Oblique Detector Resolution Should Use Local Emission Angle #3600 PR #4600 seems to address this issue, but has been outstanding for several months.
Update OSIRIS-REx Camera Model to handle new IK codes #3079 Issue has been around a while and appears to have a fairly clear solution drawn out in the description.
Get Clean Jenkins Dev Tests #4666
CLI display of -last command is incorrect #4617
Modified Camera.Plugin files not added to lib/Camera.Plugin at ninja #3834. This has been in issue for a few years, and is a nice to have.

Low Priority

Qmos - Latitude Constructor Error #4754 Nice to have, but has a workaround.
Jigsaw command line wrong in gui mode #4725
cnetcheck not excluding ignored measures resulting in false positives #4649
Automated cherrypicking fails if there are merge conflicts #4636
time strings may not be properly conditioned in things that use iTime #4370
Some cubes have invalid History BLOBs #4337. May be closeable, just requires a status check.
PushFrame camera model needs GetLocalNormal calculation to stay within the correct framelet #4256. Appears to have been resolved by #4214.
ALE: Selene driver camera version tick #397. Should be a super simple fix.
USGSCSM: UsgsAstroLsSensorModel::getReferenceDateAndTime does not return the correct reference time #333
PLIO: Expand covariance matrices when reading in ISIS control networks #158

Deferred

Data missing from S-band lunar radar images after changing projection using map2map #4585 Previously worked, but cause was not identified. Follow up with user

Missions Tickets

Products Tickets

Uncategorized Tickets

Non-Isis Tickets

Legacy Tickets

Legacy tickets are tickets that have been featured in multiple previous support sprints but have not been closed. These tickets may require significant individual attention, more context, etc., and are being listed here for discussion before they are closed by the issue bot.

Band priority is not optimized in automos #3711.
Findfeatures fails for LROC NAC images #4396. Significant work has gone into this. A solution was identified, but there are copyright concerns.
Change spiceinit to use smithed kernels by default #3482. A decision needs to be made and the issue either worked or closed. Discussion on the post seems to favor not making smithed kernels default.
Add statistical outlier rejection to jigsaw #4559
CISS2ISIS: Incorrect pixel value mapping #672

[jessemapel]

We should add DOI-USGS/usgscsm#357

We've gotten several external reports of USGSCSM not working correctly with some LRO NAC data. This is the most comprehensive and would be a good candidate for support work.

[lwellerastro]

Hi,
Here are my thoughts on some of the tickets above.

• Findfeatures - "Can't invert empty matrix" Errors #4639
  I think this one is high priority since it affects the Kaguya TC global mosaic project and because I have run into the error in one form or another a number of times over the years and for various datasets. It's not going to go away. I don't know that the scope of the problem is tractable via our shorter sprints though. Kris Becker is aware of the problem and has a fix (in mind?) so someone with a deeper knowledge of the inner workings of findfeatures ought to chat with him or have a good look at his comments in this post (particularly his initial comments). I think we need to determine sooner or later what sort of work would be necessary to address this bug and then decide if we dedicate more sprint time to it this year or make a high priority IAA proposal for next year.

I have only processed about 7% of the data through findfeatures and have run into this problem repeatedly and gotten through some of it using Kris's work around suggestions for fastgeompoints, but in other cases the problem was manifested using a smaller overlap ratio, so I had to use a bigger ratio and miss out on potentially key connections. I'm wondering when I won't be be able to work around the problem anymore.

Again, knowing nothing about how easy/hard this is, what would be immensely helpful, is having findfeatures tell what image it is puking on. When there hundreds, sometimes >1000 overlaps for an image (yes, these absolutely do exist, especially near the poles), trying to figure who to remove to get the program to run is impossible. Having that ability though, to work around the issue by simply removing the image from the fromlist would be something more than having a program fail and not having a clue where to begin.

• Jigsaw "Unable to set coefficients vector" issue with 6.0.0 #4686
  I have no skin in this one at that moment, but I think this would be a high priority.

• Qmos - Latitude Constructor Error #4754
  Low priority since there is a work around, but should eventually get fixed.

• Some cubes have invalid History BLOBs #4337
  Seems like this one should be closed since it's a done deal. I personally have been able to use blobdump to see history of older cubes that have this corruption issue.

• Findfeatures fails for LROC NAC images #4396.
  Someone else just needs to decide if this is really important or not. I'm not working on LROC, so I don't particularly care and the person who I was trying to help that exposed this problem is using other methods to get points. That works fine for smaller areas of interest, but if we ever wanted to go big on controlling LROC NAC, then I would hope to use findfeatures. If we can't, we can't.

• Change spiceinit to use smithed kernels by default #3482.
  Well, there's a lot in the comments for this post and mostly cautioning against a change. Knowledgeable folks can always script spiceinit to cksmithed=true spksmithed=true to pick up any smithed kernels available. I don't think we should be forcing a change, chaos could ensue, and it would make other knowledgeable folks have to change their pipelines to set cksmithed=false and spksmithed=false in certain situations. Seems like a documentation and education issue. Should we doing that?

[blandoplanet]

Adding to what Lynn said. Below is the prioritization I suggest for issues relevant to products. Note that I don't have complete context or knowledge on many of these so the order may need to change.

I agree with Lynn that #4639 is probably the highest priority on the products side, as it has the potential to affect on-going work, Although current work arounds exist, they may not do the trick when we bring in more data. At a minimum, understanding the scope of what it will take to fix it is essential.

I think #4396 is also a priority, although not as important as #4639. This hasn't come up with our current LROC polar work (Janet isn't using findfeatures), but I expect we will continue to control LROC NAC in selected regions (e.g., for photoclinometry and/or landing sites) and we want all of our tools available. I know there has been a lot of work on this previously. Can we determine a path forward?

#358 - there is a PR open. Can we support getting this closed? APPL group will soon be using LROC data in GXP with the Ls CSM and I'd like to address known issues if we can. I may not have enough context here.

#4214 - this looks like it's ready to go?

#4686 and #4684 - These both sound like issues that are preventing people from using our software to get their work done. Both should get fixed.

#4635 - Seems like Kelvin was making good progress. Can we get this knocked out?

#3079 - this is old. Can we get it crossed off? I think this only affects a few images with different focus positions (???) but we should be supporting them if we can.

#4585 - Not sure where to slot this. It was worked back in November but the cause wasn't identified.

#4754 - this is one that is low priority because there is a simple work around, but ideally this kind of work around wouldn't be necessary. Let's fix it if we can knock it out quickly.

#3600 - seems like this change would be an improvement but is maybe less critical than other issues (I may be wrong).

For what it's worth. I agree with Lynn on #3482. It's dangerous to change the default kernel to smithed. There are better ways to provide users with data where position and pointing are already improved (e.g., ARD).

[ladoramkershner]

Sorry to have such a short response, but a lot of these tickets are past my scope for the moment. However, I agree with Findfeatures - "Can't invert empty matrix" Errors #4639 (scoping as MVP) being prioritized. So I would like to upvote that!

[KrisBecker]

I have spent some time analyzing findfeatures as I was having some issues as well. I can't say for sure if my modifications would fix the problems you are having but I suspect it will help.

I have attached the analysis report that discuss the issues that were identified and the modifications made to address them. If you have any questions, please let me know.

We have been using this modified version of findfeatures for a several months and it works well, particularly for identifying and eliminating FastGeom errors.

I will submit a PR with these changes if you all think this will help resolve some of the problems you are seeing.

findfeatures_analysis_improvements.pdf

[lwellerastro]

Thanks @KrisBecker, this is a awesome document. I recognize so many of the problems you have documented in there including the dreadful formatting coming out of cnetcheck when it encounters nolatlon output. Moses wrote a python script to deal with so I could get through Europa. A post for cnetcheck output is long overdue addressing that. Anyway, thank you so much for sharing that. @jessemapel is going to have a look a see what he thinks about any of it being addressed via a PR from you.

[mvelascoUSGS]

Not sure if I should be adding a request to look at the budget tool here...

I am looking for some dev time to help resolve issues with the budget tool when loading the "main" proposal page. It seems that users who don't have access to a lot of proposals can view this page. Others, like me who have access to all budgets, cannot get this page to load. This is the only place that new budgets can be created. Fortunately, I was able to get Devon to create a few empty budgets that I have taken ownership so that new budgets can be created by PIs in the interim.

In addition I also have a few screenshot of some errors that, time permitting, I would appreciate someone looking into.

I am not looking to make major changes to the budget tool. With the FY23 IAA process starting in the near future, I would at least like
to have the budget tool serviceable until next steps are determined on its future. I would hope this might qualify as a "high" priority for Astro.

Thanks!

[jlaura]

Here are the service related issues that I am aware of at this time. Please add more if I missed any!

• Site (https://upcimages.wr.usgs.gov/) is down again with a 503 error.
• The web maps for the PGM website are down (https://planetarymapping.wr.usgs.gov/Target/project/Mars)
• I'm on the VPN and getting a 503 "Service Temporarily Unavailable" on https://pds-pgadmin.prod-asc.chs.usgs.gov .
• 503 on: site error: https://astrocloud-dev.prod-asc.chs.usgs.gov/
• PILOT is having trouble generating downloads and delivering them to users. This seems to be a result of the move to amazon.
• Do we know if the thermal radiance website made the transition to a new address? I think this is Greg's site(?).
  https://astrogeology.usgs.gov/tools/thermal-radiance-calculator/
• TADP portal
• Budget site (see above)

[thareUSGS]

I think a couple of those can be taken off.

• upcimages is working again. This is simply a file server for image thumbnails (and other minor information), so nothing really to debug besides making sure it is stable. Otherwise, we could transition the files to any other alternative fileserver (which has the ability to easily accept lots of new file additions from the UPC processing pipeline).
• https://astrocloud-dev.prod-asc.chs.usgs.gov/ is also working from the last sprint. It is a dev site for internal (VPN) testing only.

[thareUSGS]

For additions:

• I have seen chatter on Open Planetary that the ISIS Spice server can be very temperamental. This seems low-priority since I think it works most of the time, but folks are talking about supporting code to hit it several times until it is successful. Not really a great way to call it.
• Probably best suited for Nomenclature sprint, but there is still an issue for a single return in a search (e.g. "Abbot"). Essentially, any search that results in a single hit is redirecting to the internal domain "planetary-names.prod-asc.chs.usgs.gov". Not sure if in the code or IP rules?
• Related to the last issue, pushing to an internal address seems to happen not on just Nomenclature but other sites also. Is there something in our configs causing this? If solved, other deployed sites should be reviewed.

[jlaura]

@thareUSGS I do not understand the last bullet. Could you explain it a different way for me?

[thareUSGS]

I have seen on many of our sites that some calls (back to itself) will end up using the internal name. So something happens on "planetarynames.wr.usgs.gov/" and it will return the correct response but now on "planetary-names.prod-asc.chs.usgs.gov" (which can't be seen from the outside). I have seen it happen on PILOT too. I would only tackle this if someone figures out the "Abbot" issue on nomenclature above. I don't think was an issue prior to moving the servers so perhaps an EKS configuration (or a forwarding issue?).

[jlaura]

👍 sounds like this is a nice to have at this point and not a breaking issue then?

[jlaura]

Another issue that could very much use attention, particularly at the start of the sprint was discussed in chat. Thanks @acpaquette for bringing this up! CI performance has a significant negative impact on our ability to work efficiently, so this is a pretty major deal. I would advocate that the folks working more AWS related item work to get GitLab runners (where we can specify the build machine sizes) up and running. @Kelvinrr, @AustinSanders see here for examples (internal only, links will not work for non-USGS folks).

I was doing some fishing around in the Jenkins pipelines and I don't think the ISIS runners are actually running with 4 cores. There are two containers that are created, a jnpl container that has 4 cpus dedicated to it and the OS container. The OS containers is supposed to inherit from the jnpl container but I don't think it also inherits its resources. Thus, it seems like the OS containers themselves are running with severally limited resources

Something else I may have missed during stand up at some point, almost every test that is taking a significantly longer time are tests with fileIO specifically tests that read in cudes. This is likely a problem due to transfer times to the AWS/kubernetes executors

[jessemapel]

I would second that all the slow tests are heavily IO bound.

[jlaura]

Here is a very recent cam2map issue that might also be a good candidate to get closed sooner rather than later.

[jlaura]

Legacy Issues Prioritized

High

Band priority is not optimized in automos #3711.
Get PRs ISIS repo: 4520, 4519 iterated with the external contributors

Medium

Findfeatures fails for LROC NAC images #4396. Significant work has gone into this. A solution was identified, but there are copyright concerns.

#Deployment Issues:

High

ISIS CI Runners
TADP portal
Budget site (see above).

Med

The web maps for the PGM website are down (https://planetarymapping.wr.usgs.gov/Target/project/Mars)
Do we know if the thermal radiance website made the transition to a new address? I think this is Greg's site(?).
https://astrogeology.usgs.gov/tools/thermal-radiance-calculator/
Backlog of other static sites to be deployed

Low

PILOT is having trouble generating downloads and delivering them to users. This seems to be a result of the move to amazon.