broken svs on Bio-Formats update

[will-moore]

When testing ome/omero-web#536 on idr-testing, I noticed this issue which I think Seb mentioned was due to a Bio-Formats update (see below)

Deployed on idr-testing from branch at #537, this is returning different zoomLevelScaling from previously!?
On https://idr.openmicroscopy.org/webclient/img_detail/9840218/ that image has:

    "tiles": true,
    "tile_size": {
        "width": 240,
        "height": 240
    },
    "levels": 3,
    "zoomLevelScaling": {
        "0": 1.0,
        "1": 0.2499840367792606,
        "2": 0.029308473277568484
    },

But on idr-testing, a different zoomLevelScaling is causing problems:

From slack idr, 28th Feb
Seb: "SVS in particular is a format which went through several changes in recent versions"

Will: Ah - so that bug may not be due to my PR?

Seb:
14:30
where that bug == the resolution levels are different between deployments or that bug == the resolution levels on idr-testing are incorrect?
14:32
actually the bug is clearly in production IDR (and has been addressed in recent Bio-Formats versions)
14:33
https://idr.openmicroscopy.org/webclient/img_detail/9840219/?dataset=10450 is absolutely not a label image, it's a resolution level of the whole slide image
14:34
so this dataset will partly break as part of the OMERO.server upgrade and we might need to consider reimporting these files unfortunately

[will-moore]

As discussed in IDR meeting:

• need to fix by re-importing (will get 2 images instead of 3) and re-annotate.
• need to check for other images with this issue - find other svs images and check manually

[will-moore]

Find 'SVS' format and count images...

idr=> select * from format where value='SVS';
 id  | permissions | value | external_id 
-----+-------------+-------+-------------
 360 |         -52 | SVS   |            
(1 row)

idr=> select count(*) from image where format=360 and series=1;
 count 
-------
   556
(1 row)

Find Datasets...

idr=> select DISTINCT parent from DatasetImageLink where child in (select id from image where format=360 and series=1);
 parent 
--------
  10348
  10349
  10352
  10354
  10356
  10358
  10388
  10413
  10414
  10415
  10425
  10426
  10427
  10428
  10429
  10431
  10436
  10450
  10461
  10474
  10475
  10476
  10478
  10479
  10480
  10481
  10484
  10485
  10492
  10517
  10530
  10550
  10557
  10558
  10559
  10560
  15274
  15280
  16651
  16652
  16653
  16654
(42 rows)

[will-moore]

Found 397 Broken images:

idr0070

Dataset: 10348, 17 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839914|image-9839917|image-9839920|image-9839923|image-9839926|image-9839929|image-9839932|image-9839935|image-9839938|image-9839941|image-9839947|image-9839950|image-9839953|image-9839956|image-9839959|image-9839962|image-9839965

Dataset: 10349, 21 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839970|image-9839973|image-9839976|image-9839979|image-9839982|image-9839985|image-9839988|image-9839991|image-9839994|image-9839997|image-9840000|image-9840005|image-9840017|image-9840020|image-9840023|image-9840026|image-9840029|image-9840032|image-9840035|image-9840038|image-9840041

Dataset: 10352, 18 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9840066|image-9840069|image-9840072|image-9840075|image-9840078|image-9840081|image-9840084|image-9840087|image-9840090|image-9840093|image-9840096|image-9840099|image-9840102|image-9840105|image-9840108|image-9840111|image-9840114|image-9840117

Dataset: 10354, 12 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839873|image-9839876|image-9839879|image-9839882|image-9839885|image-9839888|image-9839891|image-9839894|image-9839897|image-9839900|image-9839903|image-9839906

Dataset: 10356, 15 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839773|image-9839776|image-9839779|image-9839782|image-9839785|image-9839788|image-9839791|image-9839794|image-9839797|image-9839800|image-9839803|image-9839806|image-9839809|image-9839812|image-9839815

Dataset: 10358, 14 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839825|image-9839828|image-9839831|image-9839836|image-9839839|image-9839842|image-9839845|image-9839848|image-9839851|image-9839854|image-9839857|image-9839860|image-9839863|image-9839866

Dataset: 10388, 3 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9838978|image-9838981|image-9838984

Dataset: 10413, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839274|image-9839277

Dataset: 10414, 3 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839283|image-9839286|image-9839289

Dataset: 10415, 1 Image
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839294

Dataset: 10425, 1 Image
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839412

Dataset: 10426, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839420|image-9839423

Dataset: 10427, 2 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839426|image-9839429

Dataset: 10428, 18 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839432|image-9839435|image-9839438|image-9839441|image-9839444|image-9839447|image-9839450|image-9839453|image-9839456|image-9839459|image-9839462|image-9839465|image-9839468|image-9839471|image-9839474|image-9839477|image-9839480|image-9839483

Dataset: 10429, 10 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839486|image-9839489|image-9839492|image-9839495|image-9839498|image-9839501|image-9839504|image-9839507|image-9839510|image-9839513

Dataset: 10431, 24 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839567|image-9839570|image-9839573|image-9839576|image-9839579|image-9839582|image-9839585|image-9839588|image-9839591|image-9839594|image-9839597|image-9839600|image-9839603|image-9839606|image-9839609|image-9839612|image-9839615|image-9839618|image-9839621|image-9839624|image-9839628|image-9839633|image-9839637|image-9839640

Dataset: 10436, 8 Images
https://idr-testing.openmicroscopy.org/webclient/?show=image-9839671|image-9839674|image-9839677|image-9839680|image-9839683|image-9839686|image-9839689|image-9839692

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840219

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840355|image-9840358|image-9840361

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840478|image-9840481|image-9840484|image-9840487|image-9840490|image-9840493|image-9840496|image-9840499|image-9840502|image-9840505|image-9840508|image-9840511|image-9840514|image-9840517|image-9840523|image-9840526|image-9840536|image-9840539|image-9840542|image-9840545

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840550|image-9840553|image-9840556|image-9840559|image-9840562|image-9840565|image-9840568|image-9840571

https://idr-testing.openmicroscopy.org/webclient/?show=image-9840579|image-9840582|image-9840585|image-9840588

10478
https://idr-testing.openmicroscopy.org/webclient/?show=image-9840625|image-9840628|image-9840631|image-9840634|image-9840637|image-9840640|image-9840643|image-9840646|image-9840649|image-9840652|image-9840655|image-9840658|image-9840661|image-9840664|image-9840667|image-9840670|image-9840673|image-9840676|image-9840679|image-9840683|image-9840685|image-9840688|image-9840691|image-9840694|image-9840697|image-9840700|image-9840703|image-9840706|image-9840709|image-9840712|image-9840715|image-9840718|image-9840721|image-9840724|image-9840727|image-9840730|image-9840733|image-9840736

10479
https://idr-testing.openmicroscopy.org/webclient/?show=image-9840740|image-9840743|image-9840746|image-9840749|image-9840752|image-9840755|image-9840758|image-9840761|image-9840764|image-9840767|image-9840770|image-9840773|image-9840776|image-9840779|image-9840782|image-9840785|image-9840788|image-9840791|image-9840794|image-9840797|image-9840800|image-9840803|image-9840806|image-9840809|image-9840812|image-9840815|image-9840818|image-9840821|image-9840824|image-9840827|image-9840830|image-9840833

10480
https://idr-testing.openmicroscopy.org/webclient/?show=image-9840845|image-9840848|image-9840851|image-9840854|image-9840857|image-9840860|image-9840863|image-9840866|image-9840869|image-9840872|image-9840875|image-9840878|image-9840881|image-9840884|image-9840887|image-9840890|image-9840893|image-9840896|image-9840899|image-9840902|image-9840905

10481
https://idr-testing.openmicroscopy.org/webclient/?show=image-9840909|image-9840912|image-9840915|image-9840918|image-9840921|image-9840924|image-9840927|image-9840930|image-9840933|image-9840936|image-9840939|image-9840942|image-9840945|image-9840948|image-9840951|image-9840954|image-9840957|image-9840960|image-9840963|image-9840966|image-9840969|image-9840972|image-9840975|image-9840978|image-9840981|image-9840984|image-9840987|image-9840990|image-9840993|image-9840996|image-9840999|image-9841002|image-9841005|image-9841008|image-9841011|image-9841014|image-9841017

10484
https://idr-testing.openmicroscopy.org/webclient/?show=image-9841070|image-9841073|image-9841076|image-9841079|image-9841082|image-9841085|image-9841088|image-9841091|image-9841094|image-9841169|image-9841172|image-9841175

10485
https://idr-testing.openmicroscopy.org/webclient/?show=image-9841097|image-9841100|image-9841103|image-9841106|image-9841109|image-9841112|image-9841115|image-9841118|image-9841121|image-9841124|image-9841127|image-9841130|image-9841133|image-9841136|image-9841139|image-9841142|image-9841145|image-9841148|image-9841151|image-9841154|image-9841157|image-9841163|image-9841166

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841222|image-9841225|image-9841228

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841448|image-9841451|image-9841454

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841528

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841712|image-9841715|image-9841718|image-9841721|image-9841724|image-9841727

10557 - None!

10558
https://idr-testing.openmicroscopy.org/webclient/?show=image-9841778|image-9841781|image-9841784|image-9841787

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841791|image-9841794|image-9841797|image-9841800|image-9841803|image-9841806|image-9841809|image-9841812

https://idr-testing.openmicroscopy.org/webclient/?show=image-9841815|image-9841818

idr0114

15274, 15280 - No svs!

idr0135

16651, 16652, 16653, 16654 - All labels OK

[will-moore]

All the svs above are from idr0070. The https://github.com/IDR/idr0070-kerwin-hdbr/blob/master/experimentA/idr0070-experimentA-filePaths.tsv has 400 svs images, so let's just re-import all of them.

Steps:

• Delete all existing svs images from idr0070
• Re-import svs images
• Update https://github.com/IDR/idr0070-kerwin-hdbr/blob/master/experimentA/idr0070-experimentA-annotation.csv to remove the 400 'label' images
• Re-annotate svs images (maybe re-annotate ALL images will be easier)

[will-moore]

To focus on idr0070, max Image ID is 9841818, so we can count etc...

$ psql -U omero -d idr -h $DBHOST -F "," -c "select count(id) from image where format=360 and series=1 and (id < 9841819)"
 count 
-------
   400
(1 row)

400 svs Image IDs in idr0070

9838978,9838981,9838984,9839274,9839277,9839283,9839286,9839289,9839293,9839412,9839420,9839423,9839426,9839429,9839432,9839435,9839438,9839441,9839444,9839447,9839450,9839453,9839456,9839459,9839462,9839465,9839468,9839471,9839474,9839477,9839480,9839483,9839486,9839489,9839492,9839495,9839498,9839501,9839504,9839507,9839510,9839513,9839567,9839570,9839573,9839576,9839579,9839582,9839585,9839588,9839591,9839594,9839597,9839600,9839603,9839606,9839609,9839612,9839615,9839618,9839621,9839624,9839628,9839633,9839637,9839640,9839671,9839674,9839677,9839680,9839683,9839686,9839689,9839692,9839773,9839776,9839779,9839782,9839785,9839788,9839791,9839794,9839797,9839800,9839803,9839806,9839809,9839812,9839815,9839825,9839828,9839831,9839836,9839839,9839842,9839845,9839848,9839851,9839854,9839857,9839860,9839863,9839866,9839873,9839876,9839879,9839882,9839885,9839888,9839891,9839894,9839897,9839900,9839903,9839906,9839914,9839917,9839920,9839923,9839926,9839929,9839932,9839935,9839938,9839941,9839947,9839950,9839953,9839956,9839959,9839962,9839965,9839970,9839973,9839976,9839979,9839982,9839985,9839988,9839991,9839994,9839997,9840000,9840005,9840017,9840020,9840023,9840026,9840029,9840032,9840035,9840038,9840041,9840066,9840069,9840072,9840075,9840078,9840081,9840084,9840087,9840090,9840093,9840096,9840099,9840102,9840105,9840108,9840111,9840114,9840117,9840219,9840355,9840358,9840361,9840478,9840481,9840484,9840487,9840490,9840493,9840496,9840499,9840502,9840505,9840508,9840511,9840514,9840517,9840523,9840526,9840536,9840539,9840542,9840545,9840550,9840553,9840556,9840559,9840562,9840565,9840568,9840571,9840574,9840579,9840582,9840585,9840588,9840625,9840628,9840631,9840634,9840637,9840640,9840643,9840646,9840649,9840652,9840655,9840658,9840661,9840664,9840667,9840670,9840673,9840676,9840679,9840682,9840685,9840688,9840691,9840694,9840697,9840700,9840703,9840706,9840709,9840712,9840715,9840718,9840721,9840724,9840727,9840730,9840733,9840736,9840740,9840743,9840746,9840749,9840752,9840755,9840758,9840761,9840764,9840767,9840770,9840773,9840776,9840779,9840782,9840785,9840788,9840791,9840794,9840797,9840800,9840803,9840806,9840809,9840812,9840815,9840818,9840821,9840824,9840827,9840830,9840833,9840845,9840848,9840851,9840854,9840857,9840860,9840863,9840866,9840869,9840872,9840875,9840878,9840881,9840884,9840887,9840890,9840893,9840896,9840899,9840902,9840905,9840909,9840912,9840915,9840918,9840921,9840924,9840927,9840930,9840933,9840936,9840939,9840942,9840945,9840948,9840951,9840954,9840957,9840960,9840963,9840966,9840969,9840972,9840975,9840978,9840981,9840984,9840987,9840990,9840993,9840996,9840999,9841002,9841005,9841008,9841011,9841014,9841017,9841070,9841073,9841076,9841079,9841082,9841085,9841088,9841091,9841094,9841097,9841100,9841103,9841106,9841109,9841112,9841115,9841118,9841121,9841124,9841127,9841130,9841133,9841136,9841139,9841142,9841145,9841148,9841151,9841154,9841157,9841163,9841166,9841169,9841172,9841175,9841222,9841225,9841228,9841448,9841451,9841454,9841528,9841712,9841715,9841718,9841721,9841724,9841727,9841772,9841775,9841778,9841781,9841784,9841787,9841791,9841794,9841797,9841800,9841803,9841806,9841809,9841812,9841815,9841818

[will-moore]

Test deleting even a single Image fails (hangs) due to Annotations.
First remove all annotations...

$  omero metadata populate --context deletemap --report --wait 300 --batch 100 --localcfg '{"ns":["openmicroscopy.org/mapr/organism", "openmicroscopy.org/mapr/antibody", "openmicroscopy.org/mapr/gene", "openmicroscopy.org/mapr/cell_line", "openmicroscopy.org/mapr/phenotype", "openmicroscopy.org/mapr/sirna", "openmicroscopy.org/mapr/compound", "openmicroscopy.org/mapr/protein"], "typesToIgnore":["Annotation"]}' --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

$ omero metadata populate --context deletemap --report --wait 300 --batch 100 --cfg experimentA/idr0070-experimentA-bulkmap-config.yml Project:1104

$ python /uod/idr/metadata/idr-utils/scripts/annotate/clean_orphaned_maps.py
INFO:omero.util.Resources:Starting
INFO:omero.util.Resources:Starting
INFO:omero.util.Resources:Halted
INFO:root:Found 0 orphaned Organism maps
INFO:root:Found 74 orphaned Antibody maps
INFO:root:Deleting 74 maps
INFO:root:Found 2 orphaned Gene maps
INFO:root:Deleting 2 maps
INFO:root:Found 0 orphaned Cell Line maps
INFO:root:Found 0 orphaned Phenotype maps
INFO:root:Found 0 orphaned siRNA maps
INFO:root:Found 0 orphaned Compound maps
INFO:root:Found 0 orphaned Protein maps
INFO:root:Found 0 orphaned Notebook maps
INFO:root:Found 0 orphaned Study Info maps
INFO:root:Found 0 orphaned Study Components maps
INFO:omero.util.Resources:Halted

Can't delete using a single Image:ID:

$ omero delete --dry-run --report Image:9838978
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Image:9838978 failed: 'graph-fail'
failed: within Fileset[3550638] may not delete Image[9838978] while Image[9838979] remains
Steps: 4
Elapsed time: 0.844 secs.
Flags: [FAILURE, CANCELLED]

Need to use Fileset ID:

$ omero delete --dry-run --report Fileset:3550638
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:3550638 Dry run performed
ok
Steps: 4
Elapsed time: 5.382 secs.
Flags: []
Deleted objects
  Instrument:93241
  Objective:90855
  ObjectiveSettings:91280-91282
  CommentAnnotation:25123922
  FilesetAnnotationLink:3550438
  DatasetImageLink:4540977-4540979
  Channel:26299350-26299358
  Image:9838977-9838979
  LogicalChannel:10609300-10609308
  OriginalFile:29671051,29671052
  Pixels:9838977-9838979
  ChannelBinding:25422660-25422668
  QuantumDef:9523332-9523334
  RenderingDef:9523332-9523334
  Thumbnail:11390582-11390584
  Fileset:3550638
  FilesetEntry:22140458
  FilesetJobLink:14306936-14306940
  IndexingJob:14317104
  JobOriginalFileLink:3568052
  MetadataImportJob:14317101
  PixelDataJob:14317102
  ThumbnailGenerationJob:14317103
  UploadJob:14317100

Max Fileset ID for idr0070 is 3552155

psql -U omero -d idr -h $DBHOST -c "select count(fileset) from image where format=360 and series=1 and (fileset < 3552156)"
 count 
-------
   400
(1 row)

Get Fileset IDs....

psql -U omero -d idr -h $DBHOST -c "select fileset from image where format=360 and series=1 and (fileset < 3552156)" >> idr0070_svs_filesetIds.txt

Need to remove whitespace from each line:

$ for r in $(cat idr0070_svs_filesetIds.txt); do echo $r | tr -d '[:space:]' >> idr0070_svs_filesetIds.csv && echo "" >> idr0070_svs_filesetIds.csv; done

Do the delete....

for r in $(cat idr0070_svs_filesetIds.csv); do omero delete --report Fileset:$r; done

[will-moore]

Now re-import....

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr0070-kerwin-hdbr/experimentA/

sudo cat idr0070-experimentA-filePaths.tsv | grep svs >> /tmp/idr0070-experimentA-filePaths_svs.tsv
sudo mv /tmp/idr0070-experimentA-filePaths_svs.tsv ./
# update bulk.yml to point at the ...svs.tsv
sudo vi idr0070-experimentA-bulk.yml

wc idr0070-experimentA-filePaths_svs.tsv
  402  1400 95150 idr0070-experimentA-filePaths_svs.tsv

As omero-server...

omero import --bulk experimentA/idr0070-experimentA-bulk.yml --file /tmp/idr0070svs.log  --errs /tmp/idr0070svs.err

On completion...
2 svs not imported because original wasn't deleted...

$ grep ClientPathExclusion /tmp/idr0070svs.err
2024-03-27 16:44:56,417 6461       [2-thread-1] INFO   .importer.exclusions.ClientPathExclusion - ClientPath match for filename: uod/idr/filesets/idr0070-kerwin-hdbr/20200414-Batch3-ftp/HDBR_SYP_IHC/1605,2,Rt hemi,10pcw,24_SYP.svs
2024-03-27 16:46:04,488 6406       [2-thread-1] INFO   .importer.exclusions.ClientPathExclusion - ClientPath match for filename: uod/idr/filesets/idr0070-kerwin-hdbr/20200422-Batch5/HDBR_MKI67_IHC/1605,2,Rt hemi,10pcw,24_KI67.svs

All files were either imported as 2 (or 1) image - none with 3 images as before:

[wmoore@test120-omeroreadwrite ~]$ grep "1 file uploaded" /tmp/idr0070svs.err | wc
    400    5200   31598
[wmoore@test120-omeroreadwrite ~]$ grep "2 images imported" /tmp/idr0070svs.err | wc
    391    5083   30889
[wmoore@test120-omeroreadwrite ~]$ grep "1 image imported" /tmp/idr0070svs.err | wc
      2      26     156

[will-moore]

Re-annotate....

Remove label rows from csv on idr-testing...

[wmoore@test120-omeroreadwrite experimentA]$ sudo cat idr0070-experimentA-annotation.csv | grep -v label > /tmp/idr0070-experimentA-annotation2.csv
[wmoore@test120-omeroreadwrite experimentA]$ sudo rm idr0070-experimentA-annotation.csv
[wmoore@test120-omeroreadwrite experimentA]$ sudo mv /tmp/idr0070-experimentA-annotation2.csv idr0070-experimentA-annotation.csv

Reannotate as normal... check_annotations....

/opt/omero/server/venv3/bin/python /uod/idr/metadata/idr-utils/scripts/annotate/check_annotations.py Project:1104 idr0070-experimentA-annotation.csv --output /tmp/errors.csv


GAP43-10PCW,"HDBR_GAP43_IHC_10PCW/1634,6,Sl 2,10pcw,37_GAP43.svs",,,,,,,,,,,,,,,,,,,,Missing annotation
GAP43-12PCW,"HDBR_GAP43_IHC_12PCW/11761,12,Sl 3,12pcw,140_GAP43.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
GAP43-CS18,"HDBR_GAP43_IHC_CS18/1262,2,Embryo_Placenta,CS18,165_GAP43.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
MKI67-10PCW,"HDBR_MKI67_IHC/1634,6,Sl 2,10pcw,37_KI67.svs",,,,,,,,,,,,,,,,,,,,Missing annotation
NKX2-2-CS19,"HDBR_NKX2-2_IHC_CS19/352,1,Head,CS19,284_NKX2-2.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
PAX6-12PCW,"HDBR_PAX6_IHC_hires_12PCW/11610,9,brain,12pcw,114_PAX6.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
PAX6-CS19,"HDBR_PAX6_IHC_hires_CS19/352,1,Head,CS19,284_PAX6.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
WNT8B-12PCW,"HDBR_WNT8B_IHC_12PCW/11610,9,brain,12pcw,110_2014-05-20 14_35_30_wnt8b.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation
WNT8B-12PCW,"HDBR_WNT8B_IHC_12PCW/11610,9,brain,12pcw,160_2014-05-20 14_17_11_wnt8b.svs [label image]",,,,,,,,,,,,,,,,,,,,Missing annotation

```

[will-moore]

Missing annotation errors from above are fixed in IDR/idr0070-kerwin-hdbr@4b08e7f as follows:

Checked all 7 of the [label image] images above and they are all working. So these are genuine label images and should be added back to the annotations.csv.
The other 2 Missing annotation errors are from 2 images that are now just a single image (no [. ] in the name) so these were re-named in the annotations.csv.

[will-moore]

Fixed - test:

(venv3) bash-4.2$ /opt/omero/server/venv3/bin/python /uod/idr/metadata/idr-utils/scripts/annotate/check_annotations.py Project:1104 idr0070-experimentA-annotation.csv --output /tmp/idr0070_errors.csv
All images are unique and have annotations.

Deleted previous OMERO.table:

(venv3) bash-4.2$ omero delete --report Annotation:25135328
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Annotation:25135328 ok
Steps: 6
Elapsed time: 0.248 secs.
Flags: []
Deleted objects
  FileAnnotation:25135328
  ProjectAnnotationLink:1667
  OriginalFile:29676048

Create OMERO.table

/opt/omero/server/OMERO.server/bin/omero metadata populate --report --batch 1000 --file idr0070-experimentA-annotation.csv Project:1104

Create Map annotations

/opt/omero/server/OMERO.server/bin/omero metadata populate --context bulkmap --batch 100 --cfg idr0070-experimentA-bulkmap-config.yml Project:1104

[will-moore]

The 3 Filesets in https://idr-testing.openmicroscopy.org/webclient/?show=dataset-10461 are still broken.
These weren't deleted & reimported for some reason...?

Fileset IDs are:
3551354, 3551355, 3551356

These are all found in idr0070_svs_filesetIds.csv above. So delete was attempted above.
Try again:

(venv3) bash-4.2$ omero delete Fileset:3551354 --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:3551354 failed: 'graph-fail'
failed: cannot read ome.model.fs.Fileset[3551354]
Steps: 6
Elapsed time: 0.032 secs.
Flags: [FAILURE, CANCELLED]

Also, none of these Filesets or Image IDs can be found in the DB via psql.
Using the Dataset ID, can see it contains different Images:

idr=> select * from DatasetImageLink where parent=10461;
   id    | permissions | version |  child   | creation_id | external_id | group_id | owner_id | update_id | parent 
---------+-------------+---------+----------+-------------+-------------+----------+----------+-----------+--------
 7310578 |         -56 |       0 | 15153328 |   389349671 |             |        3 |        2 | 389349671 |  10461
 7310579 |         -56 |       0 | 15153329 |   389349671 |             |        3 |        2 | 389349671 |  10461
 7310580 |         -56 |       0 | 15153330 |   389349700 |             |        3 |        2 | 389349700 |  10461
 7310581 |         -56 |       0 | 15153331 |   389349700 |             |        3 |        2 | 389349700 |  10461
 7310582 |         -56 |       0 | 15153332 |   389349731 |             |        3 |        2 | 389349731 |  10461
 7310583 |         -56 |       0 | 15153333 |   389349731 |             |        3 |        2 | 389349731 |  10461
(6 rows)

It turns out that the images, thumbnails, right panel etc showing in the webclient above were all cached in the browser or nginx and have all been deleted and replaced successfully!

[will-moore]

cc @francesw or @dominikl

I wonder if you could review the actions above that I have performed on idr-testing (starting at #691 (comment) above) and plan to apply to idr-next for the NGFF release?

Summary:

• removed all annotations for idr0070
• deleted and re-imported all svs images in idr0070 (400 in total)
• updated annotations.csv accordingly (mostly just removing [macro] rows but some other tweaks too - see Label images removal idr0070-kerwin-hdbr#2
• checked and re-apply annotations

You can see an example of the changes by looking at e.g.
https://idr.openmicroscopy.org/webclient/?show=dataset-10461
(and the equivalent Dataset on idr testing)