Multiple replicas in a bucket yield multiple duplicate file listings

[michael-conway]

intended behavior?

(.venv) (base) conwaymc@adimi02012643 workspace-gabble % aws s3 ls s3://drsobjects                            
2026-05-05 10:36:10    2188307 ChatGPT Image Aug 11, 2025, 03_45_52 PM.png
2026-05-05 10:36:36     194886 GA4GH Standards Matrix - By Workstream.xlsx
2026-05-05 10:40:35      87973 values.yaml
2026-05-05 10:41:03      87973 values.yaml
2026-05-05 10:35:57    6716037 wellarchitected-framework.pdf
2026-05-05 12:48:47    6716037 wellarchitected-framework.pdf
2026-05-05 12:49:30    6716037 wellarchitected-framework.pdf

where multiple replicas exist

[korydraughn]

That looks like a bug.

We'll investigate.

[korydraughn]

Can you show the resource hierarchy?
What version of iRODS are you running the S3 API against?

[michael-conway]

Hey Kory, 5.0.2

irods@irods-provider:~$ ilsresc
demoResc:unixfilesystem
test1-resc:unixfilesystem
test1-resc2:unixfilesystem
test1-resc3:unixfilesystem
irods@irods-provider:~$ 

[korydraughn]

Can you post the ils -L output?

[michael-conway]

sorry yeah

irods@irods-provider:~$ ils -L
/tempZone/home/test1/drsobjects:
  test1             0 demoResc      2188307 2026-05-05.14:36 & ChatGPT Image Aug 11, 2025, 03_45_52 PM.png
    sha2:8g0d3/taNB8bMuhErtZmCZzvCnq/1uGXZBYCWl7bGwQ=    generic    /var/lib/irods/iRODS/Vault/home/test1/drsobjects/ChatGPT Image Aug 11, 2025, 03_45_52 PM.png
  test1             0 demoResc       194886 2026-05-05.14:36 & GA4GH Standards Matrix - By Workstream.xlsx
    sha2:a6kFiuH5YGNsVzFetXnpsL8WOb9vpmWpsDv5ESxCVpI=    generic    /var/lib/irods/iRODS/Vault/home/test1/drsobjects/GA4GH Standards Matrix - By Workstream.xlsx
  test1             1 test1-resc        87973 2026-05-05.14:40 & values.yaml
    sha2:yHSLNmqXWAjG4GZOVQb1x+CY9wgubZiYLHsoX+DSLAU=    generic    /var/lib/irods/iRODS/Vault1/home/test1/drsobjects/values.yaml
  test1             2 test1-resc2        87973 2026-05-05.14:41 & values.yaml
    sha2:yHSLNmqXWAjG4GZOVQb1x+CY9wgubZiYLHsoX+DSLAU=    generic    /var/lib/irods/iRODS/Vault2/home/test1/drsobjects/values.yaml
  test1             0 demoResc      6716037 2026-05-05.14:35 & wellarchitected-framework.pdf
    sha2:BRnKLf7n1OSKWOqX7ElDnqaDXbzytOiSn4SQGRWtG/4=    generic    /var/lib/irods/iRODS/Vault/home/test1/drsobjects/wellarchitected-framework.pdf
  test1             1 test1-resc      6716037 2026-05-05.16:48 & wellarchitected-framework.pdf
    sha2:BRnKLf7n1OSKWOqX7ElDnqaDXbzytOiSn4SQGRWtG/4=    generic    /var/lib/irods/iRODS/Vault1/home/test1/drsobjects/wellarchitected-framework.pdf
  test1             2 test1-resc2      6716037 2026-05-05.16:49 & wellarchitected-framework.pdf
    sha2:BRnKLf7n1OSKWOqX7ElDnqaDXbzytOiSn4SQGRWtG/4=    generic    /var/lib/irods/iRODS/Vault2/home/test1/drsobjects/wellarchitected-framework.pdf
  C- /tempZone/home/test1/drsobjects/subfolder```

[korydraughn]

I was not able to reproduce this with the minio client. I was able to reproduce it with the aws client.

[trel]

oh? that's ... surprising that there's a difference...

[korydraughn]

After some investigation with @trel, we learned that the minio client only appears to work. It reports the entry with the latest modification time. This means the minio client can report stale replicas even though good replicas are present. This was confirmed through the use of itouch.

We are going to address this by returning the latest good replica. If no good replica exists, the latest stale replica is returned.

[alanking]

There are two ways we can make this behave:

1. Return information for the latest good replica and, failing that, the latest stale replica. This is the way suggested above and would put this client in line with clients like Metalnx (and possibly Jargon itself?).
2. Return information about the replica returned by the ObjStat API. This is the way the HTTP API's stat operation works: https://github.com/irods/irods_client_http_api/blob/0.6.0/API.md#stat-1 (see explanation below)

There's clearly a preference for 1 at this point, but I just wanted to lay out option 2 in case that seemed interesting.

---

Here's some additional info about HTTP API's stat operation:

For the HTTP API data object stat operation, the HTTP API is using the iRODS filesystem status function: https://github.com/irods/irods_client_http_api/blob/486c1b120bfc39ea405dce56ac2f09174e6536f5/endpoints/data_objects/src/main.cpp#L1896

The iRODS filesystem status function calls stat, which uses the ObjStat API: https://github.com/irods/irods/blob/e16f20424192f24db2dcc3dbaaa95cee0334a2b6/lib/filesystem/src/filesystem.cpp#L216

The ObjStat API gets information for all replicas for the requested data object and returns information about the first non-stale replica that it finds, as described in irods/irods#8993 (see also: https://github.com/irods/irods/blob/6dc24af5f43eb0576add6a6c36f300a4bc9c15fd/server/api/src/rsObjStat.cpp#L277-L315). Given that data spends astronomically more time at rest than intermediate, it is likely returning information about a good replica (although we have no guarantees about which good replica - it appears to depend on the order coming out of the catalog).

---

Finally, some musings mainly for my own benefit:

I have an implementation to fix this which is using the iRODS filesystem collection_iterator which I was hoping would get me to option 2, but it seems to be using some other means of getting data object information as it will return the info in order of replica number. It is using the OpenCollection/ReadCollection APIs to get info - yet another implementation for stat'ing a data object, apparently.

Most important thing to point out here is that none of our existing APIs or client libraries order information in the way described by option 1, so we will have to implement something in the S3 API.