Design proposal for bucket publishing (configuration)

[michael-conway]

I am working on a DRS (GA4GH) Data Repository Service implementation and am interested in adding s3 access endpoints to DRS objects. I've got the iRODS S3 Client API stood up and am working to weave it into the implementation.

I've got some provisional plans to automate user and bucket mappings by sharing a shared reference to the .json files in the current s3 api, such that a collection can be set/unset as a bucket, marked by a special AVU and causing an update to the shared file. The idea is that the s3 api engine can pick up these file changes as they are updated in the shared file.

Similarly, I am interested in the access keys, and to my mind the model would be that a User can generate an access key that is placed in a user-private location...

/zone/home/.irods-ext

would have an s3-access empty file to which AVUs could be attached. In this way a user can generate a secret key and keep it secret and it can synch via the user config.json as in the bucket mappings.

What I would really propose is to eliminate the shared file in a later iteration to read collection level special AVUs for bucket mappings, and to consult the user metadata via the mechanism above, or some other mechanism, meaning the s3 api would be metadata driven without the bucket and user config json manipulation

[trel]

I think I am understanding your request... but have a couple questions...

You're planning to build a mechanism so iRODS metadata can drive the mappings. This makes sense and one of the possible futures we envisioned with this design.

But you're proposing that later, in the future...

1. You want to have a piece of software pay attention to iRODS Collection AVUs and update the s3 bucket mapping file that the S3 API uses.

And... you want this piece of software to be the S3 API itself?

2. You want to have a piece of software generate a key for users to use the S3 API upon request, and manipulate the user mapping file that the S3 API uses.

And... you want this piece of software to be the S3 API itself, and store the key in the user account's iRODS namespace?

The current design was to separate the pieces and allow flexibility (and ingenuity), rather than tying the S3 API to mechanisms within a particular iRODS deployment.

[michael-conway]

The updating of the s3 file is the interim solution I am working on, my proposal would be to eliminate the files entirely. I'm working on a patch in the meantime where I can programatically update the files that s3 api is looking at but that would be eliminated.

I'm wondering if bucket designations as an AVU that is placed at the collection level is sufficient, and the reads of the config file switch to a periodic metadata query on the s3 api side?

The same could go for users.

[trel]

feels like a rule that could be written/provided and run whenever needed (periodic or upon request), keep it separate from any iRODS server logic. the S3 API could execute the rule.

and wouldn't want to put user secrets in AVUs... but yes, could also be queried/retrieved.

[michael-conway]

a periodic rule is a nice solution. In the long run the ability to directly interrogate avu (which is just bucket mapping, not secrets) would be clean enough. the user mappings are trickier. Those are in iRODS files, not exposed as AVUs, whether that is tight enough in the end is a question.

[trel]

agreed. yes, this feels like a plugin implementation detail rather than something that the s3 api itself would enforce.

we currently only ship a single plugin for bucket mapping (uses a file), but another (that uses AVUs) wouldn't change the architecture. seems clean.

and yes, a user mapping plugin that reads the secrets from iRODS itself should also work, but not convinced it's a design that others want... but feel free to play with it.