Enable sharded repodata with `rapidsai` channel

[jakirkham]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

Now that sharded repodata is available with Anaconda.org, we would like to enable it with the rapidsai channel. Is there something that needs to be done on the backend to enable this? If so, could we roll this out?

Also what would the RAPIDS team need to do to implement this on our side?

Why is this needed?

This would improve download times when working with the rapidsai channel. Also it would play nicer with Anaconda and Cloudflare.

What should happen?

Sharded repodata would be an available option for consumers of the rapidsai channel. Existing repodata would be maintained to provide either option to users.

Additional Context

The CDN is already enabled for rapidsai: #1055

[lwinfree]

I'll open an ticket on my team's internal board & let you know if this is something y'all can implement or if we need to do it ourselves. I'll get back to you!

[jakirkham]

Awesome! Thank you Lilly! 🙏

One thing we should confirm is that both repodata formats are supported for some time (certainly with the goal of moving completely to sharded in the future)

[jakirkham]

Should add would be happy to do a call if that would help

[lwinfree]

apologies for being slow here @jakirkham -- I'm working on an answer!

[lwinfree]

@jakirkham we think we'll have this done in a few weeks, I'll keep you posted! Once it is ready on our side, it would be good to quickly connect to talk about it & discuss how we'll monitor it too.

[jakirkham]

Hooray that's great news! Thanks Lilly 😄

Happy to

(please forgive the close/reopen noise; GH went rogue on me 😅)

[lwinfree]

hey @jakirkham this is now done! We've tested on our end, but please check it out and let us know if you detect a noticeable change in speed. (Or any issues). 🚀

[jakirkham]

Thanks Lilly! 🙏

Was away on vacation when this came in, but got back recently

Have tried this out a bit locally and it is working! Also it is much faster 🥳

As a follow up question, what would be needed for this to work on rapidsai-nightly as well? Would it need to be on CDN?