-
Notifications
You must be signed in to change notification settings - Fork 2
Using Nautilus S3
Previous tutorials have shown how to use personal cloud services to stage data on PVC on the Nautilus cluster. However, there is also an option to use S3 cloud storage provided by Nautilus.
These resources can be used for uploading datasets to Nautilus PVC, downloading results and artifacts from pods and jobs to other systems and/or your local machine, or for moving data between Kubernetes namespaces.
In order to gain access to the S3 resources, you will need to request an S3 access key and secret. To do this you need to create an account in their [matrix] server here.
Once you have created an account and verified your email address, you can send a message in the Nautilus Support channel (seen on the left sidebar) requesting access to the S3 resources. You will also need to specify which regional pool works best for you.
Here is a sample message:
Hi. I was hoping to request access to the S3 resources. The central pool would work best for me. Thank you!
Once an admin sees your request for access to S3 resources, they will send you a private message asking for your email. After you provide an email address (it is recommended to use the same email address you use for the PRP Portal), you will be sent an access ID and secret access key. These two strings will enable you to create and interact with S3 buckets provided by Nautilus
To interact with Nautilus' S3 resources from your local machine, the recommended tool is rclone. For help installing, see the rclone documentation here
Once you have installed rclone, we can configure it to interact with Nautilus' S3 resources using the rclone config command.
To setup access from your personal machine, you can follow the sample setup process below. Be sure to enter the Access ID and Access Key that you received from a Nautilus admin when prompted:
$ rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.
name> nautilus
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
\ (s3)
6 / Backblaze B2
\ (b2)
7 / Better checksums for other remotes
\ (hasher)
8 / Box
\ (box)
9 / Cache a remote
\ (cache)
10 / Citrix Sharefile
\ (sharefile)
11 / Combine several remotes into one
\ (combine)
12 / Compress a remote
\ (compress)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
18 / Google Drive
\ (drive)
19 / Google Photos
\ (google photos)
20 / HTTP
\ (http)
21 / Hadoop distributed file system
\ (hdfs)
22 / HiDrive
\ (hidrive)
23 / Hubic
\ (hubic)
24 / In memory object storage system.
\ (memory)
25 / Internet Archive
\ (internetarchive)
26 / Jottacloud
\ (jottacloud)
27 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
28 / Local Disk
\ (local)
29 / Mail.ru Cloud
\ (mailru)
30 / Mega
\ (mega)
31 / Microsoft Azure Blob Storage
\ (azureblob)
32 / Microsoft OneDrive
\ (onedrive)
33 / OpenDrive
\ (opendrive)
34 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ (swift)
35 / Pcloud
\ (pcloud)
36 / Put.io
\ (putio)
37 / QingCloud Object Storage
\ (qingstor)
38 / SSH/SFTP
\ (sftp)
39 / Sia Decentralized Cloud
\ (sia)
40 / Storj Decentralized Cloud Storage
\ (storj)
41 / Sugarsync
\ (sugarsync)
42 / Transparently chunk/split large files
\ (chunker)
43 / Union merges the contents of several upstream fs
\ (union)
44 / Uptobox
\ (uptobox)
45 / WebDAV
\ (webdav)
46 / Yandex Disk
\ (yandex)
47 / Zoho
\ (zoho)
48 / premiumize.me
\ (premiumizeme)
49 / seafile
\ (seafile)
Storage> 5
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
\ (AWS)
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ (Alibaba)
3 / Ceph Object Storage
\ (Ceph)
4 / China Mobile Ecloud Elastic Object Storage (EOS)
\ (ChinaMobile)
5 / Cloudflare R2 Storage
\ (Cloudflare)
6 / Arvan Cloud Object Storage (AOS)
\ (ArvanCloud)
7 / Digital Ocean Spaces
\ (DigitalOcean)
8 / Dreamhost DreamObjects
\ (Dreamhost)
9 / Huawei Object Storage Service
\ (HuaweiOBS)
10 / IBM COS S3
\ (IBMCOS)
11 / IDrive e2
\ (IDrive)
12 / Seagate Lyve Cloud
\ (LyveCloud)
13 / Minio Object Storage
\ (Minio)
14 / Netease Object Storage (NOS)
\ (Netease)
15 / RackCorp Object Storage
\ (RackCorp)
16 / Scaleway Object Storage
\ (Scaleway)
17 / SeaweedFS S3
\ (SeaweedFS)
18 / StackPath Object Storage
\ (StackPath)
19 / Storj (S3 Compatible Gateway)
\ (Storj)
20 / Tencent Cloud Object Storage (COS)
\ (TencentCOS)
21 / Wasabi Object Storage
\ (Wasabi)
22 / Any other S3 compatible provider
\ (Other)
provider> 22
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
1 / Enter AWS credentials in the next step.
\ (false)
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth> 1
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> YOUR_KEY_ID_HERE
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YOUR_SECRET_KEY_HERE
Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Use this if unsure.
1 | Will use v4 signatures and an empty region.
\ ()
/ Use this only if v4 signatures don't work.
2 | E.g. pre Jewel/v10 CEPH.
\ (other-v2-signature)
region>
Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint> https://s3-central.nrp-nautilus.io
Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
/ Owner gets FULL_CONTROL.
3 | The AllUsers group gets READ and WRITE access.
| Granting this on a bucket is generally not recommended.
\ (public-read-write)
/ Owner gets FULL_CONTROL.
4 | The AuthenticatedUsers group gets READ access.
\ (authenticated-read)
/ Object owner gets FULL_CONTROL.
5 | Bucket owner gets READ access.
| If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-read)
/ Both the object owner and the bucket owner get FULL_CONTROL over the object.
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-full-control)
acl> 1
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: s3
- provider: Other
- access_key_id: YOUR_KEY_ID_HERE
- secret_access_key: YOUR_SECRET_KEY_HERE
- endpoint: https://s3-central.nrp-nautilus.io
- acl: private
Keep this "nautilus" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
nautilus s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
To create a bucket, we can use the mkdir command:
rclone mkdir remote:bucket
If you have followed the above configuration and want to create a bucket called my-bucket, you would run:
rclone mkdir nautilus:my-bucket
We can view all the buckets currently visible using:
rclone lsd nautilus:
If you want to copy data from your local machine to an S3 bucket, you can use the rclone copy command:
rclone copy --progress --copy-links myDir nautilus:myBucket/myDir
To download data from S3, we can use the same command, but switching the argument order:
rclone copy --progress --copy-links nautilus:myBucket/myDir myDir
The --progress flag will enable progress to be printed to stdout and the --copy-links flag will ensure that symlinks are copied and not broken during upload/download.
Alternatively, if you want a bucket to mirror a local directory, say you have a datasets directory on your hard drive that you want to upload, you can use the sync command:
rclone sync --progress --copy-links /path/to/datasets nautilus:myBucket
Again, we can perform the same operation in reverse, i.e. a local directory mirror an S3 bucket, we can simply swap the order of the arguments:
rclone sync --progress --copy-links nautilus:myBucket /path/to/datasets
WARNING: Sync is a destructive operation. Any/all files in the destination that are not present in the source will be deleted. It is recommended to run sync with the --dry-run flag to see what files will be deleted to ensure data is not unintentionally deleted.
If you would like to use rclone is a deployed application where you do not have access to a CLI to run rclone config or you are in an application where you do not want to store your S3 credentials in plain text, rclone can be configured entirely from the environment.
This can be useful for applications such as Kubernetes, Continuous Integration, Docker Containers, moving data from other systems, or scripting.
For interaction with the Nautilus S3 from outside the cluster, we can set these environment variables:
export RCLONE_CONFIG_NAUTILUS_TYPE=s3
export RCLONE_CONFIG_NAUTILUS_ENDPOINT=https://s3-central.nrp-nautilus.io
export RCLONE_CONFIG_NAUTILUS_ACCESS_KEY_ID=YOUR_ACCESS_ID_HERE
export RCLONE_CONFIG_NAUTILUS_SECRET_ACCESS_KEY=YOUR_SECRET_HERE
export RCLONE_CONFIG_NAUTILUS_PROVIDER=Other
Once you have exported these to an environment variables, you can interact with rclone the exact same way as above:
rclone lsd nautilus:
rclone mkdir nautilus:myBucket
rclone copy ...
rclone sync
When accessing Nautilus S3 resources on the Nautilus Kubernetes cluster, you will need to change the endpoint to the internal URL: http://rook-ceph-rgw-centrals3.rook-central
There is an example in the repository of starting an rclone pod with everything except the access key ID and secret access key already exported to the environment here.
To start the pod with rclone fully configured, you will need to export the environment variables and use envsubst:
export RCLONE_CONFIG_NAUTILUS_ACCESS_KEY_ID=YOUR_ACCESS_ID_HERE
export RCLONE_CONFIG_NAUTILUS_SECRET_ACCESS_KEY=YOUR_SECRET_HERE
envsubst < rclone_pod.yml | kubectl apply -f -
After the pod has started, you can attach to it using kubectl exec. You may want to update the Kube spec file and add a PVC to the pod before starting it.
If you would like to sync a PVC to S3 via a batch job, you will need to create a bucket and then again you can use the provided kube spec file here with exported environment variables:
export RCLONE_CONFIG_NAUTILUS_ACCESS_KEY_ID=YOUR_ACCESS_ID_HERE
export RCLONE_CONFIG_NAUTILUS_SECRET_ACCESS_KEY=YOUR_SECRET_HERE
export NAUTILUS_PVC_NAME=YOUR_PVC_NAME
rclone mkdir nautilus:$NAUTILUS_PVC_NAME
envsubst < rclone_sync_to_s3.yml | kubectl apply -f -
If you have uploaded data to an S3 bucket from some other source and would like to sync an S3 bucket to a PVC via a batch job, again you can use a provided kube spec file here with exported environment variables:
export RCLONE_CONFIG_NAUTILUS_ACCESS_KEY_ID=YOUR_ACCESS_ID_HERE
export RCLONE_CONFIG_NAUTILUS_SECRET_ACCESS_KEY=YOUR_SECRET_HERE
export NAUTILUS_PVC_NAME=YOUR_PVC_NAME
envsubst < rclone_sync_from_s3.yml | kubectl apply -f -
If you have data in a PVC in a Kubernetes namespace and want to transfer the data to another PVC in another namespace, you can combine the previous two concepts and build 2 batch jobs:
- Upload from PVC to S3 using
rclone_sync_to_s3.ymlin old namespace - Download from S3 to PVC using
rclone_sync_from_s3.ymlin new namespace
Docker
Nautilus Basics
Nautilus Advanced Usage
Jupyter