Connection Refused errors when pushing images

[ohm314]

Hi @vsoch

I've been trying to setup sregistry to test it out and see if it could help for our use-case but I've been struggling to push images onto the registry.

I've followed your documentation configuring and setting everything up (I'm working with the sregistry master branch), but once I try pushing an image I see following error:

$ singularity push  -U neurodamus-neocortex__1.13-2.16.6-2.8.1-amd64.sif library://ohm314/neurodamus/neurodamus-neocortex:1.13-2.16.6-2.8.1-amd64
WARNING: Skipping container verification
FATAL:   Unable to push image to library: Put "http://127.0.0.1:9000/sregistry/neurodamus/neurodamus-neocortex%3Asha256.e8a1d69e13ff648ef2d109fb1101c5d01b8ce85f56286fd67f0ef54e2b180f2e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=obp%2F20231205%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231205T221105Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256&partNumber=1&uploadId=ZjUyNTk1ZWQtZTQwZC00YmI5LWE5Y2ItNzQzMDUzZjNkYWQyLjJkYjI0YWVkLTcyOWEtNDM2ZC04ZTM2LTA2NGIzNThjMmI3Mw&X-Amz-Signature=2569cc63f40b3ab23a768cb7628eac475809f9f2b8d0ec9440178b9fee6cdba5": dial tcp 127.0.0.1:9000: connect: connection refused

This points me to some issue with minio. Looking at the logs in docker compose, i see at startup:

sregistry-minio-1      | MinIO Object Storage Server
sregistry-minio-1      | Copyright: 2015-2023 MinIO, Inc.
sregistry-minio-1      | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
sregistry-minio-1      | Version: RELEASE.2023-12-02T10-51-33Z (go1.21.4 linux/amd64)                                                                                                 sregistry-minio-1      |
sregistry-minio-1      | Status:         1 Online, 0 Offline.
sregistry-minio-1      | S3-API: http://172.18.0.4:9000  http://127.0.0.1:9000
sregistry-minio-1      | Console: http://172.18.0.4:41875 http://127.0.0.1:41875
sregistry-minio-1      |
sregistry-minio-1      | Documentation: https://min.io/docs/minio/linux/index.html
sregistry-minio-1      | Warning: The standard parity is set to 0. This can lead to data loss.

The last messages that match the push requests are:

sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:04 +0000] "GET /v1/containers/ohm314/neurodamus/neurodamus-neocortex HTTP/1.1" 200 536 "-" "Singularity-Ce/3.11.3 (Linux
amd64) Go/1.20.4" "-"                                                                                                                                                                 sregistry-uwsgi-1      | [pid: 31|app: 0|req: 5/35] 3.215.141.12 () {32 vars in 564 bytes} [Tue Dec  5 16:11:04 2023] GET /v1/containers/ohm314/neurodamus/neurodamus-neocortex => gen
erated 536 bytes in 45 msecs (HTTP/1.1 200) 8 headers in 240 bytes (1 switches on core 1)
sregistry-uwsgi-1      | GET PushNamedContainerView
sregistry-uwsgi-1      | [pid: 39|app: 0|req: 25/36] 3.215.141.12 () {32 vars in 721 bytes} [Tue Dec  5 16:11:04 2023] GET /v1/images/ohm314/neurodamus/neurodamus-neocortex:sha256.e8
a1d69e13ff648ef2d109fb1101c5d01b8ce85f56286fd67f0ef54e2b180f2e?arch=amd64 => generated 565 bytes in 28 msecs (HTTP/1.1 200) 8 headers in 240 bytes (1 switches on core 2)
sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:04 +0000] "GET /v1/images/ohm314/neurodamus/neurodamus-neocortex:sha256.e8a1d69e13ff648ef2d109fb1101c5d01b8ce85f56286fd67
f0ef54e2b180f2e?arch=amd64 HTTP/1.1" 200 565 "-" "Singularity-Ce/3.11.3 (Linux amd64) Go/1.20.4" "-"
sregistry-uwsgi-1      | [pid: 27|app: 0|req: 2/37] 3.215.141.12 () {32 vars in 474 bytes} [Tue Dec  5 16:11:04 2023] GET /version => generated 58 bytes in 2 msecs (HTTP/1.1 200) 8 h
eaders in 239 bytes (1 switches on core 1)
sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:04 +0000] "GET /version HTTP/1.1" 200 58 "-" "Singularity-Ce/3.11.3 (Linux amd64) Go/1.20.4" "-"
sregistry-uwsgi-1      | POST RequestMultiPartPushImageFileView
sregistry-uwsgi-1      | Start multipart upload ZjUyNTk1ZWQtZTQwZC00YmI5LWE5Y2ItNzQzMDUzZjNkYWQyLjJkYjI0YWVkLTcyOWEtNDM2ZC04ZTM2LTA2NGIzNThjMmI3Mw
sregistry-uwsgi-1      | [pid: 35|app: 0|req: 6/38] 3.215.141.12 () {34 vars in 538 bytes} [Tue Dec  5 16:11:04 2023] POST /v2/imagefile/5/_multipart => generated 158 bytes in 42 mse
cs (HTTP/1.1 200) 8 headers in 231 bytes (1 switches on core 0)
sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:04 +0000] "POST /v2/imagefile/5/_multipart HTTP/1.1" 200 158 "-" "Singularity-Ce/3.11.3 (Linux amd64) Go/1.20.4" "-"
sregistry-uwsgi-1      | PUT RequestMultiPartPushImageFileView
sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:05 +0000] "PUT /v2/imagefile/5/_multipart HTTP/1.1" 200 558 "-" "Singularity-Ce/3.11.3 (Linux amd64) Go/1.20.4" "-"
sregistry-uwsgi-1      | [pid: 39|app: 0|req: 26/39] 3.215.141.12 () {34 vars in 539 bytes} [Tue Dec  5 16:11:05 2023] PUT /v2/imagefile/5/_multipart => generated 558 bytes in 12 mse
cs (HTTP/1.1 200) 8 headers in 231 bytes (1 switches on core 1)
sregistry-uwsgi-1      | PUT RequestMultiPartAbortView
sregistry-uwsgi-1      | Delete imagefile signal running.
sregistry-nginx-1      | 3.215.141.12 - - [05/Dec/2023:22:11:05 +0000] "PUT /v2/imagefile/5/_multipart_abort HTTP/1.1" 200 0 "-" "Singularity-Ce/3.11.3 (Linux amd64) Go/1.20.4" "-"
sregistry-uwsgi-1      | [pid: 39|app: 0|req: 27/40] 3.215.141.12 () {34 vars in 551 bytes} [Tue Dec  5 16:11:05 2023] PUT /v2/imagefile/5/_multipart_abort => generated 0 bytes in 15
 msecs (HTTP/1.1 200) 7 headers in 191 bytes (1 switches on core 0)

Looking at the minio trace I see:

$ ../mc admin trace -v myminio
minio:9000 [REQUEST s3.NewMultipartUpload] [2023-12-05T22:11:04.320] [Client IP: 172.18.0.5]
minio:9000 POST /sregistry/neurodamus/neurodamus-neocortex%3Asha256.e8a1d69e13ff648ef2d109fb1101c5d01b8ce85f56286fd67f0ef54e2b180f2e?uploads
minio:9000 Proto: HTTP/1.1
minio:9000 Host: minio:9000                                                                                                                                                           minio:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio:9000 Amz-Sdk-Invocation-Id: cd886676-1ccd-4300-bfbd-16c95d1196c3
minio:9000 Amz-Sdk-Request: attempt=1
minio:9000 Content-Length: 0
minio:9000 X-Amz-Date: 20231205T221104Z
minio:9000 Accept-Encoding: identity
minio:9000 Authorization: AWS4-HMAC-SHA256 Credential=obp/20231205/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e314ee04d311733a6900e8a9cf
3b1227a70f6073f84a3d860619809b2927a551
minio:9000 User-Agent: Boto3/1.28.47 md/Botocore#1.31.47 ua/2.0 os/linux#5.14.0-362.8.1.el9_3.x86_64 md/arch#x86_64 lang/python#3.9.18 md/pyimpl#CPython cfg/retry-mode#legacy Botocor
e/1.31.47
minio:9000
minio:9000 [RESPONSE] [2023-12-05T22:11:04.321] [ Duration 672µs TTFB 658.003µs ↑ 131 B  ↓ 409 B ]
minio:9000 200 OK
minio:9000 Server: MinIO
minio:9000 X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
minio:9000 X-Amz-Request-Id: 179E0F0E8A84F395
minio:9000 X-Content-Type-Options: nosniff                                                                                                                                            minio:9000 X-Xss-Protection: 1; mode=block                                                                                                                                            minio:9000 Accept-Ranges: bytes
minio:9000 Content-Length: 409
minio:9000 Content-Type: application/xml
minio:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio:9000 Vary: Origin,Accept-Encoding
minio:9000 <?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>sregistry</Bucket><Key>neurodamus/neurodamus-neocortex:sha256.e8a1d69e13ff648ef2d109fb1101c5d01
b8ce85f56286fd67f0ef54e2b180f2e</Key><UploadId>ZjUyNTk1ZWQtZTQwZC00YmI5LWE5Y2ItNzQzMDUzZjNkYWQyLjJkYjI0YWVkLTcyOWEtNDM2ZC04ZTM2LTA2NGIzNThjMmI3Mw</UploadId></InitiateMultipartUploadR
esult>
minio:9000

Some extra notes:

• At first I was getting a warning that I'm using an outdated version of minio, so i changed the section in the docker-compose.yaml file to simply pull the latest image.
• This is all running on a EC2 instance, I've opened port 80 and 9000 (I wasn't too sure that port needs to be opened) in the security group
• I'm running RHEL9 with an up to date docker
• The sregistry web UI works fine, I've created an account, a token and successfully logged in with that token from the singularity client
• I'm not using any SSL, first i just wanna get things working. Once I know what I'm doing, I'll be setting up a fresh instance elsehwere.
• I've placed some non-default minio username and password into settings.yaml, .minio.env and also the json file that the mc command points me at.
• Can't think of anything else useful?

surely I'm missing something quite obvious! I'd appreciate your help. Thanks!

[vsoch]

I would try using the version of minio that the registry was developed with - there were breaking changes that I haven't gone to work on anytime recently. I also would suspect there is some issue with EC2 - my suggestion is to get it working locally first then reproduce on EC2 and then you can figure out what EC2 setting might be the issue. I've never tested on RHEL so I'm not sure I can add feedback there, but it's containerized so (I would think) that's not the issue!

Let me know what you think of the above!

[vsoch]

also, any reason to not just use a registry that supports pushing SIF (e.g., GitHub packages?) It's expensive to run your own infra (I would know running Singularity Hub for 5 years)!

[ohm314]

Well I started off with version you had tagged in the docker-compose - I agree that's what makes most sense. But since that gave me this very same error with the additional warning of mirio being out-of-date, I thought I could maybe go to the latest version and see what happens, but you're right that was probably not the smartest move :) I'll downgrade and try one more time. I may have fixed a few things in my config in the meantime.

I'm looking for a solution where we can maintain a private registry, so I thought it could be worthwhile trying to setup our own. What I am mostly interested in is a registry that will maintain the different containers for different architectures in one place and where I then can easily pull in the right image depending on the arch that I'm running on.

Are you on slack/discord somewhere? I'd love to pick your brains on our use-case if you have some time for a little chat.

[ohm314]

OK, so I did retest with the tagged version of minio and I see the same messages in the logs.

[vsoch]

Did you test on localhost first (not on EC2)?

[ohm314]

I did not :) but getting all the social connect stuff working on my local machine seems like a nightmare.