Minio support

[guptam38-roche]

Overview:

This PR adds support for MinIO as a storage backend, expanding our storage options beyond the existing solutions. MinIO is an open-source, high-performance, S3-compatible object storage system that can be deployed anywhere.

Changes:

1. Implemented MinioStorage adapter that integrates with the MinIO client library.
2. Added configuration options for MinIO endpoints, access keys, secret keys, and bucket names.
3. Updated documentation with setup instructions for MinIO.

Test:

1. Tested through ama operations (init, clone, add, upload).
2. Verified compatibility with existing storage operations.

Steps on How to use:

1. wget https://dl.min.io/server/minio/release/linux-amd64/minio
2. chmod +x minio
3. minio server /data --console-address ":9001"  # Start Minio server

Access the MinIO Console at http://localhost:9001

Login with:

Username: minioadmin
Password: minioadmin

Enter project-bucket as the bucket name
Click "Create Bucket"

Creating a Project with MinIO Storage

1. Using the API
   Make a POST request to create a project with MinIO storage:

curl -X POST http://127.0.0.1:5000/db/project \
  -H "Content-Type: application/json" \
  -d '{
    "name": "minio_project",
    "is_active": true,
    "staging_url": "",
    "credentials_server": {
        "endpoint": "localhost:9000",
        "access_key": "minioadmin",
        "secret_key": "minioadmin",
        "secure": false
    },
    "remote_url": "minio://project-bucket",
    "credentials_user": {
        "endpoint": "localhost:9000",
        "access_key": "minioadmin",
        "secret_key": "minioadmin",
        "secure": false
    },
    "readme": "Project using MinIO for local storage",
    "description": "Test project using MinIO storage",
    "title": "MinIO Test Project",
    "status": 1
}'

[huany172-roche]

Ideally the test credentials will be included in a shared instance of the MockClient class across a test module. For example,

class MockClient:
    def __init__(self, project=None, credentials=None):
        self.project = project
        # endpoint = credentials.get('endpoint', 'localhost:9000')
    access_key = credentials.get('access_key', 'minioadmin')
    secret_key = credentials.get('secret_key', 'minioadmin')
    secure = credentials.get('secure', False)
        self._credentials = credentials

# Pytest fixture to create a shared MockClient instance for the test module
@pytest.fixture(scope="module")
def mock_client():
    # Create and return a shared instance of MockClient
    client = MockClient()
    return client

Here in get_minio_client, you can just return the Minio instance initiated from the previous keys, and remove the default values. Move these values to a test file. Create a tests folder and test for bucket cors. For example, client/amapy-plugin-s3/amapy_plugin_s3/tests/test_aws_storage.py

[huany172-roche]

Here actually the original update_cors() code was flawed, it should have leveraged the AssetStorage class's set_bucket_cors.
for example,

with created.storage(server=True):
    remote_url = Config.shared().get_storage_url() # get remote_url from the current project
    storage = StorageFactory.storage_for_url(src_url=remote_url) #check the usage of this line, it returns the AssetStorage instance
    storage.set_bucket_cors(credentials=data.get("credentials_server"),
                     bucket_name=store_url.bucket,
                     origin_url=Configs.shared().frontend_url)

@guptam38-roche Could you help to refactor this? Thanks

[huany172-roche]

Also, test it using either the api call or asset dashboard create project feature, and check MinIO admin console to verify, it should work like before the refactor.

[biswalc-roche]

@guptam38-roche please allow me to test it today