SECA Conformance Tests

A comprehensive conformance testing tool to validate a Cloud Service Provider (CSP) of the SECA API specification.

Overview

SECA Conformance ensure that CSP implementations comply with the standardized API specification for sovereign cloud services. This tool validates API endpoints, resource lifecycle management, and compliance with SECA standards across multiple cloud providers.

Requirements

• POSIX compatible environment;
• Make Build Tool;
• Git SCM;
• Go 1.24 or higher;
• [JQ] (https://jqlang.org/);
• [Docker] (https://docs.docker.com/engine/install/);
• Allure Report V2.

Installation

git clone https://github.com/eu-sovereign-cloud/conformance
cd conformance
make
make install

Configuration

The following configurations are required to run the tool. These configurations can be set as command line parameters or environment variables:

Parameter	Variable	Description	Required	Default
--provider.region.v1	PROVIDER_REGION_V1	URL of a Region V1 provider API implementation	True
--provider.authorization.v1	PROVIDER_AUTHORIZATION_V1	URL of a Authorization V1 provider API implementation. Required if you will run Authorization provider secenarios	False
--client.auth.token	CLIENT_AUTH_TOKEN	Valid JWT token to access the CSP API's	True
--client.tenant	CLIENT_TENANT	Name of the Tenant used in the secenarios	True
--client.region	CLIENT_REGION	Name of the Region used in the secenarios	True
--scenarios.filter	SCENARIOS_FILTER	Regular expression to filter scenarios to run. To know the available scenarios run the list command	False
--scenarios.additional.regions	SCENARIOS_ADDITIONAL_REGIONS	Comma-separated list of additional regions to be used in the Region provider scenarios.	False
--scenarios.users	SCENARIOS_USERS	Comma-separated list of valid CSP users. Required if you will run Authorization provider secenarios	False
--scenarios.cidr	SCENARIOS_CIDR	CIDR range available in the CSP to create network resources. Required if you will run Network provider secenarios	False
--scenarios.public.ips	SCENARIOS_PUBLIC_IPS	Public IPs range, in CIDR format, to create CSP public IP's.Required if you will run any Network provider secenarios	False
--report.results.path	REPORT_RESULTS_PATH	Path to store the tests result reports	True
--retry.base.delay	RETRY_BASE_DELAY	Initial waiting time (in seconds) after creating a resource before performing the first state check	False	5
--retry.base.interval	RETRY_BASE_INTERVAL	Time interval (in seconds) to wait between consecutive retry attempts when checking the resource state	False	30
--retry.max.attempts	RETRY_MAX_ATTEMPTS	Maximum number of retry attempts to check the resource state before timing out	False	10

Running

To execute the conformance tests, set the configuration variables and use the following command format:

secatest run \
  --provider.region.v1=$REGION_API \
  --provider.authorization.v1=$AUTHORIZATION_API \
  --client.auth.token=$TOKEN \
  --client.region=$REGION \
  --client.tenant=$TENANT \
  --scenarios.users=$USERS \
  --scenarios.cidr=$CIDR \
  --scenarios.public.ips=$PUBLIC_IPS \
  --report.results.path=$RESULTS_PATH

Example:

secatest run \
  --provider.region.v1=https://demo.secapi.cloud/providers/seca.region \
  --provider.authorization.v1=https://demo.secapi.cloud/providers/seca.authorization \
  --client.auth.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 \
  --client.region=eu-central-1 \
  --client.tenant=demo \
  --scenarios.users=user1@sdemo.secapi.cloud,user2@demo.secapi.cloud \
  --scenarios.cidr=10.1.0.0/16 \
  --scenarios.public.ips=52.93.126.1/26 \
  --report.results.path=./resports/result

Viewing Result

To see the the result report use the following command format:

secatest report $REPORTS_RESULT_PATH

Example:

secatest report ./reports/results

Your default browser will be opened, with the Allure Report viewer:

Listing Scenarios

To see the list of available test scenarios run the following command:

secatest list

Example:

secatest list

Available Test Scenarios:
Authorization.V1.LifeCycle
Compute.V1.LifeCycle
Network.V1.LifeCycle
Storage.V1.LifeCycle
Workspace.V1.LifeCycle
Foundation.V1.Usage

Run Filtering Scenarios

To execute the conformance tests, filtering the scenarios to test, use the following command format:

secatest run \
  --provider.region.v1=$REGION_API \
  --client.auth.token=$TOKEN \
  --client.region=$REGION \
  --client.tenant=$TENANT \
  --scenarios.filter=$SCENARIOS_FILTER \
  --report.results.path=$RESULTS_PATH

Example:

secatest run \
  --provider.region.v1=https://demo.secapi.cloud/providers/seca.region \
  --client.auth.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 \
  --client.region=eu-central-1 \
  --client.tenant=demo \
  --scenarios.filter=Compute.V1.LifeCycle \
  --report.results.path=./resports/result

---

💰 Funding

This open-source project is sponsored by Aruba & IONOS SE and has received public funding from the European Union NextGenerationEU within the IPCEI-CIS program.