Improve envoy charm integration tests

[orfeas-k]

Context

After bumping envoy version 1.12 -> 1.27 in #102, curling the application with header -H 'Content-Type: application/grpc-web-text' started responding with 503. The same is true for upstream envoy deployment (I applied upstream manifests and tried curling the application). That being said however, the envoy application is functional and kfp pipeline steps have no issue getting artifacts.

╰─$ curl 10.152.183.229:9090 -H 'Content-Type: application/grpc-web-text' -v                                       
*   Trying 10.152.183.229:9090...
* Connected to 10.152.183.229 (10.152.183.229) port 9090
> GET / HTTP/1.1
> Host: 10.152.183.229:9090
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/grpc-web-text
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 0
< content-type: application/grpc-web+proto
< grpc-message: upstream connect error or disconnect/reset before headers. reset reason: remote reset
< grpc-status: 14
< date: Fri, 07 Jun 2024 08:30:52 GMT
< server: envoy
< 
* Connection #0 to host 10.152.183.229 left intact

Thus, we need to find a new way of testing the application.
My guess is that we may have to use grpcurl tool or find out the proper headers that kfp-ui is using in order to get the artifacts, so we can simulate its behaviour.

Exploration I did

POST request

I noticed that most of kfp-ui requests are POST so I tried to imitate their behavior which results in 200 OK responses. However, I 'm not sure if that's close enough to what kfp-ui is doing.

# copying from browser console
# Response raw headers
HTTP/1.1 200 OK
content-type: application/grpc-web+proto
grpc-accept-encoding: identity, deflate, gzip
x-envoy-upstream-service-time: 4
date: Fri, 07 Jun 2024 08:34:24 GMT
server: istio-envoy
transfer-encoding: chunked
# Request raw headers
POST /ml_metadata.MetadataStoreService/GetArtifactsByContext HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 7
Content-Type: application/grpc-web+proto
Cookie: authservice_session=MTcxNzY2NDIxOXxOd3dBTkVGTldFdEJNMUUyVWtoVVRFSlhRVlZITmt0SVExQXpTMFpHUnpjMFRFaEVXVU5CVEVJMVdrNUVOa1ZFUTFNMFZVWldXVUU9fOzCgsyQHgeS7_mhIYhrTP6NV5j741_Sj46QS42e9VDo
Host: 10.64.140.43.nip.io
Origin: http://10.64.140.43.nip.io
Referer: http://10.64.140.43.nip.io/pipeline/
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
X-Grpc-Web: 1
X-User-Agent: grpc-web-javascript/0.1

Combining that information and information I found in random issues upstream, I tried testing with the following

╰─$ curl 10.152.183.229:9090 -H 'X-User-Agent: grpc-web-javascript/0.1' -H 'Accept-Encoding: gzip, deflate' -H 'Content-Type: application/grpc-web+proto' --data-binary 'AAAAAAA=' --compressed -v             
*   Trying 10.152.183.229:9090...
* Connected to 10.152.183.229 (10.152.183.229) port 9090
> POST / HTTP/1.1
> Host: 10.152.183.229:9090
> User-Agent: curl/8.5.0
> Accept: */*
> X-User-Agent: grpc-web-javascript/0.1
> Accept-Encoding: gzip, deflate
> Content-Type: application/grpc-web+proto
> Content-Length: 8
> 
< HTTP/1.1 200 OK
< content-type: application/grpc-web+proto
< grpc-status: 12
< x-envoy-upstream-service-time: 0
< date: Fri, 07 Jun 2024 08:45:57 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host 10.152.183.229 left intact

Get request with grpc content type

╰─$ curl 10.152.183.229:9090 -H 'Content-Type: application/grpc' -H 'Accept: application/grpc-web-text' -v          
*   Trying 10.152.183.229:9090...
* Connected to 10.152.183.229 (10.152.183.229) port 9090
> GET / HTTP/1.1
> Host: 10.152.183.229:9090
> User-Agent: curl/8.5.0
> Content-Type: application/grpc
> Accept: application/grpc-web-text
> 
< HTTP/1.1 200 OK
< content-type: application/grpc
< grpc-status: 14
< grpc-message: upstream connect error or disconnect/reset before headers. reset reason: remote reset
< date: Fri, 07 Jun 2024 08:46:25 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host 10.152.183.229 left intact

What needs to get done

1. Understand more how envoy works and how kfp-ui uses it
2. Add integration tests that imitate the requests between the two

Definition of Done

Integration tests ensure envoy is functional

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5820.

This message was autogenerated

[orfeas-k]

Note that we could add the new check as a health check to the charm, similarly to what we did for the charm in track/2.0 during #124.