Multiple flaky networking tests

[BenTheElder]

TestActorArbitraryPortAccess
TestActorDirectAccess
TestActorEgress
TestActorEgressHTTPS

/cc Lior Lieberman (@LiorLieberman) Bowei Du (@bowei) Keith Mattix II (@keithmattix)

failing on main post merge:
https://github.com/agent-substrate/substrate/actions/runs/32200086998/job/95911826556

[keithmattix]

#993 adds some retries to TestActorArbitraryPortAccess

[BenTheElder]

thank you! everything passed when that merged, watching for if the next commit does as well. unfortunately we're starting to get throttled so it may be a bit. will close if we don't see more.

[haiyanmeng]

From https://github.com/agent-substrate/substrate/actions/runs/32200086998/job/95911826556,

=== RUN   TestActorArbitraryPortAccess
    arbitraryport_test.go:51: creating actor networking-e2e/arbitraryport-1787098383986053833
    arbitraryport_test.go:51: resumed actor networking-e2e/arbitraryport-1787098383986053833
=== RUN   TestActorArbitraryPortAccess/regression:_default_port_still_reachable
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:61: default-port access through ingress succeeded; body: ok
=== RUN   TestActorArbitraryPortAccess/arbitrary_port_reachable
    arbitraryport_test.go:73: tunneled request returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
=== RUN   TestActorArbitraryPortAccess/unlisted_port_rejected
    arbitraryport_test.go:130: tunneled request to an unlisted port correctly returned HTTP 503; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorArbitraryPortAccess (42.69s)
    --- PASS: TestActorArbitraryPortAccess/regression:_default_port_still_reachable (30.50s)
    --- FAIL: TestActorArbitraryPortAccess/arbitrary_port_reachable (5.23s)
    --- PASS: TestActorArbitraryPortAccess/unlisted_port_rejected (5.04s)
=== RUN   TestActorDirectAccess
    networking_test.go:53: creating actor networking-e2e/direct-1787098426678550099
    networking_test.go:53: resumed actor networking-e2e/direct-1787098426678550099
=== RUN   TestActorDirectAccess/direct
    networking_test.go:58: direct Actor access through ate-demo-counter/counter-679c74d7df-m5hlk:80 was blocked as expected: the server is currently unable to handle the request (get pods counter-679c74d7df-m5hlk:80)
=== RUN   TestActorDirectAccess/via_ingress
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503 after 30s; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorDirectAccess (36.40s)
    --- PASS: TestActorDirectAccess/direct (0.01s)
    --- FAIL: TestActorDirectAccess/via_ingress (35.54s)
=== RUN   TestActorEgress
    networking_test.go:77: creating actor networking-e2e/egress-1787098463077668393
    networking_test.go:77: resumed actor networking-e2e/egress-1787098463077668393
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:84: Actor egress fetch returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorEgress (35.44s)
=== RUN   TestActorEgressHTTPS
    networking_test.go:96: creating actor networking-e2e/egress-https-1787098498518949911
    networking_test.go:96: resumed actor networking-e2e/egress-https-1787098498518949911
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:107: Actor HTTPS egress fetch returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorEgressHTTPS (35.44s)
FAIL
FAIL	github.com/agent-substrate/substrate/internal/e2e/suites/networking	150.107s

[keithmattix]

I will take a look

[keithmattix]

haiyanmeng Looks like that link is actually an old run from before my PR and #669 which fixes a lot of the 503s. Is there a newer failure I can take a look at?

[haiyanmeng]

Keith Mattix II (@keithmattix) , I glanced the failures at the top of https://github.com/agent-substrate/substrate/actions/workflows/pr-workflow.yaml?query=is%3Afailure, but have not found one showing the flaky networking tests.

[haiyanmeng]

I wish we could have some dashboard showing the green/red status of each test in each run.