[BUG] channel in filtergateway

[youngtaekiim]

Reproduce

1. make builder image in project root folder.
2. make install
3. cd examples && ./helloworld.sh

Expected

no error and launch helloworld container

Actual

# podman logs -f piccolo-server-apiserver 
http api listening on 0.0.0.0:47099
Err(
    GRpcStatus(
        Status {
            code: Unavailable,
            message: "tcp connect error: Connection refused (os error 111)",
            source: Some(
                tonic::transport::Error(
                    Transport,
                    ConnectError(
                        ConnectError(
                            "tcp connect error",
                            Os {
                                code: 111,
                                kind: ConnectionRefused,
                                message: "Connection refused",
                            },
                        ),
                    ),
                ),
            ),
        },
    ),
)

# podman logs -f piccolo-player-filtergateway 
FilterGatewayManager init
Piccolod gateway listening on 0.0.0.0:47002
Failed to initialize FilterGatewayManager: GRpcStatus(Status { code: Unavailable, message: "tcp connect error: Connection refused (os error 111)", source: Some(tonic::transport::Error(Transport, ConnectError(ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))) })

Received scenario handling request
Failed to send scenario: channel closed
Error handling scenario: Failed to send scenario

[youngtaekiim]

There are two issues.

First, gRPC connect error.

Second, channel closed error in filtergateway

[youngtaekiim]

@akshayg0314 Could you please review this issue?

[akshayg0314]

@akshayg0314 Could you please review this issue?

yes sure i will analyze the issue and update you

[akshaylg0314]

i am getting below error while reproducing the issue

acrn@acrn-NUC11TNHi5:~/new_ak/new/orignal_new/pullpiri$ sudo make install
mkdir -p /etc/piccolo/yaml
mkdir -p /etc/containers/systemd/piccolo/
mkdir -p /etc/containers/systemd/piccolo/etcd-data/
cp -r ./src/settings.yaml /etc/containers/systemd/piccolo/
cp -r ./containers/piccolo-. /etc/containers/systemd/piccolo/
systemctl daemon-reload
systemctl start piccolo-agent
Failed to start piccolo-agent.service: Unit piccolo-agent.service not found.
make: *** [Makefile:66: install] Error 5

[akshaylg0314]

@youngtaekiim
I executed each module manually — apiserver, filtergateway, nodeagent, and actioncontroller.
After that, I ran helloworld.sh.

I encountered a parsing error in nodeagent only — no issues related to gRPC or channel closures.

If you're seeing gRPC errors or channel closed issues, it's likely due to stale etcd data.
Please clear etcd before running tests using the following command:
"etcdctl del "" --prefix"

Error:
NodeAgentManager init NodeAgentManager successfully initialized NodeAgent listening on 0.0.0.0:47004 Received workload handling request Failed to parse scenario: spec: missing field actionat line 7 column 3 Error handling workload: Failed to parse scenario YAML Received workload handling request Failed to parse scenario: spec: missing fieldactionat line 7 column 3 Error handling workload: Failed to parse scenario YAML Received workload handling request Failed to parse scenario: spec: missing fieldactionat line 7 column 3 Error handling workload: Failed to parse scenario YAML

Please guide me regarding the error I got related to "Failed to start piccolo-agent.service: Unit piccolo-agent.service not found."
Thanks!

[akshaylg0314]

Hey @youngtaekiim, below are my findings for this issue. This issue can be resolved by using the following 2 fixes:-

When executing the services manually in the following order:

filtergateway

nodeagent

apiserver

piccolo-etcd

./helloworld.sh

The filtergateway fails to initialize due to ETCD not being available at the time of its startup. This results in the error:
Failed to initialize FilterGatewayManager: GRpcStatus(Status { code: Unavailable, message: "tcp connect error: Connection refused (os error 111)" ... Consequently, API server attempts to send the scenario to filtergateway fail with a "channel closed" error.

🔎 Root Cause:
The filtergateway tries to establish a gRPC connection with ETCD during initialization. If ETCD is not yet running, the connection attempt fails and filtergateway still runs without retrying for etcd connection. This causes all downstream requests (e.g., from apiserver) to fail.

✅ Fix Options:
Option 1: Enforce Correct Startup Order

Ensure ETCD is started before any dependent components like filtergateway, apiserver, etc.

Recommended startup order:

1. piccolo-etcd
2. filtergateway
3. nodeagent
4. apiserver
5. ./helloworld.sh

This is the quickest short-term fix and can be enforced via documentation or automation (e.g., in systemd unit dependencies or in makefile).

Option 2: Improve FilterGateway Resilience

Implement retry logic in filtergateway for ETCD connection, similar to how apiserver handles unavailable filtergateway.

This could involve:

Retrying with exponential backoff on connection failures.

Logging and continuing to retry in a background task until ETCD becomes available.

How to Validate the Fix:
Kill and restart ETCD independently, then start filtergateway → It should retry until ETCD becomes available.

Run ./helloworld.sh after components are up → it should work regardless of small delays in ETCD availability.

We can either update the Makefile to ensure ETCD is started before the other services or implement a permanent fix in filtergateway (Option 2) by adding retry logic for ETCD connection. Please share your review of this approach.

Thanks!!

[youngtaekiim]

@akshaylg0314 That's a very good point. I forgot that etcd uses gRPC internally. I confirmed that the communication error does not occur when I adjust the order between containers. However, since player pods and server pods are likely to be deployed on different nodes by default, I will think more about how to adjust the order.

[akshaylg0314]

@akshaylg0314 That's a very good point. I forgot that etcd uses gRPC internally. I confirmed that the communication error does not occur when I adjust the order between containers. However, since player pods and server pods are likely to be deployed on different nodes by default, I will think more about how to adjust the order.

Hey @youngtaekiim
Thanks for verification I think you can explore option 2 also once.

I have a query related to the vendor(dds-manager) folder in the player. When we are running code coverage for filtergateway, cargo tarpaulin is linking vendor code also from outside the filtergateway directory. Is there any dependency is there for DDS in the vendor folder for filtergateway?
What is actual use of the vendor folder where we are going to use this ?

Thanks

[youngtaekiim]

@akshaylg0314 That's a very good point. I forgot that etcd uses gRPC internally. I confirmed that the communication error does not occur when I adjust the order between containers. However, since player pods and server pods are likely to be deployed on different nodes by default, I will think more about how to adjust the order.

Hey @youngtaekiim Thanks for verification I think you can explore option 2 also once.

I have a query related to the vendor(dds-manager) folder in the player. When we are running code coverage for filtergateway, cargo tarpaulin is linking vendor code also from outside the filtergateway directory. Is there any dependency is there for DDS in the vendor folder for filtergateway? What is actual use of the vendor folder where we are going to use this ?

Thanks

The vendor folder is a module that I used last year to make my gateway (now filtergateway) work. I left it there so that other developers can refer to it for their DDS receiver implementations while I was away. It is not used at all now, so you can delete it. After deleting it, please run cargo tarpaulin and let me know the results.

[akshaylg0314]

Oh, okay @youngtaekiim — thank you so much for clarifying this! I've been trying to understand the usage and linkage of the vendor folder for the past two weeks while you were unavailable.
Can i delete this vendor folder and push the changes or you guys will do this?
thanks!!

[youngtaekiim]

Oh, okay @youngtaekiim — thank you so much for clarifying this! I've been trying to understand the usage and linkage of the vendor folder for the past two weeks while you were unavailable. Can i delete this vendor folder and push the changes or you guys will do this? thanks!!

I don't know much about cargo tarpaulin, so if there is no problem with the behavior after deletion, it would be better for you to commit it yourself.

[akshaylg0314]

Oh, okay @youngtaekiim — thank you so much for clarifying this! I've been trying to understand the usage and linkage of the vendor folder for the past two weeks while you were unavailable. Can i delete this vendor folder and push the changes or you guys will do this? thanks!!

I don't know much about cargo tarpaulin, so if there is no problem with the behavior after deletion, it would be better for you to commit it yourself.

ohh sure thanks @youngtaekiim