Skip to content

Gateway tunnel session can be removed but not disconnected #2494

New issue
New issue
Closed
#2497
Closed
Gateway tunnel session can be removed but not disconnected#2494
#2497
Assignees
richturnerpankalog

Description

@richturner
richturner
opened on Feb 4, 2026

On an edge gateway it has been observed that an active tunnel session can be removed but it does not get disconnected.

It has been confirmed that the GatewayClientConnector.activeTunnelSessions list is empty but a tunnel session is still established and will auto reconnect so it becomes impossible to terminate the tunnel session without restarting the manager.

Activity

  1. richturner
    added a parent issue on Feb 4, 2026
  2. richturner
    assigned
    richturner
    and
    pankalog
    on Feb 4, 2026

  3. richturner commented on Feb 4, 2026

    @richturner
    richturner
    on Feb 4, 2026
    MemberAuthor

    Here's some logs that might indicate an issue with two identical tunnel start requests coming in close together (<10s apart), you can see below that the two occasions where start requests were close together we then get a failure the second time:

    [root@7827c6e41bc8 app]# cat /storage/logs/openremote.log | grep "Start tunnel request"
2026-02-02 17:28:01.609  INFO    [multiThreadIoEventLoopGr..-3-1] ger.gateway.GatewayClientService.GATEWAY : Start tunnel request received: GatewayTunnelStartRequestEvent{sshHostname='staging.vdl-es.openremote.app', sshPort=2222, info=GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}}
2026-02-02 17:40:37.687  INFO    [multiThreadIoEventLoopGr..-3-1] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 17:44:39.179  INFO    [ContainerExecutor-14          ] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 17:44:46.862  INFO    [ContainerExecutor-37          ] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 17:52:48.521  INFO    [ContainerExecutor-64          ] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 17:52:57.893  INFO    [ContainerExecutor-28          ] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 18:33:10.443  INFO    [multiThreadIoEventLoopGr..-4-1] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
2026-02-02 18:33:28.643  INFO    [multiThreadIoEventLoopGr..-4-1] r.gateway.GatewayClientConnector.GATEWAY : Start tunnel request received: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}

    [root@7827c6e41bc8 app]# cat -n /storage/logs/openremote.log | grep "Gateway tunnel session failed to connect"
594040  2026-02-02 17:44:49.147  WARNING [ContainerExecutor-14          ] r.gateway.GatewayClientConnector.GATEWAY : Gateway tunnel session failed to connect: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
594600  2026-02-02 17:53:00.191  WARNING [ContainerExecutor-28          ] r.gateway.GatewayClientConnector.GATEWAY : Gateway tunnel session failed to connect: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
594627  2026-02-02 18:33:12.845  WARNING [ContainerExecutor-155         ] r.gateway.GatewayClientConnector.GATEWAY : Gateway tunnel session failed to connect: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}
594640  2026-02-02 18:33:30.984  WARNING [ContainerExecutor-126         ] r.gateway.GatewayClientConnector.GATEWAY : Gateway tunnel session failed to connect: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}, GatewayTunnelInfo{id='gw-6gfwwr6i4fai01yn2xocqx', gatewayId='3abPA6VHH5XuRUzWV1BAlR', realm='vdl-es', targetPort=443, target='10.4.0.41', type='HTTPS', assignedPort=null}

  4. richturner commented on Feb 4, 2026

    @richturner
    richturner
    on Feb 4, 2026
    MemberAuthor

    OK so problem is in GatewayClientConnector.close() where active tunnels are not terminated, see this log extract, the manager interconnect settings were modified and that means a new GatewayClientConnector instance is created without properly cleaning up the old one:

    2026-02-02 17:44:46.669  INFO    [WebService task-2             ] ger.gateway.GatewayClientService.GATEWAY : Updating/creating gateway connection: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
2026-02-02 17:44:46.688  INFO    [ContainerExecutor-12          ] ger.gateway.GatewayClientService.GATEWAY : Modified gateway client connection 'UPDATE': GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
2026-02-02 17:44:46.689  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Destroying gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
2026-02-02 17:44:46.690  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Connection status change for gateway IO client 'DISCONNECTING': GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
2026-02-02 17:44:46.694  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Connection status change for gateway IO client 'DISCONNECTED': GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
2026-02-02 17:44:46.695  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}

  5. richturner commented on Feb 4, 2026

    @richturner
    richturner
    on Feb 4, 2026
    MemberAuthor

    More useful info on when the GatewayClientConnector has been recreated:

    [root@7827c6e41bc8 app]# cat -n /storage/logs/openremote.log | grep "ing gateway IO client"
591835  2026-02-02 17:05:48.748  INFO    [ContainerExecutor-44          ] ger.gateway.GatewayClientService.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
591840  2026-02-02 17:05:48.785  INFO    [ContainerExecutor-23          ] ger.gateway.GatewayClientService.GATEWAY : Destroying gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
591845  2026-02-02 17:05:48.788  INFO    [ContainerExecutor-23          ] ger.gateway.GatewayClientService.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
592506  2026-02-02 17:20:03.906  INFO    [main                          ] ger.gateway.GatewayClientService.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
592691  2026-02-02 17:20:14.122  INFO    [ContainerExecutor-26          ] ger.gateway.GatewayClientService.GATEWAY : Destroying gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
592696  2026-02-02 17:20:14.133  INFO    [ContainerExecutor-26          ] ger.gateway.GatewayClientService.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
593100  2026-02-02 17:36:38.593  INFO    [main                          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
593540  2026-02-02 17:44:38.559  INFO    [main                          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
594031  2026-02-02 17:44:46.689  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Destroying gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
594034  2026-02-02 17:44:46.695  INFO    [ContainerExecutor-12          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
594350  2026-02-02 17:52:47.176  INFO    [main                          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
594591  2026-02-02 17:52:57.699  INFO    [ContainerExecutor-50          ] r.gateway.GatewayClientConnector.GATEWAY : Destroying gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
594594  2026-02-02 17:52:57.706  INFO    [ContainerExecutor-50          ] r.gateway.GatewayClientConnector.GATEWAY : Creating gateway IO client: GatewayConnection{localRealm='master', host='staging.vdl-es.openremote.app', port=null, realm='vdl-es', secured=true, attributeFilters=true, assetSyncRules={*=org.openremote.model.gateway.GatewayAssetSyncRule@d068e7fe}, disabled=false}
  6. richturner
    closed this as completedin #2497on Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions