sf + webhost integration problems: port binding (kestrel), primary -> secondary transition #45
Description
Hi -
I'm following the pattern shown here:
• Create web-host (often with Kestrel).
• Register service-fabric services (asynchronously)
• Run the web-host
However, the lifetime of the web-host persists beyond the sf-services, and the web-host stays open until the process terminates. Since the web-host is already initialized when CreateServiceInstanceListeners() is called, ICommunicationListener.OpenAsync (and CloseAsync) are essentially no-ops.
This is probably not the intended sf pattern; instead the listener should be started inside ICommunicationListener.OpenAsync(), and stopped in CloseAsync(). An issue I'm seeing now is that a websocket based stateful service instance went from primary -> secondary, and since all the websocket clients remained connected to the secondary, requests failed since they weren't routed to the primary (if the connections closed, the service-instance would be re-resolved, and everything would work as expected).
However, it’s not immediately clear how, or whether, I should make this change as it doesn’t seem to fit the aspnet-core web-host pattern. In particular, if I stopped (disposed) the web-host in CloseAsync(), it's problematic to restart it again since all the process initialization logic in Startup.cs will be triggered (with DI services re-registering, etc etc).
As a hack, I can detect the primary -> secondary transition and manually close all the websocket connections. But, it remains a concern that all secondaries (not just the ones which transitioned to primary) will have the port open and actively listening sf (I don't want read-only secondaries).
To compound the problems, since I use Kestrel, each secondary will exclusively bind to the port, preventing other instances (eg: primaries) from starting.
How do I best use aspnet core within service fabric to prevent, or at least mitigate, the above problems?
It seems that moving to WebListener is a good first step. At that point, the processes can at least share ports, and different service-instances can bind to specific urls (eg: with partitionIds and/or guids ) so > 1 primary/secondary can run on a node (presumably http.sys or some subsystem routes requests to the correct service-instance even if they have the same port) ala your service-registry.
Has anyone else ran into these issues, or tried using Kestrel in service fabric (in a production, or near-production, capacity)?
thanks!
--craig