service-fabric-aspnetcore | repo contains ASP.NET Core integration | Microservice library

In your given scenario, when your Client encounters a 404, the X-Service-Fabric:ResourceNotFound header isn't the only attribute your code could examine when deciding whether or not to retry some operation.

To simplistically address your concern that your client won't be able to tell the difference between a "friendly" node and a "newly arrived" node, and since you're already working with http headers, you could add a custom HTTP header to outgoing responses to identify that a request comes from your application. When the client receives a 404, you can simply check for the presence of your custom header to answer the question of whether or not it's a "legit" retry. Of course, adding a custom HTTP header just for the sake of this validation check may be more of a global solution to a local problem. Ed: goes without saying that this should not be used to make security decisions by the application

A more elegant and complex means of accomplishing the same would be to deduce the same result using a different combination of HTTP headers and response attributes (e.g., see if some other headers are expected/unexpected), but this could also be a hyper-local solution to the problem.

Respectfully, I don't think you have read the docs well enough. It is well explained here:

Services that use a dynamically-assigned port should make use of this middleware.

Services that use a fixed unique port do not have this problem in a cooperative environment. A fixed unique port is typically used for externally-facing services that need a well-known port for client applications to connect to. For example, most Internet-facing web applications will use port 80 or 443 for web browser connections. In this case, the unique identifier should not be enabled.

So summarized: when using Kestrel or WebListener you can choose to use a dynamic port or a fixed port. See the sections Use WebListener/Kestrel with a static port and Use WebListener/Kestrel with a dynamic port in the mentioned link. When you opt to use a dynamic port use ServiceFabricIntegrationOptions.UseUniqueServiceUrl, otherwise use ServiceFabricIntegrationOptions.None as the parameter for the middleware.

Now, as of the why you need this unique service url middleware in case of a dynamic port, there is a scenario that describes the possible problem:

If services use dynamically-assigned application ports, a service replica may coincidentally use the same IP:port endpoint of another service that was previously on the same physical or virtual machine. This can cause a client to mistakely connect to the wrong service. This can happen if the following sequence of events occur:

• Service A listens on 10.0.0.1:30000 over HTTP.
• Client resolves Service A and gets address 10.0.0.1:30000
• Service A moves to a different node.
• Service B is placed on 10.0.0.1 and coincidentally uses the same port 30000.
• Client attempts to connect to service A with cached address 10.0.0.1:30000.
• Client is now successfully connected to service B not realizing it is connected to the wrong service.

This can cause bugs at random times that can be difficult to diagnose