failover | cluster-fuck management | TCP library

There are multiple ways you could handle distributing the requests across your servers. Some may be more preferable than the other depending on your deployment environment.

The Fabio load balancer docs have a section on deployment configurations which I'll use as a reference.

In this model, you could configure DNS to point to the IPs of all three servers. Clients would receive all three IPs back in response to a DNS query, and randomly connect to one of the available instances.

If an IP is unhealthy, the client should retry the request to one of the other IPs, but clients may experience slower response times if a server is unavailable for an extended period of time and the client is occasionally routing requests to that unavailable IP.

You can mitigate this issue by configuring your DNS server to perform health checking of backend instances (assuming it supports it). AWS Route 53 provides this functionality (see Configuring DNS failover). If your DNS server does not support health checking, but provides an API to update records, you can use Consul Terraform Sync to automate adding/removing server IPs as the health of the Fabio instances changes in Consul.

As you mentioned the other option would be to place Fabio behind a load balancer. If you're deploying in the cloud, this could be the cloud provider's LB. The LB would give you better control over traffic routing to Fabio, provide TLS/SSL termination, and other functionality.

If you're on-premises, you could front it with any available load balancer like F5, A10, nginx, Apache Traffic Server, etc. You would need to ensure the LB is deployed in a highly available manner. Some suggestions for doing this are covered in the next section.

Whether you're running Fabio directly on the Internet, or behind a load balancer, you need to make sure the IP which clients are connecting to is highly available.

If you're deploying on-premises, one method for achieving this would be to assign a common loopback IP each of the Fabio servers (e.g., 192.0.2.10), and then use an L2 redundancy protocol like Virtual Router Redundancy Protocol (VRRP) or an L3 routing protocol like BGP to ensure the network routes requests to available instances.

Keepalived is a VRRP daemon for Linux. There can find many tutorials online for installing and configure in.

GoCast is a BGP daemon built on GoBGP which conditionally advertises IPs to the upstream network based on the state of health checks. The author of this tool published a blog post titled BGP based Anycast as a Service which walks through deploying GoCast on Nomad, and configuring it to use Consul for health information.

If you're deploying on-premises, a more simple configuration than the two aforementioned solutions might be to configure your router to install/remove static routes based on health checks to your backend instances. Cisco routers support this through their IP SLA feature. This tutorial walks through a basic setup configuration http://www.firewall.cx/cisco-technical-knowledgebase/cisco-routers/813-cisco-router-ipsla-basic.html.

As you can see, there are many ways to configure HA for Fabio or an upstream LB. Its hard to provide a good recommendation without knowing more about your environment. Hopefully one of these suggestions will be useful to you.

You absolutely have no reason to worry about failover for AAD.

From documentation:

For durability, any piece of data written to Azure AD is replicated to at least 4 and up to 13 datacenters depending on your tenant configuration. Within each data center, data is again replicated at least 9 times for durability but also to scale-out capacity to serve authentication load. To illustrate—this means that at any point in time, there are at least 36 copies of your directory data available within our service in our smallest region. For durability, writes to Azure AD are not completed until a successful commit to an out of region datacenter.

As mentioned, Azure AD itself is architected with multiple levels of internal resilience, but our principle extends even further to have resilience in all our external dependencies. This is expressed in our no single point of failure (SPOF) principle.

Given the criticality of our services we don’t accept SPOFs in critical external systems like Distributed Name Service (DNS), content delivery networks (CDN), or Telco providers that transport our multi-factor authentication (MFA), including SMS and Voice. For each of these systems, we use multiple redundant systems configured in a full active-active configuration.

Azure AD is already a massive system running on over 300,000 CPU Cores and able to rely on the massive scalability of the Azure Cloud to dynamically and rapidly scale up to meet any demand. This can include both natural increases in traffic, such as a 9AM peak in authentications in a given region, but also huge surges in new traffic served by our Azure AD B2C which powers some of the world’s largest events and frequently sees rushes of millions of new users.

Here, use this in your controller:

As it seems noone else had the same issue or using primitive objects as cache entries, thus haven't noticed the issue. Upon replicating and fortunately traced the root cause, the below points are coming up :

• Always implement Serializable / hashCode / equals for custom objects that are going to end been transmitted through a replicated/synchronized cache.
• Never put primitive arrays, as the hashcode / equals would not be calculated - efficiently-
• Dont enable eviction with remove strategy on replicated caches, as upon reaching the maximum limit, the entries are getting removed randomly - based on TinyLFU - and not based on the expired timer and never getting removed from the JVM heap.

Yes, it is possible! The context should be built like this:

You simply need to set up Azure App Service access restrictions. See here : https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions

PS : it also contains a section on how to only allow Front Door access https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions#restrict-access-to-a-specific-azure-front-door-instance

For you question "Is there a way for ADF to detect if the primary is down and auto failover to the secondary IR?", the answer is no, Data Factory doesn't have the failover feature. The shared integration runtime nodes don't affect each other.

For another question in the comment, the IR can't be stop/pause automatically, we must set it manually on the machine:

Just to extend the comment that has been added. So, basically if You use ConfluentRegistryAvroDeserializationSchema.forGeneric the data produced my the consumer isn't really String but rather GenericRecord. So, the moment You will try to use it in Your map that expects String it will fail, because your DataStream is not DataStream but rather DataStream.

Now, it works if You remove the map only because You havent specified the type when defining FlinkKafkaConsumer and your FlinkKafkaProducer, so Java will just try to cast every object to required type. Your FlinkKafkaProducer is actually FlinkKafkaProducer so there will be no problem there and thus it will work as it should.

In this particular case You don't seem to be needing Avro at all, since the data is just raw CSV.

UPDATE: Seems that You are actually processing Avro, in this case You need to change the type of Your DataStream to DataStream and all the functions You gonna write are going to work using GenericRecord not String.

So, You need something like:

The issue was with win_shell module. I tried to use ansible.windows.win_powershell and it helped. Variable began contain symbols in language which I need.

Thanks for taking time!