Micro-Services

Found a way, there is something called azurerm_api_management_api_operation_policy

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/api_management_api_operation_policy

operation id is something you can get it from api-spec file, which uniquely identifies individual apis

I assume the following from the information you provide:

• You have two Datatypes (Java classes). They should be merged together to one Java class
• You have to load this data from different sources
• Non of the classes are leading

I can provide you some example code. The code is based on the previos adoptions. This will give you an idea. It's not a simple copy and paste solution.

At first create a class with all fields you want to include in the result:

Amazon ECS has recently introduced a new feature dubbed "deployment circuit breaker" that covers that exact scenario. I would suggest you go through this blog post that has some insides about how it works including the events notifications.

It's still valid, because:

• a 0-RTT request is more computentionally expensive on both the client as well as the server side than just reusing the connection, since all private key crypto operations and certificate checks still apply
• 0-RTT requests can introduce security issues due to providing a chance for replay attacks (see https://tools.ietf.org/html/draft-ietf-quic-tls-34#section-9.2). Without using the 0-RTT feature a QUIC handshake still requires 1-RTT.

However since QUIC already provides multiplexing multiple requests on a stream the client should not be required to keep a full pool of connections around. A single connection is typically sufficient, as long as the server advertises being able to support a high enough number of streams.

Add the following property on the Auth Server:

Only push to DLQ records that cause non-retryable errors, that is: point 1 (bad format) and point 2 (bad data) in your example. For the format of the DLQ records, a good approach is to:

• push to DLQ the exact same kafka record value and key as the original one, do not wrap it inside any kind of envelope. This makes it much easier to reprocess with other tools during troubleshooting (e.g. with a new version of a deserializer or so).
• add a bunch of Kafka header to communicate meta-data about the error, a few typical examples would be:
  • original topic name, partition, offset and Kafka timestamp of this record
  • exception or error message
  • name and version of the application that failed to process that record
  • time of the error

Typically I use one single DLQ topic per service or application (not one per inbound topic, not a shared one across services). That tends to keep things independent and manageable.

Oh, and you probably want to put some monitoring and alert on the inbound traffic to the DLQ topic ;)

Point 3 (high volume) should, IMHO, be dealt with some sort of auto-scaling, not with a DLQ. Try to always over-estimate (a bit) the number of partitions of the input topic, since the maximum number of instances you can start of your service is limited by that. A too high number of messages is not going to overload your service, since the Kafka consumers are explicitly polling for more messages when they decide to, so they're never asking for more than the app can process. What happens if there is a peak of messages is simply they'll keep piling up in the upstream kafka topic.

Point 4 (connectivity) should be retried directly from the source topic, without any DLQ involved, since the error is transient. Dropping the message to a DLQ and picking up the next one is not going to solve any issue since, well, the connectivity issue will still be present and the next message would likely be dropped as well. Reading, or not reading, a record from Kafka is not making it go away, so a record stored there is easy to read again later. You can program your service to move forward to the next inbound record only if it successfully writes a resulting record to the outbound topic (see Kafka transactions: reading a topic is actually involving a write operation since the new consumer offsets need to be persisted, so you can tell your program to persist new offsets and the output records as part of the same atomic transaction).

Kafka is more like a storage system (with just 2 operations: sequential reads and sequential writes) than a messaging queue, it's good at persistence, data replication, throughput, scale... (...and hype ;) ). It tends to be really good for representing data as a sequence of events, as in "event sourcing". If the needs of this microservice setup is mostly asynchronous point-to-point messaging, and if most scenarios would rather favor super low latency and choose to drop messages rather than reprocessing old ones (as seems suggested by the 4 points listed), maybe a lossy in-memory queuing system like Redis queues is more appropriate?

Laravel has a function in its api :

Is that possible to create one application client in keycloak and share it amount all realms?

Out-of-the box this is not possible, just like users, clients are defined at the Realm level, and consequently, cannot be shared among realms.

A circuit breaker should be used when you need to allow a failing service to recover, instead of continuously hitting it with requests, despite the fact that it cannot serve them. This is a nice article that you can check for more details about how to use this pattern.

So if your service is likely to recover in a short amount of time, you can use a circuit breaker.

Otherwise, which I think it's the case here (because you want an alert to be fired when the service is down), I would focus more on the reasons why that microservice would fail, and I would try to minimize the chance of occurrence.

If you use Kubernetes, you can monitor the microservices using Prometheus. Here is a nice article that can get you started. Prometheus scrapes the Kubernetes API and exposes metrics related to pods. You can create an alert based on those.

To monitor external services, you can use Prometheus for this too. If the external service is able to expose metrics via Prometheus, then you just plug it and you're done. Otherwise, you have to write some code to check the health of that service and expose a metric based on that.