monitoring | Monitor service availability

There are many agents capable of saving metrics collected in k8s to remote Prometheus server outside the cluster, example Prometheus itself now support agent mode, exporter from Opentelemetry, or using managed Prometheus etc.

According to man7.org getifaddrs, any of the socket operations could be a cause for EBADF

ERRORS

getifaddrs() may fail and set errno for any of the errors specified for socket(2), bind(2), getsockname(2), recvmsg(2), sendto(2), malloc(3), or realloc(3).

Unrelated, but do you do freeifaddrs() somewhere?

The pathInfo function returns all the path segments for a Request. Perhaps we could define a typeclass that, given a Servant API, produced a "parser" for the list of segments, whose result would be a formatted version of the list.

The parser type could be something like:

The error is caused by using a service account that does not have an IAM role that includes the permission cloudscheduler.jobs.create. An example role is roles/cloudscheduler.admin aka Cloud Scheduler Admin. I have the feeling that you have mixed the permission of the service account that you use with Cloud Scheduler (at runtime, when a job triggers something) and the permission of the account currently creating the job (aka your account for example).

You actually need two service accounts for the job to get created. You need one that you set up yourself (can be whatever name you like and doesn't require any special permissions) and you also need the one for the default Cloud Scheduler itself ( which is managed by Google)

1. Use an existing service account to be used for the call from Cloud Scheduler to your HTTP target or you can create a new service account for this purpose. The service account must belong to the same project as the one in which the Cloud Scheduler jobs are created. This is the client service account. Use this one when specifying the service account to generate the OAuth / OICD tokens. If your target is part of Google Cloud, like Cloud Functions/Cloud Run update your client service account by granting it the necessary IAM role (Cloud function invoker for cloud functions and Cloud Run Invoker for Cloud Run).The receiving service automatically verifies the generated token. If your target is outside of Google Cloud, the receiving service must manually verify the token.

2. The other service account is the default Cloud Scheduler service account which must also be present in your project and have the Cloud Scheduler Service Agent role granted to it. This is so it can generate header tokens on behalf of your client service account to authenticate to your target. The Cloud Scheduler service account with this role granted is automatically set up when you enable the Cloud Scheduler API, unless you enabled it prior to March 19, 2019, in which case you must add the role manually.

Note : Do not remove the service-YOUR_PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com service account from your project, or its Cloud Scheduler Service Agent role. Doing so will result in 403 responses to endpoints requiring authentication, even if your job's service account has the appropriate role.

The solution is to use a Configuration deployment platform hook for any commands that change the files in the deployment directory. Note that this is different from an Application deployment platform hook.

Using the example of the collectstatic command, the best thing to do is to move it from a container command to a pair of hooks, one for standard deployments and one for configuration changes.

To do this, remove the collectstatic container command. Then, make two identical files:

• .platform/confighooks/predeploy/predeploy.sh
• .platform/hooks/predeploy/predeploy.sh

Each file should have the following code:

Yes. You have to configure docker with Cloud logging driver, create a log based metric and an alerting policy on that metric.

Solution:

Configure docker with cloud logging driver.

• SSH to your instance.

• Run this command as root: dockerd --log-driver=gcplogs. This will forward your docker logs to Google Cloud Logging.

• If running a container-optimized OS then follow these steps:

  echo '{"log-driver":"gcplogs"}' | sudo tee /etc/docker/daemon.json

  sudo systemctl restart docker

• Try starting and exiting a container.

These are the logs generated whenever we exit a container. Keep in mind the two exit messages

Create a log based metric

• Go to Logging -> Select Log-based Metrics.

• Click on Create Metric.

• Metric Type: Counter. In Details, enter a Log Metric name (e.g. mysite-container-exited)

• Under Filter Selection -> Build filter, copy the following code

  resource.type="gce_instance"

  log_name="projects/myproject-12345/logs/gcplogs-docker-driver" replace myproject-12345 with your project name.

  jsonPayload.container.name="/mysite" change mysite to your container name.

  jsonPayload.message=~"exit$" This is regex that matches exit as the last word in a line.

It should look something like this.

Create an alerting policy

• Go to Monitoring -> Select Alerting.
• Click on Create Policy.
• Click on Select a Metric and search for your metric name (e.g. mysite-container-exited).
• Select your metric and click on Apply.
• Set Rolling window to 2 min and Rolling window function to count. Click Next.
• Set Alert Trigger to Any time series violates, Threshold postion to Above threshold and Threshold value to 1. Click Next.
• Select a notification Channel to send alerts to. If none exists then click on Manage Notification Channels and create one.
• Easiest one would be to add an email for notification channel.
• After creating go back, click on Refresh icon in Notification Channels and select it. Click ok.
• Click on Save Policy.

As we have seen that two exit commands are issued per container to the logs whenever we exit a container, thats's why the threshold is set to above 1.

You can monitor more containers by creating a new metric and changing the value of jsonPayload.container.name="/mysite" to your container name.

Now try to start and stop the container. You will receive an alert on email in 2-3 minutes.

This is a bug in Spring Boot for which I've just opened an issue. Thanks for bringing it to our attention.

You can work around the problem by excluding the bean that exports the endpoints to JMX from lazy initialization. To do so, add the following bean to your application:

Currently it's not available, We have added to Azure metrics feature request to the Translator service and it will be added in the near future.

I believe that you need a combination of concatMap() and last().

concatMap does not subscribe to the next observable until the previous completes. Using it you will ensure the order of requests execution. And as it follows from the description it doesn't cancel previous subscriptions and let them finish, unlike switchMap.

last emits the last value emitted from the source on completion. Using it you will ensure that only one (last) result will be passed to the result.

Your code will look like that: