kvstore | A gRPC-based , in-memory , key-value store | Key Value Database library

You can add httpOpt as follows:

The config looks good, similar as mine. I would troubleshoot it with following steps:

1. Exec to docker container and check if the rules file is not empty cat /etc/loki/rules/rules.yaml

2. Check the logs of loki. When rules are loaded properly logs like this will pop up:

If Loki is running in single tenant mode, the required ID is fake (yes we know this might seem alarming but it’s totally fine, no it can’t be changed).

No, you'll not lose all logs from the previous year and start from scratch. The Table Manager keeps the last tables alive using the following formula:

com.apple.developer.icloud-container-identifiers is related to CloudKit, not to iCloud documents. This key is badly documented, but it is added to your Entitlements file when you select a CloudKit container in the iCloud section of the Target settings of your app in Xcode.

iCloud document storage is a little bit older, maybe because of that the documentation for com.apple.developer.ubiquity-container-identifiers is much clearer.

To give a bit of background to anyone new to Loki, as stated in the documentation: Loki does not come with any included authentication layer. Operators are expected to run an authenticating reverse proxy in front of your services, such as NGINX using basic auth or an OAuth2 proxy.

This basically means that you'll have to place something in between the client(s) and Loki to enforce e.g. basic authentication. In this case there's a Ingress (Nginx) acting as a reverse proxy with basic authentication.

To troubleshoot problems with authentication using Nginx there's a number of things to check:

• Logs of the Nginx Ingress Pod, check for authentication errors.
• That the added Kubernetes Secret contains what you expect.
• That you have configured the Ingress object with the needed annotations.

When it comes to using Nginx as Ingress and adding basic authentication this resource from the official docs is really helpful.

In general when creating Kubernetes secrets, especially from the command line using kubectl you'll have to single quote the password if it contains special characters. This is so that the special characters wont be interpreted by your shell. More info here.

I finally did work this out. It requires a compactor but gives no warning about it. Best practice is to create an AWS s3 bucket without any public access. Next create an IAM user with programmatic access only. Create an access policy which gives full access only to the bucket you created. Attach the policy to the user's permissions. You do not need to attach a policy to the bucket itself. Check if you have "/" in your URL that you escape it with %2F otherwise you will get an auth error. Note that this config is for loki v2.0.0 which was released yesterday.

Here are my complete working docker-compose and loki config files. I put them on an external network to enable prometheus monitoring.

here is my docker-compose.yaml

The reason is that rs = future.result() is actually a blocking call - see python docs. Unfortunately, executor.submit() doesn't return an awaitable object (concurrent.futures.Future is different from asyncio.Future.

You can use asyncio.wrap_future which takes concurrent.futures.Future and returns asyncio.Future (see python docs). The new Future object is awaitable thus you can convert your blocking function into an async function.

An Example:

2 ideas:

1. Before increasing the GPU count, grow batch size so that a single GPU is as busy as possible
2. Use P3 instances instead of P2. P3 is more recent, has more memory, more CUDA cores, faster memory bandwidth and NVLink inter-GPU connections. Though it's more expensive by hour, your total training cost may be much smaller if properly tuned

Also, if your problem involves sparse updates, meaning if just a small fraction of all tokens appear in a given mini-batch, you can try using token_embedding_storage_type = 'row_sparse', which I think refers to using sparse gradient updates like described here https://medium.com/apache-mxnet/learning-embeddings-for-music-recommendation-with-mxnets-sparse-api-5698f4d7d8