oom | Open Octave Project exists to provide professional level | Audio Utils library

We have setup Redis with sentinel high availability using 3 nodes. Suppose fist node is master, when we reboot first node, failover happens and second node becomes master, until this point every thing is OK. But when fist node comes back it cannot sync with master and we saw that in its config no "masterauth" is set.
Here is the error log and Generated by CONFIG REWRITE config:

I was able to run my react app locally without issues, however when i deployed app to heroku I got OOM errors. It's not the first time I deploy the app, however this time I add OKTA authentication which apparently cause this issue. Any advise on how to resolve this issue will be appreciated.

Was doing some internal testing about a clustering solution on top of infinispan/jgroups and noticed that the expired entries were never becoming eligible for GC, due to a reference on the expiration-reaper, while having more than 1 nodes in the cluster with expiration enabled / eviction disabled. Due to some system difficulties the below versions are being used :

• JDK 1.8
• Infinispan 9.4.20
• JGroups 4.0.21

In my example I am using a simple Java main scenario, placing a specific number of data, expecting them to expire after a specific time period. The expiration is indeed happening, as it can be confirmed both while accessing the expired entry and by the respective event listener(if its configured), by it looks that it is never getting removed from the available memory, even after an explicit GC or while getting close to an OOM error.

So the question is :

Is this really expected as default behavior, or I am missing a critical configuration as per the cluster replication / expiration / serialization ?

Example :

Cache Manager :

We have a kstreams app doing kstream-kstable inner join. Both the topics are high volume with 256 partitions each. kstreams App is deployed on 8 nodes with 8 GB heap each right now. We see that the heap memory keeps constantly growing and eventually OOM happens. I am not able to get the heap dump as its running in a container which gets killed when that happens. But, I have tried a few things to gain confidence that it is related to the state stores/ktable related stuff. Without the below RocksDBConfigSetter the memory gets used up pretty quick, but with the below it is slowed down to some extent. Need some guidance to proceed further , thanks

I added below 3 properties,

I’m training DeepSpeech from scratch (without checkpoint) with a language model generated using KenLM as stated in its doc. The dataset is a Common Voice dataset for Persian language.

My configurations are as follows:

1. Batch size = 2 (due to cuda OOM)
2. Learning rate = 0.0001
3. Num. neurons = 2048
4. Num. epochs = 50
5. Train set size = 7500
6. Test and Dev sets size = 5000
7. dropout for layers 1 to 5 = 0.2 (also 0.4 is experimented, same results)

Train and val losses decreases through the training process but after a few epochs val loss does not decrease anymore. Train loss is about 18 and val loss is about 40.

The predictions are all empty strings at the end of the process. Any ideas how to improve the model?

The Spark docs state:

a Spark executor exits either on failure or when the associated application has also exited. In both scenarios, all state associated with the executor is no longer needed and can be safely discarded.

However, in the scenario where the Spark cluster configuration and dataset are such that occasional executor failure OOM occurs deep into a job, it is far more preferable for the shuffle files written by the dead executor to continue to be available to the job rather than have them recomputed.

In such a scenario with the External Shuffle Service enabled, I have appeared to observe Spark continuing to fetch the afore mentioned shuffle files and only rerunning the tasks that were active at the time when the executor died. In contrast, with the External Shuffle Service disabled I have seen Spark rerun a proportion of previously completed stages to recompute lost shuffle files as expected.

So can Spark with the External Shuffle Service enabled use saved shuffle files in the event of Executor failure as I have appeared to observe? I think so, but the documentation makes me doubt it.

I am running Spark 3.0.1 with Yarn on EMR 6.2 with dynamic allocation disabled.

Also, pre-empting comments, of course it is preferable to configure the cluster so that executor OOM never occurs. However, when initially aiming to complete an expensive Spark job the optimal cluster configuration is not yet achieved. It is at this time that shuffle reuse in the face of executor failure is valuable.

We use Amazon MWAA Airflow, rarely some task as marked as "FAILED" but there is no logs at all. As if the container had been shut down without noticing us.

I have found this link: https://cloud.google.com/composer/docs/how-to/using/troubleshooting-dags#task_fails_without_emitting_logs Which explain this by OOM on the machine. But our tasks are doing almost nothing with CPU and RAM. They only do 1 HTTP call to AWS API. So very light.

On Cloudwatch, I can see that no others tasks are launched on the same container (the DAG run start by printing the container IP, so I can search this IP on all tasks).

If someone has an idea, would be great, thanks !

I'm running a kubernetes cluster of 20+ nodes. And one pod in a namespace got restarted. The pod got killed due to OOM with exit code 137 and restarted again as expected. But would like to know the node in which the pod was running earlier. Any place we could check the logs for the info? Like tiller, kubelet, kubeproxy etc...

I am new to Pytorch and am trying to transfer my previous code from Tensorflow to Pytorch due to memory issues. However, when trying to reproduce Flatten layer, some issues kept coming out.

In my DataLoader object, batch_size is mixed with the first dimension of input (in my GNN, the input unpacked from DataLoader object is of size [batch_size*node_num, attribute_num], e.g. [4*896, 32] after the GCNConv layers). Basically, if I implement torch.flatten() after GCNConv, samples are mixed together (to [4*896*32]) and there would be only 1 output from this network, while I expect #batch_size outputs. And if I use nn.Flatten() instead, nothing seems to happen (still [4*896, 32]). Should I set batch_size as the first dim of the input at the very beginning, or should I directly use view() function? I tried directly using view() and it (seemed to have) worked, although I am not sure if this is the same as Flatten. Please refer to my code below. I am currently using global_max_pool because it works (it can separate batch_size directly).

By the way, I am not sure why training is so slow in Pytorch... When node_num is raised to 13000, I need an hour to go through an epoch, and I have 100 epoch per test fold and 10 test folds. In tensorflow the whole training process only takes several hours. Same network architecture and raw input data, as shown here in another post of mine, which also described the memory issues I met when using TF.

Have been quite frustrated for a while. I checked this and this post, but it seems their problems somewhat differ from mine. Would greatly appreciate any help!

Code: