ephemeral | An ORM for nonpersisted objects | REST library

In most cases, this is happening due to excess of log messages are consuming the storage. Solution for that would be to configure the Docker logging driver to limit the amount of saved logs:

The nvidia-gpu-device-plugin should be installed in the GPU node as well. You should see nvidia-gpu-device-plugin DaemonSet in your kube-system namespace.

It should be automatically deployed by Google, but if you want to deploy it on your own, run the following command: kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml

It will install the GPU plugin in the node and afterwards your pods will be able to consume it.

I managed to come up with this:

The reason write permissions were not allowed is because I was serving files directly from the app bundle, which cannot be written to.

My solution was to copy the contents of that directory into the Documents directory and use WebDAV to write to that directory instead.

It is very important to understand the main differences between Volumes and PersistentVolumes. Both Volumes and PersistentVolumes are Kubernetes resources which provides an abstraction of a data storage facility.

• Volumes: let your pod write to a filesystem that exists as long as the pod exists. They also let you share data between containers in the same pod but data in that volume will be destroyed when the pod is restarted. Volume decouples the storage from the Container. Its lifecycle is coupled to a pod.

• PersistentVolumes: serves as a long-term storage in your Kubernetes cluster. They exist beyond containers, pods, and nodes. A pod uses a persistent volume claim to to get read and write access to the persistent volume. PersistentVolume decouples the storage from the Pod. Its lifecycle is independent. It enables safe pod restarts and sharing data between pods.

When it comes to hostPath:

A hostPath volume mounts a file or directory from the host node's filesystem into your Pod.

hostPath has its usage scenarios but in general it might not recommended due to several reasons:

• Pods with identical configuration (such as created from a PodTemplate) may behave differently on different nodes due to different files on the nodes

• The files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume

• You don't always directly control which node your pods will run on, so you're not guaranteed that the pod will actually be scheduled on the node that has the data volume.

• If a node goes down you need the pod to be scheduled on other node where your locally provisioned volume will not be available.

The hostPath would be good if for example you would like to use it for log collector running in a DaemonSet.

I recommend the Kubernetes Volumes Guide as a nice supplement to this topic.

Setting a fixed port for the IceGrid node process with IceGrid.Node.Endpoints=tcp -h OUR_IP_ADDRESS -p 10232 will only ensure that the IceGrid node process uses this port. It won't have any effect on the configuration of the servers managed by the IceGrid node.

You will need to update the server endpoint configuration to use fixed ports as well (with the -p PORT option) if you want to ensure these servers use port numbers > 10000.

Another, easier, option would be to update your operating system configuration to ensure ephemeral ports are allocated within a range that fits your policy. Updating this configuration most likely requires administrative privileges however.

The issue with AWS provider, which didn't have much options, So I have upgraded to terraform-provider-aws_3.24.0_linux_amd64.zip and now can be added specific tags for each EBS volume

Well, that's exactly how it is implemented for BATCH.

In STREAMING you don't have the full Customers table in full, as by definition it is infinite.

For BATCH, I'll just quote this post from their official blog:

Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs [...] The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs

This link also has information on input size: it can spill to the disk. Windowing is not required (but if you specify it, it will for sure help you maintain performance/deployment size requirements )

Now if you're in STREAMING mode and know that one side won't change, you can still tell Flink about it so it optimizes around that. Use JOIN

Temporal joins take an arbitrary table (left input/probe site) and correlate each row to the corresponding row’s relevant version in the versioned table (right input/build side)

However be aware that these probe side request will go right through Flink and look it up in the database if you're using JDBC (make sure you have an index on the join key)

Disk cache refers to memory that is available and used, so that the underlying file system doesn't have to be considered at all. Ephemeral or persistent storage shouldn't matter in regard to that as far as I know.

The page cache makes sure that currently unused memory is used to keep frequently accessed parts of the file system in memory. Since the memory isn't required by any other processes, it's instead used to speed up what's usually the slowest memory - the actual disk. If the whole database / index can be kept in memory instead of having to be read from disk when a query is to be resolved, the performance of most/all databases will greatly increase.