oyaml | Ordered YAML : drop-in replacement for PyYAML | YAML Processing library

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

1. For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.

2. For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

It seems that in this version the type of metadata parameter is no longer dict. From the docs:

metadata (Sequence[Tuple[str, str]]) -- Additional metadata that is provided to the method.

Try with:

the following dag is working as expected, changed:

• the cluster name (cluster_name -> cluster-name).
• path for scripts.
• Dag definition.

Additionally to the service you created to expose elastic search outside the cluster, you also need an headless service so that each node / Pod of the elastic cluster can communicate with each other.

I would do the following:

First, inside the spec of the StatefulSet, change spec.serviceName to another value, such as for example elasticsearch-headless

Second. create the new service with the following:

If a resource was stored when the newer API version (v1) did not exist yet, would this be a problem when the older API version (v1beta1) is removed?

Kubernetes supports a huge elastic deprecation system, which allows you to create, migrate and maintain API versions in time, however(jumping to your next question, you should sometimes manually upgrade API versions to up-to-date ones)

You can check Kubernetes Deprecation Policy guide, that is very important part of keeping cluster in work condition.

Main rules:

• Rule #1: API elements may only be removed by incrementing the version of the API group.
• Rule #2: API objects must be able to round-trip between API versions in a given release without information loss, with the exception of whole REST resources that do not exist in some versions.
• Rule #3: An API version in a given track may not be deprecated until a new API version at least as stable is released.
• Rule #4a: Other than the most recent API versions in each track, older API versions must be supported after their announced deprecation for a certain duration.
• Rule #4b: The "preferred" API version and the "storage version" for a given group may not advance until after a release has been made that supports both the new version and the previous version

You can check also table that describes which API versions are supported in a series of subsequent releases.

Would upgrading to Kubernetes v1.22, which removes rbac.authorization.k8s.io/v1beta1, break already created/stored resources?

I think yes and you have to do some actions according to 1.22 RBAC deprecation resources

How are resource transformations between different API versions handled?

Check What to do

• Test with deprecated APIs disabled

• Locate use of deprecated APIs

• Migrate to non-deprecated APIs

These pods are managed by cronjob controller.

Use kubectl get cronjobs to list them.

Found a solution to this one after trying a lot of different stuff, forgot to update at the time. The certs were renewed after they had already expired, I guess this stopped a synchronisation of the certs across the different components in the cluster and nothing could talk to the API.

This is a three node cluster. I cordoned the worker nodes, stopped the kubelet service on them, stopped docker containers + service, started new docker containers, started the kubelet, uncordoned the nodes and carried out the same procedure on the master node. This forced the synchronisation of certs and keys across the different components.

The issue was with the metrics server.

Metrics server was configured to use kubelet-preferred-address-types=InternalIP but worker node didn't have any InternalIP listed:

The ~/.kube/config wasn't updated with the changes.

I ran: