cloud-pipeline | Cloud agnostic genomics analysis , scientific computation | Continuous Deployment library

After some time researching the problem I've stumbled upon this Github issue. The problem was originated by a mismatch between google_cloud_pipeline_components and kubernetes_api docs. In this case, serving_container_environment_variables is typed as an Optional[dict[str, str]] whereas it should have been typed as a Optional[list[dict[str, str]]]. A similar mismatch can be found for serving_container_ports argument as well. Passing arguments following kubernetes documentation did the trick:

I'm glad you solved most of your main issues and found a workaround for model declaration.

For your input.output observation on gcs_source_uris, the reason behind it is because the way the function/class returns the value. If you dig inside the class/methods of google_cloud_pipeline_components you will find that it implements a structure that will allow you to use .outputs from the returned value of the function called.

If you go to the implementation of one of the components of the pipeline you will find that it returns an output array from convert_method_to_component function. So, in order to have that implemented in your custom class/function your function should return a value which can be called as an attribute. Below is a basic implementation of it.

Ugh... I was struggling for two days and right after posting it on stackoverflow in the end, I thought of clearing ENV variable and it worked. Somehow AWS Keys were being stored in env, not sure how?. I just cleared them by below cmd and it worked :D

I didn't realized in the first place that ConcatPlaceholder accept both Artifact and string. This is exactly what I wanted to achieve:

When kubernetes creates containers, it first looks to local images, and then will try registry(docker registry by default)

You are getting this error because:

• your image cant be found localy on your node.

• you specified imagePullPolicy: Never, so you will never try to download image from registry

You have few ways of resolving this, but all of them generally instruct you to get image locally and tag it properly.

To get image on your node you can:

• copy images from one node to another

• build image from existing Dockerfile

Once you get image, tag it and specify in the deployment