argo-workflows | Workflow engine for Kubernetes | BPM library

by argoproj Go Version: v0.0.0-dev-no-git-3 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | argo-workflows Summary

argo-workflows is a Go library typically used in Automation, BPM applications. argo-workflows has a Permissive License and it has medium support. However argo-workflows has 1276 bugs and it has 1 vulnerabilities. You can download it from GitHub.

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition). Argo is a Cloud Native Computing Foundation (CNCF) hosted project.

Support

Quality

Security

License

Reuse

Support

Quality

Security

License

Reuse

Support

argo-workflows has a medium active ecosystem.

It has 12704 star(s) with 2753 fork(s). There are 200 watchers for this library.

There were 8 major release(s) in the last 6 months.

There are 739 open issues and 4277 have been closed. On average issues are closed in 153 days. There are 43 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of argo-workflows is v0.0.0-dev-no-git-3

argo-workflows Support

Best in #BPM

Average in #BPM

argo-workflows Support

Best in #BPM

Average in #BPM

Quality

argo-workflows has 1276 bugs (1275 blocker, 0 critical, 1 major, 0 minor) and 2880 code smells.

argo-workflows Quality

Best in #BPM

Average in #BPM

argo-workflows Quality

Best in #BPM

Average in #BPM

Security

argo-workflows has 1 vulnerability issues reported (0 critical, 0 high, 1 medium, 0 low).

argo-workflows code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

argo-workflows Security

Best in #BPM

Average in #BPM

argo-workflows Security

Best in #BPM

Average in #BPM

License

argo-workflows is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

argo-workflows License

Best in #BPM

Average in #BPM

argo-workflows License

Best in #BPM

Average in #BPM

Reuse

argo-workflows releases are available to install and integrate.

Installation instructions are not available. Examples and code snippets are available.

It has 237874 lines of code, 10271 functions and 1414 files.

It has high code complexity. Code complexity directly impacts maintainability of the code.

argo-workflows Reuse

Best in #BPM

Average in #BPM

argo-workflows Reuse

Best in #BPM

Average in #BPM

Top functions reviewed by kandi - BETA

kandi has reviewed argo-workflows and discovered the below as its top functions. This is intended to give you an instant insight into argo-workflows implemented functionality, and help decide if they suit your requirements.

Checks the validity of the input value .

Calls the API .

Validate and convert input_value to required types .

Instantiate a new instance of the given class .

Validate all the properties of this schema

Calls an API call .

Return a OneOfSchema instance from the model .

Initialize an OpenApiModel from keyword arguments .

Convert a model instance to a python dictionary .

Attempts to convert the given value to an appropriate object .

Get all kandi verified functions for this library.

argo-workflows Key Features

UI to visualize and manage Workflows

Artifact support (S3, Artifactory, Alibaba Cloud OSS, HTTP, Git, GCS, raw)

Workflow templating to store commonly used Workflows in the cluster

Archiving Workflows after executing for later access

Scheduled workflows using cron

Server interface with REST API (HTTP and GRPC)

DAG or Steps based declaration of workflows

Step level input & outputs (artifacts/parameters)

Loops

Parameterization

Conditionals

Timeouts (step & workflow level)

Retry (step & workflow level)

Resubmit (memoized)

Suspend & Resume

Cancellation

K8s resource orchestration

Exit Hooks (notifications, cleanup)

Garbage collection of completed workflow

Scheduling (affinity/tolerations/node selectors)

Volumes (ephemeral/existing)

Parallelism limits

Daemoned steps

DinD (docker-in-docker)

Script steps

Event emission

Prometheus metrics

Multiple executors

Multiple pod and workflow garbage collection strategies

Automatically calculated resource usage per step

Java/Golang/Python SDKs

Pod Disruption Budget support

Single-sign on (OAuth2/OIDC)

Webhook triggering

CLI

Out-of-the box and custom Prometheus metrics

Windows container support

Embedded widgets

Multiplex log viewer

argo-workflows Examples and Code Snippets

No Code Snippets are available at this moment for argo-workflows.

Community Discussions

Trending Discussions on argo-workflows

Argo workflows is trying to save the artifact to /var/run/argo/outputs/artifacts. Where is this specified?

How do I use Argo Workflows Using Previous Step Outputs As Inputs?

Difference between namespace install vs managed namespace install in Argo Workflows?

Argo Workflow to continue processing during fan-out

How does the Argo Workflows CLI get permissions?

Dynamically provide an image name for the container template

Retrying after a settable delay in Argo Workflows

Dynamic "Fan-In" for artifact outputs in Argo?

How can I delete the Argo events launched in the Argo namespace?

Is there a way to gracefully end a pod with the Kubernetes client-go?

Trending Discussions on argo-workflows

Argo workflows is trying to save the artifact to /var/run/argo/outputs/artifacts. Where is this specified?

How do I use Argo Workflows Using Previous Step Outputs As Inputs?

Difference between namespace install vs managed namespace install in Argo Workflows?

Argo Workflow to continue processing during fan-out

How does the Argo Workflows CLI get permissions?

Dynamically provide an image name for the container template

Retrying after a settable delay in Argo Workflows

Dynamic "Fan-In" for artifact outputs in Argo?

How can I delete the Argo events launched in the Argo namespace?

Is there a way to gracefully end a pod with the Kubernetes client-go?

QUESTION

Argo workflows is trying to save the artifact to /var/run/argo/outputs/artifacts. Where is this specified?

Asked 2022-Mar-18 at 11:45

I found Argo lint today. Thank you to the Argo team!!! This is a very useful tool and has saved me tons of time. The following yaml checks out with no errors, but when I try to run it, I get the following error. How can I track down what is happening?

FATA[2022-03-14T19:36:29.512Z] workflows.argoproj.io "hello-world-s5rm5" not found

Here is the workflow

---
{
   "apiVersion": "argoproj.io/v1alpha1",
   "kind": "Workflow",
   "metadata": {
      "annotations": {
         "workflows.argoproj.io/description": "testing a linter",
         "workflows.argoproj.io/version": ">= 3.1.0"
      },
      "labels": {
         "workflows.argoproj.io/archive-strategy": "false"
      },
      "generateName": "hello-world-",
      "namespace": "sandbox"
   },
   "spec": {
      "arguments": {
         "parameters": [
            {
               "name": "msg",
               "value": "Hello there"
            }
         ]
      },
      "entrypoint": "entrypoint",
      "securityContext": {
         "fsGroup": 2000,
         "fsGroupChangePolicy": "OnRootMismatch",
         "runAsGroup": 3000,
         "runAsNonRoot": true,
         "runAsUser": 1000
      },
      "templates": [
         {
            "container": {
               "args": [
                  "cowsay Hello Test >> {{outputs.artifacts.message}}"
               ],
               "command": [
                  "sh",
                  "-c"
               ],
               "image": "docker/whalesay:latest",
               "imagePullPolicy": "IfNotPresent",
            },
            "name": "whalesay",
            "outputs": {
               "artifacts": [
                  {
                     "name": "message",
                     "path": "/tmp/output.tgz",
                     "s3": {
                        "key": "whalesay",
                     }
                  }
               ]
            },
            "retryStrategy": {
               "limit": "10"
            },
            "securityContext": {
               "fsGroup": 2000,
               "fsGroupChangePolicy": "OnRootMismatch",
               "runAsGroup": 3000,
               "runAsNonRoot": true,
               "runAsUser": 1000
            }
         },
         {
            "inputs": {
               "artifacts": [
                  {
                     "s3": {
                        "key": "whalesay",
                     },
                     "name": "data",
                     "path": "/tmp/input"
                  }
               ]
            },
            "name": "print",
            "retryStrategy": {
               "limit": "10"
            },
            "script": {
               "command": [
                  "python"
               ],
               "image": "python:alpine3.6",
               "imagePullPolicy": "IfNotPresent",
               "source": "import sys \nsys.stdout.write(\"{{inputs.artifacts.data}}\")\n\n"
            },
            "securityContext": {
               "fsGroup": 2000,
               "fsGroupChangePolicy": "OnRootMismatch",
               "runAsGroup": 3000,
               "runAsNonRoot": true,
               "runAsUser": 1000
            }
         },
         {
            "dag": {
               "tasks": [
                  {
                     "name": "whalesay",
                     "template": "whalesay"
                  },
                  {
                     "arguments": {
                        "artifacts": [
                           {
                              "from": "{{whalesay.outputs.artifacts.message}}",
                              "name": "data"
                           }
                        ]
                     },
                     "dependencies": [
                        "whalesay"
                     ],
                     "name": "print",
                     "template": "print"
                  }
               ]
            },
            "name": "entrypoint"
         }
      ]
   }
}
...

Here is the result of kubectl describe

Name:         hello-world
Namespace:    sandbox
Labels:       workflows.argoproj.io/archive-strategy=false
Annotations:  workflows.argoproj.io/description: testing a linter
              workflows.argoproj.io/version: >= 3.1.0
API Version:  argoproj.io/v1alpha1
Kind:         Workflow
Metadata:
  Creation Timestamp:  2022-03-14T19:33:19Z
  Generation:          1
  Managed Fields:
    API Version:  argoproj.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:workflows.argoproj.io/description:
          f:workflows.argoproj.io/version:
        f:labels:
          .:
          f:workflows.argoproj.io/archive-strategy:
      f:spec:
      f:status:
    Manager:         argo
    Operation:       Update
    Time:            2022-03-14T19:33:19Z
  Resource Version:  16499078
  UID:               b438cf44-241c-44bf-bb42-e470eaf4ca08
Spec:
  Arguments:
    Parameters:
      Name:    msg
      Value:   Hello there
  Entrypoint:  entrypoint
  Security Context:
    Fs Group:                2000
    Fs Group Change Policy:  OnRootMismatch
    Run As Group:            3000
    Run As Non Root:         true
    Run As User:             1000
  Templates:
    Container:
      Args:
        cowsay Hello Test >> {{outputs.artifacts.message}}
      Command:
        sh
        -c
      Image:              docker/whalesay:latest
      Image Pull Policy:  IfNotPresent
      Name:               
      Resources:
    Inputs:
    Metadata:
    Name:  whalesay
    Outputs:
      Artifacts:
        Name:  message
        Path:  /tmp/output.tgz
        s3:
          Key:  whalesay
    Retry Strategy:
      Limit:  10
    Security Context:
      Fs Group:                2000
      Fs Group Change Policy:  OnRootMismatch
      Run As Group:            3000
      Run As Non Root:         true
      Run As User:             1000
    Inputs:
      Artifacts:
        Name:  data
        Path:  /tmp/input
        s3:
          Key:  whalesay
    Metadata:
    Name:  print
    Outputs:
    Retry Strategy:
      Limit:  10
    Script:
      Command:
        python
      Image:              python:alpine3.6
      Image Pull Policy:  IfNotPresent
      Name:               
      Resources:
      Source:  import sys 
sys.stdout.write("{{inputs.artifacts.data}}")


    Security Context:
      Fs Group:                2000
      Fs Group Change Policy:  OnRootMismatch
      Run As Group:            3000
      Run As Non Root:         true
      Run As User:             1000
    Dag:
      Tasks:
        Arguments:
        Name:      whalesay
        Template:  whalesay
        Arguments:
          Artifacts:
            From:  {{whalesay.outputs.artifacts.message}}
            Name:  data
        Dependencies:
          whalesay
        Name:      print
        Template:  print
    Inputs:
    Metadata:
    Name:  entrypoint
    Outputs:
Status:
  Finished At:  
  Started At:   
Events:

UPDATE:

I have re-installed (upgraded) Argo and made some progress. The error (below) suggests that I have set up my Artifact repository wrong. I am following instructions found here to the best of my understanding.

The Google technical support folks are telling me that my GCS bucket is configured for read only. I am conversing with them on how to open the bucket for writing. Once that is done, am I correct that updating the configmap is sufficient?

https://argoproj.github.io/argo-workflows/configure-artifact- and repository/#google-cloud-storage-gcs

https://argoproj.github.io/argo-workflows/artifact-repository-ref/

Another Update:

Thanks to the Google help folks, I think I have cloud storage configured (I think), but I cannot yet confirm. I am getting the following error (full stack below.

Question: Where is the prefix "/var/run/argo/outputs/artifacts" specified? I have not encountered this before.

What is the proper way to reconcile this automatic insertion in the workflow?

 open /var/run/argo/outputs/artifacts/tmp/output.tgz.tgz: no such file or directory

                      hello-worldnztr8-4118214805 (v1:metadata.name)
      ARGO_CONTAINER_RUNTIME_EXECUTOR:    emissary
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 hello-worldnztr8
      ARGO_WORKFLOW_UID:                  4ed3e706-48d9-4d22-bf73-fcccc4a4e6d0
      ARGO_CONTAINER_NAME:                init
      ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
      ARGO_NODE_ID:                       hello-worldnztr8-4118214805
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      0001-01-01T00:00:00Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
Containers:
  wait:
    Container ID:  containerd://6593d624b0350cc51a739e19d78f39e5726a9f1dfddc7e8995b082a073f57864
    Image:         quay.io/argoproj/argoexec:v3.3.0
    Image ID:      quay.io/argoproj/argoexec@sha256:b37739320a21d1d96789082c659b96f2dcb59c51483d9852bc858f6cfddf82fb
    Port:          
    Host Port:     
    Command:
      argoexec
      wait
      --loglevel
      info
    State:          Terminated
      Reason:       Error
      Message:      open /var/run/argo/outputs/artifacts/tmp/output.tgz.tgz: no such file or directory
      Exit Code:    1
      Started:      Wed, 16 Mar 2022 20:42:36 +0000
      Finished:     Wed, 16 Mar 2022 20:42:37 +0000
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                      hello-worldnztr8-4118214805 (v1:metadata.name)
      ARGO_CONTAINER_RUNTIME_EXECUTOR:    emissary
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 hello-worldnztr8
      ARGO_WORKFLOW_UID:                  4ed3e706-48d9-4d22-bf73-fcccc4a4e6d0
      ARGO_CONTAINER_NAME:                wait
      ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
      ARGO_NODE_ID:                       hello-worldnztr8-4118214805
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      0001-01-01T00:00:00Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
  main:
    Container ID:  containerd://c1c3c014b6c975b5702da564f76a4f5026352bc1f3b57f7dc4d1738104ee7ab8
    Image:         docker/whalesay:latest
    Image ID:      sha256:c717279bbba020bf95ac72cf47b2c8abb3a383ad4b6996c1a7a9f2a7aaa480ad
    Port:          
    Host Port:     
    Command:
      /var/run/argo/argoexec
      emissary
      --
      sh
      -c
    Args:
      cowsay Hello Test >> {{outputs.artifacts.message}}
    State:          Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 16 Mar 2022 20:42:36 +0000
      Finished:     Wed, 16 Mar 2022 20:42:36 +0000
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_CONTAINER_NAME:                main
      ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
      ARGO_NODE_ID:                       hello-worldnztr8-4118214805
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      0001-01-01T00:00:00Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  var-run-argo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  
  kube-api-access-tvh6k:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  55s   default-scheduler  Successfully assigned default/hello-worldnztr8-4118214805 to gke-cluster-1-default-pool-d262cd84-va7g
  Normal  Pulled     55s   kubelet            Container image "quay.io/argoproj/argoexec:v3.3.0" already present on machine
  Normal  Created    54s   kubelet            Created container init
  Normal  Started    54s   kubelet            Started container init
  Normal  Pulled     53s   kubelet            Container image "quay.io/argoproj/argoexec:v3.3.0" already present on machine
  Normal  Created    53s   kubelet            Created container wait
  Normal  Started    53s   kubelet            Started container wait
  Normal  Pulled     53s   kubelet            Container image "docker/whalesay:latest" already present on machine
  Normal  Created    53s   kubelet            Created container main
  Normal  Started    53s   kubelet            Started container main

ANSWER

Answered 2022-Mar-18 at 11:45

The complete fix is detailed here https://github.com/argoproj/argo-workflows/issues/8168#event-6261265751

for purposes of this discussion, the output must be the explicit location (not a placeholder) e.g. /tmp/ouput

I think the standard is that you do not put the .tgz suffix in the output location, but that is not yet confirmed as there was another fix involved. Perhaps someone from the Argo team can confirm this.

Source https://stackoverflow.com/questions/71473378

QUESTION

How do I use Argo Workflows Using Previous Step Outputs As Inputs?

Asked 2022-Mar-02 at 20:49

I am trying to format my workflow per these instructions (https://argoproj.github.io/argo-workflows/workflow-inputs/#using-previous-step-outputs-as-inputs) but cannot seem to get it right. Specifically, I am trying to imitate "Using Previous Step Outputs As Inputs"

I have included my workflow below. In this version, I have added a path to the inputs.artifacts because the error requests one. The error I am now receiving is:

ATA[2022-02-28T14:14:45.933Z] Failed to submit workflow: templates.entrypoint.tasks.print1 templates.print1.inputs.artifacts.result.from not valid in inputs

Can someone please tell me how to correct this workflow so that it works?

---
{
   "apiVersion": "argoproj.io/v1alpha1",
   "kind": "Workflow",
   "metadata": {
      "annotations": {
         "workflows.argoproj.io/description": "Building from the ground up",
         "workflows.argoproj.io/version": ">= 3.1.0"
      },
      "labels": {
         "workflows.argoproj.io/archive-strategy": "false"
      },
      "name": "data-passing",
      "namespace": "sandbox"
   },
   "spec": {
      "artifactRepositoryRef": {
         "configMap": "my-config",
         "key": "data"
      },
      "entrypoint": "entrypoint",
      "nodeSelector": {
         "kubernetes.io/os": "linux"
      },
      "parallelism": 3,
      "securityContext": {
         "fsGroup": 2000,
         "fsGroupChangePolicy": "OnRootMismatch",
         "runAsGroup": 3000,
         "runAsNonRoot": true,
         "runAsUser": 1000
      },
      "templates": [
         {
            "container": {
               "args": [
                  "Hello World"
               ],
               "command": [
                  "cowsay"
               ],
               "image": "docker/whalesay:latest",
               "imagePullPolicy": "IfNotPresent"
            },
            "name": "whalesay",
            "outputs": {
               "artifacts": [
                  {
                     "name": "msg",
                     "path": "/tmp/raw"
                  }
               ]
            },
            "securityContext": {
               "fsGroup": 2000,
               "fsGroupChangePolicy": "OnRootMismatch",
               "runAsGroup": 3000,
               "runAsNonRoot": true,
               "runAsUser": 1000
            }
         },
         {
            "inputs": {
               "artifacts": [
                  {
                     "from": "{{tasks.whalesay.outputs.artifacts.msg}}",
                     "name": "result",
                     "path": "/tmp/raw"
                  }
               ]
            },
            "name": "print1",
            "script": {
               "command": [
                  "python"
               ],
               "image": "python:alpine3.6",
               "imagePullPolicy": "IfNotPresent",
               "source": "cat {{inputs.artifacts.result}}\n"
            },
            "securityContext": {
               "fsGroup": 2000,
               "fsGroupChangePolicy": "OnRootMismatch",
               "runAsGroup": 3000,
               "runAsNonRoot": true,
               "runAsUser": 1000
            }
         },
         {
            "dag": {
               "tasks": [
                  {
                     "name": "whalesay",
                     "template": "whalesay"
                  },
                  {
                     "arguments": {
                        "artifacts": [
                           {
                              "from": "{{tasks.whalesay.outputs.artifacts.msg}}",
                              "name": "result",
                              "path": "/tmp/raw"
                           }
                        ]
                     },
                     "dependencies": [
                        "whalesay"
                     ],
                     "name": "print1",
                     "template": "print1"
                  }
               ]
            },
            "name": "entrypoint"
         }
      ]
   }
}
...

ANSWER

Answered 2022-Mar-01 at 15:26

A very similar workflow from the Argo developers/maintainers can be found here:

https://github.com/argoproj/argo-workflows/blob/master/examples/README.md#artifacts

Source https://stackoverflow.com/questions/71296484

QUESTION

Difference between namespace install vs managed namespace install in Argo Workflows?

Asked 2022-Feb-09 at 15:13

I am trying to install argo workflows and looking at the documentation I can see 3 different types of installation https://argoproj.github.io/argo-workflows/installation/.

Can anybody give some clarity on the namespace install vs managed namespace install? If its a managed namespace, how can I tell the managed namespace? Should I edit the k8's manifest for deployment? What benefit it can provide compared to simple namespace install ?

ANSWER

Answered 2022-Feb-09 at 15:13

A namespace install allows Workflows to run only in the namespace where Argo Workflows is installed.

A managed namespace install allows Workflows to run only in one namespace besides the one where Argo Workflows is installed.

Using a managed namespace install might make sense if you want some users/processes to be able to run Workflows without granting them any privileges in the namespace where Argo Workflows is installed.

For example, if I only run CI/CD-related Workflows that are maintained by the same team that manages the Argo Workflows installation, it's probably reasonable to use a namespace install. But if all the Workflows are run by a separate data science team, it probably makes sense to give them a data-science-workflows namespace and run a "managed namespace install" of Argo Workflows from another namespace.

To configure a managed namespace install, edit the workflow-controller and argo-server Deployments to pass the --managed-namespace argument.

You can currently only configure one managed namespace, but in the future it may be possible to manage more than one.

Source https://stackoverflow.com/questions/71046661

QUESTION

Argo Workflow to continue processing during fan-out

Asked 2022-Feb-03 at 14:15

General question here, wondering if anyone has any ideas or experience trying to achieve something I am right now. I'm not entirely sure if its even possible in the argo workflow system...

I'm wondering if it is possible to continue a workflow regardless if a dynamic fanout has finished. By dynamic fanout I mean that B1/B2/B3 can go to B30 potentially.

I want to see if C1 can start when B1 has finished. The B stage is creating a small file which then in C stage I need to run an api request that it has finished and upload said file. But in this scenario B2/B3 still are processing.

And finally, D1 would have to wait for all of C1/2/3-C# to finish to complete

Diagram what I'm trying to achieve

#           *
#           | 
#          A1 (generates a dynamic list that can change depending on the inputs) 
#           | 
#        /  |  \ 
#       B1  B2  B3 +++ B#
#       |   |    |
#       C1         +++ C#
#       *   *   *
#        \  |  /
#         \ | /
#           D1

I was viewing https://github.com/argoproj/argo-workflows/blob/master/docs/enhanced-depends-logic.md but I cant wrap my head around if this is what I need to achieve this. Especially if the fan-out steps are dynamic.

Seems to me that it would bind C stage to the entirety of B stage and require for for B to finish

ANSWER

Answered 2022-Feb-03 at 14:15

Something like this should work:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
  templates:
    - name: main
      steps:
        - - name: A
            template: A
        - - name: B_C
            template: B_C
            arguments:
              parameters:
                - name: item
                  value: "{{item}}"
            withParam: "{{steps.A.outputs.parameters.items}}"
        - - name: D
            template: D
    - name: A
      # container or script spec here
      outputs:
        parameters:
          - name: items
            valueFrom:
              path: /tmp/items.json
    - name: B_C
      inputs:
        parameters:
          - name: item
      steps:
        - - name: B
            template: B
            arguments:
              parameters:
                - name: item
                  value: "{{inputs.parameters.item}}"
        - - name: C
            template: C
            arguments:
              artifacts:
                - name: file
                  from: "{{steps.B.outputs.artifacts.file}}"
    - name: B
      inputs:
        parameters:
          - name: item
      # container or script spec here
      outputs:
        artifacts:
          - name: file
            path: /tmp/file
    - name: C
      inputs:
        artifacts:
          - name: file
      # container or script spec here
    - name: D
      # container or script spec here

Step B_C in the main template runs instances of the B_C template in parallel.

Template B_C runs B and C in series. Once template B_C starts, it runs as quickly as possible, completely unaware of any concurrent executions of the B_C template. So C1 blocks only on B1, never on B2 or B3 or any other B#.

Once all instances of B_C are finished, the main template finally invokes the D template.

Source https://stackoverflow.com/questions/70960203

QUESTION

How does the Argo Workflows CLI get permissions?

Asked 2022-Feb-01 at 19:57

I am new to the argo universe and was trying to set up Argo Workflows https://github.com/argoproj/argo-workflows/blob/master/docs/quick-start.md#install-argo-workflows .

I have installed the argo CLI from the page : https://github.com/argoproj/argo-workflows/releases/latest . I was trying it in my minikube setup and I have my kubectl already configured to the minikube cluster. I am able to hit argo commands without any issues after putting it in my local bin folder.

How does it work? Where do the argo CLI is connecting to operate?

ANSWER

Answered 2022-Feb-01 at 18:04

The argo CLI manages two API clients. The first client connects to the Argo Workflows API server. The second connects to the Kubernetes API. Depending on what you're doing, the CLI might connect just to one API or the other.

To connect to the Kubernetes API, the CLI just uses your kube config.

To connect to the Argo server, the CLI first checks for an ARGO_TOKEN environment variable. If it's not available, the CLI falls back to using the kube config.

ARGO_TOKEN is only necessary when the Argo Server is configured to require client auth and then only if you're doing things which require access to the Argo API instead of just the Kubernetes API.

Source https://stackoverflow.com/questions/70943374

QUESTION

Dynamically provide an image name for the container template

Asked 2022-Jan-18 at 15:02

Is there a way to provide an image name for the container template dynamically based on its input parameters?

We have more than 30 different tasks each with its own image and that should be invoked identically in a workflow. The number may vary each run depending on the output of a previous task. So we don't want to or even can't just hardcode them inside workflow YAML.

An easy solution would be to provide the image field for the container depending on the input parameter and have the same template for each of these tasks. But looks like it's impossible. This workflow doesn't work:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay

  templates:
  - name: whalesay
    inputs:
      parameters:
        - name: image
          default: whalesay:latest
    container:
      image: "docker/{{image}}"
      command: [cowsay]
      args: ["hello world"]

Is there some workaround for this particular case?

Also is there a document somewhere describing in which fields one can use workflow variables? Documentation page says only:

Some fields in a workflow specification allow for variable references which are automatically substituted by Argo.

ANSWER

Answered 2022-Jan-14 at 11:29

This is a possible workaround: to use when and conditional run of a task. We need to list of all possible tasks with their container images though:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-dynamic-image-
spec:
  entrypoint: main

  templates:
  - name: main
    steps:
    - - name: fanout-step
        template: fanout
    - - name: loop-step
        template: options
        arguments:
          parameters:
          - name: code
            value: "{{item}}"
        withParam: "{{steps.code-step.outputs.parameters.codes}}"

  - name: fanout
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import json
        with open("/tmp/codes.json", 'w') as wf:
          json.dump(["foo", "bar", "buz"], wf)
    outputs:
      parameters:
        - name: codes
          valueFrom:
            path: /tmp/codes.json

  - name: foo-code
    script:
      image: python:alpine3.6
      command: [ python ]
      source: |
        print("foo-code")

  - name: bar-code
    script:
      image: python:alpine3.6
      command: [ python ]
      source: |
        print("bar-code")

  - name: buz-code
    script:
      image: python:alpine3.6
      command: [ python ]
      source: |
        print("buz-code")

  - name: missed-code
    script:
      image: python:alpine3.6
      command: [ python ]
      source: |
        print("THIS SHOULD NOT BE PRINTED")

  - name: options
    inputs:
      parameters:
        - name: code
    steps:
    - - name: foo-code-option
        template: foo-code
        when: "{{inputs.parameters.code}} == foo"
      - name: bar-code-option
        template: bar-code
        when: "{{inputs.parameters.code}} == bar"
      - name: buz-code-option
        template: buz-code
        when: "{{inputs.parameters.code}} == buz"
      - name: missed-code-option
        template: missed-code
        when: "{{inputs.parameters.code}} == missed"

Source https://stackoverflow.com/questions/70706752

QUESTION

Retrying after a settable delay in Argo Workflows

Asked 2022-Jan-12 at 23:47

One of our Argo Workflow steps may hit a rate limit and I want to be able to tell argo how long it should wait until the next retry.

Is there a way to do it?

I've seen Retries on the documentation but it only talks about retry count and backoff strategies and it doesn't look like it could be parameterized.

ANSWER

Answered 2022-Jan-12 at 23:47

As far as I know there's no built-in way to add a pause before the next retry.

However, you could build your own with Argo's exit handler feature.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: exit-handler-with-pause-
spec:
  arguments:
    parameters
    - name: pause-before-retry-seconds
      value: "60"
  entrypoint: intentional-fail
  onExit: exit-handler
  - name: intentional-fail
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo intentional failure; exit 1"]
  - name: exit-handler
    steps:
    - - name: pause
        template: pause
        when: "{{workflow.status}} != Succeeded"
  - name: pause
    container:
      image: alpine:latest
      env:
      - name: SECONDS
        value: "{{workflow.parameters.pause-before-retry-seconds}}"
      command: [sh, -c]
      args:
      - >-
        echo "Pausing before retry..."
        sleep "$SECONDS"

If the retry pause needs to be calculated within the workflow, check out the exit handler with params example.

Source https://stackoverflow.com/questions/70683231

QUESTION

Dynamic "Fan-In" for artifact outputs in Argo?

Asked 2022-Jan-10 at 14:20

I have an Argo workflow with dynamic fan-out tasks that do some map operation (in a Map-Reduce meaning context). I want to create a reducer that aggregates their results. It's possible to do that when the outputs of each mapper are small and can be put as an output parameter. See this SO question-answer for the description of how to do it.

But how to aggregate output artifacts with Argo without writing custom logic of writing them to some storage in each mapper and read from it in reducer?

ANSWER

Answered 2022-Jan-10 at 14:20

Artifacts are more difficult to aggregate than parameters.

Parameters are always text and are generally small. This makes it easy for Argo Workflows to aggregate them into a single JSON object which can then be consumed by a "reduce" step.

Artifacts, on the other hand, may be any type or size. So Argo Workflows is limited in how much it can help with aggregation.

The main relevant feature it provides is declarative repository write/read operations. You can specify, for example, an S3 prefix to write each parameter to. Then, in the reduce step, you can load everything from that prefix and perform your aggregation logic.

Argo Workflows provides a generic map/reduce example. But besides artifact writing/reading, you pretty much have to do the aggregation logic yourself.

Source https://stackoverflow.com/questions/70643465

QUESTION

How can I delete the Argo events launched in the Argo namespace?

Asked 2021-Dec-12 at 15:27

I am trying to delete (and recreate) the Argo namespace, but it won't fully delete because I tried launching an eventsource and eventbus there. Now these will not delete.

I have tried to delete them via yaml and individually - no success yet.

The frustrating result is that I cannot re-launch argo

customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": roles.rbac.authorization.k8s.io "argo-role" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": rolebindings.rbac.authorization.k8s.io "argo-binding" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": configmaps "workflow-controller-configmap" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "workflow-controller-metrics" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "workflow-controller" is forbidden: unable to create new content in namespace argo because it is being terminated

Here is what is going on in the argo namespace itself

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-10-21T14:52:51Z",
        "deletionTimestamp": "2021-12-10T18:48:17Z",
        "labels": {
            "kubernetes.io/metadata.name": "argo"
        },
        "name": "argo",
        "resourceVersion": "9222845",
        "uid": "2ce61352-74a5-40d1-b35e-fe1efa39c3af"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-12-10T18:48:23Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2021-12-10T18:48:23Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2021-12-10T18:48:23Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2021-12-10T18:48:23Z",
                "message": "Some resources are remaining: eventbus.argoproj.io has 1 resource instances, eventsources.argoproj.io has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2021-12-10T18:48:23Z",
                "message": "Some content in the namespace has finalizers remaining: eventbus-controller in 1 resource instances, eventsource-controller in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

UPDATE: Even though one test of Kubernetes shows that the eventsource persists, another test shows that it does not

This is the response to kubectl get eventsources argo

Error from server (NotFound): eventsources.argoproj.io "argo" not found

ANSWER

Answered 2021-Dec-12 at 15:27

For anyone who stumbles onto this question, it is a permissions issue. Make certain your service account has permissions to work in both namespaces (argo and argo-events).

Source https://stackoverflow.com/questions/70309339

QUESTION

Is there a way to gracefully end a pod with the Kubernetes client-go?

Asked 2021-Oct-29 at 12:01

The main question is if there is a way to finish a pod from the client-go sdk, I'm not trying to delete a pod, I just want to finish it with a Phase-Status: Completed.

In the code, I'm trying to update the pod phase but It doesn't work, It does not return an error or panic but The pod does not finish. My code:

func main() {

    // creates the in-cluster config
    config, err := rest.InClusterConfig()
    if err != nil {
        panic(err.Error())
    }
    // creates the clientset
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err.Error())
    }
    for {

        pods, err := clientset.CoreV1().Pods("ns").List(context.TODO(), metav1.ListOptions{})
        if err != nil {
            panic(err.Error())
        }
        for _, pod := range pods.Items {

            podName:= pod.Name
            if strings.Contains(strings.ToLower(podName), "single-condition") {
            
                fmt.Println("get pods metadatada")
                fmt.Println(pod.Name)
                fmt.Printf("pod.Name %s \n", pod.Name)
                fmt.Printf("Status.Phase %s \n", pod.Status.Phase)
                fmt.Printf("PodIP %s \n", pod.Status.PodIP)

                containers := pod.Status.ContainerStatuses
                if len(containers) > 0 {
                    for _ ,c := range containers {
                        fmt.Printf("c.Name %s \n", c.Name)
                        fmt.Printf("c.State %s \n", c.State)
                        fmt.Printf("c.State.Terminated %s \n", c.State.Terminated)

                        stateTerminated := c.State.Terminated
                        stateRunning := c.State.Running
                        if stateTerminated == nil && stateRunning != nil {
                            fmt.Printf("c.State.Terminated %s \n", c.State.Terminated)
                            fmt.Printf("stateRunning Reason: %s\n", reflect.TypeOf(c.State.Running))

                            getPod, getErr := clientset.CoreV1().Pods("ns").Get(context.TODO(), "single-condition-pipeline-9rqrs-1224102659" , metav1.GetOptions{})
                            if getErr != nil {
                                fmt.Println("error1")
                                panic(fmt.Errorf("Failed to get: %v", getErr))
                            }
                            fmt.Println("update values")
                            fmt.Printf(" getPodName %d \n", getPod.Name)
                            getPod.Status.Phase = "Succeeded"
                            fmt.Println("updated status phase")
                            getContainers := getPod.Status.ContainerStatuses
                            fmt.Printf("len get container %d \n", len(getContainers))
                            _, updateErr := clientset.CoreV1().Pods("argo-workflows").Update(context.TODO(), getPod, metav1.UpdateOptions{})
                            fmt.Println("commit update")
                            if updateErr != nil {
                                fmt.Println("error updated")
                                panic(fmt.Errorf("Failed to update: %v", updateErr))
                            }

                        } else {
                            fmt.Printf("c.State.Terminated %s \n", c.State.Terminated.Reason)
                            //fmt.Println("Not finished ready!!!")
                            //fmt.Printf("c.State.Running %s \n", c.State.Running)
                            //fmt.Printf("c.State.Waiting %s \n", c.State.Waiting)
                        }
                    }
                }
            }
        }
        time.Sleep(10 * time.Second)

    }
}

and some logs:

single-condition-pipeline-9rqrs-1224102659
pod.Name single-condition-pipeline-9rqrs-1224102659 
Status.Phase Running 
PodIP XXXXXXXXXXXX
c.Name main 
---------------------------------------------------------------------------------------------
c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
c.State.Terminated nil 
c.State.Terminated nil 
stateRunning Reason: *v1.ContainerStateRunning
update values
 getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
updated status phase
len get container 2 
commit update
c.Name wait 
c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
c.State.Terminated nil 
c.State.Terminated nil 
stateRunning Reason: *v1.ContainerStateRunning
update values
 getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
updated status phase
len get container 2 
---------------------------------------------------------------------------------------------
commit update
---------------------------------------------------------------------------------------------
get pods metadatada
single-condition-pipeline-9rqrs-1224102659
pod.Name single-condition-pipeline-9rqrs-1224102659 
Status.Phase Running 
PodIP XXXXXXXXXX 
c.Name main 
c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
c.State.Terminated nil 
c.State.Terminated nil 
stateRunning Reason: *v1.ContainerStateRunning
update values
 getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
updated status phase
len get container 2 
commit update
c.Name wait 
c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
c.State.Terminated nil 
c.State.Terminated nil 
stateRunning Reason: *v1.ContainerStateRunning
update values
 getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
updated status phase
len get container 2 
commit update

so here: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-status, It mentions a Patch but I don't know how to use it, so if somebody could help me or if there is another way to finish it.

ANSWER

Answered 2021-Oct-29 at 12:01

You cannot set the phase or anything else in the Pod status field, it is read only. According to the Pod Lifecycle documentation your pod will have a phase of Succeeded after "All containers in the Pod have terminated in success, and will not be restarted." So this will only happen if you can cause all of your pod's containers to exit with status code 0 and if the pod restartPolicy is set to onFailure or Never, if it is set to Always (the default) then the containers will eventually restart and your pod will eventually return to the Running phase.

In summary, you cannot do what you are attempting to do via the Kube API directly. You must:

Ensure your pod has a restartPolicy that can support the Succeeded phase.
Cause your application to terminate, possibly by sending it SIGINT or SIGTERM, or possibly by commanding it via its own API.

Source https://stackoverflow.com/questions/69765180

Community Discussions, Code Snippets contain sources that include Stack Exchange Network