argo-workflows | Workflow engine for Kubernetes | BPM library

 by   argoproj Go Version: v0.0.0-dev-no-git-3 License: Apache-2.0

kandi X-RAY | argo-workflows Summary

argo-workflows is a Go library typically used in Automation, BPM applications. argo-workflows has a Permissive License and it has medium support. However argo-workflows has 1276 bugs and it has 1 vulnerabilities. You can download it from GitHub.
Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition). Argo is a Cloud Native Computing Foundation (CNCF) hosted project.
    Support
      Quality
        Security
          License
            Reuse
            Support
              Quality
                Security
                  License
                    Reuse

                      kandi-support Support

                        summary
                        argo-workflows has a medium active ecosystem.
                        summary
                        It has 12704 star(s) with 2753 fork(s). There are 200 watchers for this library.
                        summary
                        There were 8 major release(s) in the last 6 months.
                        summary
                        There are 739 open issues and 4277 have been closed. On average issues are closed in 153 days. There are 43 open pull requests and 0 closed requests.
                        summary
                        It has a neutral sentiment in the developer community.
                        summary
                        The latest version of argo-workflows is v0.0.0-dev-no-git-3
                        argo-workflows Support
                          Best in #BPM
                            Average in #BPM
                            argo-workflows Support
                              Best in #BPM
                                Average in #BPM

                                  kandi-Quality Quality

                                    summary
                                    argo-workflows has 1276 bugs (1275 blocker, 0 critical, 1 major, 0 minor) and 2880 code smells.
                                    argo-workflows Quality
                                      Best in #BPM
                                        Average in #BPM
                                        argo-workflows Quality
                                          Best in #BPM
                                            Average in #BPM

                                              kandi-Security Security

                                                summary
                                                argo-workflows has 1 vulnerability issues reported (0 critical, 0 high, 1 medium, 0 low).
                                                summary
                                                argo-workflows code analysis shows 0 unresolved vulnerabilities.
                                                summary
                                                There are 0 security hotspots that need review.
                                                argo-workflows Security
                                                  Best in #BPM
                                                    Average in #BPM
                                                    argo-workflows Security
                                                      Best in #BPM
                                                        Average in #BPM

                                                          kandi-License License

                                                            summary
                                                            argo-workflows is licensed under the Apache-2.0 License. This license is Permissive.
                                                            summary
                                                            Permissive licenses have the least restrictions, and you can use them in most projects.
                                                            argo-workflows License
                                                              Best in #BPM
                                                                Average in #BPM
                                                                argo-workflows License
                                                                  Best in #BPM
                                                                    Average in #BPM

                                                                      kandi-Reuse Reuse

                                                                        summary
                                                                        argo-workflows releases are available to install and integrate.
                                                                        summary
                                                                        Installation instructions are not available. Examples and code snippets are available.
                                                                        summary
                                                                        It has 237874 lines of code, 10271 functions and 1414 files.
                                                                        summary
                                                                        It has high code complexity. Code complexity directly impacts maintainability of the code.
                                                                        argo-workflows Reuse
                                                                          Best in #BPM
                                                                            Average in #BPM
                                                                            argo-workflows Reuse
                                                                              Best in #BPM
                                                                                Average in #BPM
                                                                                  Top functions reviewed by kandi - BETA
                                                                                  kandi has reviewed argo-workflows and discovered the below as its top functions. This is intended to give you an instant insight into argo-workflows implemented functionality, and help decide if they suit your requirements.
                                                                                  • Checks the validity of the input value .
                                                                                    • Calls the API .
                                                                                      • Validate and convert input_value to required types .
                                                                                        • Instantiate a new instance of the given class .
                                                                                          • Validate all the properties of this schema
                                                                                            • Calls an API call .
                                                                                              • Return a OneOfSchema instance from the model .
                                                                                                • Initialize an OpenApiModel from keyword arguments .
                                                                                                  • Convert a model instance to a python dictionary .
                                                                                                    • Attempts to convert the given value to an appropriate object .
                                                                                                      Get all kandi verified functions for this library.
                                                                                                      Get all kandi verified functions for this library.

                                                                                                      argo-workflows Key Features

                                                                                                      UI to visualize and manage Workflows
                                                                                                      Artifact support (S3, Artifactory, Alibaba Cloud OSS, HTTP, Git, GCS, raw)
                                                                                                      Workflow templating to store commonly used Workflows in the cluster
                                                                                                      Archiving Workflows after executing for later access
                                                                                                      Scheduled workflows using cron
                                                                                                      Server interface with REST API (HTTP and GRPC)
                                                                                                      DAG or Steps based declaration of workflows
                                                                                                      Step level input & outputs (artifacts/parameters)
                                                                                                      Loops
                                                                                                      Parameterization
                                                                                                      Conditionals
                                                                                                      Timeouts (step & workflow level)
                                                                                                      Retry (step & workflow level)
                                                                                                      Resubmit (memoized)
                                                                                                      Suspend & Resume
                                                                                                      Cancellation
                                                                                                      K8s resource orchestration
                                                                                                      Exit Hooks (notifications, cleanup)
                                                                                                      Garbage collection of completed workflow
                                                                                                      Scheduling (affinity/tolerations/node selectors)
                                                                                                      Volumes (ephemeral/existing)
                                                                                                      Parallelism limits
                                                                                                      Daemoned steps
                                                                                                      DinD (docker-in-docker)
                                                                                                      Script steps
                                                                                                      Event emission
                                                                                                      Prometheus metrics
                                                                                                      Multiple executors
                                                                                                      Multiple pod and workflow garbage collection strategies
                                                                                                      Automatically calculated resource usage per step
                                                                                                      Java/Golang/Python SDKs
                                                                                                      Pod Disruption Budget support
                                                                                                      Single-sign on (OAuth2/OIDC)
                                                                                                      Webhook triggering
                                                                                                      CLI
                                                                                                      Out-of-the box and custom Prometheus metrics
                                                                                                      Windows container support
                                                                                                      Embedded widgets
                                                                                                      Multiplex log viewer

                                                                                                      argo-workflows Examples and Code Snippets

                                                                                                      No Code Snippets are available at this moment for argo-workflows.
                                                                                                      Community Discussions

                                                                                                      Trending Discussions on argo-workflows

                                                                                                      Argo workflows is trying to save the artifact to /var/run/argo/outputs/artifacts. Where is this specified?
                                                                                                      chevron right
                                                                                                      How do I use Argo Workflows Using Previous Step Outputs As Inputs?
                                                                                                      chevron right
                                                                                                      Difference between namespace install vs managed namespace install in Argo Workflows?
                                                                                                      chevron right
                                                                                                      Argo Workflow to continue processing during fan-out
                                                                                                      chevron right
                                                                                                      How does the Argo Workflows CLI get permissions?
                                                                                                      chevron right
                                                                                                      Dynamically provide an image name for the container template
                                                                                                      chevron right
                                                                                                      Retrying after a settable delay in Argo Workflows
                                                                                                      chevron right
                                                                                                      Dynamic "Fan-In" for artifact outputs in Argo?
                                                                                                      chevron right
                                                                                                      How can I delete the Argo events launched in the Argo namespace?
                                                                                                      chevron right
                                                                                                      Is there a way to gracefully end a pod with the Kubernetes client-go?
                                                                                                      chevron right

                                                                                                      QUESTION

                                                                                                      Argo workflows is trying to save the artifact to /var/run/argo/outputs/artifacts. Where is this specified?
                                                                                                      Asked 2022-Mar-18 at 11:45

                                                                                                      I found Argo lint today. Thank you to the Argo team!!! This is a very useful tool and has saved me tons of time. The following yaml checks out with no errors, but when I try to run it, I get the following error. How can I track down what is happening?

                                                                                                      FATA[2022-03-14T19:36:29.512Z] workflows.argoproj.io "hello-world-s5rm5" not found
                                                                                                      

                                                                                                      Here is the workflow

                                                                                                      ---
                                                                                                      {
                                                                                                         "apiVersion": "argoproj.io/v1alpha1",
                                                                                                         "kind": "Workflow",
                                                                                                         "metadata": {
                                                                                                            "annotations": {
                                                                                                               "workflows.argoproj.io/description": "testing a linter",
                                                                                                               "workflows.argoproj.io/version": ">= 3.1.0"
                                                                                                            },
                                                                                                            "labels": {
                                                                                                               "workflows.argoproj.io/archive-strategy": "false"
                                                                                                            },
                                                                                                            "generateName": "hello-world-",
                                                                                                            "namespace": "sandbox"
                                                                                                         },
                                                                                                         "spec": {
                                                                                                            "arguments": {
                                                                                                               "parameters": [
                                                                                                                  {
                                                                                                                     "name": "msg",
                                                                                                                     "value": "Hello there"
                                                                                                                  }
                                                                                                               ]
                                                                                                            },
                                                                                                            "entrypoint": "entrypoint",
                                                                                                            "securityContext": {
                                                                                                               "fsGroup": 2000,
                                                                                                               "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                               "runAsGroup": 3000,
                                                                                                               "runAsNonRoot": true,
                                                                                                               "runAsUser": 1000
                                                                                                            },
                                                                                                            "templates": [
                                                                                                               {
                                                                                                                  "container": {
                                                                                                                     "args": [
                                                                                                                        "cowsay Hello Test >> {{outputs.artifacts.message}}"
                                                                                                                     ],
                                                                                                                     "command": [
                                                                                                                        "sh",
                                                                                                                        "-c"
                                                                                                                     ],
                                                                                                                     "image": "docker/whalesay:latest",
                                                                                                                     "imagePullPolicy": "IfNotPresent",
                                                                                                                  },
                                                                                                                  "name": "whalesay",
                                                                                                                  "outputs": {
                                                                                                                     "artifacts": [
                                                                                                                        {
                                                                                                                           "name": "message",
                                                                                                                           "path": "/tmp/output.tgz",
                                                                                                                           "s3": {
                                                                                                                              "key": "whalesay",
                                                                                                                           }
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "retryStrategy": {
                                                                                                                     "limit": "10"
                                                                                                                  },
                                                                                                                  "securityContext": {
                                                                                                                     "fsGroup": 2000,
                                                                                                                     "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                                     "runAsGroup": 3000,
                                                                                                                     "runAsNonRoot": true,
                                                                                                                     "runAsUser": 1000
                                                                                                                  }
                                                                                                               },
                                                                                                               {
                                                                                                                  "inputs": {
                                                                                                                     "artifacts": [
                                                                                                                        {
                                                                                                                           "s3": {
                                                                                                                              "key": "whalesay",
                                                                                                                           },
                                                                                                                           "name": "data",
                                                                                                                           "path": "/tmp/input"
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "name": "print",
                                                                                                                  "retryStrategy": {
                                                                                                                     "limit": "10"
                                                                                                                  },
                                                                                                                  "script": {
                                                                                                                     "command": [
                                                                                                                        "python"
                                                                                                                     ],
                                                                                                                     "image": "python:alpine3.6",
                                                                                                                     "imagePullPolicy": "IfNotPresent",
                                                                                                                     "source": "import sys \nsys.stdout.write(\"{{inputs.artifacts.data}}\")\n\n"
                                                                                                                  },
                                                                                                                  "securityContext": {
                                                                                                                     "fsGroup": 2000,
                                                                                                                     "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                                     "runAsGroup": 3000,
                                                                                                                     "runAsNonRoot": true,
                                                                                                                     "runAsUser": 1000
                                                                                                                  }
                                                                                                               },
                                                                                                               {
                                                                                                                  "dag": {
                                                                                                                     "tasks": [
                                                                                                                        {
                                                                                                                           "name": "whalesay",
                                                                                                                           "template": "whalesay"
                                                                                                                        },
                                                                                                                        {
                                                                                                                           "arguments": {
                                                                                                                              "artifacts": [
                                                                                                                                 {
                                                                                                                                    "from": "{{whalesay.outputs.artifacts.message}}",
                                                                                                                                    "name": "data"
                                                                                                                                 }
                                                                                                                              ]
                                                                                                                           },
                                                                                                                           "dependencies": [
                                                                                                                              "whalesay"
                                                                                                                           ],
                                                                                                                           "name": "print",
                                                                                                                           "template": "print"
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "name": "entrypoint"
                                                                                                               }
                                                                                                            ]
                                                                                                         }
                                                                                                      }
                                                                                                      ...
                                                                                                      
                                                                                                      

                                                                                                      Here is the result of kubectl describe

                                                                                                      Name:         hello-world
                                                                                                      Namespace:    sandbox
                                                                                                      Labels:       workflows.argoproj.io/archive-strategy=false
                                                                                                      Annotations:  workflows.argoproj.io/description: testing a linter
                                                                                                                    workflows.argoproj.io/version: >= 3.1.0
                                                                                                      API Version:  argoproj.io/v1alpha1
                                                                                                      Kind:         Workflow
                                                                                                      Metadata:
                                                                                                        Creation Timestamp:  2022-03-14T19:33:19Z
                                                                                                        Generation:          1
                                                                                                        Managed Fields:
                                                                                                          API Version:  argoproj.io/v1alpha1
                                                                                                          Fields Type:  FieldsV1
                                                                                                          fieldsV1:
                                                                                                            f:metadata:
                                                                                                              f:annotations:
                                                                                                                .:
                                                                                                                f:workflows.argoproj.io/description:
                                                                                                                f:workflows.argoproj.io/version:
                                                                                                              f:labels:
                                                                                                                .:
                                                                                                                f:workflows.argoproj.io/archive-strategy:
                                                                                                            f:spec:
                                                                                                            f:status:
                                                                                                          Manager:         argo
                                                                                                          Operation:       Update
                                                                                                          Time:            2022-03-14T19:33:19Z
                                                                                                        Resource Version:  16499078
                                                                                                        UID:               b438cf44-241c-44bf-bb42-e470eaf4ca08
                                                                                                      Spec:
                                                                                                        Arguments:
                                                                                                          Parameters:
                                                                                                            Name:    msg
                                                                                                            Value:   Hello there
                                                                                                        Entrypoint:  entrypoint
                                                                                                        Security Context:
                                                                                                          Fs Group:                2000
                                                                                                          Fs Group Change Policy:  OnRootMismatch
                                                                                                          Run As Group:            3000
                                                                                                          Run As Non Root:         true
                                                                                                          Run As User:             1000
                                                                                                        Templates:
                                                                                                          Container:
                                                                                                            Args:
                                                                                                              cowsay Hello Test >> {{outputs.artifacts.message}}
                                                                                                            Command:
                                                                                                              sh
                                                                                                              -c
                                                                                                            Image:              docker/whalesay:latest
                                                                                                            Image Pull Policy:  IfNotPresent
                                                                                                            Name:               
                                                                                                            Resources:
                                                                                                          Inputs:
                                                                                                          Metadata:
                                                                                                          Name:  whalesay
                                                                                                          Outputs:
                                                                                                            Artifacts:
                                                                                                              Name:  message
                                                                                                              Path:  /tmp/output.tgz
                                                                                                              s3:
                                                                                                                Key:  whalesay
                                                                                                          Retry Strategy:
                                                                                                            Limit:  10
                                                                                                          Security Context:
                                                                                                            Fs Group:                2000
                                                                                                            Fs Group Change Policy:  OnRootMismatch
                                                                                                            Run As Group:            3000
                                                                                                            Run As Non Root:         true
                                                                                                            Run As User:             1000
                                                                                                          Inputs:
                                                                                                            Artifacts:
                                                                                                              Name:  data
                                                                                                              Path:  /tmp/input
                                                                                                              s3:
                                                                                                                Key:  whalesay
                                                                                                          Metadata:
                                                                                                          Name:  print
                                                                                                          Outputs:
                                                                                                          Retry Strategy:
                                                                                                            Limit:  10
                                                                                                          Script:
                                                                                                            Command:
                                                                                                              python
                                                                                                            Image:              python:alpine3.6
                                                                                                            Image Pull Policy:  IfNotPresent
                                                                                                            Name:               
                                                                                                            Resources:
                                                                                                            Source:  import sys 
                                                                                                      sys.stdout.write("{{inputs.artifacts.data}}")
                                                                                                      
                                                                                                      
                                                                                                          Security Context:
                                                                                                            Fs Group:                2000
                                                                                                            Fs Group Change Policy:  OnRootMismatch
                                                                                                            Run As Group:            3000
                                                                                                            Run As Non Root:         true
                                                                                                            Run As User:             1000
                                                                                                          Dag:
                                                                                                            Tasks:
                                                                                                              Arguments:
                                                                                                              Name:      whalesay
                                                                                                              Template:  whalesay
                                                                                                              Arguments:
                                                                                                                Artifacts:
                                                                                                                  From:  {{whalesay.outputs.artifacts.message}}
                                                                                                                  Name:  data
                                                                                                              Dependencies:
                                                                                                                whalesay
                                                                                                              Name:      print
                                                                                                              Template:  print
                                                                                                          Inputs:
                                                                                                          Metadata:
                                                                                                          Name:  entrypoint
                                                                                                          Outputs:
                                                                                                      Status:
                                                                                                        Finished At:  
                                                                                                        Started At:   
                                                                                                      Events:         
                                                                                                      
                                                                                                      

                                                                                                      UPDATE:

                                                                                                      I have re-installed (upgraded) Argo and made some progress. The error (below) suggests that I have set up my Artifact repository wrong. I am following instructions found here to the best of my understanding.

                                                                                                      The Google technical support folks are telling me that my GCS bucket is configured for read only. I am conversing with them on how to open the bucket for writing. Once that is done, am I correct that updating the configmap is sufficient?

                                                                                                      https://argoproj.github.io/argo-workflows/configure-artifact- and repository/#google-cloud-storage-gcs

                                                                                                      https://argoproj.github.io/argo-workflows/artifact-repository-ref/

                                                                                                      Another Update:

                                                                                                      Thanks to the Google help folks, I think I have cloud storage configured (I think), but I cannot yet confirm. I am getting the following error (full stack below.

                                                                                                      Question: Where is the prefix "/var/run/argo/outputs/artifacts" specified? I have not encountered this before.

                                                                                                      What is the proper way to reconcile this automatic insertion in the workflow?

                                                                                                       open /var/run/argo/outputs/artifacts/tmp/output.tgz.tgz: no such file or directory
                                                                                                      
                                                                                                                            hello-worldnztr8-4118214805 (v1:metadata.name)
                                                                                                            ARGO_CONTAINER_RUNTIME_EXECUTOR:    emissary
                                                                                                            GODEBUG:                            x509ignoreCN=0
                                                                                                            ARGO_WORKFLOW_NAME:                 hello-worldnztr8
                                                                                                            ARGO_WORKFLOW_UID:                  4ed3e706-48d9-4d22-bf73-fcccc4a4e6d0
                                                                                                            ARGO_CONTAINER_NAME:                init
                                                                                                            ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
                                                                                                            ARGO_NODE_ID:                       hello-worldnztr8-4118214805
                                                                                                            ARGO_INCLUDE_SCRIPT_OUTPUT:         false
                                                                                                            ARGO_DEADLINE:                      0001-01-01T00:00:00Z
                                                                                                            ARGO_PROGRESS_FILE:                 /var/run/argo/progress
                                                                                                            ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
                                                                                                            ARGO_PROGRESS_FILE_TICK_DURATION:   3s
                                                                                                          Mounts:
                                                                                                            /var/run/argo from var-run-argo (rw)
                                                                                                            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
                                                                                                      Containers:
                                                                                                        wait:
                                                                                                          Container ID:  containerd://6593d624b0350cc51a739e19d78f39e5726a9f1dfddc7e8995b082a073f57864
                                                                                                          Image:         quay.io/argoproj/argoexec:v3.3.0
                                                                                                          Image ID:      quay.io/argoproj/argoexec@sha256:b37739320a21d1d96789082c659b96f2dcb59c51483d9852bc858f6cfddf82fb
                                                                                                          Port:          
                                                                                                          Host Port:     
                                                                                                          Command:
                                                                                                            argoexec
                                                                                                            wait
                                                                                                            --loglevel
                                                                                                            info
                                                                                                          State:          Terminated
                                                                                                            Reason:       Error
                                                                                                            Message:      open /var/run/argo/outputs/artifacts/tmp/output.tgz.tgz: no such file or directory
                                                                                                            Exit Code:    1
                                                                                                            Started:      Wed, 16 Mar 2022 20:42:36 +0000
                                                                                                            Finished:     Wed, 16 Mar 2022 20:42:37 +0000
                                                                                                          Ready:          False
                                                                                                          Restart Count:  0
                                                                                                          Environment:
                                                                                                            ARGO_POD_NAME:                      hello-worldnztr8-4118214805 (v1:metadata.name)
                                                                                                            ARGO_CONTAINER_RUNTIME_EXECUTOR:    emissary
                                                                                                            GODEBUG:                            x509ignoreCN=0
                                                                                                            ARGO_WORKFLOW_NAME:                 hello-worldnztr8
                                                                                                            ARGO_WORKFLOW_UID:                  4ed3e706-48d9-4d22-bf73-fcccc4a4e6d0
                                                                                                            ARGO_CONTAINER_NAME:                wait
                                                                                                            ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
                                                                                                            ARGO_NODE_ID:                       hello-worldnztr8-4118214805
                                                                                                            ARGO_INCLUDE_SCRIPT_OUTPUT:         false
                                                                                                            ARGO_DEADLINE:                      0001-01-01T00:00:00Z
                                                                                                            ARGO_PROGRESS_FILE:                 /var/run/argo/progress
                                                                                                            ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
                                                                                                            ARGO_PROGRESS_FILE_TICK_DURATION:   3s
                                                                                                          Mounts:
                                                                                                            /var/run/argo from var-run-argo (rw)
                                                                                                            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
                                                                                                        main:
                                                                                                          Container ID:  containerd://c1c3c014b6c975b5702da564f76a4f5026352bc1f3b57f7dc4d1738104ee7ab8
                                                                                                          Image:         docker/whalesay:latest
                                                                                                          Image ID:      sha256:c717279bbba020bf95ac72cf47b2c8abb3a383ad4b6996c1a7a9f2a7aaa480ad
                                                                                                          Port:          
                                                                                                          Host Port:     
                                                                                                          Command:
                                                                                                            /var/run/argo/argoexec
                                                                                                            emissary
                                                                                                            --
                                                                                                            sh
                                                                                                            -c
                                                                                                          Args:
                                                                                                            cowsay Hello Test >> {{outputs.artifacts.message}}
                                                                                                          State:          Terminated
                                                                                                            Reason:       Error
                                                                                                            Exit Code:    2
                                                                                                            Started:      Wed, 16 Mar 2022 20:42:36 +0000
                                                                                                            Finished:     Wed, 16 Mar 2022 20:42:36 +0000
                                                                                                          Ready:          False
                                                                                                          Restart Count:  0
                                                                                                          Environment:
                                                                                                            ARGO_CONTAINER_NAME:                main
                                                                                                            ARGO_TEMPLATE:                      {"name":"whalesay","inputs":{},"outputs":{"artifacts":[{"name":"message","path":"/tmp/output.tgz","s3":{"key":"whalesay"}}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["cowsay Hello Test \u003e\u003e {{outputs.artifacts.message}}"],"resources":{},"imagePullPolicy":"IfNotPresent"},"archiveLocation":{"archiveLogs":false},"retryStrategy":{"limit":"10"},"securityContext":{"runAsUser":1000,"runAsGroup":3000,"runAsNonRoot":true,"fsGroup":2000,"fsGroupChangePolicy":"OnRootMismatch"}}
                                                                                                            ARGO_NODE_ID:                       hello-worldnztr8-4118214805
                                                                                                            ARGO_INCLUDE_SCRIPT_OUTPUT:         false
                                                                                                            ARGO_DEADLINE:                      0001-01-01T00:00:00Z
                                                                                                            ARGO_PROGRESS_FILE:                 /var/run/argo/progress
                                                                                                            ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
                                                                                                            ARGO_PROGRESS_FILE_TICK_DURATION:   3s
                                                                                                          Mounts:
                                                                                                            /var/run/argo from var-run-argo (rw)
                                                                                                            /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvh6k (ro)
                                                                                                      Conditions:
                                                                                                        Type              Status
                                                                                                        Initialized       True
                                                                                                        Ready             False
                                                                                                        ContainersReady   False
                                                                                                        PodScheduled      True
                                                                                                      Volumes:
                                                                                                        var-run-argo:
                                                                                                          Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
                                                                                                          Medium:
                                                                                                          SizeLimit:  
                                                                                                        kube-api-access-tvh6k:
                                                                                                          Type:                    Projected (a volume that contains injected data from multiple sources)
                                                                                                          TokenExpirationSeconds:  3607
                                                                                                          ConfigMapName:           kube-root-ca.crt
                                                                                                          ConfigMapOptional:       
                                                                                                          DownwardAPI:             true
                                                                                                      QoS Class:                   BestEffort
                                                                                                      Node-Selectors:              
                                                                                                      Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                                                                                                                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                                                                                                      Events:
                                                                                                        Type    Reason     Age   From               Message
                                                                                                        ----    ------     ----  ----               -------
                                                                                                        Normal  Scheduled  55s   default-scheduler  Successfully assigned default/hello-worldnztr8-4118214805 to gke-cluster-1-default-pool-d262cd84-va7g
                                                                                                        Normal  Pulled     55s   kubelet            Container image "quay.io/argoproj/argoexec:v3.3.0" already present on machine
                                                                                                        Normal  Created    54s   kubelet            Created container init
                                                                                                        Normal  Started    54s   kubelet            Started container init
                                                                                                        Normal  Pulled     53s   kubelet            Container image "quay.io/argoproj/argoexec:v3.3.0" already present on machine
                                                                                                        Normal  Created    53s   kubelet            Created container wait
                                                                                                        Normal  Started    53s   kubelet            Started container wait
                                                                                                        Normal  Pulled     53s   kubelet            Container image "docker/whalesay:latest" already present on machine
                                                                                                        Normal  Created    53s   kubelet            Created container main
                                                                                                        Normal  Started    53s   kubelet            Started container main
                                                                                                      

                                                                                                      ANSWER

                                                                                                      Answered 2022-Mar-18 at 11:45

                                                                                                      The complete fix is detailed here https://github.com/argoproj/argo-workflows/issues/8168#event-6261265751

                                                                                                      for purposes of this discussion, the output must be the explicit location (not a placeholder) e.g. /tmp/ouput

                                                                                                      I think the standard is that you do not put the .tgz suffix in the output location, but that is not yet confirmed as there was another fix involved. Perhaps someone from the Argo team can confirm this.

                                                                                                      Source https://stackoverflow.com/questions/71473378

                                                                                                      QUESTION

                                                                                                      How do I use Argo Workflows Using Previous Step Outputs As Inputs?
                                                                                                      Asked 2022-Mar-02 at 20:49

                                                                                                      I am trying to format my workflow per these instructions (https://argoproj.github.io/argo-workflows/workflow-inputs/#using-previous-step-outputs-as-inputs) but cannot seem to get it right. Specifically, I am trying to imitate "Using Previous Step Outputs As Inputs"

                                                                                                      I have included my workflow below. In this version, I have added a path to the inputs.artifacts because the error requests one. The error I am now receiving is:

                                                                                                      ATA[2022-02-28T14:14:45.933Z] Failed to submit workflow: templates.entrypoint.tasks.print1 templates.print1.inputs.artifacts.result.from not valid in inputs
                                                                                                      

                                                                                                      Can someone please tell me how to correct this workflow so that it works?

                                                                                                      ---
                                                                                                      {
                                                                                                         "apiVersion": "argoproj.io/v1alpha1",
                                                                                                         "kind": "Workflow",
                                                                                                         "metadata": {
                                                                                                            "annotations": {
                                                                                                               "workflows.argoproj.io/description": "Building from the ground up",
                                                                                                               "workflows.argoproj.io/version": ">= 3.1.0"
                                                                                                            },
                                                                                                            "labels": {
                                                                                                               "workflows.argoproj.io/archive-strategy": "false"
                                                                                                            },
                                                                                                            "name": "data-passing",
                                                                                                            "namespace": "sandbox"
                                                                                                         },
                                                                                                         "spec": {
                                                                                                            "artifactRepositoryRef": {
                                                                                                               "configMap": "my-config",
                                                                                                               "key": "data"
                                                                                                            },
                                                                                                            "entrypoint": "entrypoint",
                                                                                                            "nodeSelector": {
                                                                                                               "kubernetes.io/os": "linux"
                                                                                                            },
                                                                                                            "parallelism": 3,
                                                                                                            "securityContext": {
                                                                                                               "fsGroup": 2000,
                                                                                                               "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                               "runAsGroup": 3000,
                                                                                                               "runAsNonRoot": true,
                                                                                                               "runAsUser": 1000
                                                                                                            },
                                                                                                            "templates": [
                                                                                                               {
                                                                                                                  "container": {
                                                                                                                     "args": [
                                                                                                                        "Hello World"
                                                                                                                     ],
                                                                                                                     "command": [
                                                                                                                        "cowsay"
                                                                                                                     ],
                                                                                                                     "image": "docker/whalesay:latest",
                                                                                                                     "imagePullPolicy": "IfNotPresent"
                                                                                                                  },
                                                                                                                  "name": "whalesay",
                                                                                                                  "outputs": {
                                                                                                                     "artifacts": [
                                                                                                                        {
                                                                                                                           "name": "msg",
                                                                                                                           "path": "/tmp/raw"
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "securityContext": {
                                                                                                                     "fsGroup": 2000,
                                                                                                                     "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                                     "runAsGroup": 3000,
                                                                                                                     "runAsNonRoot": true,
                                                                                                                     "runAsUser": 1000
                                                                                                                  }
                                                                                                               },
                                                                                                               {
                                                                                                                  "inputs": {
                                                                                                                     "artifacts": [
                                                                                                                        {
                                                                                                                           "from": "{{tasks.whalesay.outputs.artifacts.msg}}",
                                                                                                                           "name": "result",
                                                                                                                           "path": "/tmp/raw"
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "name": "print1",
                                                                                                                  "script": {
                                                                                                                     "command": [
                                                                                                                        "python"
                                                                                                                     ],
                                                                                                                     "image": "python:alpine3.6",
                                                                                                                     "imagePullPolicy": "IfNotPresent",
                                                                                                                     "source": "cat {{inputs.artifacts.result}}\n"
                                                                                                                  },
                                                                                                                  "securityContext": {
                                                                                                                     "fsGroup": 2000,
                                                                                                                     "fsGroupChangePolicy": "OnRootMismatch",
                                                                                                                     "runAsGroup": 3000,
                                                                                                                     "runAsNonRoot": true,
                                                                                                                     "runAsUser": 1000
                                                                                                                  }
                                                                                                               },
                                                                                                               {
                                                                                                                  "dag": {
                                                                                                                     "tasks": [
                                                                                                                        {
                                                                                                                           "name": "whalesay",
                                                                                                                           "template": "whalesay"
                                                                                                                        },
                                                                                                                        {
                                                                                                                           "arguments": {
                                                                                                                              "artifacts": [
                                                                                                                                 {
                                                                                                                                    "from": "{{tasks.whalesay.outputs.artifacts.msg}}",
                                                                                                                                    "name": "result",
                                                                                                                                    "path": "/tmp/raw"
                                                                                                                                 }
                                                                                                                              ]
                                                                                                                           },
                                                                                                                           "dependencies": [
                                                                                                                              "whalesay"
                                                                                                                           ],
                                                                                                                           "name": "print1",
                                                                                                                           "template": "print1"
                                                                                                                        }
                                                                                                                     ]
                                                                                                                  },
                                                                                                                  "name": "entrypoint"
                                                                                                               }
                                                                                                            ]
                                                                                                         }
                                                                                                      }
                                                                                                      ...
                                                                                                      

                                                                                                      ANSWER

                                                                                                      Answered 2022-Mar-01 at 15:26

                                                                                                      A very similar workflow from the Argo developers/maintainers can be found here:

                                                                                                      https://github.com/argoproj/argo-workflows/blob/master/examples/README.md#artifacts

                                                                                                      Source https://stackoverflow.com/questions/71296484

                                                                                                      QUESTION

                                                                                                      Difference between namespace install vs managed namespace install in Argo Workflows?
                                                                                                      Asked 2022-Feb-09 at 15:13

                                                                                                      I am trying to install argo workflows and looking at the documentation I can see 3 different types of installation https://argoproj.github.io/argo-workflows/installation/.

                                                                                                      Can anybody give some clarity on the namespace install vs managed namespace install? If its a managed namespace, how can I tell the managed namespace? Should I edit the k8's manifest for deployment? What benefit it can provide compared to simple namespace install ?

                                                                                                      ANSWER

                                                                                                      Answered 2022-Feb-09 at 15:13

                                                                                                      A namespace install allows Workflows to run only in the namespace where Argo Workflows is installed.

                                                                                                      A managed namespace install allows Workflows to run only in one namespace besides the one where Argo Workflows is installed.

                                                                                                      Using a managed namespace install might make sense if you want some users/processes to be able to run Workflows without granting them any privileges in the namespace where Argo Workflows is installed.

                                                                                                      For example, if I only run CI/CD-related Workflows that are maintained by the same team that manages the Argo Workflows installation, it's probably reasonable to use a namespace install. But if all the Workflows are run by a separate data science team, it probably makes sense to give them a data-science-workflows namespace and run a "managed namespace install" of Argo Workflows from another namespace.

                                                                                                      To configure a managed namespace install, edit the workflow-controller and argo-server Deployments to pass the --managed-namespace argument.

                                                                                                      You can currently only configure one managed namespace, but in the future it may be possible to manage more than one.

                                                                                                      Source https://stackoverflow.com/questions/71046661

                                                                                                      QUESTION

                                                                                                      Argo Workflow to continue processing during fan-out
                                                                                                      Asked 2022-Feb-03 at 14:15

                                                                                                      General question here, wondering if anyone has any ideas or experience trying to achieve something I am right now. I'm not entirely sure if its even possible in the argo workflow system...

                                                                                                      I'm wondering if it is possible to continue a workflow regardless if a dynamic fanout has finished. By dynamic fanout I mean that B1/B2/B3 can go to B30 potentially.

                                                                                                      I want to see if C1 can start when B1 has finished. The B stage is creating a small file which then in C stage I need to run an api request that it has finished and upload said file. But in this scenario B2/B3 still are processing.

                                                                                                      And finally, D1 would have to wait for all of C1/2/3-C# to finish to complete

                                                                                                      Diagram what I'm trying to achieve

                                                                                                      #           *
                                                                                                      #           | 
                                                                                                      #          A1 (generates a dynamic list that can change depending on the inputs) 
                                                                                                      #           | 
                                                                                                      #        /  |  \ 
                                                                                                      #       B1  B2  B3 +++ B#
                                                                                                      #       |   |    |
                                                                                                      #       C1         +++ C#
                                                                                                      #       *   *   *
                                                                                                      #        \  |  /
                                                                                                      #         \ | /
                                                                                                      #           D1
                                                                                                      

                                                                                                      I was viewing https://github.com/argoproj/argo-workflows/blob/master/docs/enhanced-depends-logic.md but I cant wrap my head around if this is what I need to achieve this. Especially if the fan-out steps are dynamic.

                                                                                                      Seems to me that it would bind C stage to the entirety of B stage and require for for B to finish

                                                                                                      ANSWER

                                                                                                      Answered 2022-Feb-03 at 14:15

                                                                                                      Something like this should work:

                                                                                                      apiVersion: argoproj.io/v1alpha1
                                                                                                      kind: Workflow
                                                                                                      spec:
                                                                                                        templates:
                                                                                                          - name: main
                                                                                                            steps:
                                                                                                              - - name: A
                                                                                                                  template: A
                                                                                                              - - name: B_C
                                                                                                                  template: B_C
                                                                                                                  arguments:
                                                                                                                    parameters:
                                                                                                                      - name: item
                                                                                                                        value: "{{item}}"
                                                                                                                  withParam: "{{steps.A.outputs.parameters.items}}"
                                                                                                              - - name: D
                                                                                                                  template: D
                                                                                                          - name: A
                                                                                                            # container or script spec here
                                                                                                            outputs:
                                                                                                              parameters:
                                                                                                                - name: items
                                                                                                                  valueFrom:
                                                                                                                    path: /tmp/items.json
                                                                                                          - name: B_C
                                                                                                            inputs:
                                                                                                              parameters:
                                                                                                                - name: item
                                                                                                            steps:
                                                                                                              - - name: B
                                                                                                                  template: B
                                                                                                                  arguments:
                                                                                                                    parameters:
                                                                                                                      - name: item
                                                                                                                        value: "{{inputs.parameters.item}}"
                                                                                                              - - name: C
                                                                                                                  template: C
                                                                                                                  arguments:
                                                                                                                    artifacts:
                                                                                                                      - name: file
                                                                                                                        from: "{{steps.B.outputs.artifacts.file}}"
                                                                                                          - name: B
                                                                                                            inputs:
                                                                                                              parameters:
                                                                                                                - name: item
                                                                                                            # container or script spec here
                                                                                                            outputs:
                                                                                                              artifacts:
                                                                                                                - name: file
                                                                                                                  path: /tmp/file
                                                                                                          - name: C
                                                                                                            inputs:
                                                                                                              artifacts:
                                                                                                                - name: file
                                                                                                            # container or script spec here
                                                                                                          - name: D
                                                                                                            # container or script spec here
                                                                                                      

                                                                                                      Step B_C in the main template runs instances of the B_C template in parallel.

                                                                                                      Template B_C runs B and C in series. Once template B_C starts, it runs as quickly as possible, completely unaware of any concurrent executions of the B_C template. So C1 blocks only on B1, never on B2 or B3 or any other B#.

                                                                                                      Once all instances of B_C are finished, the main template finally invokes the D template.

                                                                                                      Source https://stackoverflow.com/questions/70960203

                                                                                                      QUESTION

                                                                                                      How does the Argo Workflows CLI get permissions?
                                                                                                      Asked 2022-Feb-01 at 19:57

                                                                                                      I am new to the argo universe and was trying to set up Argo Workflows https://github.com/argoproj/argo-workflows/blob/master/docs/quick-start.md#install-argo-workflows .

                                                                                                      I have installed the argo CLI from the page : https://github.com/argoproj/argo-workflows/releases/latest . I was trying it in my minikube setup and I have my kubectl already configured to the minikube cluster. I am able to hit argo commands without any issues after putting it in my local bin folder.

                                                                                                      How does it work? Where do the argo CLI is connecting to operate?

                                                                                                      ANSWER

                                                                                                      Answered 2022-Feb-01 at 18:04

                                                                                                      The argo CLI manages two API clients. The first client connects to the Argo Workflows API server. The second connects to the Kubernetes API. Depending on what you're doing, the CLI might connect just to one API or the other.

                                                                                                      To connect to the Kubernetes API, the CLI just uses your kube config.

                                                                                                      To connect to the Argo server, the CLI first checks for an ARGO_TOKEN environment variable. If it's not available, the CLI falls back to using the kube config.

                                                                                                      ARGO_TOKEN is only necessary when the Argo Server is configured to require client auth and then only if you're doing things which require access to the Argo API instead of just the Kubernetes API.

                                                                                                      Source https://stackoverflow.com/questions/70943374

                                                                                                      QUESTION

                                                                                                      Dynamically provide an image name for the container template
                                                                                                      Asked 2022-Jan-18 at 15:02

                                                                                                      Is there a way to provide an image name for the container template dynamically based on its input parameters?

                                                                                                      We have more than 30 different tasks each with its own image and that should be invoked identically in a workflow. The number may vary each run depending on the output of a previous task. So we don't want to or even can't just hardcode them inside workflow YAML.

                                                                                                      An easy solution would be to provide the image field for the container depending on the input parameter and have the same template for each of these tasks. But looks like it's impossible. This workflow doesn't work:

                                                                                                      apiVersion: argoproj.io/v1alpha1
                                                                                                      kind: Workflow
                                                                                                      metadata:
                                                                                                        generateName: hello-world-
                                                                                                      spec:
                                                                                                        entrypoint: whalesay
                                                                                                      
                                                                                                        templates:
                                                                                                        - name: whalesay
                                                                                                          inputs:
                                                                                                            parameters:
                                                                                                              - name: image
                                                                                                                default: whalesay:latest
                                                                                                          container:
                                                                                                            image: "docker/{{image}}"
                                                                                                            command: [cowsay]
                                                                                                            args: ["hello world"]
                                                                                                      

                                                                                                      Is there some workaround for this particular case?

                                                                                                      Also is there a document somewhere describing in which fields one can use workflow variables? Documentation page says only:

                                                                                                      Some fields in a workflow specification allow for variable references which are automatically substituted by Argo.

                                                                                                      ANSWER

                                                                                                      Answered 2022-Jan-14 at 11:29

                                                                                                      This is a possible workaround: to use when and conditional run of a task. We need to list of all possible tasks with their container images though:

                                                                                                      apiVersion: argoproj.io/v1alpha1
                                                                                                      kind: Workflow
                                                                                                      metadata:
                                                                                                        generateName: test-dynamic-image-
                                                                                                      spec:
                                                                                                        entrypoint: main
                                                                                                      
                                                                                                        templates:
                                                                                                        - name: main
                                                                                                          steps:
                                                                                                          - - name: fanout-step
                                                                                                              template: fanout
                                                                                                          - - name: loop-step
                                                                                                              template: options
                                                                                                              arguments:
                                                                                                                parameters:
                                                                                                                - name: code
                                                                                                                  value: "{{item}}"
                                                                                                              withParam: "{{steps.code-step.outputs.parameters.codes}}"
                                                                                                      
                                                                                                        - name: fanout
                                                                                                          script:
                                                                                                            image: python:alpine3.6
                                                                                                            command: [python]
                                                                                                            source: |
                                                                                                              import json
                                                                                                              with open("/tmp/codes.json", 'w') as wf:
                                                                                                                json.dump(["foo", "bar", "buz"], wf)
                                                                                                          outputs:
                                                                                                            parameters:
                                                                                                              - name: codes
                                                                                                                valueFrom:
                                                                                                                  path: /tmp/codes.json
                                                                                                      
                                                                                                        - name: foo-code
                                                                                                          script:
                                                                                                            image: python:alpine3.6
                                                                                                            command: [ python ]
                                                                                                            source: |
                                                                                                              print("foo-code")
                                                                                                      
                                                                                                        - name: bar-code
                                                                                                          script:
                                                                                                            image: python:alpine3.6
                                                                                                            command: [ python ]
                                                                                                            source: |
                                                                                                              print("bar-code")
                                                                                                      
                                                                                                        - name: buz-code
                                                                                                          script:
                                                                                                            image: python:alpine3.6
                                                                                                            command: [ python ]
                                                                                                            source: |
                                                                                                              print("buz-code")
                                                                                                      
                                                                                                        - name: missed-code
                                                                                                          script:
                                                                                                            image: python:alpine3.6
                                                                                                            command: [ python ]
                                                                                                            source: |
                                                                                                              print("THIS SHOULD NOT BE PRINTED")
                                                                                                      
                                                                                                        - name: options
                                                                                                          inputs:
                                                                                                            parameters:
                                                                                                              - name: code
                                                                                                          steps:
                                                                                                          - - name: foo-code-option
                                                                                                              template: foo-code
                                                                                                              when: "{{inputs.parameters.code}} == foo"
                                                                                                            - name: bar-code-option
                                                                                                              template: bar-code
                                                                                                              when: "{{inputs.parameters.code}} == bar"
                                                                                                            - name: buz-code-option
                                                                                                              template: buz-code
                                                                                                              when: "{{inputs.parameters.code}} == buz"
                                                                                                            - name: missed-code-option
                                                                                                              template: missed-code
                                                                                                              when: "{{inputs.parameters.code}} == missed"
                                                                                                      

                                                                                                      Source https://stackoverflow.com/questions/70706752

                                                                                                      QUESTION

                                                                                                      Retrying after a settable delay in Argo Workflows
                                                                                                      Asked 2022-Jan-12 at 23:47

                                                                                                      One of our Argo Workflow steps may hit a rate limit and I want to be able to tell argo how long it should wait until the next retry.

                                                                                                      Is there a way to do it?

                                                                                                      I've seen Retries on the documentation but it only talks about retry count and backoff strategies and it doesn't look like it could be parameterized.

                                                                                                      ANSWER

                                                                                                      Answered 2022-Jan-12 at 23:47

                                                                                                      As far as I know there's no built-in way to add a pause before the next retry.

                                                                                                      However, you could build your own with Argo's exit handler feature.

                                                                                                      apiVersion: argoproj.io/v1alpha1
                                                                                                      kind: Workflow
                                                                                                      metadata:
                                                                                                        generateName: exit-handler-with-pause-
                                                                                                      spec:
                                                                                                        arguments:
                                                                                                          parameters
                                                                                                          - name: pause-before-retry-seconds
                                                                                                            value: "60"
                                                                                                        entrypoint: intentional-fail
                                                                                                        onExit: exit-handler
                                                                                                        - name: intentional-fail
                                                                                                          container:
                                                                                                            image: alpine:latest
                                                                                                            command: [sh, -c]
                                                                                                            args: ["echo intentional failure; exit 1"]
                                                                                                        - name: exit-handler
                                                                                                          steps:
                                                                                                          - - name: pause
                                                                                                              template: pause
                                                                                                              when: "{{workflow.status}} != Succeeded"
                                                                                                        - name: pause
                                                                                                          container:
                                                                                                            image: alpine:latest
                                                                                                            env:
                                                                                                            - name: SECONDS
                                                                                                              value: "{{workflow.parameters.pause-before-retry-seconds}}"
                                                                                                            command: [sh, -c]
                                                                                                            args:
                                                                                                            - >-
                                                                                                              echo "Pausing before retry..."
                                                                                                              sleep "$SECONDS"
                                                                                                      

                                                                                                      If the retry pause needs to be calculated within the workflow, check out the exit handler with params example.

                                                                                                      Source https://stackoverflow.com/questions/70683231

                                                                                                      QUESTION

                                                                                                      Dynamic "Fan-In" for artifact outputs in Argo?
                                                                                                      Asked 2022-Jan-10 at 14:20

                                                                                                      I have an Argo workflow with dynamic fan-out tasks that do some map operation (in a Map-Reduce meaning context). I want to create a reducer that aggregates their results. It's possible to do that when the outputs of each mapper are small and can be put as an output parameter. See this SO question-answer for the description of how to do it.

                                                                                                      But how to aggregate output artifacts with Argo without writing custom logic of writing them to some storage in each mapper and read from it in reducer?

                                                                                                      ANSWER

                                                                                                      Answered 2022-Jan-10 at 14:20

                                                                                                      Artifacts are more difficult to aggregate than parameters.

                                                                                                      Parameters are always text and are generally small. This makes it easy for Argo Workflows to aggregate them into a single JSON object which can then be consumed by a "reduce" step.

                                                                                                      Artifacts, on the other hand, may be any type or size. So Argo Workflows is limited in how much it can help with aggregation.

                                                                                                      The main relevant feature it provides is declarative repository write/read operations. You can specify, for example, an S3 prefix to write each parameter to. Then, in the reduce step, you can load everything from that prefix and perform your aggregation logic.

                                                                                                      Argo Workflows provides a generic map/reduce example. But besides artifact writing/reading, you pretty much have to do the aggregation logic yourself.

                                                                                                      Source https://stackoverflow.com/questions/70643465

                                                                                                      QUESTION

                                                                                                      How can I delete the Argo events launched in the Argo namespace?
                                                                                                      Asked 2021-Dec-12 at 15:27

                                                                                                      I am trying to delete (and recreate) the Argo namespace, but it won't fully delete because I tried launching an eventsource and eventbus there. Now these will not delete.

                                                                                                      I have tried to delete them via yaml and individually - no success yet.

                                                                                                      The frustrating result is that I cannot re-launch argo

                                                                                                      customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
                                                                                                      customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
                                                                                                      customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
                                                                                                      customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
                                                                                                      customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io unchanged
                                                                                                      customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
                                                                                                      clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
                                                                                                      clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
                                                                                                      clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
                                                                                                      clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
                                                                                                      clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role unchanged
                                                                                                      clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
                                                                                                      clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": roles.rbac.authorization.k8s.io "argo-role" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": rolebindings.rbac.authorization.k8s.io "argo-binding" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": configmaps "workflow-controller-configmap" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "workflow-controller-metrics" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "workflow-controller" is forbidden: unable to create new content in namespace argo because it is being terminated
                                                                                                      

                                                                                                      Here is what is going on in the argo namespace itself

                                                                                                      {
                                                                                                          "apiVersion": "v1",
                                                                                                          "kind": "Namespace",
                                                                                                          "metadata": {
                                                                                                              "creationTimestamp": "2021-10-21T14:52:51Z",
                                                                                                              "deletionTimestamp": "2021-12-10T18:48:17Z",
                                                                                                              "labels": {
                                                                                                                  "kubernetes.io/metadata.name": "argo"
                                                                                                              },
                                                                                                              "name": "argo",
                                                                                                              "resourceVersion": "9222845",
                                                                                                              "uid": "2ce61352-74a5-40d1-b35e-fe1efa39c3af"
                                                                                                          },
                                                                                                          "spec": {
                                                                                                              "finalizers": [
                                                                                                                  "kubernetes"
                                                                                                              ]
                                                                                                          },
                                                                                                          "status": {
                                                                                                              "conditions": [
                                                                                                                  {
                                                                                                                      "lastTransitionTime": "2021-12-10T18:48:23Z",
                                                                                                                      "message": "All resources successfully discovered",
                                                                                                                      "reason": "ResourcesDiscovered",
                                                                                                                      "status": "False",
                                                                                                                      "type": "NamespaceDeletionDiscoveryFailure"
                                                                                                                  },
                                                                                                                  {
                                                                                                                      "lastTransitionTime": "2021-12-10T18:48:23Z",
                                                                                                                      "message": "All legacy kube types successfully parsed",
                                                                                                                      "reason": "ParsedGroupVersions",
                                                                                                                      "status": "False",
                                                                                                                      "type": "NamespaceDeletionGroupVersionParsingFailure"
                                                                                                                  },
                                                                                                                  {
                                                                                                                      "lastTransitionTime": "2021-12-10T18:48:23Z",
                                                                                                                      "message": "All content successfully deleted, may be waiting on finalization",
                                                                                                                      "reason": "ContentDeleted",
                                                                                                                      "status": "False",
                                                                                                                      "type": "NamespaceDeletionContentFailure"
                                                                                                                  },
                                                                                                                  {
                                                                                                                      "lastTransitionTime": "2021-12-10T18:48:23Z",
                                                                                                                      "message": "Some resources are remaining: eventbus.argoproj.io has 1 resource instances, eventsources.argoproj.io has 1 resource instances",
                                                                                                                      "reason": "SomeResourcesRemain",
                                                                                                                      "status": "True",
                                                                                                                      "type": "NamespaceContentRemaining"
                                                                                                                  },
                                                                                                                  {
                                                                                                                      "lastTransitionTime": "2021-12-10T18:48:23Z",
                                                                                                                      "message": "Some content in the namespace has finalizers remaining: eventbus-controller in 1 resource instances, eventsource-controller in 1 resource instances",
                                                                                                                      "reason": "SomeFinalizersRemain",
                                                                                                                      "status": "True",
                                                                                                                      "type": "NamespaceFinalizersRemaining"
                                                                                                                  }
                                                                                                              ],
                                                                                                              "phase": "Terminating"
                                                                                                          }
                                                                                                      }
                                                                                                      

                                                                                                      UPDATE: Even though one test of Kubernetes shows that the eventsource persists, another test shows that it does not

                                                                                                      This is the response to kubectl get eventsources argo

                                                                                                      Error from server (NotFound): eventsources.argoproj.io "argo" not found
                                                                                                      

                                                                                                      ANSWER

                                                                                                      Answered 2021-Dec-12 at 15:27

                                                                                                      For anyone who stumbles onto this question, it is a permissions issue. Make certain your service account has permissions to work in both namespaces (argo and argo-events).

                                                                                                      Source https://stackoverflow.com/questions/70309339

                                                                                                      QUESTION

                                                                                                      Is there a way to gracefully end a pod with the Kubernetes client-go?
                                                                                                      Asked 2021-Oct-29 at 12:01

                                                                                                      The main question is if there is a way to finish a pod from the client-go sdk, I'm not trying to delete a pod, I just want to finish it with a Phase-Status: Completed.

                                                                                                      In the code, I'm trying to update the pod phase but It doesn't work, It does not return an error or panic but The pod does not finish. My code:

                                                                                                      func main() {
                                                                                                      
                                                                                                          // creates the in-cluster config
                                                                                                          config, err := rest.InClusterConfig()
                                                                                                          if err != nil {
                                                                                                              panic(err.Error())
                                                                                                          }
                                                                                                          // creates the clientset
                                                                                                          clientset, err := kubernetes.NewForConfig(config)
                                                                                                          if err != nil {
                                                                                                              panic(err.Error())
                                                                                                          }
                                                                                                          for {
                                                                                                      
                                                                                                              pods, err := clientset.CoreV1().Pods("ns").List(context.TODO(), metav1.ListOptions{})
                                                                                                              if err != nil {
                                                                                                                  panic(err.Error())
                                                                                                              }
                                                                                                              for _, pod := range pods.Items {
                                                                                                      
                                                                                                                  podName:= pod.Name
                                                                                                                  if strings.Contains(strings.ToLower(podName), "single-condition") {
                                                                                                                  
                                                                                                                      fmt.Println("get pods metadatada")
                                                                                                                      fmt.Println(pod.Name)
                                                                                                                      fmt.Printf("pod.Name %s \n", pod.Name)
                                                                                                                      fmt.Printf("Status.Phase %s \n", pod.Status.Phase)
                                                                                                                      fmt.Printf("PodIP %s \n", pod.Status.PodIP)
                                                                                                      
                                                                                                                      containers := pod.Status.ContainerStatuses
                                                                                                                      if len(containers) > 0 {
                                                                                                                          for _ ,c := range containers {
                                                                                                                              fmt.Printf("c.Name %s \n", c.Name)
                                                                                                                              fmt.Printf("c.State %s \n", c.State)
                                                                                                                              fmt.Printf("c.State.Terminated %s \n", c.State.Terminated)
                                                                                                      
                                                                                                                              stateTerminated := c.State.Terminated
                                                                                                                              stateRunning := c.State.Running
                                                                                                                              if stateTerminated == nil && stateRunning != nil {
                                                                                                                                  fmt.Printf("c.State.Terminated %s \n", c.State.Terminated)
                                                                                                                                  fmt.Printf("stateRunning Reason: %s\n", reflect.TypeOf(c.State.Running))
                                                                                                      
                                                                                                                                  getPod, getErr := clientset.CoreV1().Pods("ns").Get(context.TODO(), "single-condition-pipeline-9rqrs-1224102659" , metav1.GetOptions{})
                                                                                                                                  if getErr != nil {
                                                                                                                                      fmt.Println("error1")
                                                                                                                                      panic(fmt.Errorf("Failed to get: %v", getErr))
                                                                                                                                  }
                                                                                                                                  fmt.Println("update values")
                                                                                                                                  fmt.Printf(" getPodName %d \n", getPod.Name)
                                                                                                                                  getPod.Status.Phase = "Succeeded"
                                                                                                                                  fmt.Println("updated status phase")
                                                                                                                                  getContainers := getPod.Status.ContainerStatuses
                                                                                                                                  fmt.Printf("len get container %d \n", len(getContainers))
                                                                                                                                  _, updateErr := clientset.CoreV1().Pods("argo-workflows").Update(context.TODO(), getPod, metav1.UpdateOptions{})
                                                                                                                                  fmt.Println("commit update")
                                                                                                                                  if updateErr != nil {
                                                                                                                                      fmt.Println("error updated")
                                                                                                                                      panic(fmt.Errorf("Failed to update: %v", updateErr))
                                                                                                                                  }
                                                                                                      
                                                                                                                              } else {
                                                                                                                                  fmt.Printf("c.State.Terminated %s \n", c.State.Terminated.Reason)
                                                                                                                                  //fmt.Println("Not finished ready!!!")
                                                                                                                                  //fmt.Printf("c.State.Running %s \n", c.State.Running)
                                                                                                                                  //fmt.Printf("c.State.Waiting %s \n", c.State.Waiting)
                                                                                                                              }
                                                                                                                          }
                                                                                                                      }
                                                                                                                  }
                                                                                                              }
                                                                                                              time.Sleep(10 * time.Second)
                                                                                                      
                                                                                                          }
                                                                                                      }
                                                                                                      

                                                                                                      and some logs:

                                                                                                      single-condition-pipeline-9rqrs-1224102659
                                                                                                      pod.Name single-condition-pipeline-9rqrs-1224102659 
                                                                                                      Status.Phase Running 
                                                                                                      PodIP XXXXXXXXXXXX
                                                                                                      c.Name main 
                                                                                                      ---------------------------------------------------------------------------------------------
                                                                                                      c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
                                                                                                      c.State.Terminated nil 
                                                                                                      c.State.Terminated nil 
                                                                                                      stateRunning Reason: *v1.ContainerStateRunning
                                                                                                      update values
                                                                                                       getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
                                                                                                      updated status phase
                                                                                                      len get container 2 
                                                                                                      commit update
                                                                                                      c.Name wait 
                                                                                                      c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
                                                                                                      c.State.Terminated nil 
                                                                                                      c.State.Terminated nil 
                                                                                                      stateRunning Reason: *v1.ContainerStateRunning
                                                                                                      update values
                                                                                                       getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
                                                                                                      updated status phase
                                                                                                      len get container 2 
                                                                                                      ---------------------------------------------------------------------------------------------
                                                                                                      commit update
                                                                                                      ---------------------------------------------------------------------------------------------
                                                                                                      get pods metadatada
                                                                                                      single-condition-pipeline-9rqrs-1224102659
                                                                                                      pod.Name single-condition-pipeline-9rqrs-1224102659 
                                                                                                      Status.Phase Running 
                                                                                                      PodIP XXXXXXXXXX 
                                                                                                      c.Name main 
                                                                                                      c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
                                                                                                      c.State.Terminated nil 
                                                                                                      c.State.Terminated nil 
                                                                                                      stateRunning Reason: *v1.ContainerStateRunning
                                                                                                      update values
                                                                                                       getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
                                                                                                      updated status phase
                                                                                                      len get container 2 
                                                                                                      commit update
                                                                                                      c.Name wait 
                                                                                                      c.State {nil &ContainerStateRunning{StartedAt:2021-10-29 04:41:51 +0000 UTC,} nil} 
                                                                                                      c.State.Terminated nil 
                                                                                                      c.State.Terminated nil 
                                                                                                      stateRunning Reason: *v1.ContainerStateRunning
                                                                                                      update values
                                                                                                       getPodName %!d(string=single-condition-pipeline-9rqrs-1224102659) 
                                                                                                      updated status phase
                                                                                                      len get container 2 
                                                                                                      commit update
                                                                                                      

                                                                                                      so here: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-status, It mentions a Patch but I don't know how to use it, so if somebody could help me or if there is another way to finish it.

                                                                                                      ANSWER

                                                                                                      Answered 2021-Oct-29 at 12:01

                                                                                                      You cannot set the phase or anything else in the Pod status field, it is read only. According to the Pod Lifecycle documentation your pod will have a phase of Succeeded after "All containers in the Pod have terminated in success, and will not be restarted." So this will only happen if you can cause all of your pod's containers to exit with status code 0 and if the pod restartPolicy is set to onFailure or Never, if it is set to Always (the default) then the containers will eventually restart and your pod will eventually return to the Running phase.

                                                                                                      In summary, you cannot do what you are attempting to do via the Kube API directly. You must:

                                                                                                      1. Ensure your pod has a restartPolicy that can support the Succeeded phase.
                                                                                                      2. Cause your application to terminate, possibly by sending it SIGINT or SIGTERM, or possibly by commanding it via its own API.

                                                                                                      Source https://stackoverflow.com/questions/69765180

                                                                                                      Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                                                                                                      Vulnerabilities

                                                                                                      No vulnerabilities reported

                                                                                                      Install argo-workflows

                                                                                                      You can download it from GitHub.

                                                                                                      Support

                                                                                                      Get started hereHow to write Argo Workflow specsHow to configure your artifact repository
                                                                                                      Find more information at:
                                                                                                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                                      Find more libraries
                                                                                                      Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                                      Save this library and start creating your kit
                                                                                                      CLONE
                                                                                                    • HTTPS

                                                                                                      https://github.com/argoproj/argo-workflows.git

                                                                                                    • CLI

                                                                                                      gh repo clone argoproj/argo-workflows

                                                                                                    • sshUrl

                                                                                                      git@github.com:argoproj/argo-workflows.git

                                                                                                    • Share this Page

                                                                                                      share link

                                                                                                      Explore Related Topics

                                                                                                      Consider Popular BPM Libraries

                                                                                                      Try Top Libraries by argoproj

                                                                                                      argo-cd

                                                                                                      by argoprojGo

                                                                                                      argo-rollouts

                                                                                                      by argoprojGo

                                                                                                      argo-events

                                                                                                      by argoprojGo

                                                                                                      gitops-engine

                                                                                                      by argoprojGo

                                                                                                      applicationset

                                                                                                      by argoprojGo

                                                                                                      Compare BPM Libraries with Highest Support

                                                                                                      airflow

                                                                                                      by apache

                                                                                                      dagster

                                                                                                      by dagster-io

                                                                                                      azkaban

                                                                                                      by azkaban

                                                                                                      conductor

                                                                                                      by Netflix

                                                                                                      prefect

                                                                                                      by PrefectHQ

                                                                                                      Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                                      Find more libraries
                                                                                                      Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                                      Save this library and start creating your kit