prefect | easiest way to orchestrate and observe your data pipelines | BPM library

 by   PrefectHQ Python Version: 2.82 License: Apache-2.0

kandi X-RAY | prefect Summary

prefect is a Python library typically used in Automation, BPM applications. prefect has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can install using 'pip install prefect' or download it from GitHub, PyPI.
From the Latin praefectus, meaning "one who is in charge", a prefect is an official who oversees a domain and makes sure that the rules are followed. Similarly, Prefect is responsible for making sure that workflows execute properly. It also happens to be the name of a roving researcher for that wholly remarkable book, The Hitchhiker's Guide to the Galaxy.
    Support
      Quality
        Security
          License
            Reuse
            Support
              Quality
                Security
                  License
                    Reuse

                      kandi-support Support

                        summary
                        prefect has a highly active ecosystem.
                        summary
                        It has 11514 star(s) with 1158 fork(s). There are 153 watchers for this library.
                        summary
                        There were 10 major release(s) in the last 6 months.
                        summary
                        There are 676 open issues and 2735 have been closed. On average issues are closed in 203 days. There are 55 open pull requests and 0 closed requests.
                        summary
                        It has a neutral sentiment in the developer community.
                        summary
                        The latest version of prefect is 2.82
                        prefect Support
                          Best in #BPM
                            Average in #BPM
                            prefect Support
                              Best in #BPM
                                Average in #BPM

                                  kandi-Quality Quality

                                    summary
                                    prefect has 0 bugs and 0 code smells.
                                    prefect Quality
                                      Best in #BPM
                                        Average in #BPM
                                        prefect Quality
                                          Best in #BPM
                                            Average in #BPM

                                              kandi-Security Security

                                                summary
                                                prefect has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
                                                summary
                                                prefect code analysis shows 0 unresolved vulnerabilities.
                                                summary
                                                There are 0 security hotspots that need review.
                                                prefect Security
                                                  Best in #BPM
                                                    Average in #BPM
                                                    prefect Security
                                                      Best in #BPM
                                                        Average in #BPM

                                                          kandi-License License

                                                            summary
                                                            prefect is licensed under the Apache-2.0 License. This license is Permissive.
                                                            summary
                                                            Permissive licenses have the least restrictions, and you can use them in most projects.
                                                            prefect License
                                                              Best in #BPM
                                                                Average in #BPM
                                                                prefect License
                                                                  Best in #BPM
                                                                    Average in #BPM

                                                                      kandi-Reuse Reuse

                                                                        summary
                                                                        prefect releases are available to install and integrate.
                                                                        summary
                                                                        Deployable package is available in PyPI.
                                                                        summary
                                                                        Build file is available. You can build the component from source.
                                                                        summary
                                                                        Installation instructions are not available. Examples and code snippets are available.
                                                                        summary
                                                                        It has 76349 lines of code, 6248 functions and 579 files.
                                                                        summary
                                                                        It has medium code complexity. Code complexity directly impacts maintainability of the code.
                                                                        prefect Reuse
                                                                          Best in #BPM
                                                                            Average in #BPM
                                                                            prefect Reuse
                                                                              Best in #BPM
                                                                                Average in #BPM
                                                                                  Top functions reviewed by kandi - BETA
                                                                                  kandi has reviewed prefect and discovered the below as its top functions. This is intended to give you an instant insight into prefect implemented functionality, and help decide if they suit your requirements.
                                                                                  • Upgrade flow table
                                                                                    • Replace block_type_name_name with a slug
                                                                                    • Print an error message
                                                                                    • Print a message and exit
                                                                                  • Build a new deployment
                                                                                    • Return aorion client
                                                                                    • Create a new Flask application
                                                                                    • Create a work queue with the given name
                                                                                  • Returns a dict of cmdclass based on cmdclass
                                                                                    • Extract the version information
                                                                                  • Create a deployment from a flow
                                                                                  • Drop all indexes
                                                                                  • Set the state of a flow run
                                                                                  • Update the block document
                                                                                  • Creates aorption client
                                                                                  • Set the deployment schedule
                                                                                  • Create a flow run in the database
                                                                                  • Set the state of a task run
                                                                                  • Load a deployment
                                                                                  • Extract version information from VCS
                                                                                  • Create a container
                                                                                  • Read block schemas
                                                                                  • Login to Google
                                                                                  • Start a work queue
                                                                                  • Run a deployed flow
                                                                                  • Start a task run
                                                                                  • Adds a type field to the model
                                                                                  • Create the versioneer config file
                                                                                  Get all kandi verified functions for this library.
                                                                                  Get all kandi verified functions for this library.

                                                                                  prefect Key Features

                                                                                  The easiest way to orchestrate and observe your data pipelines

                                                                                  prefect Examples and Code Snippets

                                                                                  prefect-dask,Running tasks on Dask,Dask annotations
                                                                                  Pythondot imgLines of Code : 43dot imgLicense : Permissive (Apache-2.0)
                                                                                  copy iconCopy
                                                                                  
                                                                                                                      import dask from prefect import flow, task from prefect_dask.task_runners import DaskTaskRunner @task def show(x): print(x) @flow(task_runner=DaskTaskRunner()) def my_flow(): with dask.annotate(priority=-10): future = show(1) # low priority task with dask.annotate(priority=10): future = show(2) # high priority task
                                                                                  import dask from prefect import flow, task from prefect_dask.task_runners import DaskTaskRunner @task def show(x): print(x) # Create a `LocalCluster` with some resource annotations # Annotations are abstract in dask and not inferred from your system. # Here, we claim that our system has 1 GPU and 1 process available per worker @flow( task_runner=DaskTaskRunner( cluster_kwargs={"n_workers": 1, "resources": {"GPU": 1, "process": 1}} ) ) def my_flow(): with dask.annotate(resources={'GPU': 1}): future = show(0) # this task requires 1 GPU resource on a worker with dask.annotate(resources={'process': 1}): # These tasks each require 1 process on a worker; because we've # specified that our cluster has 1 process per worker and 1 worker, # these tasks will run sequentially future = show(1) future = show(2) future = show(3)
                                                                                  prefect-dask,Running tasks on Dask
                                                                                  Pythondot imgLines of Code : 39dot imgLicense : Permissive (Apache-2.0)
                                                                                  copy iconCopy
                                                                                  
                                                                                                                      import time from prefect import flow, task from prefect_dask import DaskTaskRunner @task def shout(number): time.sleep(0.5) print(f"#{number}") @flow(task_runner=DaskTaskRunner) def count_to(highest_number): for number in range(highest_number): shout.submit(number) if __name__ == "__main__": count_to(10) # outputs #3 #7 #2 #6 #4 #0 #1 #5 #8 #9
                                                                                  from prefect import flow from prefect_dask.task_runners import DaskTaskRunner @flow(task_runner=DaskTaskRunner(address="http://my-dask-cluster")) def my_flow(): ...
                                                                                  # Use 4 worker processes, each with 2 threads DaskTaskRunner( cluster_kwargs={"n_workers": 4, "threads_per_worker": 2} )
                                                                                  prefect-saturn,Getting Started
                                                                                  Pythondot imgLines of Code : 23dot imgLicense : Permissive (BSD-3-Clause)
                                                                                  copy iconCopy
                                                                                  
                                                                                                                      import prefect from prefect import Flow, task from prefect_saturn import PrefectCloudIntegration @task def hello_task(): logger = prefect.context.get("logger") logger.info("hello prefect-saturn") flow = Flow("sample-flow", tasks=[hello_task]) project_name = "sample-project" integration = PrefectCloudIntegration( prefect_cloud_project_name=project_name ) flow = integration.register_flow_with_saturn(flow) flow.register( project_name=project_name, labels=["saturn-cloud"] )
                                                                                  How to organize prefect flow with the using of cmd arguments?
                                                                                  Pythondot imgLines of Code : 18dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  prefect run -p /path/to/your/flow_file.py
                                                                                  
                                                                                  from prefect import task, Flow, Parameter
                                                                                  
                                                                                  @task(log_stdout=True)
                                                                                  def hello_world(name):
                                                                                      print(f"Hello {name}!")
                                                                                  
                                                                                  with Flow("mini-example") as flow:
                                                                                      name = Parameter("name", default="world")
                                                                                      hw = hello_world(name)
                                                                                  
                                                                                  prefect run -p /path/to/your/flow_file.py --param name=Marvin
                                                                                  
                                                                                  prefect register --project yourprojectname -p /path/to/your/flow_file.py
                                                                                  
                                                                                  prefect run --name mini-example --project yourprojectname
                                                                                  
                                                                                  How to deploy a Kubernetes Job at AKS with Prefect
                                                                                  Pythondot imgLines of Code : 25dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml
                                                                                  
                                                                                  kubectl apply -f agent.yaml # optionally set: -n yournamespace 
                                                                                  
                                                                                  import prefect
                                                                                  from prefect import task, Flow
                                                                                  from prefect.run_configs import KubernetesRun
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def hello_task():
                                                                                      logger = prefect.context.get("logger")
                                                                                      logger.info("Hello world!")
                                                                                  
                                                                                  
                                                                                  with Flow("hello-flow", run_config=KubernetesRun(labels=["YOUR_LABEL"])) as flow:
                                                                                      hello_task()
                                                                                  
                                                                                  if __name__ == "__main__":
                                                                                      flow.register("YOUR_PROJECT_NAME")
                                                                                  
                                                                                  prefect register --project YOUR_PROJECT_NAME -p path/to/flow.py
                                                                                  
                                                                                  prefect run --name "hello-flow" --project YOUR_PROJECT_NAME --watch
                                                                                  
                                                                                  Prefect Task Scheduling
                                                                                  Pythondot imgLines of Code : 39dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  from prefect import task, Flow, Parameter
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def get_data(board):
                                                                                      pass
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def check_db(url):
                                                                                      pass
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def upload_raw(url, board):
                                                                                      pass
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def remove_duplicates(board):
                                                                                      pass
                                                                                  
                                                                                  
                                                                                  with Flow(name="4chan_extract") as flow:
                                                                                      board_param = Parameter(name="board_name", required=True, default="pol")
                                                                                      data = get_data(board=board_param)
                                                                                      check = check_db(
                                                                                          url="postgresql://postgres:user@localhost:5434/postgres", upstream_tasks=[data]
                                                                                      )
                                                                                      upload = upload_raw(
                                                                                          url="postgresql://postgres:user@localhost:5434/postgres",
                                                                                          board=board_param,
                                                                                          upstream_tasks=[check],
                                                                                      )
                                                                                      remove_duplicates(board=board_param, upstream_tasks=[upload])
                                                                                  
                                                                                  if __name__ == "__main__":
                                                                                      flow.visualize()
                                                                                  
                                                                                  Overwrite methods but keep decorator (python)
                                                                                  Pythondot imgLines of Code : 13dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  class Base:
                                                                                      @task
                                                                                      def test1(self):
                                                                                          return self.test1_impl()
                                                                                  
                                                                                      def test1_impl(self):
                                                                                          ...
                                                                                  
                                                                                  
                                                                                  class Child:
                                                                                      def test1_impl(self):
                                                                                          ...
                                                                                  
                                                                                  Cant Access Prefect Server Remotely
                                                                                  Pythondot imgLines of Code : 9dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  prefect server start --server-port=5003 --no-postgres-port --use-volume --expose
                                                                                  
                                                                                  [server]
                                                                                  endpoint = "http://YOUR_MACHINES_PUBLIC_IP:5003/graphql"
                                                                                  
                                                                                    [server.ui]
                                                                                      apollo_url = "http://YOUR_MACHINES_PUBLIC_IP:5003/graphql"
                                                                                  
                                                                                  
                                                                                  How to mock functions with same name, when called twice?
                                                                                  Pythondot imgLines of Code : 77dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  ├── code_file.py
                                                                                  ├── code_test.py
                                                                                  └── prefect.py
                                                                                  
                                                                                  # prefect.py
                                                                                  class Client:
                                                                                      def __init__(self, *arg, **kwargs):
                                                                                          ...
                                                                                  
                                                                                      def graphql(self, *args, **kwargs):
                                                                                          print(f"graphql got called with args: {args}, kwargs: {kwargs}")
                                                                                  
                                                                                      def create_flow(self, *args, **kwargs):
                                                                                          print(f"create_flow got called with args: {args}, kwargs: {kwargs}")
                                                                                  
                                                                                  
                                                                                  # code_file.py
                                                                                  from prefect import Client
                                                                                  
                                                                                  graphql_value1 = "graphql_value1"
                                                                                  graphql_value2 = "graphql_value2"
                                                                                  
                                                                                  client = Client(api_server="http://example.com")
                                                                                  
                                                                                  
                                                                                  def run_flow(name1, name2):
                                                                                      # some graphql queries, not shown here
                                                                                      value1 = client.graphql(graphql_value1, variables={"name": name1})
                                                                                      print("First client call return value:- ", value1)
                                                                                      value2 = client.graphql(graphql_value2, variables={"name": name2, "id": value1["data"][0]["id"]})
                                                                                      print("Second client call return value:- ", value2)
                                                                                  
                                                                                      run_id = client.create_flow(id=value2["data"][0]["id"])
                                                                                      return run_id
                                                                                  
                                                                                  # code_test.py
                                                                                  import unittest
                                                                                  from unittest.mock import patch, MagicMock, call
                                                                                  from code_file import run_flow
                                                                                  
                                                                                  
                                                                                  class TestFlow(unittest.TestCase):
                                                                                      @patch("code_file.client.graphql")
                                                                                      @patch("code_file.client.create_flow")
                                                                                      def test_flow(self, create_flow: MagicMock, graphql: MagicMock) -> None:
                                                                                          first_graphql_return_value = {"data": [{"id": 1}]}
                                                                                          second_graphl_return_value = {"data": [{"id": 2}]}
                                                                                          graphql.side_effect = [
                                                                                              first_graphql_return_value,
                                                                                              second_graphl_return_value,
                                                                                          ]
                                                                                          create_flow.return_value = "test_id"
                                                                                  
                                                                                          self.assertEqual(run_flow("name1", "name2"), "test_id")
                                                                                  
                                                                                          create_flow.assert_called_once_with(id=2)
                                                                                          graphql.assert_has_calls(
                                                                                              [
                                                                                                  call("graphql_value1", variables={"name": "name1"}),
                                                                                                  call("graphql_value2", variables={"name": "name2", "id": 1})
                                                                                              ]
                                                                                          )
                                                                                  
                                                                                  
                                                                                  python -m unittest code_test.py
                                                                                  
                                                                                  First client call return value:-  {'data': [{'id': 1}]}
                                                                                  Second client call return value:-  {'data': [{'id': 2}]}
                                                                                  .
                                                                                  ----------------------------------------------------------------------
                                                                                  Ran 1 test in 0.001s
                                                                                  
                                                                                  def test_flow(mocker):
                                                                                      # Some lines of code, not shown here
                                                                                      m = mocker.patch("code_file.client.graphql", side_effect=["value1", "value2"])
                                                                                      ...
                                                                                  
                                                                                  Appending a record using pandas
                                                                                  Pythondot imgLines of Code : 26dot imgLicense : Strong Copyleft (CC BY-SA 4.0)
                                                                                  copy iconCopy
                                                                                  columns = ['DATE', 'TIME', 'ENERGY_OUT', 'AAA', 'POWER_OUT', 'BBB', 'CCC', 'ENERGY_IN',
                                                                                             'POWER_IN', 'TEMPERATURE', 'VOLTAGE', 'v7', 'v8', 'v9', 'v10', 'v11', 'v12']
                                                                                  
                                                                                  array = [
                                                                                      ['20211002', '15:40', '27294', '3.446', '1080', '1080', '0.136', '15380', '600',
                                                                                          'NaN', '244.4', '0.000', '0.000', '0.000', '1080.000', '0.000', '100.000'],
                                                                                      ['20211002', '15:30', '27204', '3.435', '1830', '1836', '0.232', '15330', '660',
                                                                                          'NaN', '246.0', '0.000', '0.000', '0.000', '1830.000', '0.000', '100.000'],
                                                                                      ['20211002', '15:25', '27051', '3.416', '4820', '4824', '0.609', '15275', '1580',
                                                                                          'NaN', '249.2', '0.000', '0.000', '0.000', '4820.000', '0.000', '100.000']
                                                                                  ]
                                                                                  
                                                                                  df = pd.DataFrame(array, columns=columns)
                                                                                  df['DATETIME'] = pd.to_datetime(df['DATE']+' '+df['TIME'])
                                                                                  df = df.sort_values('DATETIME').set_index('DATETIME')
                                                                                  df_ffill = df.resample('5Min').ffill()
                                                                                  
                                                                                                           DATE   TIME ENERGY_OUT    AAA POWER_OUT   BBB  ...     v7     v8     v9       v10    v11      v12
                                                                                  DATETIME                                                                ...                                               
                                                                                  2021-10-02 15:25:00  20211002  15:25      27051  3.416      4820  4824  ...  0.000  0.000  0.000  4820.000  0.000  100.000
                                                                                  2021-10-02 15:30:00  20211002  15:30      27204  3.435      1830  1836  ...  0.000  0.000  0.000  1830.000  0.000  100.000
                                                                                  2021-10-02 15:35:00  20211002  15:30      27204  3.435      1830  1836  ...  0.000  0.000  0.000  1830.000  0.000  100.000
                                                                                  2021-10-02 15:40:00  20211002  15:40      27294  3.446      1080  1080  ...  0.000  0.000  0.000  1080.000  0.000  100.000
                                                                                  
                                                                                  [4 rows x 17 columns]
                                                                                  
                                                                                  Community Discussions

                                                                                  Trending Discussions on prefect

                                                                                  How to mock functions with same name, when called twice?
                                                                                  chevron right
                                                                                  Queryset takes too long to populate data
                                                                                  chevron right
                                                                                  Writing input to c executable manually trigger exploit, but python input does not
                                                                                  chevron right
                                                                                  Using dropdown filter to dynamic update Highchart
                                                                                  chevron right
                                                                                  How to deploy a Kubernetes Job at AKS with Prefect
                                                                                  chevron right
                                                                                  Delete flow definition in prefect orion
                                                                                  chevron right
                                                                                  prefect.io kubernetes agent and task execution
                                                                                  chevron right
                                                                                  Prefect Local Agent Troubleshooting
                                                                                  chevron right
                                                                                  Supervisor kills Prefect agent with SIGTERM unexpectedly
                                                                                  chevron right
                                                                                  How to solve this issue Execution failed for task ':app:compileFlutterBuildDebug'
                                                                                  chevron right

                                                                                  QUESTION

                                                                                  How to mock functions with same name, when called twice?
                                                                                  Asked 2022-Mar-26 at 07:48

                                                                                  How do I mock a function which has been called twice in the same file, with different parameters? Here is my code snippet:

                                                                                  code_file.py

                                                                                  from prefect import Client
                                                                                  client = Client(
                                                                                      api_server = 
                                                                                  )
                                                                                  def run_flow(name1, name2):
                                                                                       # some graphql queries, not shown here
                                                                                       value1 = client.graphql(graphql_value1, variables={'name': name1})
                                                                                       print("First client call return value:- ", value1)
                                                                                       value2 = client.graphql(graphql_value2, variables={'name': name2, 'id': value1['data'][0]['id']})
                                                                                       print("Second client call return value:- ", value2)
                                                                                  
                                                                                       run_id = client.create_flow(id = value2['data'][0]['id'])
                                                                                          return run_id
                                                                                  

                                                                                  code_test.py

                                                                                  # I want to mock these client calls.
                                                                                  # However when I am mocking, only the first client.graphql function is being mocked. 
                                                                                  # How do I mock the other client.graphql and the client.create_flow function?
                                                                                  
                                                                                  import unittest
                                                                                  import pytest
                                                                                  from pytest_mock import mocker
                                                                                  from unittest.mock import Mock
                                                                                  from fastapi.testclient import TestClient
                                                                                  from app.main import app
                                                                                  
                                                                                  client = TestClient(app)
                                                                                  
                                                                                  class test_flow(unittest.TestCase):
                                                                                   @patch("code_file.client.graphql", side_effect=["test1s", "test2s"])
                                                                                   @patch("code_file.client.create_flow", return_value="test_id")
                                                                                   def test_flow(self, s1):
                                                                                     # Below post call invokes run_flow function
                                                                                     response = client.post("/name1/name2")
                                                                                     assert run_flow("name1", "name2") == "test_id"
                                                                                  

                                                                                  First mock call to graphql is succeeding. The second graphql call is not getting mocked. It is trying to contact the actual server and receiving 404. How can I mock both graphql client calls?

                                                                                  ANSWER

                                                                                  Answered 2022-Mar-26 at 07:48

                                                                                  I tried to reproduce your codes, but for me client.graphql got mocked perfectly.

                                                                                  Here are my codes.

                                                                                  Folder structure

                                                                                  ├── code_file.py
                                                                                  ├── code_test.py
                                                                                  └── prefect.py
                                                                                  
                                                                                  # prefect.py
                                                                                  class Client:
                                                                                      def __init__(self, *arg, **kwargs):
                                                                                          ...
                                                                                  
                                                                                      def graphql(self, *args, **kwargs):
                                                                                          print(f"graphql got called with args: {args}, kwargs: {kwargs}")
                                                                                  
                                                                                      def create_flow(self, *args, **kwargs):
                                                                                          print(f"create_flow got called with args: {args}, kwargs: {kwargs}")
                                                                                  
                                                                                  
                                                                                  # code_file.py
                                                                                  from prefect import Client
                                                                                  
                                                                                  graphql_value1 = "graphql_value1"
                                                                                  graphql_value2 = "graphql_value2"
                                                                                  
                                                                                  client = Client(api_server="http://example.com")
                                                                                  
                                                                                  
                                                                                  def run_flow(name1, name2):
                                                                                      # some graphql queries, not shown here
                                                                                      value1 = client.graphql(graphql_value1, variables={"name": name1})
                                                                                      print("First client call return value:- ", value1)
                                                                                      value2 = client.graphql(graphql_value2, variables={"name": name2, "id": value1["data"][0]["id"]})
                                                                                      print("Second client call return value:- ", value2)
                                                                                  
                                                                                      run_id = client.create_flow(id=value2["data"][0]["id"])
                                                                                      return run_id
                                                                                  
                                                                                  # code_test.py
                                                                                  import unittest
                                                                                  from unittest.mock import patch, MagicMock, call
                                                                                  from code_file import run_flow
                                                                                  
                                                                                  
                                                                                  class TestFlow(unittest.TestCase):
                                                                                      @patch("code_file.client.graphql")
                                                                                      @patch("code_file.client.create_flow")
                                                                                      def test_flow(self, create_flow: MagicMock, graphql: MagicMock) -> None:
                                                                                          first_graphql_return_value = {"data": [{"id": 1}]}
                                                                                          second_graphl_return_value = {"data": [{"id": 2}]}
                                                                                          graphql.side_effect = [
                                                                                              first_graphql_return_value,
                                                                                              second_graphl_return_value,
                                                                                          ]
                                                                                          create_flow.return_value = "test_id"
                                                                                  
                                                                                          self.assertEqual(run_flow("name1", "name2"), "test_id")
                                                                                  
                                                                                          create_flow.assert_called_once_with(id=2)
                                                                                          graphql.assert_has_calls(
                                                                                              [
                                                                                                  call("graphql_value1", variables={"name": "name1"}),
                                                                                                  call("graphql_value2", variables={"name": "name2", "id": 1})
                                                                                              ]
                                                                                          )
                                                                                  
                                                                                  

                                                                                  Running unittest by using the command

                                                                                  python -m unittest code_test.py
                                                                                  

                                                                                  produces the following output

                                                                                  First client call return value:-  {'data': [{'id': 1}]}
                                                                                  Second client call return value:-  {'data': [{'id': 2}]}
                                                                                  .
                                                                                  ----------------------------------------------------------------------
                                                                                  Ran 1 test in 0.001s
                                                                                  

                                                                                  You'll see that the prints in prefect.Client methods do not get printed.

                                                                                  Original answer Explanation

                                                                                  From python official documentation

                                                                                  If side_effect is an iterable then each call to the mock will return the next value from the iterable.

                                                                                  When you set side_effect argument with an iterable, the mock will return next value from the iterable. More detail can be found here

                                                                                  Solution
                                                                                  def test_flow(mocker):
                                                                                      # Some lines of code, not shown here
                                                                                      m = mocker.patch("code_file.client.graphql", side_effect=["value1", "value2"])
                                                                                      ...
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/71610090

                                                                                  QUESTION

                                                                                  Queryset takes too long to populate data
                                                                                  Asked 2022-Mar-22 at 19:36

                                                                                  I have a django view that generates a form depending on POST or GET. It works well with little data on the Student model but stagnates when the data gets big. Is there a remedy I can apply?? Here's the view

                                                                                  def My_View(request,pk):
                                                                                      if request.method == 'POST':
                                                                                          try:
                                                                                              form = NewIssueForm(request.POST,school= request.user.school,pk=pk,issuer = request.user)
                                                                                              if form.is_valid():
                                                                                                  name = form.cleaned_data['borrower_id'].id
                                                                                                  form.save(commit=True)
                                                                                                  books = Books.objects.get(id=pk)
                                                                                            
                                                                                                  Books.Claimbook(books)
                                                                                                  return redirect('view_books')
                                                                                                  messages.success(request,f'Issued successfully')
                                                                                          except Exception as e:
                                                                                              messages.warning(request,f"{e}")
                                                                                              return redirect('view_books')    
                                                                                      else:
                                                                                          form = NewIssueForm(school= request.user.school,pk=pk,issuer = request.user)
                                                                                      return render(request, 'new_issue.html', {'form': form})
                                                                                  

                                                                                  Here's the form

                                                                                  class NewIssueForm(forms.ModelForm):
                                                                                      def __init__(self,*args, pk,school,issuer, **kwargs):
                                                                                          super(NewIssueForm, self).__init__(*args, **kwargs)
                                                                                          booqs = Books.objects.filter(school=school).get(id=pk)
                                                                                          self.fields['issuer'].initial = issuer
                                                                                          self.fields['borrower_id'].Student.objects.filter(school = school)
                                                                                          self.fields['book_id'].label = str(booqs.book_name) + " - " + str(booqs.reg_no)
                                                                                          self.fields['book_id'].initial = pk
                                                                                    
                                                                                      class Meta:
                                                                                          model = Issue
                                                                                          fields = ['issuer','book_id','borrower_id','due_days']
                                                                                          widgets = {
                                                                                              'book_id':forms.TextInput(attrs={"class":'form-control','type':''}),
                                                                                              'issuer':forms.TextInput(attrs={"class":'form-control','type':'hidden'}),
                                                                                              'borrower_id': Select2Widget(attrs={'data-placeholder': 'Select Student','style':'width:100%','class':'form-control'}),
                                                                                              'due_days':forms.TextInput(attrs={"class":"form-control"}),
                                                                                  

                                                                                  Student model

                                                                                  class Student(models.Model):                                    
                                                                                      school = models.ForeignKey(School, on_delete=models.CASCADE)
                                                                                      name = models.CharField(max_length=200,help_text="The student's name in full")
                                                                                      now = datetime.datetime.now()
                                                                                      YEAR = [(str(a), str(a)) for a in range(now.year-5, now.year+1)]
                                                                                      year = models.CharField(max_length=4, choices = YEAR,help_text='The year the student is admitted in school')
                                                                                      student_id = models.CharField(max_length=40,help_text = "This is the student's admission number")
                                                                                      klass = models.ForeignKey(Klass,on_delete=models.CASCADE)
                                                                                      stream = models.ForeignKey(Stream,on_delete=models.CASCADE)
                                                                                      graduated = models.BooleanField(default=False,help_text = "Tick the box to mark the student as graduated")
                                                                                      prefect = models.BooleanField(default=False,help_text = "Tick the box to select the student as a prefect")
                                                                                  
                                                                                      def __str__(self):
                                                                                          return "%s - %s - F%s %s"%(self.student_id,self.name,self.klass,self.stream)
                                                                                      class Meta:
                                                                                          verbose_name_plural = 'Students'
                                                                                          unique_together = ("school", "student_id",)
                                                                                          indexes = [models.Index(fields=['student_id']),
                                                                                                     models.Index(fields=['name', ]),
                                                                                  

                                                                                  ANSWER

                                                                                  Answered 2022-Mar-22 at 19:27

                                                                                  Can't guess your model but it seems like select_related may play a trick.

                                                                                  Source https://stackoverflow.com/questions/71577775

                                                                                  QUESTION

                                                                                  Writing input to c executable manually trigger exploit, but python input does not
                                                                                  Asked 2022-Mar-13 at 21:20

                                                                                  I have this little exploitable file:

                                                                                  #include 
                                                                                  #include 
                                                                                  #include 
                                                                                  #include 
                                                                                  
                                                                                  // gcc -z execstack -z norelro -fno-stack-protector -o format0 format0.c
                                                                                  int target;
                                                                                  
                                                                                  void vuln(char *string)
                                                                                  {
                                                                                      printf(string);
                                                                                      if (target){
                                                                                          printf("Tyes yes eys");
                                                                                      }
                                                                                  }
                                                                                  int main(int argc, char **argv)
                                                                                  
                                                                                  {
                                                                                      vuln(argv[1]);
                                                                                  
                                                                                  
                                                                                      return 0;
                                                                                  }
                                                                                  

                                                                                  It's very simple, I compile like this:

                                                                                  gcc file.c -o file -no-pie
                                                                                  

                                                                                  and then I run it like this get it to leak some values:

                                                                                  ./file %x
                                                                                  38b3fda8
                                                                                  

                                                                                  Which works prefectly. But I want to automate this a bit, using python. So I try the following:

                                                                                  $ ./form &(python -c "print('%x'*3)")
                                                                                  [1] 30633
                                                                                  %x%x%x
                                                                                  [1]+  Done                    ./form
                                                                                  

                                                                                  and this looks super weird. Firstly, the string format error is not triggered. Then it prints it's own name and some other random stuff. I also tried doing this in gdb, with the same result.

                                                                                  How do I give input with python like every other tutorial online?

                                                                                  ANSWER

                                                                                  Answered 2022-Mar-13 at 21:20

                                                                                  I think you meant:

                                                                                  ./form $(python -c "print('%x'*3)")
                                                                                  

                                                                                  What ./form &(python -c "print('%x'*3)") does is:

                                                                                  /form &
                                                                                  (python -c "print('%x'*3)")
                                                                                  

                                                                                  i.e. form is run in the background. (Process 30633) in your example. Python is run in the foreground in a subshell. (And prints out %x%x%x to your terminal)

                                                                                  Source https://stackoverflow.com/questions/71460435

                                                                                  QUESTION

                                                                                  Using dropdown filter to dynamic update Highchart
                                                                                  Asked 2022-Feb-23 at 11:03

                                                                                  currently I was trying to add a dropdown filter button on my chart to let users select a specific values from a data attribute and filter the related data shown on the chart.

                                                                                  I populated my data's attribute to the select unit, and I was trying to update the chart with parse the data from the selective value, but seems the data didn't update on the HighChart.

                                                                                  Here is my code: jsfiddle

                                                                                  const data = [
                                                                                  {
                                                                                      "Data": "aaa",
                                                                                      "Media": "1",
                                                                                      "Row": "1",
                                                                                      "Column": "3",
                                                                                      "Code": "24",
                                                                                  },
                                                                                  {
                                                                                      "Data": "aaa",
                                                                                      "Media": "2",
                                                                                      "Row": "1",
                                                                                      "Column": "1",
                                                                                      "Code": "24",
                                                                                  },
                                                                                  {
                                                                                      "Data": "aaa",
                                                                                      "Media": "3",
                                                                                      "Row": "1",
                                                                                      "Column": "3",
                                                                                      "Code": "24",
                                                                                  },
                                                                                  {
                                                                                      "Data": "aaa",
                                                                                      "Media": "4",
                                                                                      "Row": "1",
                                                                                      "Column": "2",
                                                                                      "Code": "24",
                                                                                  },
                                                                                  {
                                                                                      "Data": "aaa",
                                                                                      "Media": "1",
                                                                                      "Row": "2",
                                                                                      "Column": "1",
                                                                                      "Code": "24",
                                                                                  },
                                                                                  
                                                                                      // populate Media to dropdown list
                                                                                  $.each(data, function (index, record) {
                                                                                      $('', {
                                                                                          value: record.Media,
                                                                                          text: record.Media
                                                                                      }).appendTo("#select_media");
                                                                                  });
                                                                                  
                                                                                  // remove duplicate value from dropdown list
                                                                                  var optionValues =[];
                                                                                  $('#select_media option').each(function(){
                                                                                  if($.inArray(this.value, optionValues) >-1){
                                                                                      $(this).remove()
                                                                                  }else{
                                                                                      optionValues.push(this.value);
                                                                                  }
                                                                                  });
                                                                                  
                                                                                  // getting drop down list value
                                                                                  var mediaselect = document.getElementById("select_media").value
                                                                                  
                                                                                  
                                                                                  mediaselect_str = String(mediaselect)
                                                                                  
                                                                                  const parsedData = []
                                                                                  data.forEach(obj => {
                                                                                      if (obj.Media == mediaselect_str) {
                                                                                      parsedData.push({
                                                                                          x: parseInt(obj.Row),
                                                                                          y: parseInt(obj.Column),
                                                                                          label: [obj.Data, obj.Media, obj.Code]
                                                                                      })
                                                                                      }
                                                                                  });
                                                                                  
                                                                                  var chart = Highcharts.chart('tray_container', {
                                                                                      chart: {
                                                                                          type: 'heatmap',
                                                                                          width: 500,
                                                                                          height: 500,            
                                                                                      
                                                                                      },
                                                                                      title: {
                                                                                          text: null
                                                                                      },
                                                                                      plotOptions: {
                                                                                          heatmap: {
                                                                                              borderColor: "black",
                                                                                              paddingRight: 100
                                                                                          },
                                                                                      },
                                                                                      series: [{
                                                                                          name: null,
                                                                                          borderWidth: 1,
                                                                                          data: parsedData,
                                                                                          dataLabels: {
                                                                                              inside: true,//Show 
                                                                                              enabled: true,
                                                                                              useHTML: true,
                                                                                              opacity: 1,
                                                                                              verticalAlign:'middle',
                                                                                              align: 'center',
                                                                                              style: {
                                                                                                  textShadow: 'none',
                                                                                                  HcTextStroke: null,
                                                                                                  display: 'block'
                                                                                          },
                                                                                              formatter: function() {
                                                                                              const label = this.point.label;
                                                                                              return `${label[1]} 
                                                                                  ${label[2]}
                                                                                  ${label[0]}` } }, }] });

                                                                                  Is there anything I did wrong? Thank you.

                                                                                  =================================================================

                                                                                  Updated: Solution from JMS work prefectly, but in my real case my json data was called from somewhere so this will be a function. Based on JMS's method the result was while I make selection from dropdown it counldn't find update() function since it was nested inside my function. Sorry I'm still new in the javascript and I can't figure out how to solve it.

                                                                                  function generate_chart(data) {
                                                                                  // populate Media to dropdown list
                                                                                  $.each(data, function (index, record) {
                                                                                      $('', {
                                                                                          value: record.Media,
                                                                                          text: record.Media
                                                                                      }).appendTo("#select_media");
                                                                                  });
                                                                                  
                                                                                  // remove duplicate value from dropdown list
                                                                                  var optionValues =[];
                                                                                  $('#select_media option').each(function(){
                                                                                  if($.inArray(this.value, optionValues) >-1){
                                                                                      $(this).remove()
                                                                                  }else{
                                                                                      optionValues.push(this.value);
                                                                                  }
                                                                                  });
                                                                                  
                                                                                  update();
                                                                                  function update() {
                                                                                  // find the maximun of x & y values by row and column
                                                                                  var max_x = Math.max.apply(Math, fildata.map(function(mx) {
                                                                                      return mx.OutputRow;
                                                                                  }))
                                                                                  var max_y = Math.max.apply(Math, fildata.map(function(my) {
                                                                                      return my.OutputColumn;
                                                                                  }))
                                                                                  console.log(max_x, max_y)
                                                                                  
                                                                                  // assign x & y axis array with the maxiumn 
                                                                                  var x = Array(max_x).fill().map((element, index) => index)
                                                                                  var y = Array(max_y).fill().map((element, index) => index)
                                                                                  
                                                                                  
                                                                                  // getting drop down list value
                                                                                  var mediaselect = document.getElementById("select_media").value
                                                                                  
                                                                                  
                                                                                  mediaselect_str = String(mediaselect)
                                                                                  
                                                                                  const parsedData = []
                                                                                  data.forEach(obj => {
                                                                                      if (obj.Media == mediaselect_str) {
                                                                                      parsedData.push({
                                                                                          x: parseInt(obj.Row),
                                                                                          y: parseInt(obj.Column),
                                                                                          label: [obj.Data, obj.Media, obj.Code]
                                                                                      })
                                                                                      }
                                                                                  });
                                                                                  
                                                                                  var chart = Highcharts.chart('tray_container', {
                                                                                      chart: {
                                                                                          type: 'heatmap',
                                                                                          width: 500,
                                                                                          height: 500,            
                                                                                      
                                                                                      },
                                                                                      title: {
                                                                                          text: null
                                                                                      },
                                                                                      plotOptions: {
                                                                                          heatmap: {
                                                                                              borderColor: "black",
                                                                                              paddingRight: 100
                                                                                          },
                                                                                      },
                                                                                      series: [{
                                                                                          name: null,
                                                                                          borderWidth: 1,
                                                                                          data: parsedData,
                                                                                          dataLabels: {
                                                                                              inside: true,//Show 
                                                                                              enabled: true,
                                                                                              useHTML: true,
                                                                                              opacity: 1,
                                                                                              verticalAlign:'middle',
                                                                                              align: 'center',
                                                                                              style: {
                                                                                                  textShadow: 'none',
                                                                                                  HcTextStroke: null,
                                                                                                  display: 'block'
                                                                                          },
                                                                                              formatter: function() {
                                                                                              const label = this.point.label;
                                                                                              return `${label[1]} 
                                                                                  ${label[2]}
                                                                                  ${label[0]}` } }, }] }); }

                                                                                  ANSWER

                                                                                  Answered 2022-Feb-23 at 06:42

                                                                                  You never subscribed to changes of the dropdown, so the data will not change.

                                                                                  1. Add an onchange listener on the dropdown

                                                                                   
                                                                                  
                                                                                  
                                                                                  Put your update logic into the function update() function
                                                                                  
                                                                                  
                                                                                  function update() {
                                                                                     // getting drop down list value
                                                                                     var mediaselect = document.getElementById("select_media").value
                                                                                     mediaselect_str = String(mediaselect)
                                                                                  
                                                                                     const parsedData = []
                                                                                     data.forEach(obj => {
                                                                                         if (obj.Media == mediaselect_str) {
                                                                                         parsedData.push({
                                                                                             x: parseInt(obj.Row),
                                                                                             y: parseInt(obj.Column),
                                                                                             label: [obj.Data, obj.Media, obj.Code]
                                                                                         })
                                                                                         }
                                                                                     });
                                                                                  
                                                                                     var chart = Highcharts.chart('tray_container', {
                                                                                         chart: {
                                                                                             type: 'heatmap',
                                                                                             width: 500,
                                                                                             height: 500,            
                                                                                         
                                                                                         },
                                                                                         title: {
                                                                                             text: null
                                                                                         },
                                                                                         plotOptions: {
                                                                                             heatmap: {
                                                                                                 borderColor: "black",
                                                                                                 paddingRight: 100
                                                                                             },
                                                                                         },
                                                                                         series: [{
                                                                                             name: null,
                                                                                             borderWidth: 1,
                                                                                             data: parsedData,
                                                                                             dataLabels: {
                                                                                                 inside: true,//Show 
                                                                                                 enabled: true,
                                                                                                 useHTML: true,
                                                                                                 opacity: 1,
                                                                                                 verticalAlign:'middle',
                                                                                                 align: 'center',
                                                                                                 style: {
                                                                                                     textShadow: 'none',
                                                                                                     HcTextStroke: null,
                                                                                                     display: 'block'
                                                                                             },
                                                                                                 formatter: function() {
                                                                                                 const label = this.point.label;
                                                                                                 return `${label[1]}  ${label[2]}  ${label[0]}`
                                                                                                 }
                                                                                             },
                                                                                         }]
                                                                                     });
                                                                                    }
                                                                                  
                                                                                  
                                                                                  Call update() once yourself right at the start, to get the initial chart
                                                                                  
                                                                                  See the working fiddle here: http://jsfiddle.net/ran2vz7s/3/
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/71232246

                                                                                  QUESTION

                                                                                  How to deploy a Kubernetes Job at AKS with Prefect
                                                                                  Asked 2022-Feb-21 at 10:51

                                                                                  I'm new to Prefect and would like to deploy a Kubernetes Job at an Azure Kubernetes Service using Prefect. I've set up a Linux Container including Prefect, kubectl. I could establish a connection to the AKS cluster and by using kubectl, I could deploy a Kubernetes Job at the AKS. But how does it work with Prefect? At the Container, I've stored the .py file and a YAML file, which defines the Kubernetes Job.

                                                                                  --> kubectl apply -f deploytestcontainer.yaml --> works

                                                                                  Running attached example Prefect code also works (but the Job is not getting deployed at the AKS)

                                                                                  That's the content of "firstk8sjob.py"

                                                                                  import prefect
                                                                                  from prefect import task, Flow
                                                                                  from prefect.run_configs import KubernetesRun
                                                                                   
                                                                                  @task
                                                                                  def hello_task():
                                                                                      flow.run_config = KubernetesRun(job_template_path="deploytestcontainer.yaml")
                                                                                      logger = prefect.context.get("logger")
                                                                                      logger.info("Hello world!")
                                                                                   
                                                                                  with Flow("hello-flow") as flow:
                                                                                      hello_task()
                                                                                   
                                                                                  flow.run()
                                                                                  

                                                                                  Thank you a lot in advance for your advices!

                                                                                  ANSWER

                                                                                  Answered 2022-Feb-21 at 10:51

                                                                                  There are two steps to deploy your flow to Azure AKS:

                                                                                  1. Deploy your Kubernetes agent to this cluster
                                                                                  2. Register your flow

                                                                                  First, you need to deploy a Kubernetes agent. You can generate a manifest file using:

                                                                                  prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml
                                                                                  

                                                                                  Then you can check the file, and modify it as needed (e.g. change image version to match your desired Python and Prefect version, add environment variables when needed, etc).

                                                                                  Note that the API key is for Prefect Cloud - are you a Prefect Cloud or Prefect Server user? If you're on Prefect Server, you need additionally those env variables.

                                                                                  Once your manifest is ready, you can apply it to your AKS cluster:

                                                                                  kubectl apply -f agent.yaml # optionally set: -n yournamespace 
                                                                                  

                                                                                  Then, once the agent is running, you can deploy your flow simply by specifying the label on your KubernetesRun.

                                                                                  Note that you were setting the run config within a task which is not recommended and may be the cause of your issue. You should attach the run config to your flow object as follows:

                                                                                  import prefect
                                                                                  from prefect import task, Flow
                                                                                  from prefect.run_configs import KubernetesRun
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def hello_task():
                                                                                      logger = prefect.context.get("logger")
                                                                                      logger.info("Hello world!")
                                                                                  
                                                                                  
                                                                                  with Flow("hello-flow", run_config=KubernetesRun(labels=["YOUR_LABEL"])) as flow:
                                                                                      hello_task()
                                                                                  
                                                                                  if __name__ == "__main__":
                                                                                      flow.register("YOUR_PROJECT_NAME")
                                                                                  

                                                                                  You can also register within your CLI and get rid of this "__main__" block:

                                                                                  prefect register --project YOUR_PROJECT_NAME -p path/to/flow.py
                                                                                  

                                                                                  Once the flow is registered, you can trigger a run on AKS using:

                                                                                  prefect run --name "hello-flow" --project YOUR_PROJECT_NAME --watch
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/71188440

                                                                                  QUESTION

                                                                                  Delete flow definition in prefect orion
                                                                                  Asked 2022-Feb-14 at 16:11

                                                                                  How do I delete a flow defined in prefect orion? For example, I have multiple flows visible in the Orion's dashboard, some of which are no longer useful for me. How do I delete some (but not all) flows?

                                                                                  ANSWER

                                                                                  Answered 2022-Feb-14 at 16:11

                                                                                  The only way to do this at the moment is through the REST API: https://orion-docs.prefect.io/api-ref/orion/api/flows/#prefect.orion.api.flows.delete_flow

                                                                                  It has yet to be hooked up to the UI.

                                                                                  Source https://stackoverflow.com/questions/71092971

                                                                                  QUESTION

                                                                                  prefect.io kubernetes agent and task execution
                                                                                  Asked 2022-Feb-03 at 16:12

                                                                                  While reading kubernetes agent documentation, I am getting confused with below line

                                                                                  "Configure a flow-run to run as a Kubernetes Job."

                                                                                  Does it mean that the process which is incharge of submitting flow and communication with api server will run as kubernetes job?

                                                                                  On the other side, the use case which I am trying to solve is

                                                                                  1. Setup backend server
                                                                                  2. Execute a flow composed of 2 tasks
                                                                                  3. if k8s infra available the tasks should be executed as kubernetes jobs
                                                                                  4. if docker only infra available, the tasks should be executed as docker containers.

                                                                                  Can somebody suggest me, how to solve above scenario in prefect.io?

                                                                                  ANSWER

                                                                                  Answered 2022-Feb-03 at 16:12

                                                                                  That's exactly right. When you use KubernetesAgent, Prefect deploys your flow runs as Kubernetes jobs.

                                                                                  For #1 - you can do that in your agent YAML file as follows:

                                                                                          env:
                                                                                          - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
                                                                                            value: ''
                                                                                          - name: PREFECT__CLOUD__API
                                                                                            value: "http://some_ip:4200/graphql" # paste your GraphQL Server endpoint here
                                                                                          - name: PREFECT__BACKEND
                                                                                            value: server
                                                                                  

                                                                                  #2 - write your flow

                                                                                  #3 and #4 - this is more challenging to do in Prefect, as there is currently no load balancing mechanism aware of your infrastructure. There are some hacky solutions that you may try, but there is no first-class way to handle this in Prefect.

                                                                                  One hack would be: you build a parent flow that checks your infrastructure resources and depending on the outcome, it spins up your flow run with either DockerRun or KubernetesRun run config.

                                                                                  from prefect import Flow, task, case
                                                                                  from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
                                                                                  from prefect.run_configs import DockerRun, KubernetesRun
                                                                                  
                                                                                  
                                                                                  @task
                                                                                  def check_the_infrastructure():
                                                                                      return "kubernetes"
                                                                                  
                                                                                  
                                                                                  with Flow("parent_flow") as flow:
                                                                                      infra = check_the_infrastructure()
                                                                                      with case(infra, "kubernetes"):
                                                                                          child_flow_run_id = create_flow_run(
                                                                                              flow_name="child_flow_name", run_config=KubernetesRun()
                                                                                          )
                                                                                          k8_child_flowrunview = wait_for_flow_run(
                                                                                              child_flow_run_id, raise_final_state=True, stream_logs=True
                                                                                          )
                                                                                      with case(infra, "docker"):
                                                                                          child_flow_run_id = create_flow_run(
                                                                                              flow_name="child_flow_name", run_config=DockerRun()
                                                                                          )
                                                                                          docker_child_flowrunview = wait_for_flow_run(
                                                                                              child_flow_run_id, raise_final_state=True, stream_logs=True
                                                                                          )
                                                                                  

                                                                                  But note that this would require you to have 2 agents: Kubernetes agent and Docker agent running at all times

                                                                                  Source https://stackoverflow.com/questions/70974314

                                                                                  QUESTION

                                                                                  Prefect Local Agent Troubleshooting
                                                                                  Asked 2022-Jan-31 at 10:29

                                                                                  I am running a flow on a local agent in docker on EC2 (not ECS). Prefect Cloud is configured to provide a UI for monitoring. The flow executes every 5mins, and for about an hour or so, it does just fine. However, the flows eventually fall behind, before failing to executer entirely and I get the 'cannot find a heartbeat' error.

                                                                                  Is there a way to run the local agent continuously? Why does it suddenly stop?

                                                                                  I apologise for the simplicity of the question, but I am new to Prefect.

                                                                                  Cheers

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-31 at 10:29

                                                                                  When the local or docker agent is running within a container itself (rather than a local process), your flow runs end up deployed as containers, but not as individual containers, but rather within the agent container. You effectively have a single agent container spinning up new containers within itself (docker in docker), which may have many unintended consequences such as issues with scale and resource utilization.

                                                                                  To solve this, I would recommend running the local agent as a local process monitored by a supervisord. This documentation page provides more information.

                                                                                  If you want more environment isolation for this agent process, you can run it within a virtual environment.

                                                                                  To learn more about flow's heartbeat, check out this page and this one for running local or docker agent in a container.

                                                                                  Source https://stackoverflow.com/questions/70923824

                                                                                  QUESTION

                                                                                  Supervisor kills Prefect agent with SIGTERM unexpectedly
                                                                                  Asked 2022-Jan-24 at 12:20

                                                                                  I'm using a rapsberry pi 4, v10(buster). I installed supervisor per the instructions here: http://supervisord.org/installing.html Except I changed "pip" to "pip3" because I want to monitor running things that use the python3 kernel. I'm using Prefect, and the supervisord.conf is running the program with command=/home/pi/.local/bin/prefect "agent local start" (I tried this with and without double quotes)

                                                                                  Looking at the supervisord.log file it seems like the Prefect Agent does start, I see the ASCII art that normally shows up when I start it from the command line. But then it shows it was terminated by SIGTERM;not expected, WARN recieved SIGTERM inidicating exit request.

                                                                                  I saw this post: Supervisor gets a SIGTERM for some reason, quits and stops all its processes but I don't even have that 10Periodic file it references.

                                                                                  Anyone know why/how Supervisor processes are getting killed by sigterm?

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-24 at 12:20

                                                                                  It could be that your process exits immediately because you don’t have an API key in your command and this is required to connect your agent to the Prefect Cloud API. Additionally, it’s a best practice to always assign a unique label to your agents, below is an example with “raspberry” as a label.

                                                                                  You can also check the logs/status:

                                                                                  supervisorctl status
                                                                                  

                                                                                  Here is a command you can try, plus you can specify a directory in your supervisor config (not sure whether environment variables are needed but I saw it from other raspberry Pi supervisor user):

                                                                                  [program:prefect-agent]
                                                                                  command=prefect agent local start -l raspberry -k YOUR_API_KEY --no-hostname-label
                                                                                  directory=/home/pi/.local/bin/prefect
                                                                                  user=pi
                                                                                  environment=HOME="/home/pi/.local/bin/prefect",USER="pi"
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/70827682

                                                                                  QUESTION

                                                                                  How to solve this issue Execution failed for task ':app:compileFlutterBuildDebug'
                                                                                  Asked 2022-Jan-22 at 10:00

                                                                                  I have a already try multiple methods but failed,

                                                                                  FAILURE: Build failed with an exception.

                                                                                  • Where: Script 'C:\Users\IMBA\flutter\packages\flutter_tools\gradle\flutter.gradle' line: 1070

                                                                                  • What went wrong: Execution failed for task ':app:compileFlutterBuildDebug'.

                                                                                  Process 'command 'C:\Users\IMBA\flutter\bin\flutter.bat'' finished with non-zero exit value 1

                                                                                  • Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

                                                                                  • Get more help at https://help.gradle.org

                                                                                  BUILD FAILED in 16s Exception: Gradle task assembleDebug failed with exit code 1

                                                                                  give me a prefect suggestion

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-19 at 16:26
                                                                                  1. Delete your pubspec.lock file
                                                                                  2. go to your pubspec.yaml file and click on pub get now automatically generate file and fix your your.

                                                                                  Source https://stackoverflow.com/questions/70765974

                                                                                  Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                                                                                  Vulnerabilities

                                                                                  No vulnerabilities reported

                                                                                  Install prefect

                                                                                  You can install using 'pip install prefect' or download it from GitHub, PyPI.
                                                                                  You can use prefect like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

                                                                                  Support

                                                                                  Prefect's documentation -- including concepts, tutorials, and a full API reference -- is always available at docs.prefect.io. Instructions for contributing to documentation can be found in the development guide.
                                                                                  Find more information at:
                                                                                  Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                  Find more libraries
                                                                                  Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                  Save this library and start creating your kit
                                                                                  Install
                                                                                • PyPI

                                                                                  pip install prefect

                                                                                • CLONE
                                                                                • HTTPS

                                                                                  https://github.com/PrefectHQ/prefect.git

                                                                                • CLI

                                                                                  gh repo clone PrefectHQ/prefect

                                                                                • sshUrl

                                                                                  git@github.com:PrefectHQ/prefect.git

                                                                                • Share this Page

                                                                                  share link

                                                                                  Explore Related Topics

                                                                                  Consider Popular BPM Libraries

                                                                                  Try Top Libraries by PrefectHQ

                                                                                  server

                                                                                  by PrefectHQPython

                                                                                  prefect-dbt

                                                                                  by PrefectHQPython

                                                                                  prefect-aws

                                                                                  by PrefectHQPython

                                                                                  prefect-collection-template

                                                                                  by PrefectHQPython

                                                                                  prefect-dask

                                                                                  by PrefectHQPython

                                                                                  Compare BPM Libraries with Highest Support

                                                                                  airflow

                                                                                  by apache

                                                                                  dagster

                                                                                  by dagster-io

                                                                                  azkaban

                                                                                  by azkaban

                                                                                  conductor

                                                                                  by Netflix

                                                                                  prefect

                                                                                  by PrefectHQ

                                                                                  Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                  Find more libraries
                                                                                  Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                  Save this library and start creating your kit