prefect | easiest way to orchestrate and observe your data pipelines | BPM library

 by   PrefectHQ Python Version: 2.82 License: Apache-2.0

kandi X-RAY | prefect Summary

kandi X-RAY | prefect Summary

prefect is a Python library typically used in Automation, BPM applications. prefect has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can install using 'pip install prefect' or download it from GitHub, PyPI.

From the Latin praefectus, meaning "one who is in charge", a prefect is an official who oversees a domain and makes sure that the rules are followed. Similarly, Prefect is responsible for making sure that workflows execute properly. It also happens to be the name of a roving researcher for that wholly remarkable book, The Hitchhiker's Guide to the Galaxy.

            kandi-support Support

              prefect has a highly active ecosystem.
              It has 12152 star(s) with 1251 fork(s). There are 153 watchers for this library.
              There were 9 major release(s) in the last 6 months.
              There are 495 open issues and 3350 have been closed. On average issues are closed in 27 days. There are 35 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of prefect is 2.82

            kandi-Quality Quality

              prefect has 0 bugs and 0 code smells.

            kandi-Security Security

              prefect has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              prefect code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              prefect is licensed under the Apache-2.0 License. This license is Permissive.
              Permissive licenses have the least restrictions, and you can use them in most projects.

            kandi-Reuse Reuse

              prefect releases are available to install and integrate.
              Deployable package is available in PyPI.
              Build file is available. You can build the component from source.
              Installation instructions are not available. Examples and code snippets are available.
              It has 76349 lines of code, 6248 functions and 579 files.
              It has medium code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi has reviewed prefect and discovered the below as its top functions. This is intended to give you an instant insight into prefect implemented functionality, and help decide if they suit your requirements.
            • Upgrade flow table
            • Replace block_type_name_name with a slug
            • Print an error message
            • Print a message and exit
            • Build a new deployment
            • Return aorion client
            • Create a new Flask application
            • Create a work queue with the given name
            • Returns a dict of cmdclass based on cmdclass
            • Extract the version information
            • Create a deployment from a flow
            • Drop all indexes
            • Set the state of a flow run
            • Update the block document
            • Creates aorption client
            • Set the deployment schedule
            • Create a flow run in the database
            • Set the state of a task run
            • Load a deployment
            • Extract version information from VCS
            • Create a container
            • Read block schemas
            • Login to Google
            • Start a work queue
            • Run a deployed flow
            • Start a task run
            • Adds a type field to the model
            • Create the versioneer config file
            Get all kandi verified functions for this library.

            prefect Key Features

            No Key Features are available at this moment for prefect.

            prefect Examples and Code Snippets

            prefect-dask,Running tasks on Dask,Dask annotations
            Pythondot img1Lines of Code : 43dot img1License : Permissive (Apache-2.0)
            copy iconCopy
            import dask
            from prefect import flow, task
            from prefect_dask.task_runners import DaskTaskRunner
            def show(x):
            def my_flow():
                with dask.annotate(priority=-10):
                    future = show(1)  # lo  
            prefect-dask,Running tasks on Dask
            Pythondot img2Lines of Code : 39dot img2License : Permissive (Apache-2.0)
            copy iconCopy
            import time
            from prefect import flow, task
            from prefect_dask import DaskTaskRunner
            def shout(number):
            def count_to(highest_number):
                for number in range(highest  
            prefect-saturn,Getting Started
            Pythondot img3Lines of Code : 23dot img3License : Permissive (BSD-3-Clause)
            copy iconCopy
            import prefect
            from prefect import Flow, task
            from prefect_saturn import PrefectCloudIntegration
            def hello_task():
                logger = prefect.context.get("logger")
      "hello prefect-saturn")
            flow = Flow("sample-flow", tasks=[hello_ta  
            How to mock functions with same name, when called twice?
            Pythondot img4Lines of Code : 77dot img4License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            class Client:
                def __init__(self, *arg, **kwargs):
                def graphql(self, *args, **kwargs):
                    print(f"graphql got called with args:
            How to deploy a Kubernetes Job at AKS with Prefect
            Pythondot img5Lines of Code : 25dot img5License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml
            kubectl apply -f agent.yaml # optionally set: -n yournamespace 
            import prefect
            from prefect imp
            Prefect Task Scheduling
            Pythondot img6Lines of Code : 39dot img6License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            from prefect import task, Flow, Parameter
            def get_data(board):
            def check_db(url):
            def upload_raw(url, board):
            def remove_duplicates(board):
            with Flow(name="4chan_extract
            How to organize prefect flow with the using of cmd arguments?
            Pythondot img7Lines of Code : 18dot img7License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            prefect run -p /path/to/your/
            from prefect import task, Flow, Parameter
            def hello_world(name):
                print(f"Hello {name}!")
            with Flow("mini-example") as flow:
                name = Paramete
            Cant Access Prefect Server Remotely
            Pythondot img8Lines of Code : 9dot img8License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            prefect server start --server-port=5003 --no-postgres-port --use-volume --expose
            endpoint = "http://YOUR_MACHINES_PUBLIC_IP:5003/graphql"
                apollo_url = "http://YOUR_MACHINES_PUBLIC_IP:500
            Appending a record using pandas
            Pythondot img9Lines of Code : 26dot img9License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            columns = ['DATE', 'TIME', 'ENERGY_OUT', 'AAA', 'POWER_OUT', 'BBB', 'CCC', 'ENERGY_IN',
                       'POWER_IN', 'TEMPERATURE', 'VOLTAGE', 'v7', 'v8', 'v9', 'v10', 'v11', 'v12']
            array = [
                ['20211002', '15:40', '27294', '3.446', '1080', 
            Overwrite methods but keep decorator (python)
            Pythondot img10Lines of Code : 13dot img10License : Strong Copyleft (CC BY-SA 4.0)
            copy iconCopy
            class Base:
                def test1(self):
                    return self.test1_impl()
                def test1_impl(self):
            class Child:
                def test1_impl(self):

            Community Discussions


            How to mock functions with same name, when called twice?
            Asked 2022-Mar-26 at 07:48

            How do I mock a function which has been called twice in the same file, with different parameters? Here is my code snippet:




            Answered 2022-Mar-26 at 07:48

            I tried to reproduce your codes, but for me client.graphql got mocked perfectly.

            Here are my codes.

            Folder structure



            Queryset takes too long to populate data
            Asked 2022-Mar-22 at 19:36

            I have a django view that generates a form depending on POST or GET. It works well with little data on the Student model but stagnates when the data gets big. Is there a remedy I can apply?? Here's the view



            Answered 2022-Mar-22 at 19:27

            Can't guess your model but it seems like select_related may play a trick.



            Writing input to c executable manually trigger exploit, but python input does not
            Asked 2022-Mar-13 at 21:20

            I have this little exploitable file:



            Answered 2022-Mar-13 at 21:20


            Using dropdown filter to dynamic update Highchart
            Asked 2022-Feb-23 at 11:03

            currently I was trying to add a dropdown filter button on my chart to let users select a specific values from a data attribute and filter the related data shown on the chart.

            I populated my data's attribute to the select unit, and I was trying to update the chart with parse the data from the selective value, but seems the data didn't update on the HighChart.

            Here is my code: jsfiddle



            Answered 2022-Feb-23 at 06:42

            You never subscribed to changes of the dropdown, so the data will not change.

            1. Add an onchange listener on the dropdown



            How to deploy a Kubernetes Job at AKS with Prefect
            Asked 2022-Feb-21 at 10:51

            I'm new to Prefect and would like to deploy a Kubernetes Job at an Azure Kubernetes Service using Prefect. I've set up a Linux Container including Prefect, kubectl. I could establish a connection to the AKS cluster and by using kubectl, I could deploy a Kubernetes Job at the AKS. But how does it work with Prefect? At the Container, I've stored the .py file and a YAML file, which defines the Kubernetes Job.

            --> kubectl apply -f deploytestcontainer.yaml --> works

            Running attached example Prefect code also works (but the Job is not getting deployed at the AKS)

            That's the content of ""



            Answered 2022-Feb-21 at 10:51

            There are two steps to deploy your flow to Azure AKS:

            1. Deploy your Kubernetes agent to this cluster
            2. Register your flow

            First, you need to deploy a Kubernetes agent. You can generate a manifest file using:



            Delete flow definition in prefect orion
            Asked 2022-Feb-14 at 16:11

            How do I delete a flow defined in prefect orion? For example, I have multiple flows visible in the Orion's dashboard, some of which are no longer useful for me. How do I delete some (but not all) flows?



            Answered 2022-Feb-14 at 16:11

            The only way to do this at the moment is through the REST API:

            It has yet to be hooked up to the UI.



   kubernetes agent and task execution
            Asked 2022-Feb-03 at 16:12

            While reading kubernetes agent documentation, I am getting confused with below line

            "Configure a flow-run to run as a Kubernetes Job."

            Does it mean that the process which is incharge of submitting flow and communication with api server will run as kubernetes job?

            On the other side, the use case which I am trying to solve is

            1. Setup backend server
            2. Execute a flow composed of 2 tasks
            3. if k8s infra available the tasks should be executed as kubernetes jobs
            4. if docker only infra available, the tasks should be executed as docker containers.

            Can somebody suggest me, how to solve above scenario in



            Answered 2022-Feb-03 at 16:12

            That's exactly right. When you use KubernetesAgent, Prefect deploys your flow runs as Kubernetes jobs.

            For #1 - you can do that in your agent YAML file as follows:



            Prefect Local Agent Troubleshooting
            Asked 2022-Jan-31 at 10:29

            I am running a flow on a local agent in docker on EC2 (not ECS). Prefect Cloud is configured to provide a UI for monitoring. The flow executes every 5mins, and for about an hour or so, it does just fine. However, the flows eventually fall behind, before failing to executer entirely and I get the 'cannot find a heartbeat' error.

            Is there a way to run the local agent continuously? Why does it suddenly stop?

            I apologise for the simplicity of the question, but I am new to Prefect.




            Answered 2022-Jan-31 at 10:29

            When the local or docker agent is running within a container itself (rather than a local process), your flow runs end up deployed as containers, but not as individual containers, but rather within the agent container. You effectively have a single agent container spinning up new containers within itself (docker in docker), which may have many unintended consequences such as issues with scale and resource utilization.

            To solve this, I would recommend running the local agent as a local process monitored by a supervisord. This documentation page provides more information.

            If you want more environment isolation for this agent process, you can run it within a virtual environment.

            To learn more about flow's heartbeat, check out this page and this one for running local or docker agent in a container.



            Supervisor kills Prefect agent with SIGTERM unexpectedly
            Asked 2022-Jan-24 at 12:20

            I'm using a rapsberry pi 4, v10(buster). I installed supervisor per the instructions here: Except I changed "pip" to "pip3" because I want to monitor running things that use the python3 kernel. I'm using Prefect, and the supervisord.conf is running the program with command=/home/pi/.local/bin/prefect "agent local start" (I tried this with and without double quotes)

            Looking at the supervisord.log file it seems like the Prefect Agent does start, I see the ASCII art that normally shows up when I start it from the command line. But then it shows it was terminated by SIGTERM;not expected, WARN recieved SIGTERM inidicating exit request.

            I saw this post: Supervisor gets a SIGTERM for some reason, quits and stops all its processes but I don't even have that 10Periodic file it references.

            Anyone know why/how Supervisor processes are getting killed by sigterm?



            Answered 2022-Jan-24 at 12:20

            It could be that your process exits immediately because you don’t have an API key in your command and this is required to connect your agent to the Prefect Cloud API. Additionally, it’s a best practice to always assign a unique label to your agents, below is an example with “raspberry” as a label.

            You can also check the logs/status:



            How to solve this issue Execution failed for task ':app:compileFlutterBuildDebug'
            Asked 2022-Jan-22 at 10:00

            I have a already try multiple methods but failed,

            FAILURE: Build failed with an exception.

            • Where: Script 'C:\Users\IMBA\flutter\packages\flutter_tools\gradle\flutter.gradle' line: 1070

            • What went wrong: Execution failed for task ':app:compileFlutterBuildDebug'.

            Process 'command 'C:\Users\IMBA\flutter\bin\flutter.bat'' finished with non-zero exit value 1

            • Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

            • Get more help at

            BUILD FAILED in 16s Exception: Gradle task assembleDebug failed with exit code 1

            give me a prefect suggestion



            Answered 2022-Jan-19 at 16:26
            1. Delete your pubspec.lock file
            2. go to your pubspec.yaml file and click on pub get now automatically generate file and fix your your.


            Community Discussions, Code Snippets contain sources that include Stack Exchange Network


            No vulnerabilities reported

            Install prefect

            You can install using 'pip install prefect' or download it from GitHub, PyPI.
            You can use prefect like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.


            Prefect's documentation -- including concepts, tutorials, and a full API reference -- is always available at Instructions for contributing to documentation can be found in the development guide.
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
          • PyPI

            pip install prefect

          • CLONE
          • HTTPS


          • CLI

            gh repo clone PrefectHQ/prefect

          • sshUrl


          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Explore Related Topics

            Consider Popular BPM Libraries

            Try Top Libraries by PrefectHQ


            by PrefectHQPython


            by PrefectHQPython


            by PrefectHQJavaScript


            by PrefectHQPython


            by PrefectHQPython