flyte | flexible workflow orchestration platform | BPM library

by flyteorg Python Version: v1.7.0 License: Apache-2.0

X-Ray Key Features Code Snippets(3)Community Discussions(3)Vulnerabilities Install Support

kandi X-RAY | flyte Summary

flyte is a Python library typically used in Automation, BPM, Docker applications. flyte has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. However flyte build file is not available. You can install using 'pip install flyte' or download it from GitHub, PyPI.

Flyte is a structured programming and distributed processing platform that enables highly concurrent, scalable, and maintainable workflows for Machine Learning and Data Processing. It is a fabric that connects disparate computation backends using a type-safe data dependency graph. It records all changes to a pipeline, making it possible to rewind time. It also stores a history of all executions and provides an intuitive UI, CLI, and REST/gRPC API to interact with the computation. Flyte is more than a workflow engine -- it uses workflow as a core concept, and task (a single unit of execution) as a top-level concept. Multiple tasks arranged in a data producer-consumer order creates a workflow. Workflows and Tasks can be written in any language, with out-of-the-box support for Python, Java and Scala. Flyte was designed to manage the complexity that arises in Data and ML teams and ensure they keep up their high velocity of delivering business impacting features. One way it achieves this is by separating the control-plane from the user-plane. Thus, every organization can offer Flyte as a service to their end-users where the service is managed by folks who are more infrastructure-focused, while the users use the intuitive interface of Flytekit.

Support

Quality

Security

License

Reuse

Support

flyte has a medium active ecosystem.

It has 3486 star(s) with 355 fork(s). There are 264 watchers for this library.

There were 6 major release(s) in the last 12 months.

There are 732 open issues and 1606 have been closed. On average issues are closed in 17 days. There are 48 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of flyte is v1.7.0

Quality

flyte has no bugs reported.

Security

flyte has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

flyte is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

flyte releases are available to install and integrate.

Deployable package is available in PyPI.

flyte has no build file. You will be need to create the build yourself to build the component from source.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed flyte and discovered the below as its top functions. This is intended to give you an instant insight into flyte implemented functionality, and help decide if they suit your requirements.

Create a row of the workflow stats .
Create a row of queue metrics .
Row of the metastore read latency .
Create a row for the wf store .
Displays quota usage stats .
Create a row for resource stats .
Returns a graph containing the number of failed failures .
Return a graph of error codes .
Create a row of system errors .
Create a graph with error vs success vs error .

Get all kandi verified functions for this library.

flyte Key Features

No Key Features are available at this moment for flyte.

flyte Examples and Code Snippets

Getting started,Usage

Lines of Code : 155

License : Permissive (Apache-2.0)

Copy

import "github.com/ExpediaGroup/flyte-client"

go build -v ./flyte

# start godoc server
godoc -http=:6060
# navigate to
http://localhost:6060/pkg/github.com/ExpediaGroup/flyte-client

func handle(message json.RawMessage) Event {
        ...

Ways to Define the Integration-Task Example

Python

Lines of Code : 30

License : Permissive (Apache-2.0)

Copy

import pandas as pd
from flytekit import Resources, kwtypes, task, workflow
from flytekit.types.file import CSVFile
from flytekitplugins.great_expectations import GreatExpectationsTask

file_task_object = GreatExpectationsTask(
   name="great_expecta

Remote Cluster

Python

Lines of Code : 11

License : Permissive (Apache-2.0)

Copy

export FLYTE_CONFIG=config/remote.config  # point to the remote cluster
export VERSION=v1
./docker_build_and_tag.sh  # build and push your docker image

uvicorn app.workflows.app:app --reload

fklearn deploy example.app.main:model -i "ghcr.io/unionai

Community Discussions

Trending Discussions on flyte

Flyte 0.16.2: Error loading Blob - How to get Types.Blob.fetch() to work in task decorated function?

Pass blobs between tasks in Flyte

What are the differences between airflow and Kubeflow pipeline?

QUESTION

Flyte 0.16.2: Error loading Blob - How to get Types.Blob.fetch() to work in task decorated function?

Asked 2021-Apr-06 at 17:42

I have a Flyte task function like this:

...

ANSWER

Answered 2021-Apr-06 at 17:42

Could you please give a bit more information about the code? This is flytekit version 0.15.x? I'm a bit confused since that version shouldn't have the @task decorator. It should only have @python_task which is an older API. If you want to use the new python native typing API you should install flytekit==0.17.0 instead.

Also, could you point to the documentation you're looking at? We've updated the docs a fair amount recently, maybe there's some confusion around that. These are the examples worth looking at. There's also two new Python classes, FlyteFile and FlyteDirectory that have replaced the Blob class in flytekit (though that remains what the IDL type is called).

(would've left this as a comment but I don't have the reputation to yet.)

Some code to help with fetching outputs and reading from a file output

Source https://stackoverflow.com/questions/66957164

QUESTION

Pass blobs between tasks in Flyte

Asked 2020-Oct-22 at 05:18

I'm trying to create a Flyte workflow that needs to pass data between several tasks. I looked at one of the examples in the documentation, but trying to recreate the blob-passing as minimally as possible I still can't get it to work.

Here's my workflow definition, in full (my real use case produces a lot more data, of course):

...

ANSWER

Answered 2020-Oct-22 at 05:18

I am assuming you are running this on a local sandbox environment (you are using minio, which is test blob store that we deploy in the sandbox environment). Can you please share your flytekit.config file that you used to register the workflow.

So Flyte automatically stores intermediate data in a bucket (S3 / GCS) based on how you configure it.

The prefix setting is used to automatically upload the data to the configured bucket and prefix https://github.com/lyft/flytesnacks/blob/b980963e48eac4ab7e4a9a3e58b353ad523cee47/cookbook/sandbox.config#L7

Versions prior to v0.7.0 - the shard formatter setting in the config is used- https://github.com/lyft/flytesnacks/blob/b980963e48eac4ab7e4a9a3e58b353ad523cee47/cookbook/sandbox.config#L14-L17

Please also tell us what version of Flyte you are running. Please join the slack channel and I can help you get started. Sorry for all the troubles

Source https://stackoverflow.com/questions/64467317

QUESTION

What are the differences between airflow and Kubeflow pipeline?

Asked 2019-Nov-28 at 08:12

Machine learning platform is one of the buzzwords in business, in order to boost develop ML or Deep learning.

There are a common part workflow orchestrator or workflow scheduler that help users build DAG, schedule and track experiments, jobs, and runs.

There are many machine learning platform that has workflow orchestrator, like Kubeflow pipeline, FBLearner Flow, Flyte

My question is what are the main differences between airflow and Kubeflow pipeline or other ML platform workflow orchestrator?

And airflow supports different language API and has large community, can we use airflow to build our ML workflow ?

...

ANSWER

Answered 2019-Nov-28 at 08:12

You can definitely use Airflow to orchestrate Machine Learning tasks, but you probably want to execute ML tasks remotely with operators.

For example, Dailymotion uses the KubernetesPodOperator to scale Airflow for ML tasks.

If you don't have the resources to setup a Kubernetes cluster yourself, you can use a ML platforms like Valohai that have an Airflow operator.

When doing ML on production, ideally you want to also version control your models to keep track of the data, code, parameters and metrics of each execution.

You can find more details on this article on Scaling Apache Airflow for Machine Learning Workflows

Source https://stackoverflow.com/questions/59046257

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install flyte

You can install using 'pip install flyte' or download it from GitHub, PyPI.
You can use flyte like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.