fugue | unified interface for distributed computing

fugue is a Python library typically used in Big Data, Spark applications. fugue has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install fugue' or download it from GitHub, PyPI.

The simplest way to use Fugue is the transform() function. This lets users parallelize the execution of a single function by bringing it to Spark or Dask. In the example below, the map_letter_to_food() function takes in a mapping and applies it on a column. This is just pandas and Python so far (without Fugue). Now, the map_letter_to_food() function is brought to the Spark execution engine by invoking the transform function of Fugue. The output schema, params and engine are passed to the transform() call. The schema is needed because it's a requirement on Spark. A schema of "*" below means all input columns are in the output. This syntax is simpler, cleaner, and more maintainable than the PySpark equivalent. At the same time, no edits were made to the original pandas-based function to bring it to Spark. It is still usable on pandas DataFrames. Because the Spark execution engine was used, the returned df is now a Spark DataFrame. Fugue transform() also supports DaskExecutionEngine and the pandas-based NativeExecutionEngine. A SQL-based language capable of expressing end-to-end workflows. The map_letter_to_food() function above is used in the SQL expression below. This is how to use a Python-defined transformer along with the standard SQL SELECT statement. For FugueSQL, we can change the engine by passing it to the run() method: fsql(query,mapping=map_dict_str).run("spark").