Spark Libraries - Page 2

State of the Art Natural Language Processing

Scala 3279Updated: 2 y ago License: Permissive (Apache-2.0)

Koalas: pandas API on Apache Spark

Python 3268Updated: 2 y ago License: Permissive (Apache-2.0)

A Cloud Native Batch System (Project under CNCF)

Go 3081Updated: 2 y ago License: Permissive (Apache-2.0)

A better compressed bitset in Java

Java 3057Updated: 2 y ago License: Permissive (Apache-2.0)

Interactive and Reactive Data Science using Scala and Spark.

JavaScript 3051Updated: 4 y ago License: Permissive (Apache-2.0)

A GPU-powered real-time analytics storage and query engine.

Go 2910Updated: 2 y ago License: Permissive (Apache-2.0)

REST job server for Apache Spark

Scala 2820Updated: 2 y ago License: Proprietary (Proprietary)

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Scala 2812Updated: 2 y ago License: Permissive (Apache-2.0)

The flexibility of Python with the scale and performance of modern SQL.

Python 2789Updated: 2 y ago License: Permissive (Apache-2.0)

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Python 2734Updated: 3 y ago License: Permissive (Apache-2.0)

Python clone of Spark, a MapReduce alike framework in Python

Python 2693Updated: 2 y ago License: Permissive (BSD-3-Clause)

Spark: The Definitive Guide's Code Repository

Scala 2584Updated: 2 y ago License: Proprietary (Proprietary)

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Java 2446Updated: 2 y ago License: Permissive (MIT)

Mirror of Apache Flume

Java 2423Updated: 2 y ago License: Permissive (Apache-2.0)

Microsoft Machine Learning for Apache Spark

Scala 2371Updated: 4 y ago License: Permissive (MIT)

Notebooks using the Hugging Face libraries 🤗

Jupyter Notebook 2352Updated: 2 y ago License: Permissive (Apache-2.0)

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Go 2344Updated: 2 y ago License: Permissive (Apache-2.0)

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Rust 2318Updated: 2 y ago License: Permissive (Apache-2.0)

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

Scala 2186Updated: 2 y ago License: Permissive (BSD-3-Clause)

Python library for creating data pipelines with chain functional programming

Python 2168Updated: 2 y ago License: Permissive (MIT)

A new arguably faster implementation of Apache Spark from scratch in Rust

Rust 2166Updated: 2 y ago License: Permissive (Apache-2.0)

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Java 2129Updated: 2 y ago License: Permissive (Apache-2.0)

Elasticsearch Java Rest Client.

Java 2120Updated: 2 y ago License: Permissive (Apache-2.0)

Streaming MapReduce with Scalding and Storm

Scala 2102Updated: 4 y ago License: Permissive (Apache-2.0)

Apache Parquet

Java 2042Updated: 2 y ago License: Permissive (Apache-2.0)

A Flexible, Fast, Federated(3F) SQL Analysis Middleware for Multiple Data Sources

Java 2005Updated: 2 y ago License: Permissive (MIT)

Compile-time Language Integrated Queries for Scala

Scala 1992Updated: 3 y ago License: Permissive (Apache-2.0)

Deep Learning Pipelines for Apache Spark

Python 1968Updated: 2 y ago License: Permissive (Apache-2.0)

Easy Machine Learning is a general-purpose dataflow-based system for easing the process of applying machine learning algorithms to real world tasks.

Java 1958Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Hadoop docker image

Shell 1940Updated: 2 y ago License: No License (No License)

基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法

Java 1921Updated: 2 y ago License: Permissive (Apache-2.0)

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

C# 1905Updated: 2 y ago License: Permissive (MIT)

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Java 1902Updated: 2 y ago License: Permissive (Apache-2.0)

DataStax Spark Cassandra Connector

Scala 1902Updated: 2 y ago License: Permissive (Apache-2.0)

深圳地铁大数据客流分析系统🚇🚄🌟

Scala 1871Updated: 2 y ago License: Proprietary (Proprietary)

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Java 1819Updated: 4 y ago License: Permissive (Apache-2.0)

Production Ready Data Integration Product, documentation：

Java 1819Updated: 3 y ago License: Permissive (Apache-2.0)

BlazingSQL is a lightweight, GPU accelerated, SQL engine for Python. Built on RAPIDS cuDF.

C++ 1808Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Drill is a distributed MPP query layer for self describing data

Java 1801Updated: 2 y ago License: Permissive (Apache-2.0)

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Java 1798Updated: 2 y ago License: Permissive (Apache-2.0)

jdbi is designed to provide convenient tabular data access in Java; including templated SQL, parameterized and strongly typed queries, and Streams integration

Java 1782Updated: 2 y ago License: Permissive (Apache-2.0)

Mirror of Apache Kudu

C++ 1709Updated: 2 y ago License: Permissive (Apache-2.0)

A large-scale entity and relation database supporting aggregation of properties

Java 1700Updated: 2 y ago License: Permissive (Apache-2.0)

ApacheCN 开源组织：公告、介绍、成员、活动、交流方式

CSS 1694Updated: 2 y ago License: Proprietary (Proprietary)

Elassandra = Elasticsearch + Apache Cassandra

Java 1667Updated: 2 y ago License: Permissive (Apache-2.0)

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

Python 1622Updated: 2 y ago License: Permissive (Apache-2.0)

生产环境的海量数据计算产品，文档地址：

Java 1601Updated: 4 y ago License: Permissive (Apache-2.0)

Home of the community managed version of Presto, the distributed SQL query engine for big data, under the auspices of the Presto Software Foundation.

Java 1595Updated: 4 y ago License: Permissive (Apache-2.0)

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Python 1584Updated: 2 y ago License: Permissive (Apache-2.0)

My solution to the book A Collection of Data Science Take-Home Challenges

Jupyter Notebook 1575Updated: 2 y ago License: Permissive (MIT)