Spark Libraries - Page 1

Free and Open, Distributed, RESTful Search Engine

Java 64134Updated: 2 y ago License: Proprietary (Proprietary)

Apache Spark - A unified analytics engine for large-scale data processing

Scala 35985Updated: 2 y ago License: Permissive (Apache-2.0)

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Python 25193Updated: 2 y ago License: Proprietary (Proprietary)

Mirror of Apache Kafka

Java 25123Updated: 2 y ago License: Permissive (Apache-2.0)

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

C++ 24228Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Flink

Java 21401Updated: 2 y ago License: Permissive (Apache-2.0)

Your window into the Elastic Stack

TypeScript 18535Updated: 2 y ago License: Proprietary (Proprietary)

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Python 16581Updated: 2 y ago License: Permissive (Apache-2.0)

The official home of the Presto distributed SQL query engine for big data

Java 14796Updated: 2 y ago License: Permissive (Apache-2.0)

Open source platform for the machine learning lifecycle

Python 14591Updated: 2 y ago License: Permissive (Apache-2.0)

大数据入门指南 :star:

Java 13977Updated: 2 y ago License: No License (No License)

Apache Hadoop

Java 13594Updated: 2 y ago License: Permissive (Apache-2.0)

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

Java 13540Updated: 2 y ago License: Permissive (Apache-2.0)

PredictionIO, a machine learning server for developers and ML engineers.

Scala 12509Updated: 4 y ago License: Permissive (Apache-2.0)

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.

Java 12434Updated: 3 y ago License: Permissive (Apache-2.0)

Data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes.

Python 10896Updated: 2 y ago License: Proprietary (Proprietary)

A simple expressive web framework for java. Spark has a kotlin DSL https://github.com/perwendel/spark-kotlin

Java 9473Updated: 2 y ago License: Permissive (Apache-2.0)

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

Java 7963Updated: 2 y ago License: Permissive (Apache-2.0)

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

Python 7914Updated: 2 y ago License: Permissive (MIT)

NoSQL data store using the seastar framework, compatible with Apache Cassandra

C++ 7734Updated: 3 y ago License: Strong Copyleft (AGPL-3.0)

A curated list of applied machine learning and data science notebooks and libraries across different industries (by @firmai)

Jupyter Notebook 6825Updated: 2 y ago License: No License (No License)

A Flexible and Powerful Parameter Server for large-scale machine learning

Java 6665Updated: 2 y ago License: Proprietary (Proprietary)

Pentaho Data Integration ( ETL ) a.k.a Kettle

Java 6649Updated: 2 y ago License: Permissive (Apache-2.0)

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

Jupyter Notebook 6315Updated: 2 y ago License: Permissive (Apache-2.0)

Alluxio, data orchestration for analytics and machine learning in the cloud

Java 6268Updated: 2 y ago License: Permissive (Apache-2.0)

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Java 6070Updated: 2 y ago License: Permissive (Apache-2.0)

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Scala 6067Updated: 2 y ago License: Permissive (Apache-2.0)

Data-Centric Pipelines and Data Versioning

Go 5930Updated: 2 y ago License: Permissive (Apache-2.0)

cuDF - GPU DataFrame Library

C++ 5565Updated: 2 y ago License: Permissive (Apache-2.0)

Open-source distributed computation and storage platform. Real-time Stream Processing Unconference. Save Your Spot https://hazelcast.com/lp/unconference/

Java 5409Updated: 2 y ago License: Proprietary (Proprietary)

The Metadata Platform for the Modern Data Stack

Java 4881Updated: 3 y ago License: Permissive (Apache-2.0)

Apache HBase

Java 4880Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Hive

Java 4852Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Pinot - A realtime distributed OLAP datastore

Java 4617Updated: 2 y ago License: Permissive (Apache-2.0)

The open big data serving engine. https://vespa.ai

Java 4455Updated: 2 y ago License: Permissive (Apache-2.0)

A better notebook for Scala (and more)

Jupyter Notebook 4421Updated: 2 y ago License: Permissive (Apache-2.0)

Feature Store for Machine Learning

Python 4397Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Iceberg

Java 4377Updated: 2 y ago License: Permissive (Apache-2.0)

LIBSVM -- A Library for Support Vector Machines

Java 4315Updated: 2 y ago License: Permissive (BSD-3-Clause)

Simple and Distributed Machine Learning

Scala 4302Updated: 2 y ago License: Permissive (MIT)

Upserts, Deletes And Incremental Processing on Big Data.

Java 4276Updated: 2 y ago License: Permissive (Apache-2.0)

Fast, distributed, secure AI for Big Data

Jupyter Notebook 4229Updated: 2 y ago License: Permissive (Apache-2.0)

Build data pipelines, the easy way 🛠️

TypeScript 3867Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

Example code from Learning Spark book

Java 3837Updated: 2 y ago License: Permissive (MIT)

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

Python 3781Updated: 3 y ago License: Permissive (Apache-2.0)

Source-agnostic distributed change data capture system

Java 3499Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Kylin

Java 3484Updated: 2 y ago License: Permissive (Apache-2.0)

lakeFS - Data version control for your data lake | Git for data

Go 3443Updated: 2 y ago License: Permissive (Apache-2.0)

A Scala API for Cascading

Scala 3426Updated: 2 y ago License: Permissive (Apache-2.0)

酷玩 Spark: Spark 源代码解析、Spark 类库等

Scala 3399Updated: 2 y ago License: No License (No License)