Big Data Libraries - Page 1

Apache Spark - A unified analytics engine for large-scale data processing

Scala 35985Updated: 2 y ago License: Permissive (Apache-2.0)

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Python 25193Updated: 2 y ago License: Proprietary (Proprietary)

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

C++ 24228Updated: 2 y ago License: Permissive (Apache-2.0)

Parsing gigabytes of JSON per second

C++ 16984Updated: 2 y ago License: Permissive (Apache-2.0)

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Python 16581Updated: 2 y ago License: Permissive (Apache-2.0)

📊 Cube — The Semantic Layer for Building Data Applications

Rust 15712Updated: 2 y ago License: Proprietary (Proprietary)

The official home of the Presto distributed SQL query engine for big data

Java 14796Updated: 2 y ago License: Permissive (Apache-2.0)

大数据入门指南 :star:

Java 13977Updated: 2 y ago License: No License (No License)

Free Data Engineering course!

Jupyter Notebook 13895Updated: 2 y ago License: No License (No License)

DataX是阿里云DataWorks数据集成的开源版本。

Java 13598Updated: 2 y ago License: Proprietary (Proprietary)

Apache Hadoop

Java 13594Updated: 2 y ago License: Permissive (Apache-2.0)

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

Java 13540Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Druid: a high performance real-time analytics database.

Java 12668Updated: 2 y ago License: Permissive (Apache-2.0)

PredictionIO, a machine learning server for developers and ML engineers.

Scala 12509Updated: 4 y ago License: Permissive (Apache-2.0)

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

C++ 11870Updated: 2 y ago License: Permissive (Apache-2.0)

Data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes.

Python 10896Updated: 2 y ago License: Proprietary (Proprietary)

A simple expressive web framework for java. Spark has a kotlin DSL https://github.com/perwendel/spark-kotlin

Java 9473Updated: 2 y ago License: Permissive (Apache-2.0)

open source training courses about distributed database and distributed systems

Rust 8866Updated: 2 y ago License: No License (No License)

Apache Doris is an easy-to-use, high performance and unified analytics database.

Java 8520Updated: 2 y ago License: Permissive (Apache-2.0)

Mirror of Apache Cassandra

Java 8020Updated: 2 y ago License: Permissive (Apache-2.0)

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

Java 7963Updated: 2 y ago License: Permissive (Apache-2.0)

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

Python 7914Updated: 2 y ago License: Permissive (MIT)

NoSQL data store using the seastar framework, compatible with Apache Cassandra

C++ 7734Updated: 3 y ago License: Strong Copyleft (AGPL-3.0)

Apache Beam is a unified programming model for Batch and Streaming data processing.

Java 6930Updated: 2 y ago License: Permissive (Apache-2.0)

Mirror of Apache Storm

Java 6456Updated: 2 y ago License: Permissive (Apache-2.0)

Alluxio, data orchestration for analytics and machine learning in the cloud

Java 6268Updated: 2 y ago License: Permissive (Apache-2.0)

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Java 6070Updated: 2 y ago License: Permissive (Apache-2.0)

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Scala 6067Updated: 2 y ago License: Permissive (Apache-2.0)

At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.

HTML 6053Updated: 2 y ago License: Proprietary (Proprietary)

Data-Centric Pipelines and Data Versioning

Go 5930Updated: 2 y ago License: Permissive (Apache-2.0)

Open-source distributed computation and storage platform. Real-time Stream Processing Unconference. Save Your Spot https://hazelcast.com/lp/unconference/

Java 5409Updated: 2 y ago License: Proprietary (Proprietary)

A Swift library that uses the Accelerate framework to provide high-performance functions for matrix math, digital signal processing, and image manipulation.

Swift 5083Updated: 2 y ago License: Permissive (MIT)

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Java 5076Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Mesos

C++ 5037Updated: 2 y ago License: Permissive (Apache-2.0)

The Metadata Platform for the Modern Data Stack

Java 4881Updated: 3 y ago License: Permissive (Apache-2.0)

Apache HBase

Java 4880Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Hive

Java 4852Updated: 2 y ago License: Permissive (Apache-2.0)

DataX集成可视化页面，选择数据源即可一键生成数据同步任务，支持RDBMS、Hive、HBase、ClickHouse、MongoDB等数据源，批量创建RDBMS数据同步任务，集成开源调度系统，支持分布式、增量同步数据、实时查看运行日志、监控执行器资源、KILL运行进程、数据源信息加密等。

Java 4762Updated: 2 y ago License: Permissive (MIT)

Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

Python 4693Updated: 2 y ago License: Proprietary (Proprietary)

Apache Pinot - A realtime distributed OLAP datastore

Java 4617Updated: 2 y ago License: Permissive (Apache-2.0)

🚀SQL stream processing with Postgres-like experience. 🪄More than a modern alternative to Apache Flink.

Rust 4463Updated: 2 y ago License: Permissive (Apache-2.0)

The open big data serving engine. https://vespa.ai

Java 4455Updated: 2 y ago License: Permissive (Apache-2.0)

A better notebook for Scala (and more)

Jupyter Notebook 4421Updated: 2 y ago License: Permissive (Apache-2.0)

Feature Store for Machine Learning

Python 4397Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Iceberg

Java 4377Updated: 2 y ago License: Permissive (Apache-2.0)

LIBSVM -- A Library for Support Vector Machines

Java 4315Updated: 2 y ago License: Permissive (BSD-3-Clause)

Simple and Distributed Machine Learning

Scala 4302Updated: 2 y ago License: Permissive (MIT)

Upserts, Deletes And Incremental Processing on Big Data.

Java 4276Updated: 2 y ago License: Permissive (Apache-2.0)

Fast, distributed, secure AI for Big Data

Jupyter Notebook 4229Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Doris(Incubating) is an MPP-based interactive SQL data warehousing for reporting and analysis.

C++ 4197Updated: 3 y ago License: Permissive (Apache-2.0)