Big Data Libraries - Page 2

Example code from Learning Spark book

Java 3837Updated: 1 y ago License: Permissive (MIT)

Apache NiFi

Java 3824Updated: 10 mo ago License: Permissive (Apache-2.0)

:helicopter::rocket:基于Flink实现的商品实时推荐系统。flink统计商品热度，放入redis缓存，分析日志信息，将画像标签和实时记录放入Hbase。在用户发起推荐请求后，根据用户画像重排序热度榜，并结合协同过滤和标签两个推荐模块为新生成的榜单的每一个产品添加关联产品，最后返回新的用户列表。

Java 3824Updated: 10 mo ago License: No License (No License)

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

Python 3781Updated: 2 y ago License: Permissive (Apache-2.0)

Sub-second search & analytics engine on cloud storage

Rust 3763Updated: 10 mo ago License: Proprietary (Proprietary)

CrateDB is a distributed SQL database for storing and analyzing massive amounts of data in real-time. Built on top of Lucene.

Java 3692Updated: 10 mo ago License: Permissive (Apache-2.0)

The Hunting ELK

Jupyter Notebook 3530Updated: 10 mo ago License: Strong Copyleft (GPL-3.0)

Source-agnostic distributed change data capture system

Java 3499Updated: 10 mo ago License: Permissive (Apache-2.0)

Apache Kylin

Java 3484Updated: 12 mo ago License: Permissive (Apache-2.0)

A Scala API for Cascading

Scala 3426Updated: 1 y ago License: Permissive (Apache-2.0)

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

TypeScript 3416Updated: 10 mo ago License: Permissive (MIT)

酷玩 Spark: Spark 源代码解析、Spark 类库等

Scala 3399Updated: 1 y ago License: No License (No License)

Koalas: pandas API on Apache Spark

Python 3268Updated: 11 mo ago License: Permissive (Apache-2.0)

StreamPark, Make stream processing easier! easy-to-use streaming application development framework and operation platform

Java 3259Updated: 10 mo ago License: Permissive (Apache-2.0)

Linear algebra library for Rust.

Rust 3242Updated: 10 mo ago License: Permissive (Apache-2.0)

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Go 3219Updated: 10 mo ago License: Permissive (Apache-2.0)

Apache Pinot (Incubating) - A realtime distributed OLAP datastore

Java 3139Updated: 3 y ago License: Permissive (Apache-2.0)

NumPy and Pandas interface to Big Data

Python 3133Updated: 1 y ago License: Permissive (BSD-3-Clause)

RisingWave: the next-generation streaming database in the cloud.

Rust 3116Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

Java 3083Updated: 10 mo ago License: Permissive (Apache-2.0)

A better compressed bitset in Java

Java 3057Updated: 1 y ago License: Permissive (Apache-2.0)

Interactive and Reactive Data Science using Scala and Spark.

JavaScript 3051Updated: 3 y ago License: Permissive (Apache-2.0)

:helicopter::rocket:基于Flink实现的商品实时推荐系统。flink统计商品热度，放入redis缓存，分析日志信息，将画像标签和实时记录放入Hbase。在用户发起推荐请求后，根据用户画像重排序热度榜，并结合协同过滤和标签两个推荐模块为新生成的榜单的每一个产品添加关联产品，最后返回新的用户列表。

Java 2920Updated: 3 y ago License: No License (No License)

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Python 2880Updated: 10 mo ago License: Permissive (MIT)

REST job server for Apache Spark

Scala 2820Updated: 10 mo ago License: Proprietary (Proprietary)

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Scala 2812Updated: 10 mo ago License: Permissive (Apache-2.0)

The flexibility of Python with the scale and performance of modern SQL.

Python 2789Updated: 10 mo ago License: Permissive (Apache-2.0)

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

Java 2762Updated: 1 y ago License: Permissive (Apache-2.0)

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Python 2734Updated: 2 y ago License: Permissive (Apache-2.0)

Python clone of Spark, a MapReduce alike framework in Python

Python 2693Updated: 1 y ago License: Permissive (BSD-3-Clause)

turbo.js - perform massive parallel computations in your browser with GPGPU.

JavaScript 2600Updated: 1 y ago License: Permissive (MIT)

Apache Nutch is an extensible and scalable web crawler

Java 2584Updated: 11 mo ago License: Permissive (Apache-2.0)

Spark: The Definitive Guide's Code Repository

Scala 2584Updated: 11 mo ago License: Proprietary (Proprietary)

Performance-portable, length-agnostic SIMD with runtime dispatch

C++ 2580Updated: 10 mo ago License: Permissive (Apache-2.0)

Run MapReduce jobs on Hadoop or Amazon Web Services

Python 2546Updated: 3 y ago License: Proprietary (Proprietary)

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Java 2446Updated: 11 mo ago License: Permissive (MIT)

Mirror of Apache Flume

Java 2423Updated: 10 mo ago License: Permissive (Apache-2.0)

Package gocql implements a fast and robust Cassandra client for the Go programming language.

Go 2414Updated: 11 mo ago License: Permissive (BSD-3-Clause)

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

TypeScript 2368Updated: 10 mo ago License: Permissive (Apache-2.0)

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Go 2344Updated: 11 mo ago License: Permissive (Apache-2.0)

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Rust 2318Updated: 1 y ago License: Permissive (Apache-2.0)

Apache Geode

Java 2199Updated: 11 mo ago License: Permissive (Apache-2.0)

Python library for creating data pipelines with chain functional programming

Python 2168Updated: 11 mo ago License: Permissive (MIT)

A new arguably faster implementation of Apache Spark from scratch in Rust

Rust 2166Updated: 10 mo ago License: Permissive (Apache-2.0)

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Java 2129Updated: 11 mo ago License: Permissive (Apache-2.0)

Compile-time Language Integrated Queries for Scala

Scala 2125Updated: 10 mo ago License: Permissive (Apache-2.0)

Streaming MapReduce with Scalding and Storm

Scala 2102Updated: 3 y ago License: Permissive (Apache-2.0)

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

C++ 2093Updated: 10 mo ago License: Strong Copyleft (AGPL-3.0)

Dinky is an out of the box one-stop real-time computing platform dedicated to the construction and practice of Unified Streaming & Batch and Unified Data Lake & Data Warehouse. Based on Apache Flink, Dinky provides the ability to connect many big data frameworks including OLAP and Data Lake.

Java 2093Updated: 10 mo ago License: Permissive (Apache-2.0)

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Scala 2091Updated: 3 y ago License: Permissive (Apache-2.0)