Hadoop Libraries - Page 1

Apache Spark - A unified analytics engine for large-scale data processing

Scala 35985Updated: 2 y ago License: Permissive (Apache-2.0)

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Python 25193Updated: 2 y ago License: Proprietary (Proprietary)

Mirror of Apache Kafka

Java 25123Updated: 2 y ago License: Permissive (Apache-2.0)

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

C++ 24228Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Flink

Java 21401Updated: 2 y ago License: Permissive (Apache-2.0)

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Python 16581Updated: 2 y ago License: Permissive (Apache-2.0)

The official home of the Presto distributed SQL query engine for big data

Java 14796Updated: 2 y ago License: Permissive (Apache-2.0)

大数据入门指南 :star:

Java 13977Updated: 2 y ago License: No License (No License)

DataX是阿里云DataWorks数据集成的开源版本。

Java 13598Updated: 2 y ago License: Proprietary (Proprietary)

Apache Hadoop

Java 13594Updated: 2 y ago License: Permissive (Apache-2.0)

PredictionIO, a machine learning server for developers and ML engineers.

Scala 12509Updated: 4 y ago License: Permissive (Apache-2.0)

open source training courses about distributed database and distributed systems

Rust 8866Updated: 2 y ago License: No License (No License)

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

Java 7963Updated: 2 y ago License: Permissive (Apache-2.0)

Alluxio, data orchestration for analytics and machine learning in the cloud

Java 6268Updated: 2 y ago License: Permissive (Apache-2.0)

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Scala 6067Updated: 2 y ago License: Permissive (Apache-2.0)

At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.

HTML 6053Updated: 2 y ago License: Proprietary (Proprietary)

Open-source distributed computation and storage platform. Real-time Stream Processing Unconference. Save Your Spot https://hazelcast.com/lp/unconference/

Java 5409Updated: 2 y ago License: Proprietary (Proprietary)

Apache Mesos

C++ 5037Updated: 2 y ago License: Permissive (Apache-2.0)

Apache HBase

Java 4880Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Hive

Java 4852Updated: 2 y ago License: Permissive (Apache-2.0)

DataX集成可视化页面，选择数据源即可一键生成数据同步任务，支持RDBMS、Hive、HBase、ClickHouse、MongoDB等数据源，批量创建RDBMS数据同步任务，集成开源调度系统，支持分布式、增量同步数据、实时查看运行日志、监控执行器资源、KILL运行进程、数据源信息加密等。

Java 4762Updated: 2 y ago License: Permissive (MIT)

Apache Pinot - A realtime distributed OLAP datastore

Java 4617Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Ignite

Java 4469Updated: 2 y ago License: Permissive (Apache-2.0)

Feature Store for Machine Learning

Python 4397Updated: 2 y ago License: Permissive (Apache-2.0)

LIBSVM -- A Library for Support Vector Machines

Java 4315Updated: 2 y ago License: Permissive (BSD-3-Clause)

Fast, distributed, secure AI for Big Data

Jupyter Notebook 4229Updated: 2 y ago License: Permissive (Apache-2.0)

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

Python 3781Updated: 3 y ago License: Permissive (Apache-2.0)

CrateDB is a distributed SQL database for storing and analyzing massive amounts of data in real-time. Built on top of Lucene.

Java 3692Updated: 2 y ago License: Permissive (Apache-2.0)

A Scala API for Cascading

Scala 3426Updated: 2 y ago License: Permissive (Apache-2.0)

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Go 3219Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Pinot (Incubating) - A realtime distributed OLAP datastore

Java 3139Updated: 4 y ago License: Permissive (Apache-2.0)

A better compressed bitset in Java

Java 3057Updated: 2 y ago License: Permissive (Apache-2.0)

A distributed and coördination-free log management system

Go 2968Updated: 2 y ago License: Permissive (Apache-2.0)

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Python 2880Updated: 2 y ago License: Permissive (MIT)

The flexibility of Python with the scale and performance of modern SQL.

Python 2789Updated: 2 y ago License: Permissive (Apache-2.0)

Python clone of Spark, a MapReduce alike framework in Python

Python 2693Updated: 2 y ago License: Permissive (BSD-3-Clause)

Apache Nutch is an extensible and scalable web crawler

Java 2584Updated: 2 y ago License: Permissive (Apache-2.0)

Run MapReduce jobs on Hadoop or Amazon Web Services

Python 2546Updated: 4 y ago License: Proprietary (Proprietary)

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Java 2446Updated: 2 y ago License: Permissive (MIT)

A high performance implementation of HDBSCAN clustering.

Jupyter Notebook 2430Updated: 2 y ago License: Permissive (BSD-3-Clause)

Mirror of Apache Flume

Java 2423Updated: 2 y ago License: Permissive (Apache-2.0)

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

TypeScript 2368Updated: 2 y ago License: Permissive (Apache-2.0)

Streaming MapReduce with Scalding and Storm

Scala 2102Updated: 4 y ago License: Permissive (Apache-2.0)

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

C++ 2093Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

Node.js bindings for librdkafka

JavaScript 1969Updated: 2 y ago License: Permissive (MIT)

Easy Machine Learning is a general-purpose dataflow-based system for easing the process of applying machine learning algorithms to real world tasks.

Java 1958Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Hadoop docker image

Shell 1940Updated: 2 y ago License: No License (No License)

Apache Ambari simplifies provisioning, managing, and monitoring of Apache Hadoop clusters.

Java 1925Updated: 2 y ago License: Permissive (Apache-2.0)

基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法

Java 1921Updated: 2 y ago License: Permissive (Apache-2.0)

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

Java 1902Updated: 2 y ago License: Permissive (Apache-2.0)