Hadoop Libraries - Page 2

深圳地铁大数据客流分析系统🚇🚄🌟

Scala 1871Updated: 10 mo ago License: Proprietary (Proprietary)

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Java 1819Updated: 3 y ago License: Permissive (Apache-2.0)

Production Ready Data Integration Product, documentation：

Java 1819Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Drill is a distributed MPP query layer for self describing data

Java 1801Updated: 1 y ago License: Permissive (Apache-2.0)

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Java 1798Updated: 1 y ago License: Permissive (Apache-2.0)

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads

Java 1748Updated: 10 mo ago License: Permissive (Apache-2.0)

AI on Hadoop

Java 1725Updated: 1 y ago License: Permissive (Apache-2.0)

Run Hadoop Custer within Docker Containers

Shell 1724Updated: 1 y ago License: Permissive (Apache-2.0)

Mirror of Apache Kudu

C++ 1709Updated: 10 mo ago License: Permissive (Apache-2.0)

Python interface to Hive and Presto. 🐝

Python 1609Updated: 11 mo ago License: Proprietary (Proprietary)

生产环境的海量数据计算产品，文档地址：

Java 1601Updated: 3 y ago License: Permissive (Apache-2.0)

Home of the community managed version of Presto, the distributed SQL query engine for big data, under the auspices of the Presto Software Foundation.

Java 1595Updated: 3 y ago License: Permissive (Apache-2.0)

Apache Atlas

Java 1549Updated: 11 mo ago License: Permissive (Apache-2.0)

MongoDB Connector for Hadoop

Java 1521Updated: 1 y ago License: No License (No License)

winutils.exe hadoop.dll and hdfs.dll binaries for hadoop windows

Shell 1481Updated: 10 mo ago License: No License (No License)

Java 1444Updated: 11 mo ago License: Permissive (Apache-2.0)

Write any function in minutes – whether to run a simple job that cleans up a database or build a more complex architecture. Creating functions is easier than ever before, whatever your chosen OS, platform, or development method.

cloud_api 1382Updated: Current License: Proprietary (Proprietary)

Apache Parquet

Java 1357Updated: 11 mo ago License: Permissive (Apache-2.0)

HiBench is a big data benchmark suite.

Java 1351Updated: 1 y ago License: Proprietary (Proprietary)

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

Scala 1343Updated: 1 y ago License: Permissive (Apache-2.0)

TBase is an enterprise-level distributed HTAP database. Through a single database cluster to provide users with highly consistent distributed database services and high-performance data warehouse services, a set of integrated enterprise-level solutions is formed.

C 1321Updated: 11 mo ago License: Proprietary (Proprietary)

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

Java 1302Updated: 1 y ago License: Permissive (Apache-2.0)

A cluster computing framework for processing large-scale geospatial data

Java 1302Updated: 1 y ago License: Permissive (Apache-2.0)

Distributed deep learning on Hadoop and Spark clusters.

Jupyter Notebook 1265Updated: 1 y ago License: Permissive (Apache-2.0)

cluster data collected from production clusters in Alibaba for cluster management research

Jupyter Notebook 1256Updated: 11 mo ago License: No License (No License)

KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.

Scala 1185Updated: 1 y ago License: Permissive (Apache-2.0)

Apache Spark 官方文档中文版

JavaScript 1184Updated: 11 mo ago License: Proprietary (Proprietary)

Scalable, fault-tolerant application-layer sharding for Node.js applications

JavaScript 1177Updated: 1 y ago License: Permissive (MIT)

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

Shell 1147Updated: 10 mo ago License: Permissive (MIT)

StreamSets Data Collector - Continuous big data and cloud platform ingest infrastructure

Java 1145Updated: 3 y ago License: Permissive (Apache-2.0)

450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

Python 1101Updated: 10 mo ago License: Proprietary (Proprietary)

Distributed Stream and Batch Processing

Java 1054Updated: 1 y ago License: Proprietary (Proprietary)

Python module that allows one to easily write and run Hadoop programs.

Python 1044Updated: 1 y ago License: No License (No License)

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

Java 1041Updated: 1 y ago License: Permissive (Apache-2.0)

Make stream processing easier! Flink & Spark development scaffold, The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Java 1031Updated: 2 y ago License: Permissive (Apache-2.0)

MapReduce, Spark, Java, and Scala for Data Algorithms Book

Java 996Updated: 2 y ago License: Proprietary (Proprietary)

Apache Accumulo

Java 979Updated: 10 mo ago License: Permissive (Apache-2.0)

Apache Impala

C++ 969Updated: 10 mo ago License: Permissive (Apache-2.0)

A free tutorial for Apache Spark.

Jupyter Notebook 966Updated: 1 y ago License: Proprietary (Proprietary)

学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等

Python 951Updated: 1 y ago License: Strong Copyleft (GPL-2.0)

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

Scala 943Updated: 1 y ago License: Permissive (Apache-2.0)

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

Java 935Updated: 3 y ago License: No License (No License)

Mirror of Apache Sqoop

Java 933Updated: 1 y ago License: Permissive (Apache-2.0)

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.

Java 912Updated: 10 mo ago License: Permissive (Apache-2.0)

A connector for Spark that allows reading and writing to/from Redis cluster

Scala 908Updated: 11 mo ago License: Permissive (BSD-3-Clause)

R interface for Apache Spark

R 906Updated: 10 mo ago License: Permissive (Apache-2.0)

Distributed machine learning platform

C++ 893Updated: 3 y ago License: Proprietary (Proprietary)

定期更新Hadoop生态圈中常用大数据组件文档 重心依次为: Flink Solr Sparksql ES Scala Kafka Hbase/phoenix Redis Kerberos (项目包含hadoop思维导图 印象笔记 Scala版本简单demo 常用工具类 去敏后的train code 持续更新!!!)

Java 853Updated: 1 y ago License: No License (No License)

电商用户行为分析大数据平台

Java 847Updated: 11 mo ago License: Permissive (Apache-2.0)

☕️ Java Security，安全编码和代码审计

Java 841Updated: 11 mo ago License: No License (No License)