Spark Libraries - Page 3

Distributed Deep learning with Keras & Spark

Python 1560Updated: 2 y ago License: Permissive (MIT)

Dinky is an out of the box one-stop real-time computing platform dedicated to the construction and practice of Unified Streaming & Batch and Unified Data Lake & Data Warehouse. Based on Apache Flink, Dinky provides the ability to connect many big data frameworks including OLAP and Data Lake.

Java 1550Updated: 2 y ago License: Permissive (Apache-2.0)

MongoDB Connector for Hadoop

Java 1521Updated: 2 y ago License: No License (No License)

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Jupyter Notebook 1521Updated: 2 y ago License: Proprietary (Proprietary)

A Scala kernel for Jupyter

Scala 1516Updated: 1 y ago License: Permissive (BSD-3-Clause)

Open-source graph database, built for real-time streaming data, compatible with Neo4j.

C++ 1494Updated: 1 y ago License: Proprietary (Proprietary)

Code to accompany Advanced Analytics with Spark from O'Reilly Media

Scala 1485Updated: 2 y ago License: Proprietary (Proprietary)

MLeap: Deploy ML Pipelines to Production

Scala 1449Updated: 2 y ago License: Permissive (Apache-2.0)

Lightning-fast cluster computing in Java, Scala and Python.

Scala 1423Updated: 2 y ago License: No License (No License)

Base classes to use when writing tests with Spark

Scala 1414Updated: 2 y ago License: Permissive (Apache-2.0)

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Python 1383Updated: 2 y ago License: Permissive (Apache-2.0)

High performance data store solution

Scala 1359Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Parquet

Java 1357Updated: 2 y ago License: Permissive (Apache-2.0)

HiBench is a big data benchmark suite.

Java 1351Updated: 2 y ago License: Proprietary (Proprietary)

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

Scala 1343Updated: 2 y ago License: Permissive (Apache-2.0)

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

Java 1302Updated: 2 y ago License: Permissive (Apache-2.0)

A cluster computing framework for processing large-scale geospatial data

Java 1302Updated: 2 y ago License: Permissive (Apache-2.0)

GeoMesa is a suite of tools for working with big geo-spatial data in a distributed fashion.

Scala 1302Updated: 2 y ago License: Permissive (Apache-2.0)

Distributed deep learning on Hadoop and Spark clusters.

Jupyter Notebook 1265Updated: 2 y ago License: Permissive (Apache-2.0)

SQL-based streaming analytics platform at scale

Java 1219Updated: 2 y ago License: Permissive (Apache-2.0)

Jupyter magics and kernels for working with remote Spark clusters

Python 1213Updated: 2 y ago License: Proprietary (Proprietary)

Example project implementing best practices for PySpark ETL jobs and applications.

Python 1195Updated: 1 y ago License: No License (No License)

Repository of notes, code and notebooks in Python for the book Pattern Recognition and Machine Learning by Christopher Bishop

Jupyter Notebook 1193Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

Dremio - the missing link in modern data

Java 1190Updated: 2 y ago License: Permissive (Apache-2.0)

KillrWeather is a reference application (work in progress) showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments.

Scala 1185Updated: 2 y ago License: Permissive (Apache-2.0)

A registry of publicly available datasets on AWS

Python 1184Updated: 2 y ago License: Permissive (Apache-2.0)

Apache Spark 官方文档中文版

JavaScript 1184Updated: 2 y ago License: Proprietary (Proprietary)

A library for time series analysis on Apache Spark

Scala 1175Updated: 2 y ago License: Permissive (Apache-2.0)

SQL-based streaming analytics platform at scale

Java 1147Updated: 4 y ago License: Permissive (Apache-2.0)

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

Shell 1147Updated: 1 y ago License: Permissive (MIT)

StreamSets Data Collector - Continuous big data and cloud platform ingest infrastructure

Java 1145Updated: 4 y ago License: Permissive (Apache-2.0)

PySpark + Scikit-learn = Sparkit-learn

Python 1135Updated: 2 y ago License: Permissive (Apache-2.0)

450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...

Python 1101Updated: 1 y ago License: Proprietary (Proprietary)

scala、spark使用过程中，各种测试用例以及相关资料整理

Scala 1083Updated: 2 y ago License: No License (No License)

(Deprecated) Scikit-learn integration package for Apache Spark

Python 1072Updated: 2 y ago License: Permissive (Apache-2.0)

Distributed Stream and Batch Processing

Java 1054Updated: 2 y ago License: Proprietary (Proprietary)

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Python 1048Updated: 2 y ago License: Permissive (MIT)

Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

Java 1041Updated: 2 y ago License: Permissive (Apache-2.0)

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Python 1038Updated: 2 y ago License: Proprietary (Proprietary)

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

Scala 1033Updated: 2 y ago License: Proprietary (Proprietary)

Make stream processing easier! Flink & Spark development scaffold, The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Java 1031Updated: 3 y ago License: Permissive (Apache-2.0)

PySpark-Tutorial provides basic algorithms using PySpark

Jupyter Notebook 1009Updated: 2 y ago License: Proprietary (Proprietary)

MapReduce, Spark, Java, and Scala for Data Algorithms Book

Java 996Updated: 3 y ago License: Proprietary (Proprietary)

Development in Shark has been ended.

Scala 993Updated: 4 y ago License: Permissive (Apache-2.0)

Livy is an open source REST interface for interacting with Apache Spark from anywhere

Scala 990Updated: 2 y ago License: No License (No License)

Python 989Updated: 2 y ago License: Permissive (MIT)

Apache Accumulo

Java 979Updated: 1 y ago License: Permissive (Apache-2.0)

Apache Impala

C++ 969Updated: 1 y ago License: Permissive (Apache-2.0)

A free tutorial for Apache Spark.

Jupyter Notebook 966Updated: 2 y ago License: Proprietary (Proprietary)

Parallel ML System - Bosen project

C++ 958Updated: 4 y ago License: Permissive (BSD-3-Clause)