incubator-gobblin | distributed data integration framework that simplifies

by apache Java Version: gobblin_0.11.0 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(4)Vulnerabilities Install Support

kandi X-RAY | incubator-gobblin Summary

incubator-gobblin is a Java library typically used in Telecommunications, Media, Media, Entertainment, Big Data, Docker, Kafka, Spark, Hadoop applications. incubator-gobblin has build file available, it has a Permissive License and it has medium support. However incubator-gobblin has 340 bugs and it has 6 vulnerabilities. You can download it from GitHub.

Apache Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources: databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop. Apache Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc. Gobblin ingests data from different data sources in the same execution framework, and manages metadata of different sources all in one place. This, combined with other features such as auto scalability, fault tolerance, data quality assurance, extensibility, and the ability of handling data model evolution, makes Gobblin an easy-to-use, self-serving, and efficient data ingestion framework.

Support

Quality

Security

License

Reuse

Support

incubator-gobblin has a medium active ecosystem.

It has 1819 star(s) with 669 fork(s). There are 176 watchers for this library.

It had no major release in the last 12 months.

incubator-gobblin has no issues reported. There are 99 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of incubator-gobblin is gobblin_0.11.0

Quality

incubator-gobblin has 340 bugs (64 blocker, 24 critical, 192 major, 60 minor) and 6086 code smells.

Security

incubator-gobblin has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

incubator-gobblin code analysis shows 6 unresolved vulnerabilities (6 blocker, 0 critical, 0 major, 0 minor).

There are 83 security hotspots that need review.

License

incubator-gobblin is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

incubator-gobblin releases are available to install and integrate.

Build file is available. You can build the component from source.

Installation instructions are available. Examples and code snippets are not available.

incubator-gobblin saves you 164815 person hours of effort in developing the same functionality from scratch.

It has 168698 lines of code, 13399 functions and 2377 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of incubator-gobblin

Get all kandi verified functions for this library.

incubator-gobblin Key Features

No Key Features are available at this moment for incubator-gobblin.

incubator-gobblin Examples and Code Snippets

No Code Snippets are available at this moment for incubator-gobblin.

Community Discussions

Trending Discussions on incubator-gobblin

Apache gobblin build failed

Error: Could not find or load main class org.apache.gobblin.runtime.cli.GobblinCli

Gobblin: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource

I'm trying to install Apache Gobblin. How can I install it using Gradle?

QUESTION

Apache gobblin build failed

Asked 2020-May-06 at 17:11

I'm new to gobblin. I try to build a distribution using master branch of the project. I'm getting bellow error while following the instruction.

...

ANSWER

Answered 2020-May-06 at 17:11

Current Gobblin build scripts use features that are present in JDK 8, but were removed in newer JDK versions. Gradle can use the latest JDK installed on your machine, e.g. JDK 13. As a result, the build process can fail.

As a workaround, you can tell Gradle to use JDK 8.

For example, on Windows, this can be achieved by making a change in gradle.properties (given that you have jre1.8.0_202 installed):

Source https://stackoverflow.com/questions/61634373

QUESTION

Error: Could not find or load main class org.apache.gobblin.runtime.cli.GobblinCli

Asked 2020-Mar-29 at 19:09

I am new to gobblin. I build gobblin from incubator-gobblin GitHub master branch. Now I am tring wikipedia example from getting started guide but getting following error.

WARN: HADOOP_HOME is not defined. Gobblin Hadoop libs will be used in classpath. Error: Could not find or load main class org.apache.gobblin.runtime.cli.GobblinCli

with --show-classpath it gives /mnt/c/users/name/incubator-gobblin/conf/classpath:: How can I solve it? Please let me know if anyone know the solution.

...

ANSWER

Answered 2020-Feb-04 at 18:18

Make sure that you run this command in incubator-gobblin/build/gobblin-distribution/distributions/gobblin-dist and not in incubator-gobblin/gobblin-distribution

Source https://stackoverflow.com/questions/59998664

QUESTION

Gobblin: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource

Asked 2020-Mar-06 at 12:23

I am trying mysql to hdfs data ingestion using gobblin. While running mysql-to-gobblin.pull using steps below:

1) start hadoop:
sbin\start-all.cmd

2) start mysql service:
sudo service mysql start

3) set GOBBLIN_WORK_DIR:
export GOBBLIN_WORK_DIR=/mnt/c/users/name/incubator-gobblin/GOBBLIN_WORK_DIR

4) set GOBBLIN_JOB_CONFIG_DIR
export GOBBLIN_JOB_CONFIG_DIR=/mnt/c/users/name/incubator-gobblin/GOBBLIN_JOB_CONFIG_DIR

5) Start standalone
bin/gobblin.sh service standalone start --jars /mnt/C/Users/name/incubator-gobblin/build/gobblin-sql/libs/gobblin-sql-0.15.0.jar

gives below error

...

ANSWER

Answered 2020-Mar-06 at 12:23

solution is to add this jar or dependency to get rid of Caused by: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource

Source https://stackoverflow.com/questions/60350951

QUESTION

I'm trying to install Apache Gobblin. How can I install it using Gradle?

Asked 2020-Jan-15 at 22:08

I want to install Apache Gobblin on my MacOS X. For this, I downloaded version 0.14.0 and followed the steps here.

Install Gobblin

The first thing I did was this:

...

ANSWER

Answered 2018-Dec-14 at 09:59

Just digged a little into the codes. Are you sure that Java 9 is supported by their build scripts?

Look at the line you have issue with: globalDependencies.gradle:44. See ToolProvider.getSystemToolClassLoader(). Now let's look at its docs for Java 9:

Deprecated. This method is subject to removal in a future version of Java SE. Use the system tool provider or service loader mechanisms to locate system tools as well as user-installed tools. Returns a class loader that may be used to load system tools, or null if no such special loader is provided.

Implementation Requirements:

This implementation always returns null.

See that? It always returns null!

Things were different in Java 8, though:

Returns the class loader for tools provided with this platform. This does not include user-installed tools. Use the service provider mechanism for locating user installed tools.

So the script is calling getURLs on a null object and obviously throws an NPE. It probably needs to be fixed!

Source https://stackoverflow.com/questions/53769473

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install incubator-gobblin

Extract the archive file to your local directory.
Download gradle-wrapper.jar (version 2.13) and place it in the gradle/wrapper folder. See 'Instructions to download gradle wrapper' above.
Skip tests and build the distribution: Run ./gradlew build -x findbugsMain -x test -x rat -x checkstyleMain The distribution will be created in build/gobblin-distribution/distributions directory. (or)
Run tests and build the distribution (requires Maven): Run ./gradlew build The distribution will be created in build/gobblin-distribution/distributions directory.

Support

Gobblin documentation Getting started guide Gobblin architectureCommunity Slack: Sign up or login with existing account to apache-gobblin space on SlackList of companies known to use GobblinSample projectHow to build Gobblin from source codeIssue tracker - Apache Jira

Find more information at: