sqoop | Mirror of Apache Sqoop

by apache Java Version: release-1.4.7-rc0 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | sqoop Summary

sqoop is a Java library typically used in Big Data, Spark, Hadoop applications. sqoop has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub.

This is the Sqoop (SQL-to-Hadoop) tool. Sqoop allows easy imports and exports of data sets between databases and HDFS.

Support

Quality

Security

License

Reuse

Support

sqoop has a highly active ecosystem.

It has 933 star(s) with 585 fork(s). There are 74 watchers for this library.

It had no major release in the last 6 months.

sqoop has no issues reported. There are 22 open pull requests and 0 closed requests.

It has a negative sentiment in the developer community.

The latest version of sqoop is release-1.4.7-rc0

Quality

sqoop has no bugs reported.

Security

sqoop has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

sqoop is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

sqoop releases are not available. You will need to build from source code and install.

Build file is available. You can build the component from source.

Top functions reviewed by kandi - BETA

kandi has reviewed sqoop and discovered the below as its top functions. This is intended to give you an instant insight into sqoop implemented functionality, and help decide if they suit your requirements.

Imports a table
Returns the column names for the specified table
Extract the database name from a connect string
Returns the SQL command to copy a table to stdout
Create connection manager
Create any necessary Oracle tables for the given Oracle job
Parse JDBC thin connection string
Initialize mapper connection details
Fetches the password from the given configuration
Restores a previously saved job data
Returns the primary key of the specified table
Returns a query to select the rows from the database
Sets output format
Creates a connection to the database
Configure the input format to use
Apply the command line options
Setup the output table
Initialize defaults
Create export changes table
Set up CopyManager
Gets the input splits
Splits the specified results into a list of splits
Executes the SQL command
Sets the configuration
Returns the SELECT query
Get all input splits for a table

Get all kandi verified functions for this library.

sqoop Key Features

No Key Features are available at this moment for sqoop.

sqoop Examples and Code Snippets

No Code Snippets are available at this moment for sqoop.

Community Discussions

Trending Discussions on sqoop

Why sqoop is throwing error while import?

Apache Atlas: curl: (7) Failed to connect to localhost port 21000: Connection refused

Importing data from SQL Server to HIVE using SQOOP

can't build Apache Atlas

sqoop export for hive views

Insert into and replace old records with new

Google Dataproc to SQL Server(based on centos 7) connection error?

Fetch previous date with sqoop

sqoop script error "... is not a valid DFS filename"

Sqoop and Avro depedency issue in Dataproc Spark 3.1

QUESTION

Why sqoop is throwing error while import?

Asked 2021-Apr-20 at 03:35

I have just installed sqoop and trying to import table from mysql but it is throwing below error. I am new to sqoop.

...

ANSWER

Answered 2021-Apr-20 at 03:35

You may not have commons-lang-2.6.jar in lib directory on sqoop home

then you can use sqoop list-databases --connect jdbc:mysql://localhost:3306/test --username root -P command test.

Source https://stackoverflow.com/questions/67168603

QUESTION

Apache Atlas: curl: (7) Failed to connect to localhost port 21000: Connection refused

Asked 2021-Apr-03 at 17:06

I'm trying to run apache atlas on my local. There are several problem I have faced to. First, for clearance of how I have build the apache atlas I will describe the steps:

git clone https://github.com/apache/atlas
cd atlas
mvn clean install -DskipTests -X
mvn clean package -Pdist -DskipTests

It has been built without any error. Here is the project structure:

...

ANSWER

Answered 2021-Apr-03 at 17:06

After struggling with Apache Atlas for a while, I found 3.0.0 Snapshot version very buggy! Therefore I have decided to build and install Apache Atlas 2.1.0 RC3.

Prerequisite:

Make sure you have installed java on your machine. In case it is not installed on your computer, you can install it using the following command in Linux:

sudo apt-get install openjdk-8-jre

Then JAVA_HOME should be set:

Source https://stackoverflow.com/questions/66563413

QUESTION

Importing data from SQL Server to HIVE using SQOOP

Asked 2021-Mar-31 at 11:59

I am able to successfully import data from SQL Server to HDFS using sqoop. However, when it tries to link to HIVE I get an error. I am not sure I understand the error correctly

...

ANSWER

Answered 2021-Mar-31 at 11:55

There is no such thing as schema inside the database in Hive. Database and schema mean the same thing and can be used interchangeably.

So, the bug is in using database.schema.table. Use database.table in Hive.

Read the documentation: Create/Drop/Alter/UseDatabase

Source https://stackoverflow.com/questions/66884281

QUESTION

can't build Apache Atlas

Asked 2021-Mar-07 at 13:43

I'm trying to build Apache Atlas from the main repository. As it is described in the README.md file, after cloning the repository and changing the current directory to atlas, I am trying to build using mvn clean install command. Unfortunately, since they closed the issue part of the repository, I will explain my problem here.

Build Process

Do the following items in your terminal:

git clone https://github.com/apache/atlas.git
cd atlas
export MAVEN_OPTS="-Xms2g -Xmx2g"
mvn clean install

After running the last command, I face to the following error:

...

ANSWER

Answered 2021-Mar-07 at 13:43

Since the community of Apache Atlas is quite small these days, I want to write the complete story of what has happened and how it works properly, now.

As it seems in the first error trace, the problem is with maven-surefire-plugin:2.18.1:test. In the root directory of project (repository) there is a file named pom.xml which contains the necessary libraries and frameworks to work with. It seems that several tests are wrong and because of this problem, the building phase had error. In order to skip tests and not to exit from building process we have to add the -DskipTests when we want to build:

Source https://stackoverflow.com/questions/66511228

QUESTION

sqoop export for hive views

Asked 2021-Mar-01 at 15:43

I am trying to sqoop hive view to SQL server database however i'm getting "object not found error". Does sqoop export works for hive views?

...

ANSWER

Answered 2021-Mar-01 at 15:43

Unfortunately, this is not possible to do using sqoop export, even if --hcatalog-table specified, it works only with tables and if not in HCatalog mode, it supports only exporting from directories, also no queries are supported in sqoop-export.

You can load your view data into table:

Source https://stackoverflow.com/questions/66423244

QUESTION

Insert into and replace old records with new

Asked 2021-Feb-26 at 04:14

I have a table which takes data with sqoop and every day it will be truncated.

This tblSqoop at the beginning has these values :

...

ANSWER

Answered 2021-Feb-26 at 04:14

could you pls truncate the table and reload tblMaxed using this ? (Explanation is in the code)

Source https://stackoverflow.com/questions/66368499

QUESTION

Google Dataproc to SQL Server(based on centos 7) connection error?

Asked 2021-Feb-19 at 18:15

I got stuck into an issue which already has wasted 3 days of mine. I have a dataproc cluster 1.5 and i also did setup SQL Server on google VM running centos 7 OS. But i am unable to connect SQL Server through pyspark from dataproc cluster. You can find the error snapshot in the attachment. SSL encryption is disabled on SQL server. I can access SQL server through sqlcmd(installed on dataproc cluster) and also through PYMSSQL library from dataproc cluster. But not with pyspark. The same error occurs while trying to access MSSQL from Sqoop as well. Kindly, guide me i have tried all possible solution available on internet but still no luck for me. Thanks in advance. My Connection String is:

...

ANSWER

Answered 2021-Feb-19 at 18:15

This could happen because Dataproc uses Conscrypt by default to improve performance.

Depending on MS SQL JDBC deriver version that you use it can have bugs that lead to failures when Conscrypt is used.

To workaround this issue try to disable Conscrypt during Dataproc cluster creation via cluster properties:

Source https://stackoverflow.com/questions/66255107

QUESTION

Fetch previous date with sqoop

Asked 2021-Feb-19 at 11:46

I want to put in oozie some sqoop commands in order to be executed everyday and fetch data for previous date:

The table has a column date_prof and it has values like:

...

ANSWER

Answered 2021-Feb-19 at 11:29

convert date_prof to date:

Source https://stackoverflow.com/questions/66276068

QUESTION

sqoop script error "... is not a valid DFS filename"

Asked 2021-Feb-03 at 18:10

*running this in a Linux environment via putty from windows.

I have a sqoop script, trying to copy a table from oracle to hive. I get an error regarding my destination path.../hdfs://myserver/apps/hive/warehouse/new_schema/new_table is not a valid DFS filename

Can anyone please tell me if my destination path looks correct? I am not trying to setup a file, I just want to copy a table from oracle to hive and put it in a scheme that already exists in hive. Below is my script.

...

ANSWER

Answered 2021-Feb-03 at 18:10

I think what's causing your that error is the "/" before your HDFS path. The correct path should be:

Source https://stackoverflow.com/questions/65726413

QUESTION

Sqoop and Avro depedency issue in Dataproc Spark 3.1

Asked 2021-Jan-31 at 04:38

I am upgrading from spark 2.4.7 to spark 3.1 in GCP Dataproc. I am doing sqoop import and loading the data to the Parquet file. The code is running fine on the Spark 2.4.7 version but giving the below error in Spark 3.1.

...

ANSWER

Answered 2021-Jan-31 at 04:38

This exception is caused by SQOOP-3485 issue. We will fix it in future release of Dataproc 2.0 image in 2 weeks.

Meanwhile you can try to workaround it by adding org.codehaus.jackson:jackson-mapper-asl:1.9.13 jar to Sqoop and/or your application classpath.

Source https://stackoverflow.com/questions/65953995

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install sqoop

You can download it from GitHub.
You can use sqoop like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the sqoop component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

Sqoop ships with additional documentation: a user guide and a manual page. Asciidoc sources for both of these are in src/docs/. Run ant docs to build the documentation. It will be created in build/docs/. If you got Sqoop in release form, documentation will already be built and available in the docs/ directory. For more information on compiling see COMPILING.adoc.

Find more information at: