sqoop | The GraphQL Engine powered by Gloo | GraphQL library

I assume you are in a linux environment so you should be able to use the run-one command ( ubuntu run-one ) in conjunction with you bash script in a crontab.

e.g.

The issue could be solved after following a tip received about a typo.

Then, I just changed the word postgres for postgresql as follows:

They are both "working" directories, one has code generated objects, the other has temporary files from HDFS work

--bindir

Output directory for compiled objects

--temporary-rootdir

HDFS directory for temporary files created during import (overrides default "_sqoop")

For more information check out the SqoopUserGuide: https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html

Till now, We've solved it. Pay attention to the following three points:

• Sync trust store file, named jks file, to each computing nodes. If you are using Aws or other cloud services, you should put the trust store file to all nodes, including task nodes if you're using instance fleet.
• The attribute trustCertificateKeyStoreUrl should be valid url which could start with file:/, for instance, trustCertificateKeyStoreUrl=file:/home/hadoop/ssl/clientkeystore.jks
• Set attribute useSSL as true, or it won't transform using SSL to encrypt connection.

Seems that your PostgreSQL server has SSL enabled, but the client side (Dataproc VMs) are not configured with the server certificate or its root CA.

1. With ssl=true the client side will verify the server certificate, you can use a Dataproc init action to import the server certificate to Dataproc VMs:

The error shows that the dates are not arriving as nulls. Rather, they're arriving as blank strings. You can address this a few different ways.

The cleanest way is to use the TRY_TO_DATE function on your COPY INTO statement for that column. This function will return database null when trying to convert a blank string into a date:

https://docs.snowflake.com/en/sql-reference/functions/try_to_date.html#try-to-date

You can use spark to copy data from s3 to HDFS.

Read this blog for more details.

Solution to your first problem - --hcatalog-database mydb --hcatalog-table airquality and remove --export dir parameter.

Sqoop export cannot replace data. Pls issue a sqoop eval statement before loading main table to truncate it.

Sqoop has been retired as an Apache project, and Spark can replace Sqoop. You should not be using them together.

In any case, that pysqoop library appears to be broken and no one has seemed to commit a PR to fix it it in over a year (although, I did). It obviously doesn't have unit-tests to catch that error, so I would not suggest using it in its current form.

Here's an example of sections copied from the Spark documentation for reading JDBC databases and writing to HDFS as Parquet