embulk-output-bigquery | Embulk output plugin to load/insert data | Dashboard library
kandi X-RAY | embulk-output-bigquery Summary
kandi X-RAY | embulk-output-bigquery Summary
load data into Google BigQuery as batch jobs for big amount of data
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of embulk-output-bigquery
embulk-output-bigquery Key Features
embulk-output-bigquery Examples and Code Snippets
Community Discussions
Trending Discussions on embulk-output-bigquery
QUESTION
https://github.com/embulk/embulk-output-bigquery
I'm using this plugin very well. But I have something being curious. What is difference between delete_in_advance and replace in mode?
Situation is like below. I'm migrationing to bigquery. But the job is full dump so I think delete_in_advance is more closer than replace. But I got same result about two of both. Please let me know what is difference.
...ANSWER
Answered 2020-Nov-27 at 00:10https://github.com/embulk/embulk-output-bigquery/issues/124#issuecomment-734256930
delete_in_advance: The table would become empty after (1) and before (2). This is easy implementation, but usually not desired. If another batch which reads the table while embulk is running > with delete_in_advance, the another batch may reads an empty table or a table with a few rows on the gap timing between (1) and (2).
(1) delete
(2) insert replace: In the case of replace, because we use an atomic operation, there is no gap timing which a table becomes empty or with a few rows. The behavior is very expected to build robust data > engineering.
(1) creates temp table
(2) replace target table with the temp table with an atomic operation replace mode is recommended.
QUESTION
I want to transfer data from GCS to BigQuery by embulk and digdag.
But error occurs.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
.......
Error: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
↓ Details
command :
...ANSWER
Answered 2020-Jul-25 at 16:33As the documentation [1], if we have 401- Unauthorized error then there could be many reasons, please have a related list of reasons listed below [followed the link 1], which could be helpful for troubleshooting:
Reason:AuthenticationRequiredRequesterPays
Access to a Requester Pays bucket requires authentication.
Reason: authError
This error indicates a problem with the authorization provided in the request to Cloud Storage. The following are some situations where that will occur: The OAuth access token has expired and needs to be refreshed. This can be avoided by refreshing the access token early, but code can also catch this error, refresh the token and retry automatically. Multiple non-matching authorizations were provided; choose one mode only. The OAuth access token's bound project does not match the project associated with the provided developer key. The Authorization header was of an unrecognized format or uses an unsupported credential type.
reason:lockedDomainExpired
When downloading content from a cookie-authenticated site, e.g., using the Storage Browser, the response will redirect to a temporary domain. This error will occur if access to said domain occurs after the domain expires. Issue the original request again, and receive a new redirect.
Reason: push.webhookUrlUnauthorized
Requests to storage.objects.watchAll will fail unless you verify you own the domain.
Reason: required
Access to a non-public method that requires authorization was made, but none was provided in the Authorization header or through other means.
[1] https://cloud.google.com/storage/docs/json_api/v1/status-codes#401_Unauthorized
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install embulk-output-bigquery
On a UNIX-like operating system, using your system’s package manager is easiest. However, the packaged Ruby version may not be the newest one. There is also an installer for Windows. Managers help you to switch between multiple Ruby versions on your system. Installers can be used to install a specific or multiple Ruby versions. Please refer ruby-lang.org for more information.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page