Data-Pipeline | Data pipeline is a tool to run Data loading pipelines | Cloud Storage library

5.4.5

This includes version Spring-Kafka 2.3.6

No it does not; the 5.4.x versions of spring-integration-kafka require 2.6.x; that method was added to the properties in 2.5.

See the project page for compatible versions.

https://spring.io/projects/spring-kafka

If you are using Spring Boot, it will bring in all of the right versions, and you should not specify versions at all in your pom.

For problem 3, it looks like you have a producer factory declared somewhere that is not compatible with the kafka template bean.

This is just a guess but it feels like this is a caching issue. The org.gradle.api.internal.java.JavaLibrary type exists in Gradle 3.2.1 but it doesn’t seem to exist in Gradle 6.8.1 – I checked here.

So I’d suggest trying to clear some caches, maybe starting by removing the .gradle/ and build/ directories of your project(s). My hunch is that that will already help. If it doesn’t, then try to remove (or temporarily rename) the cache/ directory in your Gradle user home (usually under ~/.gradle/) or even entirely remove/rename your Gradle user home.

Looking more closely at the stacktrace, we can see a mention of com.github.jengelman.gradle.plugins.shadow.ShadowPlugin. Is it possible that the com.github.johnrengelman.shadow plugin is applied to your actual build somewhere? At least it’s not mentioned in the build.gradle of your question …

I could imagine that you’re using an old version of that plugin which is not compatible with Gradle 6.8.1. For example, version 1.2.4 of the plugin depends on org.gradle.api.internal.java.JavaLibrary. If you’re indeed using an old version of the plugin, then the solution would probably be to upgrade to a recent version.

If you’re applying any other external plugins that are not mentioned in your question, then they could also be the culprits or result in similar issues.

When you import (or add) something into your project, a .dvc file is created with that lists that something (in this case the raw/ dir) as an "output".

DVC doesn't allow overlapping outputs among .dvc files or dvc.yaml stages, meaning that your "menu_items" stage shouldn't write to raw/ since it's already under the control of raw.dvc.

Can you make a separate directory for the pipeline outputs? E.g. use processed/menu_items/restaurant.jsonl

As per the AWS Data Pipeline documentation on AWS found below, the custom AMI must have Linux installed. This, therefore, cannot be completed currently on a Windows EC2 and must be completed on a Linux EC2.

https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-custom-ami.html

If using directly the raw path (gs://test_buk/dir/), you might be getting an error when escaping special characters in the regex. That might be the reason for which you do not get any input file into the pipeline that matches your filter.

I suggest instead that you use ".*" to math the initial part (given that you are also specifying the path, no additional files in other folders will match the filter).

Therefore, I would use the following expressions depending on the group of files you want to use (feel free to change the extension of the files):

path = gs://test_buk/dir/

regex path filter = .*201909.*\.csv or .*201910.*\.csv

If you would like to know more about the regex used, you can take a look at (1)

ANSWERING TO MYSELF

The issue was that I set the Terminate After field because I got the warning message on the image below which suggested to do that, so I set Terminate After 1 hour and the reason I used that time is because the file to be imported was only 9,6 MB. How much time does it need to process such as small file?.

So the import process of that small file last about 5 hours.

Findings:

To improve the import time I increased the myDDBWriteThroughputRatiovalue to 0.95 from 0.25, At the beginning I didn't touch that parameter because it was the default value from the template, and AWS documentation sometimes simplify a lot stuff that in many cases you have to discover by trial and error.

After changing that value the import last about an hour, which is way better that 5 hours but still slow, because we are talking about only 9,6 MB

Then this that I saw in the logs is in status 'WAITING' and reason 'Cluster ready after last step completed.' which worried me a bit due to I'm new to using this tool and I didn't quite get message, is simply the following as explained ny someone from AWS

*

If you see that the EMR Cluster is in Waiting, Cluster ready after last steps, it means that cluster had executed the first request it has received and is waiting to execute the next request/activity on the cluster.

These are all my findings, hopefully this will helps someone else.

The statement df.file_path denotes that you want access the file_path column in your dataframe table. It seams that you dataframe object does not contain this column. With df.head() you can check if you dataframe object contains the needed fields.

No need to worry. These are containers that you have run, though, probably whilst testing/debugging something.

They can safely be deleted with docker rm , or to clear them all, docker rm $(docker ps -aq)

Dataproc allows users to create clusters, whereas the driver and executor settings in Cloud Data Fusion allow users to adjust how much of the cluster resources a pipeline run will use.

As such, creating a Dataproc cluster with 3 workers and 1 master will create 4 VMs with the memory and CPUs specified in the Dataproc configuration, whereas the setting the driver/executor CPUs and memory dictates how much of each master/worker VMs CPUs and memory resources a data pipeline job running on the cluster will use.