BigData | 💎🔥Big data study notes

I am new to Spark and am trying to run on a hadoop cluster a simple spark jar file built through maven in intellij. But I am getting classnotfoundexception in all the ways I tried to submit the application through spark-submit.

My pom.xml:

I really just need another set of eyes on this code. As you can see, I am searching for files with the pattern "*45Fall...". But every time I run it, it pulls up the files "*45Sum..." and one time pulled up the "*45Win..." files. It seems totally random and the code clearly asks for Fall. I'm confused.

What I am doing is importing all files with "Fall_2040301", (there are many other numbers associated with "Fall" as well as many other names associated with "*Fall_2040301", as well as Win, Spr, and Sum). I am truncating them at 56 lines by removing the last 84 lines, and binding them together so that I can write them out as a group.

How can I initialize a ConstraintVerifier in Kotlin without using Drools and Quarkus? I already added the optaplanner-test JAR and the Maven Dependency for Optaplanner 8.6.0.Final and tried it the following way:

I'm working on converting a large database for storage in an HDF5 file. To get familiar with H5Py (version 3.2.1) and HDF5, I read the docs for H5Py and wrote a small script that stores random data in an HDF5 file, shown below.

Im am trying to code this .conf file with more scalability, and my idea is to, in order to have multi index in elasticsearch, split the path and get the last position to have the csv name and set it to the type and index in elasticsearch.

I am running an Apache Jena Fuseki server als the SPARQL endpoint that I can connect to when using the application normally. Everything works and I get the output from the resulting query.

But When I try to run my test with Springboot, Junit5 (I assume) and MockMVC it always get stuck on the following part:

I am trying to download the xlsx file from following website: https://upx.world/bigdata And I want click the arrow on the lower right corner

Here is my code (modified from the internet but doesn't work:

I'm using ScalaPB (version 0.11.1) and plugin sbt-protoc (version 1.0.3) to try to compile an old project with ProtocolBuffers in Scala 2.12. Reading the documentation, I want to set the file property preserve_unknown_fields to false. But my question is, where? Where do I need to set this flag? On the .proto file?

I've also tried to include the flag as a package-scoped option by creating a package.proto file next to my other .proto file, with the following content (as it is specified here):

I am currently reading this article and apache beam docs https://medium.com/@mohamed.t.esmat/apache-beam-bites-10b8ded90d4c

Every thing I have read is about N files. In our use case, we receive a pubsub event of ONE new file each time to kick off a job. I don't need to scale per file as I could use cloudrun for that. I need to scale with number of lines in files. ie. a 100 line file and a 100,000,000 line file, I would like to see processed in ~roughly the same time.

If I follow the above article and I give it ONE file instead of many, behind the scenes, how will apache beam scale. How will it know to use 1 node for 100 line vs. perhaps 1000 nodes for the 1,000,000 line file. After all, it doesn't know how many lines are in the file to begin with.

Does dataflow NOT scale with # of lines in the file? i was thinking perhaps node 1 would read rows 0-99 and node 2 would read/discard 0-99 and then read 100-199.

Does anyone know what is happening under the hood so I don't end up wasting hours of test time trying to figure out if it scales with respect to # of lines in a file?

EDIT: Related question but not same question - How to read large CSV with Beam?

I think dataflow may be bottlenecked by one node reading in the whole file which I could do on just a regular computer BUT I am really wondering if it could be better than this.

Another way to say this is behind the scenes, what is this line actually doing