bigdata | repository contains an introduction to Big Data | Azure library

Imagine a situation where you have a sizable List which you need to search through. When would you convert it to a Dictionary and when would you just query the list?

I am aware that querying a List is an O(n) operation, while querying a Dictionary (or Lookup/Hashset) is an O(1). However, I am entirely unsure of the efficiency of converting any O(n) collection into an O(1) collection. Isn't the efficiency of that conversion O(n) itself? Would that mean that converting a List into a Dictionary is entirely pointless unless you query it at least three times?

While we're at it, what's your thought process when you're deciding on a specific collection? What do you consider, and what do you find to be best practices?

E.g. (using my phone to write this, disregard syntax)

In order to try the Kafka stream I did this :

I've been trying to read from a .csv file on many ways, utilizing SparkContext object. I found it possible through scala.io.Source.fromFile function, but I want to use spark object. Everytime I run function textfile for org.apache.spark.SparkContext I get the same error:

I've previously splitted my bigdata:

I am trying to alter several columns on HIVE external table from double to decimal. I have dropped, recreated the table, and ran msck repair statement. However, I am unable to select the table neither from Hive nor Impala as it returns these error:

Possible steps to reproduce：

1. Run spark.sql multiple times, get DataFrame list [d1, d2, d3, d4]
2. Combine DataFrame list [d1, d2, d3, d4] to a DataFrame d5 by calling Dataset#unionByName
3. Run d5.groupBy("c1").pivot("c2").agg(concat_ws(", ", collect_list("value")))，produce DataFrame d6
4. DataFrame d6 join another DataFrame d7
5. Call function like count to trigger spark job
6. Exception happend

stack trace:

I'm trying to do a Flink (1.12.1) batch job, with the following steps:

• Custom SourceFunction to connect with MongoDB
• Do any flatmaps and maps to transform some data
• Sink it in other MongoDB

I'm trying to run it in a StreamExecutionEnvironment , with RuntimeExexutionMode.BATCH, but the application throws a exception because detect my source as UNBOUNDED... And I can't set it BOUNDED ( it must finish after collect all documents in the mongo collection )

The exception:

So lets just say I have a big data coming in that looks like this

I am trying to connect to Kafka. When I run a simple JAR file, I get the following error: