mongo-spark | The MongoDB Spark Connector

I am using the MongoDB Spark Connector to import data from MongoDB and then perform some SQL queries. I will describe the whole process before getting into the actual problem in case I have done something wrong since it's the first time I am using these tools.

I initialize spark-shell with the specific Collection, including the connector package:

I am new to Apache Spark and I am using Scala and Mongodb to learn it. https://docs.mongodb.com/spark-connector/current/scala-api/ I am trying to read the RDD from my MongoDB database, my notebook script as below:

I am trying to connect to MongoDB to write a collection. The spark session was created correctly but whe I try to insert the data into Mongo I get an error in:

How can I write to mongo using spark considering the following scenarios :

1. If the document is present, just update the matching fields with newer value and if the field is absent, add the new field. (The replaceDocument parameter if false will update the matching records but not add the new unmatched fields while if set to true, my old fields can get lost.)
2. I want to keep a datafield as READ-ONLY, example there are two fields, first_load_date and updated_on. first_load_date should never change, it is the day that record is created in mongo, and updated_on is when new fields are added or older ones replaced.
3. If document is absent, insert.

Main problem is replaceDocument = True will lead to loss of older fields not present in newer row, while False, will take care of matched but now the newer incoming fields.

I am using Mongo-Spark-Connector 2.4.1

I am getting the following error when attempting to run sbt run to run my Scala code:

insecure HTTP request is unsupported 'http://repo.typesafe.com/typesafe/releases'; switch to HTTPS or opt-in as ("typesafe-releases" at "http://repo.typesafe.com/typesafe/releases").withAllowInsecureProtocol(true), or by using allowInsecureProtocol in repositories file

This is strange because it was working perfectly fine last week and I have changed nothing in the code. I have tried adding ("typesafe-releases" at "http://repo.typesafe.com/typesafe/releases").withAllowInsecureProtocol(true) in my build.sbt file and resolver file, installing Java11, deleting my project folder, and completely reclone my code from the repository but nothing is working. I am using Visual Studios but have also tried on IntelliJ and get the same error.

Any advice would be greatly appreciated, as I have changed nothing and now suddenly my code doesn't compile anymore. Further details:

sbt.version = 1.4.0

Scala code runner version 2.12.10

My current built.sbt (please note that I did not have the resolve part added before, when my code was working fine. It was added as an attempt to resolve the issue but did not work):

I am trying to load a large Mongo collection into Apache Spark using the Scala Mongo connector.

I am using the following versions:

To start things off I created a jar file using this How to build jars from IntelliJ properly?.

My Jar files path is

Is there any way to transfer/copy my existing env (which has everything already installed) to the server?

Anyone can tell me how to use jars and packages .

1. I'm working on web aplication.
2. For Engine side spark-mongo

bin/spark-submit --properties-file config.properties --packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.1,com.crealytics:spark-excel_2.11:0.13.1 /home/PycharmProjects/EngineSpark.py 8dh1243sg2636hlf38m

• I'm using above command but it's downloading each time from maven repository(jar & packages).
• So now my concern is if i'm offline it gives me error
• its good if their any way to download it only once so no need to download each time
• any suggestion how to deal with it.