transformations | This tool will help you understand how input

I answer as I know things on each part, possibly disregarding a few of your assertions:

I have four questions. Suppose in spark I have 3 worker nodes. Each worker node has 3 executors and each executor has 3 cores. Each executor has 5 gb memory. (Total 6 executors, 27 cores and 15gb memory). What will happen if: >>> I would use 1 Executor, 1 Core. That is the generally accepted paradigm afaik.

• I have 30 data partitions. Each partition is of size 6 gb. Optimally, the number of partitions must be equal to number of cores, since each core executes one partition/task (One task per partition). Now in this case, how will each executor-core will process the partition since partition size is greater than the available executor memory? Note: I'm not calling cache() or persist(), it's simply that I'm applying some narrow transformations like map() and filter() on my rdd. >>> The number of partitions being the same of number of cores is not true. You can service 1000 partitions with 10 cores, processing one at a time. What if you have 100K partition and on-prem? Unlikely you will get 100K Executors. >>> Moving on and leaving Driver-side collect issues to one side: You may not have enough memory for a given operation on an Executor; Spark can spill to files to disk at the expense of speed of processing. However, the partition size should not exceed a maximum size, was beefed up some time ago. Using multi-core Executors failure can occur, i.e. OOM's, also a result of GC-issues, a difficult topic.

• Will spark automatically try to store the partitions on disk? (I'm not calling cache() or persist() but merely just transformations are happening after an action is called) >>> Not if it can avoid it, but when memory is tight, eviction / spilling to disk can and will occur, and in some cases re-computation from source or last checkpoint will occur.

• Since I have partitions (30) greater than the number of available cores (27) so at max, my cluster can process 27 partitions, what will happen to the remaining 3 partitions? Will they wait for the occupied cores to get freed? >>> They will be serviced by a free Executor at a point in time.

• If I'm calling persist() whose storage level is set to MEMORY_AND_DISK, then if partition size is greater than memory, it will spill data to the disk? On which disk this data will be stored? The worker node's external HDD? >>> Yes, and it will be spilled to the local file system. I think you can configure for HDFS via a setting, but local disks are faster.

This an insightful blog: https://medium.com/swlh/spark-oom-error-closeup-462c7a01709d

You will find your artifacts here:

You got single file because you have in VSBuild /p:PackageAsSingleFile=true

Also you may consider using a newer task Publish Pipeline Artifact. If not please check DownloadBuildArtifacts task here

The good thing about your solution is that it recognises that when all values are the same (smaller === bigger), that the sum should be calculated and return.

However, it is not so good that you subtract the smallest from the largest to replace the largest value. You have an interest in making these values as small as possible, so this is like the worst choice you could make. Using any other pair for the subtraction would already be an improvement.

Also:

• Having to scan the whole array with each recursive call, is time consuming. It makes your solution O(𝑛²).
• findIndex is really (inefficient) overkill for what indexOf could do here.
• If you have decided on the pair to use for subtraction, then why not consider what would happen if you subtracted as many times as possible? You could consider what this means in terms of division and remainder...
• You can avoid the excessive stack usage by just replacing the recursive call with a loop (while (true))

For finding a better algorithm, think of what it means when the array ends up with only 2 in it. This must mean that there was no odd number in the original input. Similarly, if it were 3, then this means the input consisted only of numbers that divide by 3. If you go on like this, you'll notice that the value that remains in the array is a common devisor. With this insight you should be able to write a more efficient algorithm.

Try this approach:

Don't use java.util.Date, use java.time.OffsetDateTime

The data flow requires a compute context, which is Spark. When you use a query in the transformation, that query will get executed from that Spark cluster, which essentially gets pushed down into the database engine for resolution.

When taking type constructors with bounds as type parameters, make sure you use actual arguments instead of wildcards. Using T[a, b] <: Transformation[a, b] instead of T[_, _] <: Transformation[_, _] lets it compile (Scastie). The former takes a type constructor that, when given two types, gives a type that is a subtype of Transformation[a, b] for some a and b that we don't know. By not using wildcards (and ignoring the actual parameters of T), you're letting the compiler know precisely what a T[a, b] is a subtype of.

There is no function can get the last modified time in the data flow expression.

As a workaround, you can create a Get Metadata activity to get that and then pass it's value to a parameter in your data flow.

The expression:@activity('Get Metadata1').output.lastModified

Assuming that the decimal can only contain valid time values you could do something like this: