s3select | s3select makes s3 select querying API | SQL Database library

Setup: latest (5.29) AWS EMR, spark, 1 master 1 node.

step 1. I have used S3Select to parse a file & collect all file keys for pulling from S3. step 2. Use pyspark iterate the keys in a loop and do the following

spark .read .format("s3selectCSV") .load(key) .limit(superhighvalue) .show(superhighvalue)

It took be x number of minutes.

When I increase the cluster to 1 master and 6 nodes, I am not seeing difference in time. It appears to me that I am not using the increased core nodes.
Everything else, config wise are defaults out of the box, I am not setting anything.

So, my question is does cluster size matters to read and inspect (say log or print) data from S3 using EMR, Spark?

I have a strange issue while querying from Presto (AWS EMR). I was using Presto 0.194 and everything was ok, after I upgraded to 0.224, I cannot run my queries. I'm using LDAP authentication for presto and also file base authorization for Hive using a authorization.json file. I'm using the same json file which was working fine in the old version. Any help would highly appreciated.

Error: Query 20191005_104119_00006_3snge failed: Access Denied: View owner 'username' cannot create view that selects from ...

config.propertis:

I am using S3 Select to read csv file from S3 Bucket and outputting as CSV. In the output I only see rows, but not headers. How do I get output with headers included.

I'm trying to catch the data form a S3 object. I'm using a S3 Select feature as below:

boto3 version : 1.7.59