PyAthena | Python DB API 2.0 ( PEP | AWS library

I want to show the relations between tables in a database stored in Amazon Web Services. My database name is news. From this answer, I run this Python code in Amazon SageMaker

I run a query from AWS Athena console and takes 10s. The same query run from Sagemaker using PyAthena takes 155s. Is PyAthena slowing it down or is the data transfer from Athena to sagemaker so time consuming?

What could I do to speed this up?

I tried to run a Glue job in python-shell by adding external dependencies (like pyathena, pytest,etc ..) as python egg file/ whl file in the job configurations as mentioned in the AWS documentation https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html.

The Glue job is configured under VPC having no internet and its execution resulted in the below error.

WARNING: The directory '/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')'

I even tried modifying my python script with the below code

Which one is faster pyathena or boto3 to query AWS Athena schemas using python script?

Currently I am using pyathena to query Athena schemas but it's quite slow and I know there is another option of boto3 but before starting need some experts advice.

I'm using Pyathena to run basic queries:

My requirement is to use python script to read data from AWS Glue Database into a dataframe. When I researched I fought the library - "awswrangler". I'm using the below code to connect and read data:

I have an Athena query that I run every day from my local Ubuntu machine. It runs fine most times.

I am trying to create my own custom Sagemaker Framework that runs a custom python script to train a ML model using the entry_point parameter.

Following the Python SDK documentation (https://sagemaker.readthedocs.io/en/stable/estimators.html), I wrote the simplest code to run a training job just to see how it behaves and how Sagemaker Framework works.

My problem is that I don't know how to properly build my Docker container in order to run the entry_point script.

I added the train.py script into the container that only logs the folders and files paths as well as the variables in the containers environment.

I was able to run the training job, but I couldn't find any reference of the entry_point script neither in environment variable nor the files in the container.

Here is the code I used:

• Custom Sagemaker Framework Class:

I am using pyathena library to query schemas and storing it in pandas dataframe. I've a list which contains atleast 30,000 items.

eg. l1 = [1,2,3,4..... 29999,30000]

Now I want to pass this list items in sql query. Since I cannot pass all 30,000 list items at a time, therefore, I divided list into 30 chunks and passing each chunk in loop, as shown below:

Note: I tried it to divide it in fewer chunks but 1000 items per chunks seems best option.