boto3 | Amazon Web Services Software Development Kit | AWS library

This happens, during the ground-truth-merge job, when the spark can't find any data either in '/opt/ml/processing/groundtruth/' or '/opt/ml/processing/input_data/' directories. And that can happen when either you haven't sent any requests to the sagemaker endpoint or there are no ground truths.

I got this error because, the folder /opt/ml/processing/input_data/ of the docker volume mapped to the monitoring container had no data to process. And that happened because, the thing that facilitates entire process, including fetching data couldn't find any in S3. and that happened because, there was an extra slash(/) in the directory to which endpoint's captured-data will be saved. to elaborate, while creating the endpoint, I had mentioned the directory as s3:////, while it should have just been s3:///. so, while the thing that copies data from S3 to docker volume tried to fetch data of that hour, the directory it tried to extract the data from was s3://////////(notice the two slashes). So, when I created the endpoint-configuration again with the slash removed in S3 directory, this error wasn't present and ground-truth-merge operation was successful as part of model-quality-monitoring.

I am answering this question because, someone read the question and upvoted it. meaning, someone else has faced this problem too. so, I have mentioned what worked for me. And I wrote this, so that StackExchange doesn't think I am spamming the forum with questions.

If you can call the Lambda function from API Gateway, then your question title "how to connect an aws api gateway to a private lambda function inside a vpc" is already complete and working.

It appears that your actual problem is simply accessing Secrets Manager from inside a Lambda function running in a VPC.

It's also strange that you are assigning a "db" security group to the Lambda function. What are the inbound/outbound rules of this Security Group?

It is entirely unclear why you created a VPC endpoint. What are we supposed to make of service_name = "foo"? What is service "foo"? How is this VPC endpoint related to the Lambda function in any way? If this is supposed to be a VPC endpoint for Secrets Manager, then the service name should be "com.amazonaws.YOUR-REGION.secretsmanager".

If you need more help you need to edit your question to provide the following: The inbound and outbound rules of any relevant security groups, and the Lambda function code that is trying to call SecretsManager.

Update: After clarifications in comments and the updated question, I think the problem is you are missing any subnet assignments for the VPC Endpoint. Also, since you are adding a VPC policy with full access, you can just leave that out entirely, as the default policy is full access. I suggest changing the VPC endpoint to the following:

I got the conda working by modifying the script as below, emr python versions were colliding with the conda version.:

I'm wondering if I can do the same with mock.patch()?

Sure, by using mock.patch.object():

You can use typing.get_args to get the arguments passed in to typing.Literal. In this case, you'll need to combine it with typing.cast so you can signal to "mypy" that the string value that the function returns is an acceptable Literal value.

Ok, basically by reading the ssm session manager plugin source code I came up with the following simplified reimplementation that is capable of just grabbing the command output: (you need to pip install websocket-client construct)

A quick solution is to use logging instead of pprint:

This is bit tricky as CloudFormation, and hence CDK, doesn't allow cross account/cross stage references because CloudFormation export doesn't work cross account as far as my understanding goes. All these patterns of "centralised" resources fall into that category - ie. resource in one account (or a stage in CDK) referenced by other stages.

If the resource is created outside the context of CDK (like via console), then you might as well hardcode the names/arns/etc. throughout the CDK code where its used and that should be sufficient.

1. For resources that have the ability to hold resource based policies, it's simpler as you can just attach the cross-account access permissions to them directly - again, offline via console since you are maintaining it manually anyway. Each time you add a stage (account) to your pipeline, you will need to go to the resource and add cross-account permissions manually.
2. For resources that don't have resource based policies, like SSM for eg., things are a bit roundabout as you will need to create a Role that can be assumed cross-account and then access the resource. In that case you will have to separately maintain the IAM Role too and manually update the trust policy to other accounts as you add stages to your CDK pipeline. Then, as usual hardcode the role arn in your CDK code, assume it in some CustomResource lambda and use it.

It gets more interesting if the creation is also done in the CDK code itself (ie. managed by CloudFormation - not done separately via console/aws-cli etc.). In this case, many times you wouldn't "know" the exact ARNs as the physical-id would be generated by CloudFormation and likely be a part of the ARN. Even influencing the physical-id yourself (like by hardcoding the bucket name) might not solve it in all cases. Eg. KMS ARNs and SecretManager ARNs append unique-ids or some sort of hashes to the end of the ARN.

Instead of trying to work all that out, it would be best left untouched and let CFn generate whatever random name/arn it chooses. To then reference these constructs/ARNs, just put them into SSM Parameters in the source/central account. SSM doesn't have resource based policy that I know of. So additionally create a role in cdk that trusts the accounts in your cdk code. Once done, there is no more maintenance - each time you add new environments/accounts to CDK (assuming its a cdk pipeline here), the "loop" construct that you will create will automatically add the new account into the trust relationship.

Now all you need to do is to distribute this role-arn and the SSM Parameternames to other stages. Choose an explicit role-name and SSM Parameters. The manual ARN construction given a rolename is pretty straightforward. So distribute that and SSM Parameters around your CDK code to other stages (compile time strings instead of references). In target stages, create custom-resource(s) (AWSCustomResource) backed by AwsSdkCall lambda to simply assume this role-arn and make the SDK call to retrieve the SSM Parameter values. These values can be anything, like your KMS ARNs, SecretManager's full ARNs etc. which you couldn't easily guess. Now simply use these.

Roundabout way to do a simple thing, but so far that is all I could do to get this to work.

I have the same issue in my CI/CD using GitLab-CI. The awscli version 1.22.0 have this problem. I solved temporally the problem changed in my gitlab-ci file the line:

pip install awscli --upgrade --user

By:

pip install awscli==1.21.12 --user

Because when you call latest, the version that comes is 1.22.0