ssh2-sftp-client | a client for SSH2 SFTP

I figured it out. Not sure where exactly the problem was but this will successfully pull a file from S3 and then upload to SFTP server:

Worked with version 6.0.0 of ssh2-sftp-client

Cloud Run works only until you have a HTTP request active.

You proably don't have an active request during this on Cloud Run, as outside of the active request the CPU is throttled.

Best for this pipeline is Cloud Workflows and regular Compute Engine instances.

You can setup a Workflow to start a Compute Engine for this task, and stop once it finished doing the steps.

I am the author of article: Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows it will guide you how to setup.

Executing the Workflow can be triggered by Cloud Scheduler or by HTTP ping.

I hope it will be useful to others, but I have found the solution. I always defined the mock by default, but then I invoked the mockImplementation() method on the instance of the ssh2-sftp-client library class, mocking the single functions. For each test I overwrite the return value of each function.

Here an example:

There seem to be multiple issues in your code:

1. The loop through files should be executed in the first then block itself.
2. sftp.fastGet returns a promise, hence its an asynchronous, operation, and executing asynchronous operations inside a forEach loop is not a good idea.

I would recommend updating your code with following changes:

What I believe is currently happening is that since you're not returning a Promise from your second then(), the third then() gets resolved immediately (with a value of undefined), which causes your connection to get terminated immediately (before your fastGet()s have time to finish).

To fix this, you would need to explicitly return a Promise from your second then() that only gets resolved once all of the files have been transferred. You may also want to consider rejecting that promise if at least one of the transfers fails.

The issue could be that your Lambda function does not have access to the SFTP host.

Is this SFTP server hosted outside of AWS or within your account ?

If it's hosted in the internet and not by you on AWS, you can do the following:

1. Create a public VPC [Skip this if you already have a public VPC]
2. Add a NAT Gateway in that public VPC [Also skip this if you already have a NAT Gateway]
3. In the VPC where your Lambda function resides, add a route table which redirects traffic to the NAT Gateway by setting the destination as 0.0.0.0/0 and the target as your NAT Gateway which we created in step 2.

You can find a detailed answer on how to add internet connection to your Lambda function by visiting this AWS article: How do I give internet access to my Lambda function in a VPC?

If the SFTP server is already hosted by you in AWS, you can simply add access to it by using AWS security group.

Check this article on how to connect Lambda with another AWS service (it's using RDS but you can follow the same logic with your SFTP server): Configuring a Lambda function to access Amazon RDS in an Amazon VPC

Also, make sure you set the Lambda timeout to be long enough to allow connecting and exchange data with an external SFTP server.

One good way to automate the running of scripts on EC2 is to use SSM Run Command. If you set up each EC2 instance correctly, then it becomes a managed instance and you can trigger the running of scripts across a fleet of EC2 instances, chosen by tags, for example.

In short, when you have a "chain" of promises (a.then(() => b.then(() =>...) and you want to do something at the end of them, you simply add that thing to the last .then callback.

However, if you're making multiple promises, you need to wait until all of them are finished, which you can do using Promise.all (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all):