ftp-client | A rust FTP client implementation | FTP library

Did not found the solution.

The ugly solution was to downloaded the whole directory from SFTP instead of downloading file one by one.

I figured it out. Not sure where exactly the problem was but this will successfully pull a file from S3 and then upload to SFTP server:

So, I managed to understand what was going on, (thanks to a packet sniffing tool called wireshark), it was basically a typical case of an encoding problem. For the ftp clients, what was going on is basically ftp only receives latin-1 encoding, but node converts it to utf8 behind the scenes. So for example my terminal was set to utf-8, so it kept sending utf-8 messages to the the latin-1 server, which of course it didn't know how to interpret.

But I discovered a command (in unix) that allows you to convert utf-8 to latin-1, and you use use it like this on the terminal >luit -encoding ISO-8859-1 ftp this translates utf-8 to latin-1, and then it succeeded in listing the file.

Now for the case in node-js, the only thing I had to do was to stop node-js from converting the string to UTF-8, and to convert it to latin-1 instead. the Library however had hardcoded the utf-8 string, so, I had to options, modify the library or search for an alternative that supports such encoding.

Surprisingly, I had to do the later one, since I didn't have luck on finding another ftp library, I had the good fortune of finding @icetee/ftp, which is a fork of the original ftp package. Basically I had to make a fork and modify this line. And now everything works for my server, hopefully, there will be a setting on the future for this package. And that's how I solved it.

Worked with version 6.0.0 of ssh2-sftp-client

The RFCs stand as a final approved documentation. In your case the HTTP is under the HTTP Working Group so some new features which some browsers already support are being discussed in this group. Expanding the idea, some security headers present in HTTP may be from other groups and just referenced in HTTP RCFs. The Content Security Policy is documented in the RFC 7762 not that it's tagged as informational.

1. Each area has its Working groups, in this case HTTP is nested in ART (Applications And Real-Time Area). Each of those groups compile, revise and publish new specifications. As an example you can see HTTP(httpbis) charter

2. There's two options, implement based on the RCFs and its references or follow the Working Group directives and references. Using only RFCs is more secure and interoperable but will eventually be outdated until a new RFC is published.

3. The only way is to implement what is documented under the RFCs. It's part o the internet concept, new features or standards will take a while to be fully documented and it's up to developer research and implement those.

Cloud Run works only until you have a HTTP request active.

You proably don't have an active request during this on Cloud Run, as outside of the active request the CPU is throttled.

Best for this pipeline is Cloud Workflows and regular Compute Engine instances.

You can setup a Workflow to start a Compute Engine for this task, and stop once it finished doing the steps.

I am the author of article: Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows it will guide you how to setup.

Executing the Workflow can be triggered by Cloud Scheduler or by HTTP ping.

I hope it will be useful to others, but I have found the solution. I always defined the mock by default, but then I invoked the mockImplementation() method on the instance of the ssh2-sftp-client library class, mocking the single functions. For each test I overwrite the return value of each function.

Here an example:

Julien, all the responses from the server follow a pattern.. you need to wait for a \r\n sequence from the server. You need to filter TELNET protocol escape sequences (that start with the byte 0xff or 0xfe, I don't remember from my head) and the codes must be always at the beginning of a line. Also the numbers with a - instead of a space indicate that the message is larger and you should expect another line (the last line of each message has a numeric code followed by a space)

This is essential to maintain yourself synchronized with the server.... or you'll start receiving responses to different commands than the one you think.

From the output it is not clear what you try to do, as the only thing you show is the login part to the server.

Your command lines also have (this is mandatory by the protocol) to end in a sequence \r\n (in that order) you cannot do else, or you can reach a server that doesn't understand you.

Check RFC-959 - File Transfer Protocol for details on how the ftp protocol works. The transfers are normally handled in a new, different TCP connection, so you normally have to manage the control connection plus a series of data transfers that run in parallel with it.

Simply search the dependency on google or go to https://mvnrepository.com/artifact/org.mockftpserver/MockFtpServer/2.7.1 and click under gradle tab.

Meanwhile, to answer your question, this should work