ipproxy | 代理IP提取工具 - A simple tool to crawl proxy ip | Proxy library

The requests.get call will get you the whole file anyway. You'd need to implement your own HTTP code, down to the socket level, to be able to process the content as it gets in, in a plain HTTP Get method.

The only way of getting partial results and slice the download is to add HTTP "range" request headers, if the server providing the file support then. (requests can let you set these headers).

The good news is that requests can do that for you under the hood - you can set stream=True parameter when calling requests, and it even will let you iterate the contents line by line. Check the documentation on that part.

Here is more or less what requests does under the hood so that you can get your contents line by line:

It will get reasobale sized chunks of your data, - but certainly not equest one line at a time (think ~80 bytes versus 100.000 bytes), because otherwise it'd need a new HTTP request for each line,and the overhead for each request is not trivial, even if made over the same TCP connection.

Anyway, as CSV being a text format, neither requests nor any other software could know the size of the lines, and even less the exact size of the "next" line to be read - before setting the range headers accordingly.

So, for this to work, ther have to have to be Python code to:

• accept a request for a "new line" of the CSV if there are buffered text lines, yield the next line,
• otherwise make an HTTP request for the next 100KB or so
• Concatenate the downloaded data to the remainder of the last downloaded line
• split the downloaded data at the last line-feed in the binary data,
• save the remainder of the last line
• convert your binary buffer to text, (you'd have to take care of multi-byte character boundaries in a multi-byte encoding (like utf-8) - but cutting at newlines may save you that)
• yield the next text line