socks5 | Full TCP | Proxy library

I want to set proxies to my crawler. I'm using requests module and Beautiful Soup. I have found a list of API links that provide free proxies with 4 types of protocols.

All proxies with 3/4 protocols work (HTTP, SOCKS4, SOCKS5) except one, and thats proxies with HTTPS protocol. This is my code:

I've configured shadowsocks system by running ss-server on VPS and ss-local on my client machine. Then I made a simple SOCKS5 client which connects to ss-local and resolve SOCKS request using C. All work well, when I run ss-tunnel instead of ss-local, my SOCKS5 client can't connect to ss-tunnel.

TCP connection terminates as soon as it established. I'm not sure what is the proper reason.

I am trying to implement a Socks5 server that could relay both HTTP and HTTPS traffic.

As the RFC1928 mentions, the following steps to establish a connection and forward the data must be taken :

1. Client sends a greeting message to the proxy.
2. Client & proxy authentication (assuming it is successful).
3. Client sends a request to the proxy to connect to the destination.
4. The proxy connects to the destination and sends back a response to the client to indicate a successful open tunnel.
5. The proxy reads the data from the client and forwards it to the destination.
6. The proxy reads the data from the destination and forwards it to the client.

So far, the proxy works as it should. It is able to relay HTTP traffic using its basic data forwarding mechanism. However, any request from the client to an HTTPS website will be aborted because of SSL/TLS encryption.

Is there another sequence/steps that should be followed to be able to handle SSL/TLS (HTTPS) traffic?

When using local forwarding we are free to pick destination port:

ssh -L 8000:10.10.1.11:80 user@192.168.56.102 - 80 is destination port - we explicitly specify which port to forward to.

Take a look at diagram in this article. If I understand correctly, in dynamic forwarding destination port is somewhat implicit (?):

ssh -D 8080 user@192.168.56.102 - 8080 is local host port, but still it somehow forwards to 80 for us.

Questions:

• How SOCKS5 proxy knows which port to forward traffic to?
• Does SOCKS5 proxy somehow sniff destination from the protocol of the traffic it proxies?
• Can that destination port be configured?
• Where is it configured - on my local machine, or remote that does final forwarding for us? Nowhere?
• Is it a predefined fixed set of protocols/ports? Smth like http=80 destination port?
• Any other internal details of forwarding you can add to clarify the picture.

thanks.

PS. Here https://unix.stackexchange.com/questions/213213/difference-between-local-port-forwarding-and-dynamic-port-forwarding#comment401063_213219 i found some hint to what i'm interested in, pity there are no details provided...

I want to understand the theory of exposing clients behind NATs via a public server, for example to use the client as a proxy (preferably SOCKS5). It would be ideal to find a ready solution but theory would good enough.

To point out what I mean. The client behind the NAT does not directly listen on TCP. It just opens up a TCP connection to the public server which should expose the client. The SOCKS5 proxy on client behind NAT is not directly exposed so the dialed TCP connection should somehow behave like a TCP listener.

If a client behind NAT connects to a public server, how do other users connect and how should I forward the requests?

I think basically it's the same technique that TeamViewer is using:

"When establishing a session, TeamViewer determines the optimal type of connection. After the handshake through our master servers, a direct connection via UDP or TCP is established in 70% of all cases (even behind standard gateways, NATs and firewalls). The rest of the connections are routed through our highly redundant router network via TCP or https tunneling. You do not have to open any ports in order to work with TeamViewer."

But how exactly?

As socks5 rfc says,

A UDP association terminates when the TCP connection that the UDP ASSOCIATE request arrived on terminates.

I wonder, doesn't "the TCP connection that the UDP ASSOCIATE request arrived on" just terminate when it timeouts? As there is no more data need to be sent in that TCP connection.

Should the client send meaningless data in that TCP connection just to keep it alive while it need the UDP association?

I need to get the proxy from the proxy.txt file:

Trying to build an iteration that goes through weblinks in order to extract the integer that is in the p tag that follows the p tag that contains the words "AGGREGATE AMOUNT BENEFICIALLY OWNED BY EACH REPORTING PERSON". The target pages have close enough patterns but are not identical. That AGGREGATE AMOUNT sentence is not formatted identically (sometimes capitalized, sometimes not, and spacing might not be identical). I included 3 examples below on the section I am trying to extract the integer value from.

Below are the steps I am trying to achieve:

1-Iterate through links Some links: https://www.sec.gov/Archives//edgar/data/29669/000119312521353247/d228949dsc13da.htm https://www.sec.gov/Archives//edgar/data/1482018/000119312521353071/d258656dsc13d.htm https://www.sec.gov/Archives//edgar/data/1590854/000119312521353064/d210475dsc13d.htm https://www.sec.gov/Archives//edgar/data/1021944/000119312521353050/d243715dsc13g.htm https://www.sec.gov/Archives//edgar/data/1609492/000119312521353003/d251372dsc13da.htm ...

2-Find p tag that contains partial words "AGGREGATE AMOUNT BENEFICIALLY", which can be small caps , titled, or all caps

  Aggregate Amount Beneficially Owned by Each Reporting Person:

3-Once the p tag found, continue to look in the following p tags until we get to the first p tag where the string (if exists), stripped of its commas is an integer (not empty).This can be the immediate p tag that follows, or the one after as in the code below.

10,927,100*

10,927,100 is the number I need, although it needs to be stripped from its commas (I know we can use the code below), but the string needs to be stripped from any other characters that are not numbers as well... int(shares.replace(',', '')

example1:

I'm using boost::asio to handle network communication between my program and remote server. To establish a connection with a server I do the following sequence of operations: