urllib | Request HTTP URLs in a complex world | HTTP library

The time it takes to read a document over the internet is really not related to the speed of your computer, at least in most cases. The most important determinant is the speed of your internet connection. Another important determinant is the speed with which the remote server responds to your request, which will depend in part on how many other requests the remote server is currently trying to handle.

It's also possible that the slow-down is not due to either of the above causes, but rather to measures taken by the remote server to limit scraping or to avoid congestion. It's very common for servers to deliberately reduce responsiveness to clients which make frequent requests, or even to deny the requests entirely. Or to reduce the speed of data transmission to everyone, which is another way of controlling server load. In that case, there's not much you're going to be able to do to speed up reading the requests.

From my machine, it takes a bit under 30 seconds to download the 12MB document. Since I'm in Perú it's possible that the speed of the internet connection is a factor, but I suspect that it's not the only issue. However, the data transmission does start reasonably quickly.

If the problem were related to the speed of data transfer between your machine and the server, you could speed things up by using a streaming parser (a phrase you can search for). A streaming parser reads its input in small chunks and assembles them on the fly into tokens, which is basically what you are trying to do. But the streaming parser will deal transparently with the most difficult part, which is to avoid tokens being split between two chunks. However, the nature of the SEC document, which taken as a whole is not very pure HTML, might make it difficult to use standard tools.

Since the part of the document you want to analyse is well past the middle, at least in the example you presented, you won't be able to reduce the download time by much. But that might still be worthwhile.

The basic approach you describe is workable, but you'll need to change it a bit in order to cope with the search strings being split between chunks, as you noted. The basic idea is to append successive chunks until you find the string, rather than just looking at them one at a time.

I'd suggest first identifying the entire document and then deciding whether it's the document you want. That reduces the search issue to a single string, the document terminator (\n\n; the newlines are added to reduce the possibility of false matches).

Here's a very crude implementation, which I suggest you take as an example rather than just copying it into your program. The function docs yields successive complete documents from a url; the caller can use that to select the one they want. (In the sample code, the first matching document is used, although there are actually two matches in the complete file. If you want all matches, then you will have to read the entire input, in which case you won't have any speed-up at all, although you might still have some savings from not having to parse everything.)

Try using. You will get the response. Certain websites are secured and only respond to certain user-agents only.

Replace:

There are anti-scraping measures. If those do not affect you then you can use css classes to target the li elements to loop over, and the title/values for each specification:

Perhaps there is a problem with your path? (apologies I'm not on windows to test). From memory, Windows paths use \ characters instead of /. Additionally, you may need two backticks after the drive path (C:\\).

c:\\Users\91940\AppData\Local\...

Based on the provided target data structure above, you appear to be using a dictionary. It isn't clear what you would like your keys to be so I would probably suggest using a list/array.

I would suggest a slightly different way to dissect the problem.One potential implementation would be to iterate over each row (paragraph

of the table (div

From here, if the link(s) are present you could then query the external data source (or read from a different location on the page) to collect the respective data. In the example below, I choose to do this in a different iteration of data to help make the code a bit more readable.

I have not used the BeautifulSoap4 library before. I apologise if my solution isn't the most elegant regarding the libraries usage.

EDIT:

Minimal working code based on @Weeble answer.

It uses yarl with encoded=True to stop requoting %3A to :

Couple of things. You are trying to mix bs4 syntax with selenium which is causing your current error. Additionally, you are targeting potentially dynamic values. Finally, there are anti-scraping measures which may later impact on your work.

Ignoring the last, a more robust, syntax appropriate version, might be:

Here's an example of how to split a URL and get the query parameters: