financial | dependency TypeScript/JavaScript financial library

Reshape the data to wide format twice, once for position and the other for prestige_1, and join the two results.

So if the key for os.getenv() is invalid, it returns the default values that you pass as the second parameter. If you don't set this default value, it returns a None. Possible Fixes:

As of 6.0 you can simply append the CSVs to same table one by one given the table has designated timestamp and partitioned it will work.

If your CSVs are huge I think batching them in transactions with few million rows will be better than offloading billions at once.

Depending of how much data you have and your box memory you need to partition in a way that single partition fits memory several times. So you choose if you want daily or monthly partitions.

Once you decide with partitioning you can speed up the upload if you able to upload day by day batches (or month by month) from all CSVs.

You will not need to rebuild the table every time you add an instrument, table will be rewritten automatically partition by partition when you insert records out of order.

The time it takes to read a document over the internet is really not related to the speed of your computer, at least in most cases. The most important determinant is the speed of your internet connection. Another important determinant is the speed with which the remote server responds to your request, which will depend in part on how many other requests the remote server is currently trying to handle.

It's also possible that the slow-down is not due to either of the above causes, but rather to measures taken by the remote server to limit scraping or to avoid congestion. It's very common for servers to deliberately reduce responsiveness to clients which make frequent requests, or even to deny the requests entirely. Or to reduce the speed of data transmission to everyone, which is another way of controlling server load. In that case, there's not much you're going to be able to do to speed up reading the requests.

From my machine, it takes a bit under 30 seconds to download the 12MB document. Since I'm in Perú it's possible that the speed of the internet connection is a factor, but I suspect that it's not the only issue. However, the data transmission does start reasonably quickly.

If the problem were related to the speed of data transfer between your machine and the server, you could speed things up by using a streaming parser (a phrase you can search for). A streaming parser reads its input in small chunks and assembles them on the fly into tokens, which is basically what you are trying to do. But the streaming parser will deal transparently with the most difficult part, which is to avoid tokens being split between two chunks. However, the nature of the SEC document, which taken as a whole is not very pure HTML, might make it difficult to use standard tools.

Since the part of the document you want to analyse is well past the middle, at least in the example you presented, you won't be able to reduce the download time by much. But that might still be worthwhile.

The basic approach you describe is workable, but you'll need to change it a bit in order to cope with the search strings being split between chunks, as you noted. The basic idea is to append successive chunks until you find the string, rather than just looking at them one at a time.

I'd suggest first identifying the entire document and then deciding whether it's the document you want. That reduces the search issue to a single string, the document terminator (\n\n; the newlines are added to reduce the possibility of false matches).

Here's a very crude implementation, which I suggest you take as an example rather than just copying it into your program. The function docs yields successive complete documents from a url; the caller can use that to select the one they want. (In the sample code, the first matching document is used, although there are actually two matches in the complete file. If you want all matches, then you will have to read the entire input, in which case you won't have any speed-up at all, although you might still have some savings from not having to parse everything.)

Just concatenate the quotes to your column

You may use case_when with grepl and a regex alternation:

I was able to resolve this issue by putting the paragraph contents inside a wordwrap function in php. Example: wordwrap('content string',100)

If version = "ACT" an DATE is not current year i.e 07-2020 to 06-2021 then replace ACT with ACTA

You want a case expression:

Try this: