arbitrage | Javascript library that introduces R idioms | Runtime Evironment library

Copy of my answer on Chess.SE, in case someone is looking here for an answer.

Yes, it's possible to obtain the data you want. Chess.com has a REST API which is described in the following news post:

https://www.chess.com/news/view/published-data-api

You can use the following URL to get a list of monthly archives of a players games:

That's because the website is using JavaScript to display these details and BeautifulSoup does not interact with JS on it's own.

First try to find out if the element you want to scrape is present in the page source, if so you can scrape, pretty much everything! In your case the button/span tag's were not in the page source(meaning hidden or it's pulled through a script)

No tag in the page source :

So I suggest using Selenium as the solution, and I tried a basic scrape of the website.

Here is the code I used :

You are not really trying to reshape (that would be adding columns based on the content of rows), here you just want to summarize the tag values for each company. So that can be done easily with dplyr:

Nice application! You can either write out all of your discrete equations yourself with m.options.IMODE=3 or else let Gekko manage the time dimension for you. When you include an objective or constraint, it applies them to all of the time points that you specify. With m.options.IMODE=6, there is no need to add the set indices in Gekko such as [t]. Here is a simplified model:

We first explain why the code in the question got the results that it did.

Note that there is an implicit constraint of x >= 0.

Regarding the subject that refers to a equality constraint there are no equality constraints in the code shown in the question.

Regarding the minimum, in the lp arguments > means >= so clearly x=0 is feasible. Given that all components of the objective vector are positive it leads to a minimum of 0. From ?lp

const.dir: Vector of character strings giving the direction of the constraint: each value should be one of "<," "<=," "=," "==," ">," or ">=". (In each pair the two values are identical.)

Regarding the maximum, there are no upper bound constraints on the solution and the objective vector's components are all positive so there is no finite solution. Whether the linear program was successful or not should always be shown prior to displaying the solution and if it was unsuccessful then the solution should not be displayed since it has no meaning.

Regarding the code, cbind already produces a matrix so it is pointless to convert it to a data frame and then back to a matrix. Also the objective can be expressed as a plain vector and the rhs of the constraints can be written as a scalar which will be recycled to the appropriate length. We can write the code in the question equivalently as:

Roughly - you need to do a df.groupby(["REF", "Product"]).count(), this will give you a table that just contains ref, product, and a count. From there you can project/join in RAW_TRADE_TYPE - one of "Flat" "Spread" or "Trio", using where() or select().

You can join() this to the original table. From there you have all the information you need to calculate the columns as you describe.

Ok, there is a logic problem, not with the code.

Lets take a closer look:

Quite a few questions there!

"Why does Selenium with Python have such a delay to issue commands"

I don't have any direct referencable evidence for this but the selenium delay is probably down to the amount of work it has to do. It was created for testing not for performance. The difference in that is that the most valuable part of a test is that MUST run reliably, if it's slower and does more checks in order to make it more reliable than so be it. You still get a massive performance gain over manual testing, and that's what we're replacing.

This is where you'll need the reliability - and i see you mention this need in your question too. It's great if you have a trading script that can complete an action in <1s - but what if it fails 5% of the time? How much will that cause problems?

Selenium also needs to render the GUI then, depending on how you identify your objects it needs to scan the entire DOM for what you want. Generic locators will logically take longer.

Selenium's greater power is the simplicity and the provided ability synchronisation. There's lots of talk in lots of SO articles about webdriverwait (which is great for in-page script sync), but there's also good recognition of page load wait-times. i.e. no point pushing the button before all the source is loaded.

"Why does Puppeteer seem to have a lower delay when interacting with the same browser?" Puppeteer works with chrome devtools - selenium works with the DOM. So the two are interacting in a different way. Obviously, puppeteer is only for chrome, but selenium can go to almost any browser. It might not be a problem for you if you don't care about cross-browser capability.

However - how much faster is it really?

In your execution timings, you might also want to factor in the whole life cycle of what you aim to do - namely, include: browser load time, and page render time. From your story, you want to buy or sell something. So, you kick off the script - how long does it take from pressing GO to the end result. You most likely will need synchronisation and that's where puppeteer might not be as strong (or take you much longer to get a handle on).

You say you want to "interact with any webpage with as little delay as possible", but you have to get to the page first and get it in the right state. I wouldn't be concerned about milliseconds per action. Kicking open chrome isn't the quickest action in itself and your overall millisecond test becomes >seconds to minutes.

When you have full selenium script taking 65 seconds and a puppeteer script taking 64 seconds, the overall consideration changes :-)

Some other options to think about if you really need speed:

• Try running selenium headless - no gui, less resources, runs faster. (google it) :-)
• Try other browsers
• If you're all about speed - Drop the GUI entirely. Create an API script or start to investigate the use of performance scripts (for example jmeter or loadrunner).
• Another option that isn't selenium or puppeteer is javascript directly in the browser
• Try your speed tests with other free web tools like cypress or Microsoft's playwright - they interact with JS in the browser directly and brag about some excellent inbuilt stability features.

If it was me, and it was about a single performant action - my first port of call would be doing it at the API level. There's lots of resources and practice sites out there to help get you started and there's no browser to render, there's barely anything more than the network delay. That's the real way to keep it as a sub-second script.

Final thought, when you ask "can it be improved": if you have some code, there may be other ways to make it faster. Share your script so far and i'm sure folks will love to chip in their 2 cents on how you can streamline it :-)

This is what I would do (not seeing the full code it is hard to write a full solution (also this will eat up a lot of memory, but again - in order to optimize this I would need to see the code; what I write has a benefit of being simple to implement and not require locking).

So the recommendation is:

1. create cr2_vec = [copy(cr1) for i in 0:499]
2. in outer loop write cr2 = cr2_vec[k]
3. Then do all the processing on cr2
4. after the @threads loop finishes take cr2_vec, which will have updated data frames and from each data frame of this vector take the columns that are needed and add it to your original cr1 data frame.

A more advanced solution would be not to use a vector of data frames, but just a single data frame, as you do, but after the computing is done use a lock and within a lock update the global cr1 with only the computed columns (you need to use lock to avoid race condition).

EDIT

An example of a more efficient implementation: