RSelenium | An R client for Selenium Remote WebDriver | Functional Testing library

The page does a POST request that you can mimic/simplify. To keep dynamic you need to first grab an api key and application id from a source js file, then pass those in the subsequent POST request.

In the following I simply extract the urls from each request. I set the querystring for the POST to have the max of 20 results per page. After an initial request, in which I retrieve the number of pages, I then map a function across the page numbers, extracting urls from the POST response for each; altering the page param.

You end up with a list of urls for all the projects you can then visit to extract info from; or, potentially make xmlhttp requests to.

N.B. Code can be re-factored a little as tidy up.

The solution could be trycatch

Below an example.

I did it.

Below a possible solution.

tl;dr The problem is not the css selectors. It's the encoding. Specify encoding = 'latin1'

Below a possibile solution.

I was able to fix this thanks to this info I found. You can visit the link for more detailed information.

rsDriver has a boolean argument check that is TRUE by default. Its official description:

If TRUE check the versions of selenium available and the versions of associated drivers (chromever, geckover, phantomver, iedrver). If new versions are available they will be downloaded.

This downloads both chromedriver_mac64.zip and chromedriver_mac64_m1.zip. The chromedriver installed from the m1 file on Macs. What I did to fix the problem was go to

~/Library/Application Support/binman_chromedriver/mac64/89.0.4389.23

and then delete the m1 file and chromedriver. Then unzip chromedriver_mac64.zip to get the correct chromedriver. Go back to R and run rsDriver with the argument check set to false:

This site calls an API to get the list of manufacturers: http://www.cdms.net/labelssds/Home/ManList?Keys=

On the products page, it also uses another API with the manufacturer ID, for example: http://www.cdms.net/labelssds/Home/ProductList?manId=537

You just need to loop through the Lst array and append the result to a dataframe. For instance, the following code get all the products for the first 5 manufacturers :

Set the connection parameters correctly for the operating system you are using and the browser. If the image is based on chrome browser and the default configuration of the driver is firefox.

To change this you will need to add the browserName = "chrome". I suggest to always add the server's IP address for clarity so, add remoteServerAddr = <>

The way to find out what are the default parameters for the driver is to inspect the instantiated remote driver remDr will show you what is the default configuration for the driver.