remax | opinionated yet dead-simple non | State Container library

The main problem you have are locators. 1 First, compare the locators I use and the ones in your code. 2 Second, Add explicit waits from selenium.webdriver.support import expected_conditions as EC 3 Third, remove unnecessary code.

You could use Array.prototype.reduce() method. Traverse the array and make parent as key and based on that key sum the occupiedStock.

You can play around with scale_fill_gradientn.

I think this gets you pretty close as a starting point:

Try the below approach using requests and json. I have created the script with the API URL which is fetched by inspecting the network calls in chrome which are triggering on page load and then creating a dynamic form data to traverse on each and every page to get the data.

What exactly script is doing:

1. First script will create a form data to query the the API call where page_no, query string and max values per facet(numbers of results to show) are dynamic where parameter page_no will increment by 1 upon completion of each traversal.

2. Requests will get the data from the created form data and URL using POST method which will then pass to JSON to parse it and load in json format.

3. Then from the parsed data script will traverse on the json object where data is actually present.

4. Finally looping on all the batch of each and every page data one by one and printing.

Right now script is displaying few information you can access more information form the json object like i have done below.

Maybe you should change "get_data('1')" by "get_data(str(page))"?

First thing, if you use from bs4 import BeautifulSoup, don't use import bs4 too.

Second, write soup = BeautifulSoup(page,'html.parser)

Then use price = soup.find_all('h3',{'class':'price})

After this, you should have in "price" all the prices, but you still need to refine, as in that form you will copy all that code from the h3s.

EDIT

You can use urllib to save the image on your computer from the url using this code:

I'm not exactly sure what your issue is since I'm not manually testing your code, but assuming you have the proper xpaths and id's for your elements, I would guess that you're trying to get a .text attribute from a list object (a list of web elements). So you need to add the .text attribute to each individual element. For example, if the xpath in

name = driver.find_element_by_xpath('''//*[@id="MainContent"]/div[1]/div[2]/div/div[1]/div[1]/div[1]/div[1]/h2/a''') agent_name.append(name.text)

finds all the name elements on the page for 'Joe Smith, Bob Jones, etc...', you want to add loop that allows you to add the .text attribute to each element. For example:

names = driver.find_element_by_xpath('''//*[@id="MainContent"]/div[1]/div[2]/div/div[1]/div[1]/div[1]/div[1]/h2/a''') for name in names: agent_name.append(name.text)

This should at least populate your lists. If this doesn't work I would double check that the things you're trying to scrape are indeed text attributes in the html (ie. not images), and ensure your element identifiers are correct and that you're following the recommendations/syntax in the docs for python selenium.