random_user_agent | get list of user agents | Crawler library

Your IP is still the same for all the requests. You could look into using python requests with tor which of course takes a bit longer though, because the request get's routed over TOR. I am not familiar with applying proxying over TOR with selenium but I bet there are a lot of tutorials you can find.

Walmart probably has this captcha mechanism in place for a reason though, so maybe look for another option of getting the data.

ReadTimeout exceptions are commonly caused by the following

1. Making too many requests in a givin time period
2. Making too many requests at the same time
3. Using too much bandwidth, either on your end or theirs

It looks like your are making 1 request every 2 seconds. For some websites this is fine, others could be call this a denial-of-service attack. Google for example will slow down or block requests that occur to frequently.

Some sites will also limit the requests if you don't provide the right information in the header, or if they think your a bot.

To solve this try the following:

1. Increase the time between requests. For Google, 30-45 seconds works for me if I am not using an API
2. Decrease the number of concurrent requests.
3. Have a look at the network requests that occur when you visit the site in your browser, and try to mimic them.
4. Use a package like selenium to make your activity look less like a bot.

The error might be that json_data is a string and not a dict type as json.dumps(str(soup)) returns a string.Since json_data is string, we cannot do json_data['Results'] and to access any element of string, we need to pass the index and hence the error.

To get Results from the response, the code is shown below: