job-hunter | Job scraper and applier for indeed.com | Bot library

2+2=4 under a given set of assumptions and conditions. Browsers and scrapers unfortunately aren't that predictable, with random delays, page throttling, changing pages, varying support levels for different technologies, etc.

In your current case the reason for the window_opened_by error could have been not having Capybara.default_max_wait_time set long enough (how long Capybara will wait for the window to open), however if you try the search manually you'll see that indeed no longer opens the job description in a new window if the current window is wide enough to show it in a right panel. Basically the code you're trying to use is no longer fully compatible with indeed.com due to changes in how indeed.com works. You could fix this by setting the drivers window size to a size where indeed.com will always open a new window, or by setting the window size big enough job descriptions open on the same page and rewriting the code to not look for a new window.

As for the no '#fj' issue, the easiest way to debug that is to put

Required portions/redirects of the page are loaded from 'indeed.com' rather than 'www.indeed.com' - Change your capybara-webkit configuration to allow indeed.com.

Additionally, if you want to know what the driver is actually seeing, you're generally better off using save_and_open_screenshot rather that save_and_open_page (The latter will show you the html structure but not how capybara-webkit has rendered it)

click_link takes parameters to identify the link it's going to click on. To click on the "date" link in the div with id of refineresults you would do

Your code only allows the URL http://www.indeed.com , but that URL redirects to https://www.indeed.com and also hits http://indeed.com. Therefore your page load is being blocked. Change to config.allow_url("indeed.com") and it should be able to find the input.