responsibly | Mitigating Bias and Fairness of Machine Learning | Artificial Intelligence library

Because scrapy DOES import it, based on the project name in your config. All you need to do is turn your "counselor" folder into a module by adding a __init__.py. It doesn't need any content; you can just add a line with # for convenience.

Even in the age of virtual memory and terabyte RAM, streams are more efficient than strings. When I originally posted this question I accidentally didn't notice the HTTPSession object already had a outputStream member. So the first step is to escalate it. Add this to IHTTPSession:

It is important to run spider from scrapy crawl command so it will see whole project configuration correctly. Also, you need to make sure that custom middleware is listed in SPIDER_MIDDLEWARES dict and assigned order number. Main entry point for middleware is from_crawler method, which should receive crawler instance. Then you can write your middleware processing logic here by following rules mentioned here.

It doesn't find all of them because youi haven't requested him to do so. find will only return the first occurence. If you want to scrape all the

tags in the

Are you getting the correct data when you test in Scrapy shell ? It's worth trying out your selectors in scrapy shell before commiting them to a script.

I've not looked in detail into your CSS selectors but there's a lot of for loops for essentially all you need to do is loop over the tr's. So finding a CSS selector that gets you all the rows instead of looping over the whole table and work your way down is probably more efficient.

Update:

Since you asked about the for loop

Bootstrap gives you some built in media queries by which you can define your layout.

• Extra small devices (portrait phones, less than 576px) No media query since this is the default in Bootstrap
• Small devices (landscape phones, 576px and up) @media (min-width: 576px) { ... }
• Medium devices (tablets, 768px and up) @media (min-width: 768px) { ... }
• Large devices (desktops, 992px and up) @media (min-width: 992px) { ... }
• Extra large devices (large desktops, 1200px and up) @media (min-width: 1200px) { ... }

You can use mobile first approach to design your layout. Starting from 0px to 575px will be stated first than as you move higher in the resolution you can use media queries to handle your layout. The class col-sm-6 gives you ability to use 6 cols in a row. If you want that col to shrink at large desktop design to 4, you can add col-lg-4 besides col-sm-6. In this way your design will become responsive.

As far as the tab panel is consider, it depends upon your design. You can also apply nested technique or media list approach to handle your design on different resolution. Please consider bootstrap documentation for complete guidance. You can change col-sm-auto to col-sm-6 as well.

When the input is invalid, the call to

Your first object does not have the property title, trying to toLowerCase() that is throwing the error.

You can check if the property in object exists or not before using toLowerCase():

You wanna crawl this instead of this because it loads data from that one so you get empty lists because the data you want does not exists in that page.