DataSpider | A spider library of several data sources | Crawler library

I'm trying to create a scrapy script with the intent on gaining information on individual posts on the medium website. Now, unfortunately, it requires 3 depths of links. Each year link, and each month within that year and then each day within the months links.

I've got as far as managing to get each individual link for every year, every month in that year and every day. However I just can't seem to get scrapy to deal with the individual day pages.

I'm not entirely sure whether I'm confusing using rules and using functions with callbacks to get the links. There isn't much guidance on how to recursively deal with this type of pagination. I've tried using functions and response.follow by itself without being able to get it to run.

The parse_item function dictionary is required because several articles on the individual day pages have several different ways of classifying the title annoyingly. So i created a function to grab the title regardless of the actual XPATH needed to grab the title.

The last function get_tag is needed because on each individual article that is where the tags are to grab.

I'd appreciate any insight into how to get the last step and getting the individual links to go through the parse_item function, the shell o. I should say there are no obvious errors than I can see in the shell.

Any further information necessary just let me know.

Thanks!

CODE: