twitter_scraper | Scrap real time posts from twitter through the streaming api | REST library

you do not need anything more just do in this way

Three things for the first issue you encounter:

• first of all, every API has its limits and one like Twitter would be expected to monitor its use and eventually stop a user from retrieving data if the user is asking for more than the limits. Trying to overcome the limitations of the API might not be the best idea and might result in being banned from accessing the site or other things (I'm taking guesses here as I don't know what's the policy of Twitter on the matter). That said, the documentation on the library you're using states :

  With Twitter's Search API you can only sent 180 Requests every 15 minutes. With a maximum number of 100 tweets per Request this means you can mine for 4 x 180 x 100 = 72.000 tweets per hour. By using TwitterScraper you are not limited by this number but by your internet speed/bandwith and the number of instances of TwitterScraper you are willing to start.

• then, the function you're using, query_tweets_from_user() has a limit argument which you can set to an integer. One thing you can try is changing that argument and seeing whether you get what you want or not.

• finally, if the above does not work, you could be subsetting your time range in two, three ore more subsets if needed, collect the data separately and merge them together afterwards.

The second issue you mention might be due to many different things so I'll just take a broad guess here. For me, either setting pages=100 is too high and by one way or another the program or the API is unable to retrieve the data, or you're trying to look at a hundred pages when there is less than a hundred in pages to look for reality, which results in the program trying to parse an empty document.

It seems like this library lacks proper documentation, but the GitHub page provides some usage examples to help you get started.

Any number of things could cause the error to appear, but if the cause is not permanent, then retrying an occasional failed API call could make the script work alright.

According to the Tweepy docs the API client constructor accepts a retry_count parameter which defaults to 0. Try setting retry_count to something above 0 and see if your script is able to complete successfully, something like this:

Here def __init__(self, scraped_tweets_filename): self.fetched_tweets_filename = fetched_tweets_filename you should have def __init__(self, scraped_tweets_filename): self.fetched_tweets_filename = scraped_tweets_filename