TwitterScraper | Twitter Scraper - Scrape tweets for a user or a # hashtag | Scraper library

Where the bottle-neck is depends on a bunch of different variables.

If you're doing a lot of data manipulation, but you don't have a lot of CPU time allocated to the program (i.e. there are too many users for your processor to handle), it could slow down there.

If you don't have sufficient memory, and you're trying to parse and return a lot of data, it could slow down there.

Because you're also talking to Twitter, whatever the bandwidth restrictions are between your server and the twitter server will affect the speed at which you can retrieve results from their API, and so the time it takes your program to respond to a user.

There's also the connection between yourself and the user. If that's slow, it could affect your program.

Since dfs is a list of tuples, with each tuple being (DataFrame, str), you only want to concatenate the first of each element of dfs.

You may achieve this using:

Three things for the first issue you encounter:

• first of all, every API has its limits and one like Twitter would be expected to monitor its use and eventually stop a user from retrieving data if the user is asking for more than the limits. Trying to overcome the limitations of the API might not be the best idea and might result in being banned from accessing the site or other things (I'm taking guesses here as I don't know what's the policy of Twitter on the matter). That said, the documentation on the library you're using states :

  With Twitter's Search API you can only sent 180 Requests every 15 minutes. With a maximum number of 100 tweets per Request this means you can mine for 4 x 180 x 100 = 72.000 tweets per hour. By using TwitterScraper you are not limited by this number but by your internet speed/bandwith and the number of instances of TwitterScraper you are willing to start.

• then, the function you're using, query_tweets_from_user() has a limit argument which you can set to an integer. One thing you can try is changing that argument and seeing whether you get what you want or not.

• finally, if the above does not work, you could be subsetting your time range in two, three ore more subsets if needed, collect the data separately and merge them together afterwards.

The second issue you mention might be due to many different things so I'll just take a broad guess here. For me, either setting pages=100 is too high and by one way or another the program or the API is unable to retrieve the data, or you're trying to look at a hundred pages when there is less than a hundred in pages to look for reality, which results in the program trying to parse an empty document.

I had to make some guesses based on the info you gave. But this is what I did:

Your problem appears to be the line:

It should be :