get_tweets | get_latest_tweets

You're mutating a single dict, not adding to a list.

We can refactor your code to a handful of simpler functions that process tweepy? Tweets into dicts and others that yield processed tweet dicts for a given user.

Instead of printing the tweets at the end, you could now list.append them - or even simpler, just tweets = list(process_tweets_for_users(usernames)) :)

If you have the Tweet ID, you can get to the final URL like this:

Append it to https://twitter.com/twitter/statuses/ (+ the id)

Follow the HTTP redirect.

It sounds like you're not able to get the ID itself - that will be in the tweet.id value for each Tweet - currently you're just saving tweet.text

Pip defaults to installing Python packages to a system directory which requires root access.

Do you have root permissions? If so, please try to run sudo pip install.... Otherwise, consider installing the dependency to your home directory instead which doesn't require any special privileges:

-- edit -----

Ok So apparently you need to remove the : string typing in those lines:

Twitter's API returns the raw Tweet data without any parsing. This data includes shortened URLs because that's how the Tweet is represented. Twitter itself simply parses and displays the original URL. The link itself is even still the shortened one.

Tweet objects have an entities attribute, which provides an entities object with a urls field that is an array of URL objects, representing the URLs included in the text of the Tweet, or an empty array if no links are present. Each URL object includes a display_url field with the original URL pasted/typed into the Tweet and an indices field that is an array of integers representing offsets within the Tweet text where the URL begins and ends. You can use these fields to replace the shortened URL.

Three things for the first issue you encounter:

• first of all, every API has its limits and one like Twitter would be expected to monitor its use and eventually stop a user from retrieving data if the user is asking for more than the limits. Trying to overcome the limitations of the API might not be the best idea and might result in being banned from accessing the site or other things (I'm taking guesses here as I don't know what's the policy of Twitter on the matter). That said, the documentation on the library you're using states :

  With Twitter's Search API you can only sent 180 Requests every 15 minutes. With a maximum number of 100 tweets per Request this means you can mine for 4 x 180 x 100 = 72.000 tweets per hour. By using TwitterScraper you are not limited by this number but by your internet speed/bandwith and the number of instances of TwitterScraper you are willing to start.

• then, the function you're using, query_tweets_from_user() has a limit argument which you can set to an integer. One thing you can try is changing that argument and seeing whether you get what you want or not.

• finally, if the above does not work, you could be subsetting your time range in two, three ore more subsets if needed, collect the data separately and merge them together afterwards.

The second issue you mention might be due to many different things so I'll just take a broad guess here. For me, either setting pages=100 is too high and by one way or another the program or the API is unable to retrieve the data, or you're trying to look at a hundred pages when there is less than a hundred in pages to look for reality, which results in the program trying to parse an empty document.

The error occurs only in python 3.7, because of an incompatible change, async is a reserved keyword since this version.

The solution is to keep using python 3.6 until the library is adapted to the new version, there is an already closed issue:

Since it involves two different queries: {date: ...} and {date: ..., firm: ...} depending on the existence of firm in the input, you would have to check if firm is not None in get_tweets and execute the proper query.

For example:

The error comes from the fact that your results function only returns two things: the page results.html and its title; but in your page results.html you try to recover data that is not defined in the function but that comes from another function (login).

One possible solution would be: