Retweeter | Retweeter accesses a Twitter account

you need one more join on users table for getting original poster name

You can fine-tune BERT, and you can use BERT to do retweet prediction, but you need more architecture in order to predict if user i will retweet tweet j.

Here is an architecture off the top of my head.

At a high level:

1. Create a dense vector representation (embedding) of user i (perhaps containing something about the user's interests, such as sports).
2. Create an embedding of tweet j.
3. Create an embedding of the combination of the first two embeddings together, such as with concatenation or hadamard product.
4. Feed this embedding through a NN that performs binary classification to predict retweet or non-retweet.

Let's break this architecture down by item.

To create an embedding of user i, you will need to create some kind of neural network that accepts whatever features you have about the user and produces a dense vector. This part is the most difficult component of the architecture. This area is not in my wheelhouse, but a quick google search for "user interest embedding" brings up this research paper on an algorithm called StarSpace. It suggests that it can "obtain highly informative user embeddings according to user behaviors", which is what you want.

To create an embedding of tweet j, you can use any type of neural network that takes tokens and produces a vector. Research prior to 2018 would have suggested using an LSTM or a CNN to produce the vector. However, BERT (as you mentioned in your post) is the current state-of-the-art. It takes in text (or text indices) and produces a vector for each token; one of those tokens should have been the prepended [CLS] token, which commonly is taken to be the representation of the whole sentence. This article provides a conceptual overview of the process. It is in this part of the architecture that you can fine-tune BERT. This webpage provides concrete code using PyTorch and the Huggingface implementation of BERT to do this step (I've gone through the steps and can vouch for it). In the future, you'll want to google for "BERT single sentence classification".

To create an embedding representing the combination of user i and tweet j, you can do one of many things. You can simply concatenate them together into one vector; so if user i is an M-dimensional vector and tweet j is an N-dimensional vector, then the concatenation produces an (M+N)-dimensional vector. An alternative approach is to compute the hadamard product (element-wise multiplication); in this case, both vectors must have the same dimension.

To make the final classification of retweet or not-retweet, build a simple NN that takes the combination vector and produces a single value. Here, since you are doing binary classification, a NN with a logistic (sigmoid) function would be appropriate. You can interpret the output as the probability of retweeting, so a value above 0.5 would be to retweet. See this webpage for basic details on building a NN for binary classification.

In order to get this whole system to work, you need to train it all together end-to-end. That is, you have to get all the pieces hooked up first and train it rather than training the components separately.

Your input dataset would look something like this:

Protected Retweets are shown as part of the count you see, but you're unable to obtain them or their Retweeters through the API (unless that protected account follows you).

To outline this, you can see that https://twitter.com/AmericaTalks/status/1090408203882360832 has 7 Retweets right now. If you click to see who Retweeted, it'll show 6 accounts, and at the bottom, it'll say "1 user has asked not to be shown in this view. Learn More". The API will also return only the 6 Retweet(er)s.

Note, in your code, you define count, but use countj. This will result in a NameError.
Also, API.retweets returns a list of Status objects, so you can just do len(api.retweets(1090392302130888704)), instead of looping through them to count them.

Unfortunately, it looks like the Tweet you were trying to investigate has been deleted, so it's impossible to check this now. However, the issue in your code here is that you need to send the Tweet ID as a string, not as an integer. So, for example:

Your schema is not mapped properly ... please see these posts if you want to manually construct schema (which is recommended if the data doesn't change):

PySpark: How to Update Nested Columns?

https://docs.databricks.com/_static/notebooks/complex-nested-structured.html

Also, if your JSON is multi-line (like your example) then you can ...

1. read json via multi-line option to get Spark to infer schema
2. then save nested schema
3. then read data back in with the correct schema mapping to avoid triggering a Spark job

Both the purrr and jsonlite packages have a flatten() function. The jsonlite version would probably work best for your purposes, as I'm assuming the Twitter API returns a JSON object (and the purrr:flatten only removes one layer of recursion at a time).

Info here: https://rdrr.io/cran/jsonlite/man/flatten.html

Twitter4J is a Java client library that interfaces with the Twitter REST API. To understand the right call to use it's best to understand the underlying REST API.

Looking at the Twitter Rest API we can see an API that returns a list of users who have retweeted a particular tweet, GET statuses/retweeters/ids.

In your code the Twitter4J API you're using, getRetweets(), does not return the IDs of users who retweeted.

Looking at the Twitter4J Twitter4J API docs we find getRetweeterIds(statusId) that returns the list of user IDs that retweed a particular tweet indicated by statusId.

The issue is that you are running ids.length async calls and those will finish some time in the future. You have to render your page only when they are all done. But, your for loop is synchronous so you are calling res.render() before any of them have finished. In addition, your T.get() calls may finish in any order (if that matters).

I would normally use promises for coordinating multiple asynchronous operations since it is a very, very good tool for that. But, if you aren't using promises, here's a simple technique to test when you have all your results back: