Word-Embedding | Flair pre-train Word Embedding | Natural Language Processing library

Okay let's do this.

First you need to understand that BERT has 13 layers. The first layer is basically just the embedding layer that BERT gets passed during the initial training. You can use it but probably don't want to since that's essentially a static embedding and you're after a dynamic embedding. For simplicity I'm going to only use the last hidden layer of BERT.

Here you're using two words: "New" and "York". You could treat this as one during preprocessing and combine it into "New-York" or something if you really wanted. In this case I'm going to treat it as two separate words and average the embedding that BERT produces.

This can be described in a few steps:

1. Tokenize the inputs
2. Determine where the tokenizer has word_ids for New and York (suuuuper important)
3. Pass through BERT
4. Average
5. Cosine similarity

First, what you need to import: from transformers import AutoTokenizer, AutoModel

Now we can create our tokenizer and our model:

The problem is in your LSTM layer. It is not returning a sequence of the same length. You must set return_sequences=True when stacking layer so that the second layer has a three-dimensional sequence input. After adding return_sequences = True parameter in your LSTM layer, it will give you around 90% accuracy for sure.

There's a couple of issues with the matching of topics, so you'll need to expand the match_topic() method I used, but I added some logic to see what didn't match at the end.

The results variable contains a list of dict which you can easily save as a JSON file.

Check the inline comments for the reasoning of the logic I used.

Sidenote:

I would slightly restructure the JSON if I were you. Putting the topic as a key/value pair under the GPO and CAP keys makes more sense to me than having a Topic key with a separate GPO and CAP key/value pair...

Structure wise, both Dense layer and Embedding layer are hidden layers with neurons in it. The difference is in the way they operate on the given inputs and weight matrix.

A Dense layer performs operations on the weight matrix given to it by multiplying inputs to it ,adding biases to it and applying activation function to it. Whereas Embedding layer uses the weight matrix as a look-up dictionary.

The Embedding layer is best understood as a dictionary that maps integer indices (which stand for specific words) to dense vectors. It takes integers as input, it looks up these integers in an internal dictionary, and it returns the associated vectors. It’s effectively a dictionary lookup.

This is not really an AllenNLP issue since you are using a tensorflow-based implementation of ELMo.

That said, I think the problem is that ELMo embeds tokens, not characters. You are getting 48 embeddings because the string has 48 tokens.

Technically the input will be 1D, but that doesn't matter.

The internal architecture of your neural network will take care of recognizing the different words. You could for example have a convolution with a stride equal to the embedding size.

You can flatten a 2D input to become 1D and it will work fine. This is the way you'd normally do it with word embeddings.

I don't think performing Random Forest classifier on the 3-dimensional input will be possible, but as an alternative way, you can use sentence embedding instead of word embedding. Therefore your input data will be 2-dimensional ((n_samples, n_features)) as this classifier expected.
There are many ways to get the sentence embedding vector, including Doc2Vec and SentenceBERT, but the most simple and commonly used method is to make an element-wise average over all the word embedding vectors.
In your provided example, the embedding length was considered as 3. Suppose that the sentence is "I like dogs". So the sentence embedding vector will be computed as follow:

Are you taking into account ConceptNet Numberbatch's format? As shown in the project's GitHub, it looks like this:

/c/en/absolute_value -0.0847 -0.1316 -0.0800 -0.0708 -0.2514 -0.1687 -...

/c/en/absolute_zero 0.0056 -0.0051 0.0332 -0.1525 -0.0955 -0.0902 0.07...

This format means that fille will not be found, but /c/fr/fille will.

This would depend on a number of factors, such as the precise SQL DB you intend to use and how you store this embedding. For instance, PostgreSQL allows to store query and retrieve JSON variables ( https://www.postgresqltutorial.com/postgresql-json/ ) ; Other options as SQLite would allow to store string representations of JSONs or pickle objects - that would be OK for storing, but would make querying the elements inside the vector impossible.