whispered | use development environment , you can use elasticsearch | Continuous Deployment library

The [CLS] Token somehow represents the input sequence, but how exactly is difficult to say. The language is of course an important characteristic of a sentence, probably more than meaning. BERT is a pretrained model which tries to model such characteristics as meaning, structure and also language. If you want to have a model, which helps you identify if two sentences of different language mean the same thing, I can think of two different approaches:

1. approach: You can train a classifier (SVM, logistic Regression or even some neuronal nets such as CNN) on that task. Inputs: two [CLS]-Token, Output: Same meaning, or not same meaning. As training data, you could choose [CLS]-Token-pairs of sentences of different language which are either of the same meaning or not. To get meaningful results, you would need a lot of such sentence pairs. Luckily you can either generate them via google translate, or use a parallel texts such as the bible which exists in a lot of languages, and extract sentence pairs from there.

2. approach: Fine-tune the bert model on exactly that task: As in the previous approach, you need a lot of training data. A sample input to the BERT model would look like that: A cat jumped from the trees and startled the tourists [SEP] חתול קפץ מהעץ והבהיל את התיירים

   To classify if those sentences are of the same meaning, you would add a classification layer on top of the [CLS]-Token and Fine tune the whole Model on that task.

Note: I have never worked with a multilingual BERT-model, those approaches are what comes to my mind to accomplish the mentioned task. If you try those approaches, I would be interested to know how they perform .