pre-training | Training Buys Better Robustness and Uncertainty Estimates | Cybersecurity library

For those who are interested, it is called weight tying or joint input-output embedding. There are two papers that argue for the benefit of this approach:

• Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation
• Using the Output Embedding to Improve Language Models

Okay, I wasn't able to calculate the standard deviation as planned, but did it using the code below. The grayscale imagenet's train dataset mean and standard deviation are (round it as much as you like):

Mean: 0.44531356896770125

Standard Deviation: 0.2692461874154524

Edit: process multiple entries in the column words.

Your questions 1 and 2 are answered in the code. Actually quite simple (assuming the data format is correctly represented by what shown in the screenshot). Digest:

Q1: apply the splitting function on the column and unpack by .tolist() such that separate columns can be created. See this post also.

Q2: Use list comprehension to unpack the extra list layer and retain only non-empty edges.

Q3: Yes and no. Yes because pandas is good at organizing data with heterogeneous types. For example, lists, dict, int and float can be present at different columns. Several I/O functions, such as pd.read_csv() or pd.read_json(), are also very handy.

However, there is overhead in data access, and that is especially costly for iterating over rows (records). Therefore, the transformed data that feeds directly into your model is usually converted into numpy.array or more efficient formats. Such a format conversion task is the data scientist's sole responsibility.

I make up my own sample dataset. Irrelevant columns were ignored (as I am not obliged to and shouldn't do).

This is done because they want to pre-train a bidirectional model. Most of the time the network will see a sentence with a [MASK] token, and its trained to predict the word that is supposed to be there. But in fine-tuning, which is done after pre-training (fine-tuning is the training done by everyone who wants to use BERT on their task), there are no [MASK] tokens! (unless you specifically do masked LM).

This mismatch between pre-training and training (sudden disappearence of the [MASK] token) is softened by them, with a probability of 15% the word is not replaced by [MASK]. The task is still there, the network has to predict the token, but it actually gets the answer already as input. This might seem counterintuitive but makes sense when combined with the [MASK] training.

A regular expression along with findall() method can be used for finding all the intersting links form the given html content.

BeautifulSoup offers an easy way to read table from html.

The above goal of reading pdf links form a table inside a given html content can be achieved by using regex along with BeautifulSoup.

Working example using regex and along with BeatifulSoup

You need to drop the double quotes of None:

You are aware that the training is non-deterministic anyway, right? Did you try to rerun each case a couple of times? Also the baseline? Maybe the baseline itself is an outlier.

Also, changing the computation graph, even if this will be a no-op, can also have an effect. Unfortunately it can be sensitive.

You might want to try setting deterministic_train = True in your config. This might make it a bit more deterministic. Maybe you get the same result then in each of your cases. This might make it a bit slower, though.

The order of parameter initialization might be different as well. The order depends on the order of when the layers are created. Maybe compare that in the log. It is always the same random initializer, but would use a different seed offset then, so you would get another initialization. You could play around by explicitly setting random_seed in the config, and see how much variance you get by that. Maybe all these values are within this range.

For a more in-depth debugging, you could really compare directly the computation graph (in TensorBoard). Maybe there is a difference which you did not notice. Also, maybe make a diff on the log output during net construction, for the case pretrain vs baseline. There should be no diff.

(As this is maybe a mistake, for now only as a side comment: Of course, different RETURNN versions might have some different behavior. So this should be the same.)

Another note: You do not need this tf.reduce_sum in your loss. Actually that might not be such a good idea. Now it will forget about number of frames, and number of seqs. If you just do not use tf.reduce_sum, it should also work, but now you get the correct normalization.

Another note: Instead of your lambda, you can also use loss_scale, which is simpler, and you get the original value in the log.

So basically, you could write it this way:

BERT can be viewed as a language encoder, which is trained on a humongous amount of data to learn the language well. As we know, the original BERT model was trained on the entire English Wikipedia and Book corpus, which sums to 3,300M words. BERT-base has 109M model parameters. So, if you think you have large enough data to train BERT, then the answer to your question is yes.

However, when you said "still achieve a good result", I assume you are comparing against the original BERT model. In that case, the answer lies in the size of the training data.

I am wondering why do you prefer to train BERT from scratch instead of fine-tuning it? Is it because you are afraid of the domain adaptation issue? If not, pre-trained BERT is perhaps a better starting point.

Please note, if you want to train BERT from scratch, you may consider a smaller architecture. You may find the following papers useful.

• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

If you want to use your current setup, it will have no problem running a transformer model. You can reduce memory use by reducing the batch size, but at the cost of slower runs.

Alternatively, test your algorithm on google Colab which is free. Then open a GCP account, google will provide $300 dollars of free credits. Use this to create a GPU cloud instance and then run your algorithm there.

You probably want to use Albert or Distilbert from HuggingFace Transformers. Albert and Distilbert are both compute and memory optimized. HuggingFace has lot's of excellent examples.

Rule of thumb you want to avoid Language Model training from scratch. If possible fine tune the language model or better yet skip it and go straight to the training the classifier. Also, HuggingFace and others have MedicalBert, ScienceBert, and other specialized pretrained models.