It allows the model to learn a bidirectional representation of the sentence. Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes they correspond to sentences that were next to each other in the original text, sometimes not.

The BERT model used in this tutorial ( bert-base-uncased ) has a vocabulary size V of 30522. With the embedding size of 768, the total size of the word embedding table is ~ 4 (Bytes/FP32) * 30522 * 768 = 90 MB.

This is a Sentence Pair Classification model built upon a Text Embedding model from [Hugging Face](https://huggingface.co/distilbert-base-uncased ). It takes a pair of sentences as input and classifies the input pair to 'entailment' or 'no-entailment'.

BERT, which stands for Bidirectional Encoder Representations from Transformers, is based on transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection.