RoBERTa TensorFlow Tests

Correct slow test
Style
2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00 · 2020-02-04 18:05:22 -05:00
82 changed files with 229 additions and 1582 deletions
@@ -10,7 +10,7 @@ jobs:
        parallelism: 1
        steps:
            - checkout
-            - run: sudo pip install .[sklearn,tf-cpu,torch,testing]
+            - run: sudo pip install .[sklearn,tf,torch,testing]
            - run: sudo pip install codecov pytest-cov
            - run: python -m pytest -n 8 --dist=loadfile -s -v ./tests/ --cov
            - run: codecov
@@ -26,10 +26,8 @@ jobs:
        parallelism: 1
        steps:
            - checkout
-            - run: sudo pip install .[mecab,sklearn,tf-cpu,torch,testing]
+            - run: sudo pip install .[mecab,sklearn,tf,torch,testing]
            - run: python -m pytest -n 8 --dist=loadfile -s -v ./tests/
-            - no_output_timeout: 4h
-
    run_tests_torch:
        working_directory: ~/transformers
        docker:
@@ -54,7 +52,7 @@ jobs:
        parallelism: 1
        steps:
            - checkout
-            - run: sudo pip install .[sklearn,tf-cpu,testing]
+            - run: sudo pip install .[sklearn,tf,testing]
            - run: sudo pip install codecov pytest-cov
            - run: python -m pytest -n 8 --dist=loadfile -s -v ./tests/ --cov
            - run: codecov
@@ -198,13 +198,11 @@ Follow these steps to start contributing:
 3. To indicate a work in progress please prefix the title with `[WIP]`. These
   are useful to avoid duplicated work, and to differentiate it from PRs ready
   to be merged;
-4. Make sure existing tests pass;
-5. Add high-coverage tests. No quality test, no merge. 
- - If you are adding a new model, make sure that you use `ModelTester.all_model_classes = (MyModel, MyModelWithLMHead,...)`, which triggers the common tests.
- - If you are adding new `@slow` tests, make sure they pass using `RUN_SLOW=1 python -m pytest tests/test_my_new_model.py`. 
-CircleCI does not run them. 
+4. Make sure pre-existing tests still pass;
+5. Add high-coverage tests. No quality test, no merge;
 6. All public methods must have informative docstrings;

+
 ### Tests

 You can run 🤗 Transformers tests with `unittest` or `pytest`.
@@ -24,8 +24,6 @@

 🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch.

-[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/0)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/0)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/1)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/1)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/2)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/2)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/3)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/3)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/4)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/4)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/5)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/5)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/6)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/6)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/7)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/7)
-
 ### Features

 - As easy to use as pytorch-transformers
@@ -195,7 +193,7 @@ MODELS = [(BertModel,       BertTokenizer,       'bert-base-uncased'),
          (TransfoXLModel,  TransfoXLTokenizer,  'transfo-xl-wt103'),
          (XLNetModel,      XLNetTokenizer,      'xlnet-base-cased'),
          (XLMModel,        XLMTokenizer,        'xlm-mlm-enfr-1024'),
-          (DistilBertModel, DistilBertTokenizer, 'distilbert-base-cased'),
+          (DistilBertModel, DistilBertTokenizer, 'distilbert-base-uncased'),
          (RobertaModel,    RobertaTokenizer,    'roberta-base'),
          (XLMRobertaModel, XLMRobertaTokenizer, 'xlm-roberta-base'),
         ]
@@ -495,22 +493,19 @@ Your model will then be accessible through its identifier, a concatenation of yo
 "username/pretrained_model"
 ```

-**Please add a README.md model card** to the repo under `model_cards/` with: model description, training params (dataset, preprocessing, hyperparameters), evaluation results, intended uses & limitations, etc.
-
-Your model now has a page on huggingface.co/models 🔥
-
 Anyone can load it from code:
 ```python
 tokenizer = AutoTokenizer.from_pretrained("username/pretrained_model")
 model = AutoModel.from_pretrained("username/pretrained_model")
 ```

-List all your files on S3:
+Finally, list all your files on S3:
 ```shell
 transformers-cli s3 ls
+# List all your S3 objects.
 ```

-You can also delete unneeded files:
+You can also delete files:

 ```shell
 transformers-cli s3 rm …
@@ -194,41 +194,3 @@ h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend{
    src: url(./Calibre-Thin.otf);
    font-weight:400;
 }
-
-
-/**
- * Nav Links to other parts of huggingface.co
- */
- div.menu {
-    position: absolute;
-    top: 0;
-    right: 0;
-    padding-top: 20px;
-    padding-right: 20px;
-    z-index: 1000;
-}
-div.menu a {
-    font-size: 14px;
-    letter-spacing: 0.3px;
-    text-transform: uppercase;
-    color: white;
-    -webkit-font-smoothing: antialiased;
-    background: linear-gradient(0deg, #6671ffb8, #9a66ffb8 50%);
-    padding: 10px 16px 6px 16px;
-    border-radius: 3px;
-    margin-left: 12px;
-    position: relative;
-}
-div.menu a:active {
-    top: 1px;
-}
-@media (min-width: 768px) and (max-width: 1750px) {
-    .wy-breadcrumbs {
-        margin-top: 32px;
-    }
-}
-@media (max-width: 768px) {
-    div.menu {
-        display: none;
-    }
-}
@@ -58,16 +58,6 @@ function addGithubButton() {
    document.querySelector(".wy-side-nav-search .icon-home").insertAdjacentHTML('afterend', div);
 }

-function addHfMenu() {
-    const div = `
-    <div class="menu">
-        <a href="/welcome">🔥 Sign in</a>
-        <a href="/models">🚀 Models</a>
-    </div>
-    `;
-    document.body.insertAdjacentHTML('afterbegin', div);
-}
-
 /*!
 * github-buttons v2.2.10
 * (c) 2019 なつき
@@ -84,7 +74,6 @@ function onLoad() {
    addCustomFooter();
    addGithubButton();
    parseGithubButtons();
-    addHfMenu();
 }

 window.addEventListener("load", onLoad);
@@ -63,7 +63,7 @@ XNLI
 `The Cross-Lingual NLI Corpus (XNLI) <https://www.nyu.edu/projects/bowman/xnli/>`__ is a benchmark that evaluates
 the quality of cross-lingual text representations. 
 XNLI is crowd-sourced dataset based on `MultiNLI <http://www.nyu.edu/projects/bowman/multinli/>`: pairs of text are labeled with textual entailment 
-annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).
+annotations for 15 different languages (including both high-ressource language such as English and low-ressource languages such as Swahili).

 It was released together with the paper
 `XNLI: Evaluating Cross-lingual Sentence Representations <https://arxiv.org/abs/1809.05053>`__
@@ -23,9 +23,6 @@ Tips:

 - This implementation is the same as :class:`~transformers.BertModel` with a tiny embeddings tweak as well as a
  setup for Roberta pretrained models.
- RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
-  different pre-training scheme.
- RoBERTa doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `</s>`)
 - `Camembert <./camembert.html>`__ is a wrapper around RoBERTa. Refer to this page for usage examples.

 RobertaConfig
@@ -22,9 +22,6 @@ and XNLI benchmarks. We will make XLM-R code, data, and models publicly availabl

 Tips:

- XLM-R is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
-  not require `lang` tensors to understand which language is used, and should be able to determine the correct
-  language from the input ids.
 - This implementation is the same as RoBERTa. Refer to the `documentation of RoBERTa <./roberta.html>`__ for usage
  examples as well as the information relative to the inputs and outputs.

@@ -26,22 +26,19 @@ Your model will then be accessible through its identifier, a concatenation of yo
 "username/pretrained_model"
 ```

-**Please add a README.md model card** to the repo under `model_cards/` with: model description, training params (dataset, preprocessing, hyperparameters), evaluation results, intended uses & limitations, etc.
-
-Your model now has a page on huggingface.co/models 🔥
-
 Anyone can load it from code:
 ```python
 tokenizer = AutoTokenizer.from_pretrained("username/pretrained_model")
 model = AutoModel.from_pretrained("username/pretrained_model")
 ```

-List all your files on S3:
+Finally, list all your files on S3:
 ```shell
 transformers-cli s3 ls
+# List all your S3 objects.
 ```

-You can also delete unneeded files:
+You can also delete files:

 ```shell
 transformers-cli s3 rm …
@@ -179,14 +179,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   |                                                            | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint, with an additional linear layer.                 |
 |                   |                                                            | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__)                                     |
 |                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-|                   | ``distilbert-base-cased``                                  | | 6-layer, 768-hidden, 12-heads, 65M parameters                                                                                       |
-|                   |                                                            | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint                                                     |
-|                   |                                                            | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__)                                     |
-|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-|                   | ``distilbert-base-cased-distilled-squad``                  | | 6-layer, 768-hidden, 12-heads, 65M parameters                                                                                       |
-|                   |                                                            | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint, with an additional question answering layer.       |
-|                   |                                                            | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__)                                     |
-|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 |                   | ``distilgpt2``                                             | | 6-layer, 768-hidden, 12-heads, 82M parameters                                                                                       |
 |                   |                                                            | | The DistilGPT2 model distilled from the GPT2 model `gpt2` checkpoint.                                                               |
 |                   |                                                            | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__)                                     |
@@ -209,7 +209,7 @@ past = None
 for i in range(100):
    print(i)
    output, past = model(context, past=past)
-    token = torch.argmax(output[..., -1, :])
+    token = torch.argmax(output[0, :])

    generated += [token.tolist()]
    context = token.unsqueeze(0)
@@ -17,14 +17,15 @@ pip install -r ./examples/requirements.txt
 | Section                    | Description                                                                                                                                                |
 |----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | [TensorFlow 2.0 models on GLUE](#TensorFlow-2.0-Bert-models-on-GLUE) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. 
-| [Language Model training](#language-model-training) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
-| [Language Generation](#language-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
-| [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
-| [SQuAD](#squad) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
-| [Multiple Choice](#multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
-| [Named Entity Recognition](#named-entity-recognition) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
+| [Language Model fine-tuning](#language-model-fine-tuning) | Fine-tuning the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
+| [Language Generation](#language-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet.                                         |
+| [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision.                              |
+| [SQuAD](#squad) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training.                                                                                  |
+| [Multiple Choice](#multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. 
+| [Named Entity Recognition](#named-entity-recognition) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training.                                                                                  |
 | [XNLI](#xnli) | Examples running BERT/XLM on the XNLI benchmark. |
-| [Adversarial evaluation of model performances](#adversarial-evaluation-of-model-performances) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |
+| [Adversarial evaluation of model performances](#adversarial-evaluation-of-model-performances) | Testing a model with adversarial evaluation of natural language
+inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |

 ## TensorFlow 2.0 Bert models on GLUE

@@ -48,16 +49,16 @@ Quick benchmarks from the script (no other modifications):

 Mixed precision (AMP) reduces the training time considerably for the same hardware and hyper-parameters (same batch size was used).

-## Language model training
+## Language model fine-tuning

-Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).
+Based on the script [`run_lm_finetuning.py`](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py).

-Fine-tuning (or training from scratch) the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT 
+Fine-tuning the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT 
 to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa 
 are fine-tuned using a masked language modeling (MLM) loss.

 Before running the following example, you should get a file that contains text on which the language model will be
-trained or fine-tuned. A good example of such text is the [WikiText-2 dataset](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/).
+fine-tuned. A good example of such text is the [WikiText-2 dataset](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/).

 We will refer to two different files: `$TRAIN_FILE`, which contains text for training, and `$TEST_FILE`, which contains
 text that will be used for evaluation.
@@ -71,7 +72,7 @@ the tokenization). The loss here is that of causal language modeling.
 export TRAIN_FILE=/path/to/dataset/wiki.train.raw
 export TEST_FILE=/path/to/dataset/wiki.test.raw

-python run_language_modeling.py \
+python run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
@@ -99,7 +100,7 @@ We use the `--mlm` flag so that the script may change its loss function.
 export TRAIN_FILE=/path/to/dataset/wiki.train.raw
 export TEST_FILE=/path/to/dataset/wiki.test.raw

-python run_language_modeling.py \
+python run_lm_finetuning.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
@@ -153,7 +154,7 @@ between different runs. We report the median on 5 runs (with different seeds) fo
 Some of these results are significantly different from the ones reported on the test set
 of GLUE benchmark on the website. For QQP and WNLI, please refer to [FAQ #12](https://gluebenchmark.com/faq) on the webite.

-Before running any one of these GLUE tasks you should download the
+Before running anyone of these GLUE tasks you should download the
 [GLUE data](https://gluebenchmark.com/tasks) by running
 [this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
 and unpack it to some directory `$GLUE_DIR`.
@@ -195,7 +196,7 @@ since the data processor for each task inherits from the base class DataProcesso
 The following examples fine-tune BERT on the Microsoft Research Paraphrase Corpus (MRPC) corpus and runs in less 
 than 10 minutes on a single K-80 and in 27 seconds (!) on single tesla V100 16GB with apex installed.

-Before running any one of these GLUE tasks you should download the
+Before running anyone of these GLUE tasks you should download the
 [GLUE data](https://gluebenchmark.com/tasks) by running
 [this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
 and unpack it to some directory `$GLUE_DIR`.
@@ -700,7 +701,7 @@ macro avg     0.8712    0.8774    0.8740     13869

 Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/run_xnli.py).

-[XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).
+[XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-ressource language such as English and low-ressource languages such as Swahili).

 #### Fine-tuning on XNLI

@@ -31,10 +31,8 @@ Here are the results on the dev sets of GLUE:

 | Model                     | Macro-score                    | CoLA | MNLI | MRPC | QNLI | QQP  | RTE  | SST-2| STS-B| WNLI              |
 | :---:                     |    :---:                       | :---:| :---:| :---:| :---:| :---:| :---:| :---:| :---:| :---:             |
-| BERT-base-uncased         |  **74.9**                      | 49.2 | 80.8 | 87.4 | 87.5 | 86.4 | 61.7 | 92.0 | 83.8 | 45.1              |
-| DistilBERT-base-uncased   |  **74.3**                      | 43.6 | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7 | 81.2 | 56.3              |
-| BERT-base-cased           |  **78.2**                      | 58.2 | 83.9 | 87.8 | 91.0 | 89.2 | 66.1 | 91.7 | 89.2 | 46.5              |
-| DistilBERT-base-cased     |  **75.9**                      | 47.2 | 81.5 | 85.6 | 88.2 | 87.8 | 60.6 | 90.4 | 85.5 | 56.3              |
+| BERT-base-uncased         |  **77.6**                      | 49.2 | 80.8 | 87.4 | 87.5 | 86.4 | 61.7 | 92.0 | 83.8 | 45.1              |
+| DistilBERT-base-uncased   |  **76.8**                      | 43.6 | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7 | 81.2 | 56.3              |
 | ---                       |    ---                         |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  ---              |
 | RoBERTa-base (reported)   |  **83.2**/**86.4**<sup>2</sup> | 63.6 | 87.6 | 90.2 | 92.8 | 91.9 | 78.7 | 94.8 | 91.2 | 57.7<sup>3</sup>  |
 | DistilRoBERTa<sup>1</sup> |  **79.0**/**82.3**<sup>2</sup> | 59.3 | 84.0 | 86.6 | 90.8 | 89.4 | 67.9 | 92.5 | 88.3 | 52.1              |
@@ -65,9 +63,7 @@ This part of the library has only be tested with Python3.6+. There are few speci
 Transformers includes five pre-trained Distil* models, currently only provided for English and German (we are investigating the possibility to train and release a multilingual version of DistilBERT):

 - `distilbert-base-uncased`: DistilBERT English language model pretrained on the same data used to pretrain Bert (concatenation of the Toronto Book Corpus and full English Wikipedia) using distillation with the supervision of the `bert-base-uncased` version of Bert. The model has 6 layers, 768 dimension and 12 heads, totalizing 66M parameters.
- `distilbert-base-uncased-distilled-squad`: A finetuned version of `distilbert-base-uncased` finetuned using (a second step of) knwoledge distillation on SQuAD 1.0. This model reaches a F1 score of 79.8 on the dev set (for comparison, Bert `bert-base-uncased` version reaches a 82.3 F1 score).
- `distilbert-base-cased`: DistilBERT English language model pretrained on the same data used to pretrain Bert (concatenation of the Toronto Book Corpus and full English Wikipedia) using distillation with the supervision of the `bert-base-cased` version of Bert. The model has 6 layers, 768 dimension and 12 heads, totalizing 65M parameters.
- `distilbert-base-cased-distilled-squad`: A finetuned version of `distilbert-base-cased` finetuned using (a second step of) knwoledge distillation on SQuAD 1.0. This model reaches a F1 score of 87.1 on the dev set (for comparison, Bert `bert-base-cased` version reaches a 88.7 F1 score).
+- `distilbert-base-uncased-distilled-squad`: A finetuned version of `distilbert-base-uncased` finetuned using (a second step of) knwoledge distillation on SQuAD 1.0. This model reaches a F1 score of 86.9 on the dev set (for comparison, Bert `bert-base-uncased` version reaches a 88.5 F1 score).
 - `distilbert-base-german-cased`: DistilBERT German language model pretrained on 1/2 of the data used to pretrain Bert using distillation with the supervision of the `bert-base-german-dbmdz-cased` version of German DBMDZ Bert. For NER tasks the model reaches a F1 score of 83.49 on the CoNLL-2003 test set (for comparison, `bert-base-german-dbmdz-cased` reaches a 84.52 F1 score), and a F1 score of 85.23 on the GermEval 2014 test set (`bert-base-german-dbmdz-cased` reaches a 86.89 F1 score).
 - `distilgpt2`: DistilGPT2 English language model pretrained with the supervision of `gpt2` (the smallest version of GPT2) on [OpenWebTextCorpus](https://skylion007.github.io/OpenWebTextCorpus/), a reproduction of OpenAI's WebText dataset. The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 124M parameters for GPT2). On average, DistilGPT2 is two times faster than GPT2.
 - `distilroberta-base`: DistilRoBERTa English language model pretrained with the supervision of `roberta-base` solely on [OpenWebTextCorpus](https://skylion007.github.io/OpenWebTextCorpus/), a reproduction of OpenAI's WebText dataset (it is ~4 times less training data than the teacher RoBERTa). The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 125M parameters for RoBERTa-base). On average DistilRoBERTa is twice as fast as Roberta-base.
@@ -76,8 +72,8 @@ Transformers includes five pre-trained Distil* models, currently only provided f
 Using DistilBERT is very similar to using BERT. DistilBERT share the same tokenizer as BERT's `bert-base-uncased` even though we provide a link to this tokenizer under the `DistilBertTokenizer` name to have a consistent naming between the library models.

 ```python
-tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-model = DistilBertModel.from_pretrained('distilbert-base-cased')
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+model = DistilBertModel.from_pretrained('distilbert-base-uncased')

 input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
 outputs = model(input_ids)
@@ -85,7 +81,6 @@ last_hidden_states = outputs[0]  # The last hidden-state is the first element of
 ```

 Similarly, using the other Distil* models simply consists in calling the base classes with a different pretrained checkpoint:
- DistilBERT uncased: `model = DistilBertModel.from_pretrained('distilbert-base-uncased')`
 - DistilGPT2: `model = GPT2Model.from_pretrained('distilgpt2')`
 - DistilRoBERTa: `model = RobertaModel.from_pretrained('distilroberta-base')`
 - DistilmBERT: `model = DistilBertModel.from_pretrained('distilbert-base-multilingual-cased')`
@@ -179,7 +174,7 @@ Happy distillation!

 ## Citation

-If you find the resource useful, you should cite the following paper:
+If you find the ressource useful, you should cite the following paper:

 ```
@inproceedings{sanh2019distilbert,
@@ -1,15 +0,0 @@
-{
-	"activation": "gelu",
-	"attention_dropout": 0.1,
-	"dim": 768,
-	"dropout": 0.1,
-	"hidden_dim": 3072,
-	"initializer_range": 0.02,
-	"max_position_embeddings": 512,
-	"n_heads": 12,
-	"n_layers": 6,
-	"sinusoidal_pos_embds": true,
-	"tie_weights_": true,
-	"vocab_size": 28996
-  }
-  
@@ -310,7 +310,7 @@ def evaluate(args, model, tokenizer, prefix=""):
        eval_dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)

        # multi-gpu eval
-        if args.n_gpu > 1 and not isinstance(model, torch.nn.DataParallel):
+        if args.n_gpu > 1:
            model = torch.nn.DataParallel(model)

        # Eval!
@@ -86,9 +86,6 @@ MODEL_CLASSES = {
 class TextDataset(Dataset):
    def __init__(self, tokenizer: PreTrainedTokenizer, args, file_path: str, block_size=512):
        assert os.path.isfile(file_path)
-
-        block_size = block_size - (tokenizer.max_len - tokenizer.max_len_single_sentence)
-
        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(
            directory, args.model_type + "_cached_lm_" + str(block_size) + "_" + filename
@@ -121,7 +118,7 @@ class TextDataset(Dataset):
        return len(self.examples)

    def __getitem__(self, item):
-        return torch.tensor(self.examples[item], dtype=torch.long)
+        return torch.tensor(self.examples[item])


 class LineByLineTextDataset(Dataset):
@@ -133,15 +130,15 @@ class LineByLineTextDataset(Dataset):
        logger.info("Creating features from dataset file at %s", file_path)

        with open(file_path, encoding="utf-8") as f:
-            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
+            lines = [line for line in f.read().splitlines() if len(line) > 0]

-        self.examples = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size)["input_ids"]
+        self.examples = tokenizer.batch_encode_plus(lines, max_length=block_size)["input_ids"]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
-        return torch.tensor(self.examples[i], dtype=torch.long)
+        return torch.tensor(self.examples[i])


 def load_and_cache_examples(args, tokenizer, evaluate=False):
@@ -198,12 +195,7 @@ def _rotate_checkpoints(args, checkpoint_prefix="checkpoint", use_mtime=False) -

 def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> Tuple[torch.Tensor, torch.Tensor]:
    """ Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. """
-
-    if tokenizer.mask_token is None:
-        raise ValueError(
-            "This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer."
-        )
-
+    inputs = inputs.clone().type(dtype=torch.long)
    labels = inputs.clone()
    # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
    probability_matrix = torch.full(labels.shape, args.mlm_probability)
@@ -713,10 +705,10 @@ def main():
        )

    if args.block_size <= 0:
-        args.block_size = tokenizer.max_len
+        args.block_size = tokenizer.max_len_single_sentence
        # Our input block size will be the max possible for the model
    else:
-        args.block_size = min(args.block_size, tokenizer.max_len)
+        args.block_size = min(args.block_size, tokenizer.max_len_single_sentence)

    if args.model_name_or_path:
        model = model_class.from_pretrained(
@@ -160,10 +160,7 @@ def train(args, train_dataset, model, tokenizer, labels, pad_token_label_id):
    # Check if continuing training from a checkpoint
    if os.path.exists(args.model_name_or_path):
        # set global_step to gobal_step of last saved checkpoint from model path
-        try:
-            global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
-        except ValueError:
-            global_step = 0
+        global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
        epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
        steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)

@@ -459,7 +459,7 @@ def main():
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
+    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.")
    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
    parser.add_argument(
@@ -1,7 +1,3 @@
---
-language: swedish
---
-
 # Swedish BERT Models

 The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
@@ -1,7 +1,3 @@
---
-language: swedish
---
-
 # Swedish BERT Models

 The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
@@ -1,7 +1,3 @@
---
-language: swedish
---
-
 # Swedish BERT Models

 The National Library of Sweden / KBLab releases three pretrained language models based on BERT and ALBERT. The models are trained on aproximately 15-20GB of text (200M sentences, 3000M tokens) from various sources (books, news, government publications, swedish wikipedia and internet forums) aiming to provide a representative BERT model for Swedish text. A more complete description will be published later on.
@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # UmBERTo Commoncrawl Cased

 [UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
@@ -106,13 +102,13 @@ All of the original datasets are publicly available or were released with the ow

 ## Authors

-**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)
-**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)
-**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)
+**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
+**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
+**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>

 ## About Musixmatch AI
-![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)
-We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)
+![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)<br>
+We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
 Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)


@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # UmBERTo Wikipedia Uncased

 [UmBERTo](https://github.com/musixmatchresearch/umberto) is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at [github.com/huggingface/transformers](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1)
@@ -106,12 +102,12 @@ All of the original datasets are publicly available or were released with the ow

 ## Authors

-**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)
-**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)
-**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)
+**Loreto Parisi**: `loreto at musixmatch dot com`, [loretoparisi](https://github.com/loretoparisi)<br>
+**Simone Francia**: `simone.francia at musixmatch dot com`, [simonefrancia](https://github.com/simonefrancia)<br>
+**Paolo Magnani**: `paul.magnani95 at gmail dot com`, [paulthemagno](https://github.com/paulthemagno)<br>

 ## About Musixmatch AI
-![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)
-We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)
+![Musxmatch Ai mac app icon-128](https://user-images.githubusercontent.com/163333/72244273-396aa380-35ee-11ea-894b-4ea48230c02b.png)<br>
+We do Machine Learning and Artificial Intelligence @[musixmatch](https://twitter.com/Musixmatch)<br>
 Follow us on [Twitter](https://twitter.com/musixmatchai) [Github](https://github.com/musixmatchresearch)

@@ -1,90 +0,0 @@
-## Albert xxlarge version 1 language model fine-tuned on SQuAD2.0
-
-### with the following results:
-
-```
-exact: 85.65653162637918
-f1: 89.260458954177
-total': 11873
-HasAns_exact': 82.6417004048583
-HasAns_f1': 89.8598902096736
-HasAns_total': 5928
-NoAns_exact': 88.66274179983179
-NoAns_f1': 88.66274179983179
-NoAns_total': 5945
-best_exact': 85.65653162637918
-best_exact_thresh': 0.0
-best_f1': 89.2604589541768
-best_f1_thresh': 0.0
-```
-
-### from script:
-
-```
-python -m torch.distributed.launch --nproc_per_node=2 ${RUN_SQUAD_DIR}/run_squad.py \
--model_type albert \
--model_name_or_path albert-xxlarge-v1 \
--do_train \
--train_file ${SQUAD_DIR}/train-v2.0.json \
--predict_file ${SQUAD_DIR}/dev-v2.0.json \
--version_2_with_negative \
--num_train_epochs 3 \
--max_steps 8144 \
--warmup_steps 814 \
--do_lower_case \
--learning_rate 3e-5 \
--max_seq_length 512 \
--doc_stride 128 \
--save_steps 2000 \
--per_gpu_train_batch_size 1 \
--gradient_accumulation_steps 24 \
--output_dir ${MODEL_PATH}
-
-CUDA_VISIBLE_DEVICES=0 python ${RUN_SQUAD_DIR}/run_squad.py \
--model_type albert \
--model_name_or_path ${MODEL_PATH} \
--do_eval \
--train_file ${SQUAD_DIR}/train-v2.0.json \
--predict_file ${SQUAD_DIR}/dev-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--max_seq_length 512 \
--per_gpu_eval_batch_size 48 \
--output_dir ${MODEL_PATH}
-```
-
-### using the following system & software:
-
-```
-OS/Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
-GPU/CPU: 2 x NVIDIA 1080Ti / Intel i7-8700
-Transformers: 2.3.0
-PyTorch: 1.4.0
-TensorFlow: 2.1.0
-Python: 3.7.6
-```
-
-### Inferencing / prediction works with the current Transformers v2.4.1
-
-### Access this albert_xxlargev1_sqd2_512 fine-tuned model with "tried & true" code:
-
-```python
-config_class, model_class, tokenizer_class = \
-        AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer
-
-model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
-config = config_class.from_pretrained(model_name_or_path)
-tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True)
-model = model_class.from_pretrained(model_name_or_path, config=config)
-```
-
-### or the AutoModels (AutoConfig, AutoTokenizer & AutoModel) should also work, however I have yet to use them in my app & confirm:
-
-```python
-from transformers import AutoConfig, AutoTokenizer, AutoModel
-
-model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
-config = AutoConfig.from_pretrained(model_name_or_path)
-tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)
-model = AutoModel.from_pretrained(model_name_or_path, config=config)
-```
@@ -1,71 +0,0 @@
---
-language: german
-thumbnail: https://thumb.tildacdn.com/tild3162-6462-4566-b663-376630376138/-/format/webp/Screenshot_from_2020.png
---
-
-# German BERT
-
-## Overview
-**Language model:** bert-base-cased   
-**Language:** German  
-**Training data:** Wiki, OpenLegalData, News (~ 12GB)  
-**Eval data:** Conll03 (NER), GermEval14 (NER), GermEval18 (Classification), GNAD (Classification)  
-**Infrastructure**: 1x TPU v2  
-**Published**: Jun 14th, 2019
- 
-## Details
- We trained using Google's Tensorflow code on a single cloud TPU v2 with standard settings.
- We trained 810k steps with a batch size of 1024 for sequence length 128 and 30k steps with sequence length 512. Training took about 9 days.
- As training data we used the latest German Wikipedia dump (6GB of raw txt files), the OpenLegalData dump (2.4 GB) and news articles (3.6 GB).
- We cleaned the data dumps with tailored scripts and segmented sentences with spacy v2.1. To create tensorflow records we used the recommended sentencepiece library for creating the word piece vocabulary and tensorflow scripts to convert the text to data usable by BERT.
-
-See https://deepset.ai/german-bert for more details
-
-## Hyperparameters
-
-```
-batch_size = 1024
-n_steps = 810_000
-max_seq_len = 128 (and 512 later)
-learning_rate = 1e-4
-lr_schedule = LinearWarmup
-num_warmup_steps = 10_000
-```
-
-## Performance
-
-During training we monitored the loss and evaluated different model checkpoints on the following German datasets:
-
- germEval18Fine: Macro f1 score for multiclass sentiment classification
- germEval18coarse: Macro f1 score for binary sentiment classification
- germEval14: Seq f1 score for NER (file names deuutf.\*)
- CONLL03: Seq f1 score for NER
- 10kGNAD: Accuracy for document classification
-
-Even without thorough hyperparameter tuning, we observed quite stable learning especially for our German model. Multiple restarts with different seeds produced quite similar results.
-  
-![performancetable](https://thumb.tildacdn.com/tild3162-6462-4566-b663-376630376138/-/format/webp/Screenshot_from_2020.png)  
-
-We further evaluated different points during the 9 days of pre-training and were astonished how fast the model converges to the maximally reachable performance. We ran all 5 downstream tasks on 7 different model checkpoints - taken at 0 up to 840k training steps (x-axis in figure below). Most checkpoints are taken from early training where we expected most performance changes. Surprisingly, even a randomly initialized BERT can be trained only on labeled downstream datasets and reach good performance (blue line, GermEval 2018 Coarse task, 795 kB trainset size).
-
-![checkpointseval](https://thumb.tildacdn.com/tild6335-3531-4137-b533-313365663435/-/format/webp/deepset_checkpoints.png)  
-
-## Authors
-Branden Chan: `branden.chan [at] deepset.ai`
-Timo Möller: `timo.moeller [at] deepset.ai`
-Malte Pietsch: `malte.pietsch [at] deepset.ai`
-Tanay Soni: `tanay.soni [at] deepset.ai`
-
-## About us
-![deepset logo](https://raw.githubusercontent.com/deepset-ai/FARM/master/docs/img/deepset_logo.png)
-
-We bring NLP to the industry via open source!  
-Our focus: Industry specific language models & large scale QA systems.  
-  
-Some of our work: 
- [German BERT (aka "bert-base-german-cased")](https://deepset.ai/german-bert)
- [FARM](https://github.com/deepset-ai/FARM)
- [Haystack](https://github.com/deepset-ai/haystack/)
-
-Get in touch:
-[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Website](https://deepset.ai)  
@@ -1,20 +0,0 @@
---
-thumbnail: https://raw.githubusercontent.com/JetRunner/BERT-of-Theseus/master/bert-of-theseus.png
---
-
-# BERT-of-Theseus
-See our paper ["BERT-of-Theseus: Compressing BERT by Progressive Module Replacing"](http://arxiv.org/abs/2002.02925).
-
-BERT-of-Theseus is a new compressed BERT by progressively replacing the components of the original BERT.
-
-![BERT of Theseus](https://github.com/JetRunner/BERT-of-Theseus/blob/master/bert-of-theseus.png?raw=true)
-
-## Load Pretrained Model on MNLI
-
-We provide a 6-layer pretrained model on MNLI as a general-purpose model, which can transfer to other sentence classification tasks, outperforming DistillBERT (with the same 6-layer structure) on six tasks of GLUE (dev set).
-
-| Method          | MNLI | MRPC | QNLI | QQP  | RTE  | SST-2 | STS-B |
-|-----------------|------|------|------|------|------|-------|-------|
-| BERT-base       | 83.5 | 89.5 | 91.2 | 89.8 | 71.1 | 91.5  | 88.9  |
-| DistillBERT     | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7  | 81.2  |
-| BERT-of-Theseus | 82.1 | 87.5 | 88.8 | 88.8 | 70.1 | 91.8  | 87.8  |
@@ -1,7 +1,3 @@
---
-language: german
---
-
 # 🤗 + 📚 dbmdz German BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,61 +0,0 @@
---
-language: german
-tags:
-  - "historic german"
---
-
-# 🤗 + 📚 dbmdz BERT models
-
-In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
-Library open sources German Europeana BERT models 🎉
-
-# German Europeana BERT
-
-We use the open source [Europeana newspapers](http://www.europeana-newspapers.eu/)
-that were provided by *The European Library*. The final
-training corpus has a size of 51GB and consists of 8,035,986,369 tokens.
-
-Detailed information about the data and pretraining steps can be found in
-[this repository](https://github.com/stefan-it/europeana-bert).
-
-## Model weights
-
-Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
-compatible weights are available. If you need access to TensorFlow checkpoints,
-please raise an issue!
-
-| Model                                      | Downloads
-| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------
-| `dbmdz/bert-base-german-europeana-cased`   | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-cased/config.json)   • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-cased/pytorch_model.bin)   • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-cased/vocab.txt)
-
-## Results
-
-For results on Historic NER, please refer to [this repository](https://github.com/stefan-it/europeana-bert).
-
-## Usage
-
-With Transformers >= 2.3 our German Europeana BERT models can be loaded like:
-
-```python
-from transformers import AutoModel, AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-europeana-cased")
-model = AutoModel.from_pretrained("dbmdz/bert-base-german-europeana-cased")
-```
-
-# Huggingface model hub
-
-All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
-
-# Contact (Bugs, Feedback, Contribution and more)
-
-For questions about our BERT models just open an issue
-[here](https://github.com/dbmdz/berts/issues/new) 🤗
-
-# Acknowledgments
-
-Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
-Thanks for providing access to the TFRC ❤️
-
-Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
-it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -1,61 +0,0 @@
---
-language: german
-tags:
-  - "historic german"
---
-
-# 🤗 + 📚 dbmdz BERT models
-
-In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
-Library open sources German Europeana BERT models 🎉
-
-# German Europeana BERT
-
-We use the open source [Europeana newspapers](http://www.europeana-newspapers.eu/)
-that were provided by *The European Library*. The final
-training corpus has a size of 51GB and consists of 8,035,986,369 tokens.
-
-Detailed information about the data and pretraining steps can be found in
-[this repository](https://github.com/stefan-it/europeana-bert).
-
-## Model weights
-
-Currently only PyTorch-[Transformers](https://github.com/huggingface/transformers)
-compatible weights are available. If you need access to TensorFlow checkpoints,
-please raise an issue!
-
-| Model                                      | Downloads
-| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------
-| `dbmdz/bert-base-german-europeana-uncased` | [`config.json`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-uncased/config.json) • [`pytorch_model.bin`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-uncased/pytorch_model.bin) • [`vocab.txt`](https://cdn.huggingface.co/dbmdz/bert-base-german-europeana-uncased/vocab.txt)
-
-## Results
-
-For results on Historic NER, please refer to [this repository](https://github.com/stefan-it/europeana-bert).
-
-## Usage
-
-With Transformers >= 2.3 our German Europeana BERT models can be loaded like:
-
-```python
-from transformers import AutoModel, AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-europeana-uncased")
-model = AutoModel.from_pretrained("dbmdz/bert-base-german-europeana-uncased")
-```
-
-# Huggingface model hub
-
-All models are available on the [Huggingface model hub](https://huggingface.co/dbmdz).
-
-# Contact (Bugs, Feedback, Contribution and more)
-
-For questions about our BERT models just open an issue
-[here](https://github.com/dbmdz/berts/issues/new) 🤗
-
-# Acknowledgments
-
-Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC).
-Thanks for providing access to the TFRC ❤️
-
-Thanks to the generous support from the [Hugging Face](https://huggingface.co/) team,
-it is possible to download both cased and uncased models from their S3 storage 🤗
@@ -1,7 +1,3 @@
---
-language: german
---
-
 # 🤗 + 📚 dbmdz German BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # 🤗 + 📚 dbmdz BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # 🤗 + 📚 dbmdz BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # 🤗 + 📚 dbmdz BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,7 +1,3 @@
---
-language: italian
---
-
 # 🤗 + 📚 dbmdz BERT models

 In this repository the MDZ Digital Library team (dbmdz) at the Bavarian State
@@ -1,104 +0,0 @@
-# roberta-base for QA 
-
-## Overview
-**Language model:** roberta-base  
-**Language:** English  
-**Downstream-task:** Extractive QA  
-**Training data:** SQuAD 2.0  
-**Eval data:** SQuAD 2.0  
-**Code:**  See [example](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering.py) in [FARM](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering.py)  
-**Infrastructure**: 4x Tesla v100
-
-## Hyperparameters
-
-```
-batch_size = 50
-n_epochs = 3
-base_LM_model = "roberta-base"
-max_seq_len = 384
-learning_rate = 3e-5
-lr_schedule = LinearWarmup
-warmup_proportion = 0.2
-doc_stride=128
-max_query_length=64
-``` 
-
-## Performance
-Evaluated on the SQuAD 2.0 dev set with the [official eval script](https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/).
-```
-"exact": 78.49743114629833,
-"f1": 81.73092721240889
-```
-
-## Usage
-
-### In Transformers
-```python
-from transformers.pipelines import pipeline
-from transformers.modeling_auto import AutoModelForQuestionAnswering
-from transformers.tokenization_auto import AutoTokenizer
-
-model_name = "deepset/roberta-base-squad2"
-
-# a) Get predictions
-nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
-QA_input = {
-    'question': 'Why is model conversion important?',
-    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
-}
-res = nlp(QA_input)
-
-# b) Load model & tokenizer
-model = AutoModelForQuestionAnswering.from_pretrained(model_name)
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-```
-
-### In FARM
-
-```python
-from farm.modeling.adaptive_model import AdaptiveModel
-from farm.modeling.tokenization import Tokenizer
-from farm.infer import Inferencer
-
-model_name = "deepset/roberta-base-squad2"
-
-# a) Get predictions
-nlp = Inferencer.load(model_name, task_type="question_answering")
-QA_input = [{"questions": ["Why is model conversion important?"],
-             "text": "The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks."}]
-res = nlp.inference_from_dicts(dicts=QA_input, rest_api_schema=True)
-
-# b) Load model & tokenizer
-model = AdaptiveModel.convert_from_transformers(model_name, device="cpu", task_type="question_answering")
-tokenizer = Tokenizer.load(model_name)
-```
-
-### In haystack
-For doing QA at scale (i.e. many docs instead of single paragraph), you can load the model also in [haystack](https://github.com/deepset-ai/haystack/):
-```python
-reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
-# or 
-reader = TransformersReader(model="deepset/roberta-base-squad2",tokenizer="deepset/roberta-base-squad2")
-```
-
-
-## Authors
-Branden Chan: `branden.chan [at] deepset.ai`
-Timo Möller: `timo.moeller [at] deepset.ai`
-Malte Pietsch: `malte.pietsch [at] deepset.ai`
-Tanay Soni: `tanay.soni [at] deepset.ai`
-
-## About us
-![deepset logo](https://raw.githubusercontent.com/deepset-ai/FARM/master/docs/img/deepset_logo.png)
-
-We bring NLP to the industry via open source!  
-Our focus: Industry specific language models & large scale QA systems.  
-  
-Some of our work: 
- [German BERT (aka "bert-base-german-cased")](https://deepset.ai/german-bert)
- [FARM](https://github.com/deepset-ai/FARM)
- [Haystack](https://github.com/deepset-ai/haystack/)
-
-Get in touch:
-[Twitter](https://twitter.com/deepset_ai) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Website](https://deepset.ai)
-
@@ -1,48 +0,0 @@
---
-language: french
---
-
-# flaubert-base-uncased-squad
-
-## Description
-
-A baseline model for question-answering in french ([flaubert](https://github.com/getalp/Flaubert) model fine-tuned on [french-translated SQuAD 1.1 dataset](https://github.com/Alikabbadj/French-SQuAD))
-
-## Training hyperparameters
-
-```shell
-python3 ./examples/run_squad.py \
--model_type flaubert \
--model_name_or_path flaubert-base-uncased \
--do_train \
--do_eval \
--do_lower_case \
--train_file SQuAD-v1.1-train_fr_ss999_awstart2_net.json \
--predict_file SQuAD-v1.1-dev_fr_ss999_awstart2_net.json \
--learning_rate 3e-5 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir output \
--per_gpu_eval_batch_size=3 \
--per_gpu_train_batch_size=3
-``` 
-
-## Evaluation results
-
-```shell
-{"f1": 68.66174806561969, "exact_match": 49.299692063176714}
-```
-
-## Usage
-
-```python
-from transformers import pipeline
-
-nlp = pipeline('question-answering', model='fmikaelian/flaubert-base-uncased-squad', tokenizer='fmikaelian/flaubert-base-uncased-squad')
-
-nlp({
-    'question': "Qui est Claude Monet?",
-    'context': "Claude Monet, né le 14 novembre 1840 à Paris et mort le 5 décembre 1926 à Giverny, est un peintre français et l’un des fondateurs de l'impressionnisme."
-})
-```
@@ -1,7 +1,3 @@
---
-language: dutch
---
-
 # Multilingual + Dutch SQuAD2.0

 This model is the multilingual model provided by the Google research team with a fine-tuned dutch Q&A downstream task.
@@ -1,40 +0,0 @@
---
-language: esperanto
-thumbnail: https://huggingface.co/blog/assets/EsperBERTo-thumbnail-v2.png
---
-
-# EsperBERTo: RoBERTa-like Language model trained on Esperanto
-
-**Companion model to blog post https://huggingface.co/blog/how-to-train** 🔥
-
-## Training Details
-
- current checkpoint: 566000
- machine name: `galinette`
-
-
-![](https://huggingface.co/blog/assets/EsperBERTo-thumbnail-v2.png)
-
-## Example pipeline
-
-```python
-from transformers import TokenClassificationPipeline, pipeline
-
-
-MODEL_PATH = "./models/EsperBERTo-small-pos/"
-
-nlp = pipeline(
-    "ner",
-    model=MODEL_PATH,
-    tokenizer=MODEL_PATH,
-)
-# or instantiate a TokenClassificationPipeline directly.
-
-nlp("Mi estas viro kej estas tago varma.")
-
-# {'entity': 'PRON', 'score': 0.9979867339134216, 'word': ' Mi'}
-# {'entity': 'VERB', 'score': 0.9683094620704651, 'word': ' estas'}
-# {'entity': 'VERB', 'score': 0.9797462821006775, 'word': ' estas'}
-# {'entity': 'NOUN', 'score': 0.8509314060211182, 'word': ' tago'}
-# {'entity': 'ADJ', 'score': 0.9996201395988464, 'word': ' varma'}
-```
@@ -1,59 +0,0 @@
---
-language: esperanto
-thumbnail: https://huggingface.co/blog/assets/EsperBERTo-thumbnail-v2.png
---
-
-# EsperBERTo: RoBERTa-like Language model trained on Esperanto
-
-**Companion model to blog post https://huggingface.co/blog/how-to-train** 🔥
-
-## Training Details
-
- current checkpoint: 566000
- machine name: `galinette`
-
-
-![](https://huggingface.co/blog/assets/EsperBERTo-thumbnail-v2.png)
-
-## Example pipeline
-
-```python
-from transformers import pipeline
-
-fill_mask = pipeline(
-    "fill-mask",
-    model="julien-c/EspertBERTo-small",
-    tokenizer="julien-c/EspertBERTo-small"
-)
-
-fill_mask("Jen la komenco de bela <mask>.")
-
-# This is the beginning of a beautiful <mask>.
-# =>
-
-# {
-#     'score':0.06502299010753632
-#     'sequence':'<s> Jen la komenco de bela vivo.</s>'
-#     'token':1099
-# }
-# {
-#     'score':0.0421181358397007
-#     'sequence':'<s> Jen la komenco de bela vespero.</s>'
-#     'token':5100
-# }
-# {
-#     'score':0.024884626269340515
-#     'sequence':'<s> Jen la komenco de bela laboro.</s>'
-#     'token':1570
-# }
-# {
-#     'score':0.02324388362467289
-#     'sequence':'<s> Jen la komenco de bela tago.</s>'
-#     'token':1688
-# }
-# {
-#     'score':0.020378097891807556
-#     'sequence':'<s> Jen la komenco de bela festo.</s>'
-#     'token':4580
-# }
-```
@@ -1,10 +1,3 @@
---
-tags:
- ci
---
-
-## Dummy model used for unit testing and CI
-

 ```python
 import json
@@ -1,7 +0,0 @@
-# ArXiv-NLP GPT-2 checkpoint
-
-This is a GPT-2 small checkpoint for PyTorch. It is the official `gpt2-small` fine-tuned to ArXiv paper on the computational linguistics field.
-
-## Training data
-
-This model was trained on a subset of ArXiv papers that were parsed from PDF to txt. The resulting data is made of 80MB of text from the computational linguistics (cs.CL) field.
@@ -1,7 +0,0 @@
-# ArXiv GPT-2 checkpoint
-
-This is a GPT-2 small checkpoint for PyTorch. It is the official `gpt2-small` finetuned to ArXiv paper on physics fields.
-
-## Training data
-
-This model was trained on a subset of ArXiv papers that were parsed from PDF to txt. The resulting data is made of 130MB of text, mostly from quantum physics (quant-ph) and other physics sub-fields.
@@ -1,91 +0,0 @@
---
-language: spanish
-thumbnail: https://i.imgur.com/jgBdimh.png
---
-
-# BETO (Spanish BERT) + Spanish SQuAD2.0
-
-This model is provided by [BETO team](https://github.com/dccuchile/beto) and fine-tuned on [SQuAD-es-v2.0](https://github.com/ccasimiro88/TranslateAlignRetrieve) for **Q&A** downstream task.
-
-## Details of the language model('dccuchile/bert-base-spanish-wwm-cased')
-
-Language model ([**'dccuchile/bert-base-spanish-wwm-cased'**](https://github.com/dccuchile/beto/blob/master/README.md)):
-
-BETO is a [BERT model](https://github.com/google-research/bert) trained on a [big Spanish corpus](https://github.com/josecannete/spanish-corpora). BETO is of size similar to a BERT-Base and was trained with the Whole Word Masking technique. Below you find Tensorflow and Pytorch checkpoints for the uncased and cased versions, as well as some results for Spanish benchmarks comparing BETO with [Multilingual BERT](https://github.com/google-research/bert/blob/master/multilingual.md) as well as other (not BERT-based) models.
-
-## Details of the downstream task (Q&A) - Dataset
-[SQuAD-es-v2.0](https://github.com/ccasimiro88/TranslateAlignRetrieve)
-
-| Dataset                | # Q&A |
-| ---------------------- | ----- |
-| SQuAD2.0 Train         | 130 K |
-| SQuAD2.0-es-v2.0       | 111 K |
-| SQuAD2.0 Dev           | 12  K |
-| SQuAD-es-v2.0-small Dev| 69  K |
-
-## Model training
-
-The model was trained on a Tesla P100 GPU and 25GB of RAM with the following command:
-
-```bash
-export SQUAD_DIR=path/to/nl_squad
-python transformers/examples/run_squad.py \
-  --model_type bert \
-  --model_name_or_path dccuchile/bert-base-spanish-wwm-cased \
-  --do_train \
-  --do_eval \
-  --do_lower_case \
-  --train_file $SQUAD_DIR/train_nl-v2.0.json \
-  --predict_file $SQUAD_DIR/dev_nl-v2.0.json \
-  --per_gpu_train_batch_size 12 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 2.0 \
-  --max_seq_length 384 \
-  --doc_stride 128 \
-  --output_dir /content/model_output \
-  --save_steps 5000 \
-  --threads 4 \
-  --version_2_with_negative 
-```
-
-## Results:
-
-
-  | Metric               | # Value |
-| ---------------------- | ----- |
-| **Exact**              | **76.50**50 |
-| **F1**                 | **86.07**81 |
-
-```json
-{
-  "exact": 76.50501430594491,
-  "f1": 86.07818773108252,
-  "total": 69202,
-  "HasAns_exact": 67.93020719738277,
-  "HasAns_f1": 82.37912207996466,
-  "HasAns_total": 45850,
-  "NoAns_exact": 93.34104145255225,
-  "NoAns_f1": 93.34104145255225,
-  "NoAns_total": 23352,
-  "best_exact": 76.51223953064941,
-  "best_exact_thresh": 0.0,
-  "best_f1": 86.08541295578848,
-  "best_f1_thresh": 0.0
-}
-```
-
-### Model in action (in a Colab Notebook)
-<details>
-
-1.  Set the context and ask some questions:
-
-![Set context and questions](https://media.giphy.com/media/mCIaBpfN0LQcuzkA2F/giphy.gif)
-
-2.  Run predictions:
-
-![Run the model](https://media.giphy.com/media/WT453aptcbCP7hxWTZ/giphy.gif)
-</details>
-
-> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
-
-> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -1,108 +0,0 @@
---
-language: spanish
-thumbnail: https://i.imgur.com/jgBdimh.png
---
-
-# BETO (Spanish BERT) + Spanish SQuAD2.0 + distillation using 'bert-base-multilingual-cased' as teacher
-
-This model is a fine-tuned on [SQuAD-es-v2.0](https://github.com/ccasimiro88/TranslateAlignRetrieve) and **distilled** version of [BETO](https://github.com/dccuchile/beto) for **Q&A**.
-
-Distillation makes the model **smaller, faster, cheaper and lighter** than [bert-base-spanish-wwm-cased-finetuned-spa-squad2-es](https://github.com/huggingface/transformers/blob/master/model_cards/mrm8488/bert-base-spanish-wwm-cased-finetuned-spa-squad2-es/README.md)
-
-This model was fine-tuned on the same dataset but using **distillation** during the process as mentioned above (and one more train epoch).
-
-The **teacher model** for the distillation was `bert-base-multilingual-cased`. It is the same teacher used for `distilbert-base-multilingual-cased` AKA [**DistilmBERT**](https://github.com/huggingface/transformers/tree/master/examples/distillation) (on average is twice as fast as **mBERT-base**).
-
-## Details of the downstream task (Q&A) - Dataset
-
-<details>
-
-[SQuAD-es-v2.0](https://github.com/ccasimiro88/TranslateAlignRetrieve)
-
-| Dataset                 | # Q&A |
-| ----------------------- | ----- |
-| SQuAD2.0 Train          | 130 K |
-| SQuAD2.0-es-v2.0        | 111 K |
-| SQuAD2.0 Dev            | 12 K  |
-| SQuAD-es-v2.0-small Dev | 69 K  |
-
-</details>
-
-## Model training
-
-The model was trained on a Tesla P100 GPU and 25GB of RAM with the following command:
-
-```bash
-!export SQUAD_DIR=/path/to/squad-v2_spanish \
-&& python transformers/examples/distillation/run_squad_w_distillation.py \
-  --model_type bert \
-  --model_name_or_path dccuchile/bert-base-spanish-wwm-cased \
-  --teacher_type bert \
-  --teacher_name_or_path bert-base-multilingual-cased \
-  --do_train \
-  --do_eval \
-  --do_lower_case \
-  --train_file $SQUAD_DIR/train-v2.json \
-  --predict_file $SQUAD_DIR/dev-v2.json \
-  --per_gpu_train_batch_size 12 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 3.0 \
-  --max_seq_length 384 \
-  --doc_stride 128 \
-  --output_dir /content/model_output \
-  --save_steps 5000 \
-  --threads 4 \
-  --version_2_with_negative
-```
-
-## Results:
-
-| Metric    | # Value     |
-| --------- | ----------- |
-| **Exact** | **82.40**65 |
-| **F1**    | **90.50**36 |
-
-```json
-{
-  "exact": 82.40657784457096,
-  "f1": 90.50369643376753,
-  "total": 69202,
-  "HasAns_exact": 75.13413304253,
-  "HasAns_f1": 87.35521920631561,
-  "HasAns_total": 45850,
-  "NoAns_exact": 96.68550873586845,
-  "NoAns_f1": 96.68550873586845,
-  "NoAns_total": 23352,
-  "best_exact": 82.40657784457096,
-  "best_exact_thresh": 0.0,
-  "best_f1": 90.50369643376902,
-  "best_f1_thresh": 0.0
-}
-```
-
-## Comparison:
-
-|                              Model                              | f1 score  |
-| :-------------------------------------------------------------: | :-------: |
-|       bert-base-spanish-wwm-cased-finetuned-spa-squad2-es       |   86.07   |
-| **distill**-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es | **90.50** |
-
-So, yes, this version is even more accurate.
-
-### Model in action (in a Colab Notebook)
-
-<details>
-
-1.  Set the context and ask some questions:
-
-![Set context and questions](https://media.giphy.com/media/mCIaBpfN0LQcuzkA2F/giphy.gif)
-
-2.  Run predictions:
-
-![Run the model](https://media.giphy.com/media/WT453aptcbCP7hxWTZ/giphy.gif)
-
-</details>
-
-> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
-
-> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -1,93 +0,0 @@
---
-language: greek
-thumbnail: https://github.com/nlpaueb/GreekBERT/raw/master/greek-bert-logo.png
---
-
-# GreekBERT
-
-A Greek version of BERT pre-trained language model.
-
-<img src="https://github.com/nlpaueb/GreekBERT/raw/master/greek-bert-logo.png" width="600"/> 
-
-
-## Pre-training corpora
-
-The pre-training corpora of `bert-base-greek-uncased-v1` include:
-
-* The Greek part of [Wikipedia](https://el.wikipedia.org/wiki/Βικιπαίδεια:Αντίγραφα_της_βάσης_δεδομένων),
-* The Greek part of [European Parliament Proceedings Parallel Corpus](https://www.statmt.org/europarl/), and
-* The Greek part of [OSCAR](https://traces1.inria.fr/oscar/), a cleansed version of [Common Crawl](https://commoncrawl.org).
-
-Future release will also include:
-
-* The entire corpus of Greek legislation, as published by the [National Publication Office](http://www.et.gr),  
-* The entire corpus of EU legislation (Greek translation), as published in [Eur-Lex](https://eur-lex.europa.eu/homepage.html?locale=en).
-
-## Pre-training details
-
-* We trained BERT using the official code provided in Google BERT's github repository (https://github.com/google-research/bert). We then used [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) conversion script to convert the TF checkpoint and vocabulary in the desirable format in order to be able to load the model in two lines of code for both PyTorch and TF2 users.
-* We released a model similar to the English `bert-base-uncased` model (12-layer, 768-hidden, 12-heads, 110M parameters).
-* We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
-* We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!
-
-
-## Requirements
-
-We published `bert-base-greek-uncased-v1` as part of [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) repository. So, you need to install the transfomers library through pip along with PyTorch or Tensorflow 2.
-
-```
-pip install transfomers
-pip install (torch|tensorflow)
-```
-
-## Pre-process text (Deaccent - Lower)
-
-In order to use `bert-base-greek-uncased-v1`, you have to pre-process texts to lowercase letters and remove all Greek diacritics.
-
-```python
-
-import unicodedata
-
-def strip_accents_and_lowercase(s):
-   return ''.join(c for c in unicodedata.normalize('NFD', s)
-                  if unicodedata.category(c) != 'Mn').lower()
-
-accented_string = "Αυτή είναι η Ελληνική έκδοση του BERT."
-unaccented_string = strip_accents_and_lowercase(accented_string)
-
-print(unaccented_string) # αυτη ειναι η ελληνικη εκδοση του bert.
-
-```
-
-## Load Pretrained Model 
-
-```python
-from transformers import AutoTokenizer, AutoModel
-
-tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
-model = AutoModel.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
-```
-
-## Evaluation on downstream tasks
-
-TBA
-
-## Author
-
-Ilias Chalkidis on behalf of [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr)
-
-| Github: [@ilias.chalkidis](https://github.com/seolhokim) | Twitter: [@KiddoThe2B](https://twitter.com/KiddoThe2B) |
-
-## About Us
-
-[AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr) develops algorithms, models, and systems that allow computers to process and generate natural language texts.
-
-The group's current research interests include:
-* question answering systems for databases, ontologies, document collections, and the Web, especially biomedical question answering,
-* natural language generation from databases and ontologies, especially Semantic Web ontologies,
-text classification, including filtering spam and abusive content,
-* information extraction and opinion mining, including legal text analytics and sentiment analysis,
-* natural language processing tools for Greek, for example parsers and named-entity recognizers,
-machine learning in natural language processing, especially deep learning.
-
-The group is part of the Information Processing Laboratory of the Department of Informatics of the Athens University of Economics and Business.
@@ -1,49 +0,0 @@
---
-language:
- english
- dutch
- german
- french
- italian
- spanish
---
-
-# bert-base-multilingual-uncased-sentiment
-
-This a bert-base-multilingual-uncased model finetuned for sentiment analysis on product reviews in six languages: English, Dutch, German, French, Spanish and Italian. It predicts the sentiment of the review as a number of stars (between 1 and 5).
-
-This model is intended for direct use as a sentiment analysis model for product reviews in any of the six languages above, or for further finetuning on related sentiment analysis tasks.
-
-## Training data
-
-Here is the number of product reviews we used for finetuning the model: 
-
-| Language | Number of reviews |
-| -------- | ----------------- |
-| English  | 150k           |
-| Dutch    | 80k            |
-| German   | 137k           |
-| French   | 140k           |
-| Italian  | 72k            |
-| Spanish  | 50k            |
-
-## Accuracy
-
-The finetuned model obtained the following accuracy on 5,000 held-out product reviews in each of the languages:
-
- Accuracy (exact) is the exact match on the number of stars.
- Accuracy (off-by-1) is the percentage of reviews where the number of stars the model predicts differs by a maximum of 1 from the number given by the human reviewer. 
-
-
-| Language | Accuracy (exact) | Accuracy (off-by-1) |
-| -------- | ---------------------- | ------------------- |
-| English  | 67%                 | 95%
-| Dutch    | 57%                 | 93%
-| German   | 61%                 | 94%
-| French   | 59%                 | 94%
-| Italian  | 59%                 | 95%
-| Spanish  | 58%                 | 95%
-
-## Contact 
-
-Contact [NLP Town](https://www.nlp.town) for questions, feedback and/or requests for similar models.
@@ -1,51 +0,0 @@
---
-language: german
-thumbnail: kfold.png
---
-
-# German BERT for literary texts
-
-This German BERT is based on `bert-base-german-dbmdz-cased`, and has been adapted to the domain of literary texts by fine-tuning the language modeling task on the [Corpus of German-Language Fiction](https://figshare.com/articles/Corpus_of_German-Language_Fiction_txt_/4524680/1). Afterwards the model was fine-tuned for named entity recognition on the [DROC](https://gitlab2.informatik.uni-wuerzburg.de/kallimachos/DROC-Release) corpus, so you can use it to recognize protagonists in German novels.
-
-
-# Stats
-
-## Language modeling
-
-The [Corpus of German-Language Fiction](https://figshare.com/articles/Corpus_of_German-Language_Fiction_txt_/4524680/1) consists of 3,194 documents with 203,516,988 tokens or 1,520,855 types. The publication year of the texts ranges from the 18th to the 20th century:
-
-![years](prosa-jahre.png)
-
-
-### Results
-
-After one epoch:
-
-| Model            | Perplexity |
-| ---------------- | ---------- |
-| Vanilla BERT     | 6.82       |
-| Fine-tuned BERT  | 4.98       |
-
-
-## Named entity recognition
-
-The provided model was also fine-tuned for two epochs on 10,799 sentences for training, validated on 547 and tested on 1,845 with three labels: `B-PER`, `I-PER` and `O`.
-
-
-## Results
-
-| Dataset | Precision | Recall | F1   |
-| ------- | --------- | ------ | ---- |
-| Dev     | 96.4      | 87.3   | 91.6 |
-| Test    | 92.8      | 94.9   | 93.8 |
-
-The model has also been evaluated using 10-fold cross validation and compared with a classic Conditional Random Field baseline described in [Jannidis et al.](https://opus.bibliothek.uni-wuerzburg.de/opus4-wuerzburg/frontdoor/deliver/index/docId/14333/file/Jannidis_Figurenerkennung_Roman.pdf) (2015):
-
-![kfold](kfold.png)
-
-
-# References
-
-Markus Krug, Lukas Weimer, Isabella Reger, Luisa Macharowsky, Stephan Feldhaus, Frank Puppe, Fotis Jannidis, [Description of a Corpus of Character References in German Novels](http://webdoc.sub.gwdg.de/pub/mon/dariah-de/dwp-2018-27.pdf), 2018.
-
-Fotis Jannidis, Isabella Reger, Lukas Weimer, Markus Krug, Martin Toepfer, Frank Puppe, [Automatische Erkennung von Figuren in deutschsprachigen Romanen](https://opus.bibliothek.uni-wuerzburg.de/opus4-wuerzburg/frontdoor/deliver/index/docId/14333/file/Jannidis_Figurenerkennung_Roman.pdf), 2015.
@@ -63,7 +63,6 @@ extras = {}
 extras["mecab"] = ["mecab-python3"]
 extras["sklearn"] = ["scikit-learn"]
 extras["tf"] = ["tensorflow"]
-extras["tf-cpu"] = ["tensorflow-cpu"]
 extras["torch"] = ["torch"]

 extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
@@ -101,7 +101,6 @@ from .pipelines import (
    PipelineDataFormat,
    QuestionAnsweringPipeline,
    TextClassificationPipeline,
-    TokenClassificationPipeline,
    pipeline,
 )
 from .tokenization_albert import AlbertTokenizer
@@ -1,48 +0,0 @@
-import math
-
-import torch
-import torch.nn.functional as F
-
-
-def swish(x):
-    return x * torch.sigmoid(x)
-
-
-def _gelu_python(x):
-    """ Original Implementation of the gelu activation function in Google Bert repo when initially created.
-        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
-        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
-        This is now written in C in torch.nn.functional
-        Also see https://arxiv.org/abs/1606.08415
-    """
-    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
-
-
-gelu = getattr(F, "gelu", _gelu_python)
-
-
-def gelu_new(x):
-    """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
-        Also see https://arxiv.org/abs/1606.08415
-    """
-    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
-
-
-ACT2FN = {
-    "relu": F.relu,
-    "swish": swish,
-    "gelu": gelu,
-    "tanh": F.tanh,
-    "gelu_new": gelu_new,
-}
-
-
-def get_activation(activation_string):
-    if activation_string in ACT2FN:
-        return ACT2FN[activation_string]
-    else:
-        raise KeyError(
-            "function {} not found in ACT2FN mapping {} or torch.nn.functional".format(
-                activation_string, list(ACT2FN.keys())
-            )
-        )
@@ -25,8 +25,6 @@ logger = logging.getLogger(__name__)
 DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "distilbert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-config.json",
    "distilbert-base-uncased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-distilled-squad-config.json",
-    "distilbert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-config.json",
-    "distilbert-base-cased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad-config.json",
    "distilbert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased-config.json",
    "distilbert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json",
    "distilbert-base-uncased-finetuned-sst-2-english": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-config.json",
@@ -60,7 +58,7 @@ class DistilBertConfig(PretrainedConfig):
                Number of attention heads for each attention layer in the Transformer encoder.
            dim (:obj:`int`, optional, defaults to 768):
                Dimensionality of the encoder layers and the pooler layer.
-            hidden_dim (:obj:`int`, optional, defaults to 3072):
+            intermediate_size (:obj:`int`, optional, defaults to 3072):
                The size of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
            dropout (:obj:`float`, optional, defaults to 0.1):
                The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
@@ -277,7 +277,7 @@ MODEL_CLASSES = {
        DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
    ),
-    "distilbert-base-distilled-squad": (
+    "distilbert-base-uncased-distilled-squad": (
        DistilBertConfig,
        TFDistilBertForQuestionAnswering,
        DistilBertForQuestionAnswering,
@@ -147,14 +147,7 @@ def squad_convert_example_to_features(example, max_seq_length, doc_stride, max_q
        )

        if tokenizer.pad_token_id in encoded_dict["input_ids"]:
-            if tokenizer.padding_side == "right":
-                non_padded_ids = encoded_dict["input_ids"][: encoded_dict["input_ids"].index(tokenizer.pad_token_id)]
-            else:
-                last_padding_id_position = (
-                    len(encoded_dict["input_ids"]) - 1 - encoded_dict["input_ids"][::-1].index(tokenizer.pad_token_id)
-                )
-                non_padded_ids = encoded_dict["input_ids"][last_padding_id_position + 1 :]
-
+            non_padded_ids = encoded_dict["input_ids"][: encoded_dict["input_ids"].index(tokenizer.pad_token_id)]
        else:
            non_padded_ids = encoded_dict["input_ids"]

@@ -628,7 +621,7 @@ class SquadExample(object):
        self.doc_tokens = doc_tokens
        self.char_to_word_offset = char_to_word_offset

-        # Start and end positions only has a value during evaluation.
+        # Start end end positions only has a value during evaluation.
        if start_position_character is not None and not is_impossible:
            self.start_position = char_to_word_offset[start_position_character]
            self.end_position = char_to_word_offset[
@@ -8,16 +8,13 @@ import fnmatch
 import json
 import logging
 import os
-import shutil
 import sys
-import tarfile
 import tempfile
 from contextlib import contextmanager
 from functools import partial, wraps
 from hashlib import sha256
 from typing import Optional
 from urllib.parse import urlparse
-from zipfile import ZipFile, is_zipfile

 import boto3
 import requests
@@ -206,15 +203,7 @@ def filename_to_url(filename, cache_dir=None):


 def cached_path(
-    url_or_filename,
-    cache_dir=None,
-    force_download=False,
-    proxies=None,
-    resume_download=False,
-    user_agent=None,
-    extract_compressed_file=False,
-    force_extract=False,
-    extract_dir=None,
+    url_or_filename, cache_dir=None, force_download=False, proxies=None, resume_download=False, user_agent=None
 ) -> Optional[str]:
    """
    Given something that might be a URL (or might be a local path),
@@ -226,11 +215,6 @@ def cached_path(
        force_download: if True, re-dowload the file even if it's already cached in the cache dir.
        resume_download: if True, resume the download if incompletly recieved file is found.
        user_agent: Optional string or dict that will be appended to the user-agent on remote requests.
-        extract_compressed_file: if True and the path point to a zip or tar file, extract the compressed
-            file in a folder along the archive.
-        force_extract: if True when extract_compressed_file is True and the archive was already extracted,
-            re-extract the archive and overide the folder where it was extracted.
-        extract_dir: specify a directory to extract the archive to (overwrite the default created directory).

    Return:
        None in case of non-recoverable file (non-existent or inaccessible url + no cache on disk).
@@ -245,7 +229,7 @@ def cached_path(

    if is_remote_url(url_or_filename):
        # URL, so get it from the cache (downloading if necessary)
-        output_path = get_from_cache(
+        return get_from_cache(
            url_or_filename,
            cache_dir=cache_dir,
            force_download=force_download,
@@ -255,7 +239,7 @@ def cached_path(
        )
    elif os.path.exists(url_or_filename):
        # File, and it exists.
-        output_path = url_or_filename
+        return url_or_filename
    elif urlparse(url_or_filename).scheme == "":
        # File, but it doesn't exist.
        raise EnvironmentError("file {} not found".format(url_or_filename))
@@ -263,37 +247,6 @@ def cached_path(
        # Something unknown
        raise ValueError("unable to parse {} as a URL or as a local path".format(url_or_filename))

-    if extract_compressed_file and os.path.isfile(output_path):
-        if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
-            return output_path
-
-        # Path where we extract compressed archives
-        # We avoid '.' in dir name and add "-extracted" at the end: "./model.zip" => "./model-zip-extracted/"
-        output_dir, output_file = os.path.split(output_path)
-        output_extract_dir_name = output_file.replace(".", "-") + "-extracted"
-        output_path_extracted = os.path.join(output_dir, output_extract_dir_name)
-
-        if not os.path.isdir(output_path_extracted) or force_extract:
-            # Extract file with a lock to prevent parallel extractions
-            lock_path = output_path + ".lock"
-            with FileLock(lock_path):
-                shutil.rmtree(output_path_extracted, ignore_errors=True)
-                os.makedirs(output_path_extracted)
-                if is_zipfile(output_path):
-                    with ZipFile(output_path, "r") as zip_file:
-                        zip_file.extractall(output_path_extracted)
-                        zip_file.close()
-                elif tarfile.is_tarfile(output_path):
-                    tar_file = tarfile.open(output_path)
-                    tar_file.extractall(output_path_extracted)
-                    tar_file.close()
-                else:
-                    raise EnvironmentError("Archive format of {} could not be identified".format(output_path))
-
-        return output_path_extracted
-
-    return output_path
-

 def split_s3_path(url):
    """Split a full s3 path into the bucket name and path."""
@@ -117,13 +117,7 @@ def load_tf_weights_in_albert(model, config, tf_checkpoint_path):
        name = name.split("/")

        # Ignore the gradients applied by the LAMB/ADAM optimizers.
-        if (
-            "adam_m" in name
-            or "adam_v" in name
-            or "AdamWeightDecayOptimizer" in name
-            or "AdamWeightDecayOptimizer_1" in name
-            or "global_step" in name
-        ):
+        if "adam_m" in name or "adam_v" in name or "global_step" in name:
            logger.info("Skipping {}".format("/".join(name)))
            continue

@@ -24,7 +24,6 @@ import torch
 from torch import nn
 from torch.nn import CrossEntropyLoss, MSELoss

-from .activations import gelu, gelu_new, swish
 from .configuration_bert import BertConfig
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import PreTrainedModel, prune_linear_layer
@@ -87,10 +86,7 @@ def load_tf_weights_in_bert(model, config, tf_checkpoint_path):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
-        if any(
-            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
-            for n in name
-        ):
+        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            logger.info("Skipping {}".format("/".join(name)))
            continue
        pointer = model
@@ -130,6 +126,26 @@ def load_tf_weights_in_bert(model, config, tf_checkpoint_path):
    return model


+def gelu(x):
+    """ Original Implementation of the gelu activation function in Google Bert repo when initially created.
+        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
+        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+        Also see https://arxiv.org/abs/1606.08415
+    """
+    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
+
+
+def gelu_new(x):
+    """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
+        Also see https://arxiv.org/abs/1606.08415
+    """
+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+
+def swish(x):
+    return x * torch.sigmoid(x)
+
+
 def mish(x):
    return x * torch.tanh(nn.functional.softplus(x))

@@ -714,8 +730,8 @@ class BertModel(BertPreTrainedModel):
                seq_ids = torch.arange(seq_length, device=device)
                causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
                causal_mask = causal_mask.to(
-                    attention_mask.dtype
-                )  # causal and attention masks must have same type with pytorch version < 1.3
+                    torch.long
+                )  # not converting to long will cause errors with pytorch version < 1.3
                extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
            else:
                extended_attention_mask = attention_mask[:, None, None, :]
@@ -27,7 +27,6 @@ import torch
 import torch.nn as nn
 from torch.nn import CrossEntropyLoss

-from .activations import gelu
 from .configuration_distilbert import DistilBertConfig
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import PreTrainedModel, prune_linear_layer
@@ -39,8 +38,6 @@ logger = logging.getLogger(__name__)
 DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
    "distilbert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-pytorch_model.bin",
    "distilbert-base-uncased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-distilled-squad-pytorch_model.bin",
-    "distilbert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-pytorch_model.bin",
-    "distilbert-base-cased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad-pytorch_model.bin",
    "distilbert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased-pytorch_model.bin",
    "distilbert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-pytorch_model.bin",
    "distilbert-base-uncased-finetuned-sst-2-english": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-pytorch_model.bin",
@@ -48,6 +45,8 @@ DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {


 # UTILS AND BUILDING BLOCKS OF THE ARCHITECTURE #
+def gelu(x):
+    return 0.5 * x * (1.0 + torch.erf(x / math.sqrt(2.0)))


 def create_sinusoidal_embeddings(n_pos, dim, out):
@@ -441,8 +440,8 @@ class DistilBertModel(DistilBertPreTrainedModel):
        from transformers import DistilBertTokenizer, DistilBertModel
        import torch

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = DistilBertModel.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = DistilBertModel.from_pretrained('distilbert-base-uncased')

        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids)
@@ -545,8 +544,8 @@ class DistilBertForMaskedLM(DistilBertPreTrainedModel):
        from transformers import DistilBertTokenizer, DistilBertForMaskedLM
        import torch

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = DistilBertForMaskedLM.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = DistilBertForMaskedLM.from_pretrained('distilbert-base-uncased')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids, masked_lm_labels=input_ids)
        loss, prediction_scores = outputs[:2]
@@ -620,8 +619,8 @@ class DistilBertForSequenceClassification(DistilBertPreTrainedModel):
        from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
        import torch

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
        labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids, labels=labels)
@@ -712,8 +711,8 @@ class DistilBertForQuestionAnswering(DistilBertPreTrainedModel):
        from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering
        import torch

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
        start_positions = torch.tensor([1])
        end_positions = torch.tensor([3])
@@ -799,8 +798,8 @@ class DistilBertForTokenClassification(DistilBertPreTrainedModel):
        from transformers import DistilBertTokenizer, DistilBertForTokenClassification
        import torch

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids, labels=labels)
@@ -231,7 +231,7 @@ class FlaubertModel(XLMModel):
            inputs_embeds = self.embeddings(input_ids)

        tensor = inputs_embeds + self.position_embeddings(position_ids).expand_as(inputs_embeds)
-        if langs is not None and self.use_lang_emb and self.config.n_langs > 1:
+        if langs is not None and self.use_lang_emb:
            tensor = tensor + self.lang_embeddings(langs)
        if token_type_ids is not None:
            tensor = tensor + self.embeddings(token_type_ids)
@@ -24,7 +24,6 @@ import torch
 import torch.nn as nn
 from torch.nn import CrossEntropyLoss

-from .activations import gelu_new
 from .configuration_gpt2 import GPT2Config
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import Conv1D, PreTrainedModel, SequenceSummary, prune_conv1d_layer
@@ -96,6 +95,10 @@ def load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path):
    return model


+def gelu(x):
+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+
 class Attention(nn.Module):
    def __init__(self, nx, n_ctx, config, scale=False):
        super().__init__()
@@ -203,7 +206,7 @@ class MLP(nn.Module):
        nx = config.n_embd
        self.c_fc = Conv1D(n_state, nx)
        self.c_proj = Conv1D(nx, n_state)
-        self.act = gelu_new
+        self.act = gelu
        self.dropout = nn.Dropout(config.resid_pdrop)

    def forward(self, x):
@@ -25,7 +25,6 @@ import torch
 import torch.nn as nn
 from torch.nn import CrossEntropyLoss

-from .activations import gelu_new, swish
 from .configuration_openai import OpenAIGPTConfig
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import Conv1D, PreTrainedModel, SequenceSummary, prune_conv1d_layer
@@ -115,7 +114,15 @@ def load_tf_weights_in_openai_gpt(model, config, openai_checkpoint_folder_path):
    return model


-ACT_FNS = {"relu": nn.ReLU, "swish": swish, "gelu": gelu_new}
+def gelu(x):
+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+
+def swish(x):
+    return x * torch.sigmoid(x)
+
+
+ACT_FNS = {"relu": nn.ReLU, "swish": swish, "gelu": gelu}


 class Attention(nn.Module):
@@ -79,10 +79,7 @@ def load_tf_weights_in_t5(model, config, tf_checkpoint_path):
        name = txt_name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
-        if any(
-            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
-            for n in name
-        ):
+        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            logger.info("Skipping {}".format("/".join(name)))
            tf_weights.pop(txt_name, None)
            continue
@@ -33,8 +33,6 @@ logger = logging.getLogger(__name__)
 TF_DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
    "distilbert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-tf_model.h5",
    "distilbert-base-uncased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-distilled-squad-tf_model.h5",
-    "distilbert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-tf_model.h5",
-    "distilbert-base-cased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad-tf_model.h5",
    "distilbert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-tf_model.h5",
    "distilbert-base-uncased-finetuned-sst-2-english": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-tf_model.h5",
 }
@@ -80,6 +78,8 @@ class TFEmbeddings(tf.keras.layers.Layer):
            embeddings_initializer=get_initializer(config.initializer_range),
            name="position_embeddings",
        )
+        if config.sinusoidal_pos_embds:
+            raise NotImplementedError

        self.LayerNorm = tf.keras.layers.LayerNormalization(epsilon=1e-12, name="LayerNorm")
        self.dropout = tf.keras.layers.Dropout(config.dropout)
@@ -563,8 +563,8 @@ class TFDistilBertModel(TFDistilBertPreTrainedModel):
        import tensorflow as tf
        from transformers import DistilBertTokenizer, TFDistilBertModel

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = TFDistilBertModel.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')
        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
        outputs = model(input_ids)
        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
@@ -637,8 +637,8 @@ class TFDistilBertForMaskedLM(TFDistilBertPreTrainedModel):
        import tensorflow as tf
        from transformers import DistilBertTokenizer, TFDistilBertForMaskedLM

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = TFDistilBertForMaskedLM.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = TFDistilBertForMaskedLM.from_pretrained('distilbert-base-uncased')
        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
        outputs = model(input_ids)
        prediction_scores = outputs[0]
@@ -701,8 +701,8 @@ class TFDistilBertForSequenceClassification(TFDistilBertPreTrainedModel):
        import tensorflow as tf
        from transformers import DistilBertTokenizer, TFDistilBertForSequenceClassification

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
        outputs = model(input_ids)
        logits = outputs[0]
@@ -759,8 +759,8 @@ class TFDistilBertForTokenClassification(TFDistilBertPreTrainedModel):
        import tensorflow as tf
        from transformers import DistilBertTokenizer, TFDistilBertForTokenClassification

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = TFDistilBertForTokenClassification.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = TFDistilBertForTokenClassification.from_pretrained('distilbert-base-uncased')
        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
        outputs = model(input_ids)
        scores = outputs[0]
@@ -818,8 +818,8 @@ class TFDistilBertForQuestionAnswering(TFDistilBertPreTrainedModel):
        import tensorflow as tf
        from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering

-        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')
-        model = TFDistilBertForQuestionAnswering.from_pretrained('distilbert-base-cased')
+        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+        model = TFDistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')
        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
        outputs = model(input_ids)
        start_scores, end_scores = outputs[:2]
@@ -198,7 +198,7 @@ class TFBlock(tf.keras.layers.Layer):

 class TFGPT2MainLayer(tf.keras.layers.Layer):
    def __init__(self, config, *inputs, **kwargs):
-        super().__init__(*inputs, **kwargs)
+        super().__init__(config, *inputs, **kwargs)
        self.output_hidden_states = config.output_hidden_states
        self.output_attentions = config.output_attentions
        self.num_hidden_layers = config.n_layer
@@ -303,9 +303,11 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin):
            elif os.path.isfile(pretrained_model_name_or_path + ".index"):
                archive_file = pretrained_model_name_or_path + ".index"
            else:
-                archive_file = hf_bucket_url(
-                    pretrained_model_name_or_path, postfix=(WEIGHTS_NAME if from_pt else TF2_WEIGHTS_NAME)
-                )
+                archive_file = hf_bucket_url(pretrained_model_name_or_path, postfix=TF2_WEIGHTS_NAME)
+                if from_pt:
+                    raise EnvironmentError(
+                        "Loading a TF model from a PyTorch checkpoint is not supported when using a model identifier name."
+                    )

            # redirect to the cache, if necessary
            try:
@@ -90,9 +90,9 @@ class ProjectedAdaptiveLogSoftmax(nn.Module):
                labels :: [len*bsz]
            Return:
                if labels is None:
-                    out :: [len*bsz x n_tokens] log probabilities of tokens over the vocabulary
-                else:
                    out :: [len*bsz] Negative log likelihood
+                else:
+                    out :: [len*bsz x n_tokens] log probabilities of tokens over the vocabulary
            We could replace this implementation by the native PyTorch one
            if their's had an option to set bias on all clusters in the native one.
            here: https://github.com/pytorch/pytorch/blob/dbe6a7a9ff1a364a8706bf5df58a1ca96d2fd9da/torch/nn/modules/adaptive.py#L138
@@ -18,14 +18,12 @@

 import logging
 import os
-import typing

 import torch
 from torch import nn
 from torch.nn import CrossEntropyLoss
 from torch.nn import functional as F

-from .activations import get_activation
 from .configuration_utils import PretrainedConfig
 from .file_utils import (
    DUMMY_INPUTS,
@@ -423,9 +421,11 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
                )
                archive_file = pretrained_model_name_or_path + ".index"
            else:
-                archive_file = hf_bucket_url(
-                    pretrained_model_name_or_path, postfix=(TF2_WEIGHTS_NAME if from_tf else WEIGHTS_NAME)
-                )
+                archive_file = hf_bucket_url(pretrained_model_name_or_path, postfix=WEIGHTS_NAME)
+                if from_tf:
+                    raise EnvironmentError(
+                        "Loading a PyTorch model from a TF checkpoint is not supported when using a model identifier name."
+                    )

            # redirect to the cache, if necessary
            try:
@@ -1380,15 +1380,15 @@ class SequenceSummary(nn.Module):
                - 'attn' => Not implemented now, use multi-head attention
            summary_use_proj: Add a projection after the vector extraction
            summary_proj_to_labels: If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
-            summary_activation: 'tanh' or another string => add an activation to the output, Other => no activation. Default
+            summary_activation: 'tanh' => add a tanh activation to the output, Other => no activation. Default
            summary_first_dropout: Add a dropout before the projection and activation
            summary_last_dropout: Add a dropout after the projection and activation
    """

-    def __init__(self, config: PretrainedConfig):
+    def __init__(self, config):
        super().__init__()

-        self.summary_type = getattr(config, "summary_type", "last")
+        self.summary_type = config.summary_type if hasattr(config, "summary_type") else "last"
        if self.summary_type == "attn":
            # We should use a standard multi-head attention module with absolute positional embedding for that.
            # Cf. https://github.com/zihangdai/xlnet/blob/master/modeling.py#L253-L276
@@ -1403,10 +1403,9 @@ class SequenceSummary(nn.Module):
                num_classes = config.hidden_size
            self.summary = nn.Linear(config.hidden_size, num_classes)

-        activation_string = getattr(config, "summary_activation", None)
-        self.activation = (
-            get_activation(activation_string) if activation_string else Identity()
-        )  # type: typing.Callable
+        self.activation = Identity()
+        if hasattr(config, "summary_activation") and config.summary_activation == "tanh":
+            self.activation = nn.Tanh()

        self.first_dropout = Identity()
        if hasattr(config, "summary_first_dropout") and config.summary_first_dropout > 0:
@@ -26,7 +26,6 @@ from torch import nn
 from torch.nn import CrossEntropyLoss, MSELoss
 from torch.nn import functional as F

-from .activations import gelu
 from .configuration_xlm import XLMConfig
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import PreTrainedModel, SequenceSummary, SQuADHead, prune_linear_layer
@@ -56,6 +55,17 @@ def create_sinusoidal_embeddings(n_pos, dim, out):
    out.requires_grad = False


+def gelu(x):
+    """
+    GELU activation
+    https://arxiv.org/abs/1606.08415
+    https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.py#L14
+    https://github.com/huggingface/transformers/blob/master/modeling.py
+    """
+    # return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+    return 0.5 * x * (1.0 + torch.erf(x / math.sqrt(2.0)))
+
+
 def get_masks(slen, lengths, causal, padding_mask=None):
    """
    Generate hidden states mask, and optionally an attention mask.
@@ -18,13 +18,13 @@


 import logging
+import math

 import torch
 from torch import nn
 from torch.nn import CrossEntropyLoss, MSELoss
 from torch.nn import functional as F

-from .activations import gelu_new, swish
 from .configuration_xlnet import XLNetConfig
 from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_utils import PoolerAnswerClass, PoolerEndLogits, PoolerStartLogits, PreTrainedModel, SequenceSummary
@@ -183,7 +183,20 @@ def load_tf_weights_in_xlnet(model, config, tf_path):
    return model


-ACT2FN = {"gelu": gelu_new, "relu": torch.nn.functional.relu, "swish": swish}
+def gelu(x):
+    """ Implementation of the gelu activation function.
+        XLNet is using OpenAI GPT's gelu (not exactly the same as BERT)
+        Also see https://arxiv.org/abs/1606.08415
+    """
+    cdf = 0.5 * (1.0 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+    return x * cdf
+
+
+def swish(x):
+    return x * torch.sigmoid(x)
+
+
+ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish}


 XLNetLayerNorm = nn.LayerNorm
@@ -537,7 +550,7 @@ XLNET_INPUTS_DOCSTRING = r"""
        token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
            Segment token indices to indicate first and second portions of the inputs.
            Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1``
-            corresponds to a `sentence B` token. The classifier token should be represented by a ``2``.
+            corresponds to a `sentence B` token

            `What are token type IDs? <../glossary.html#token-type-ids>`_
        input_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
@@ -649,9 +649,6 @@ class NerPipeline(Pipeline):
        return answers


-TokenClassificationPipeline = NerPipeline
-
-
 class QuestionAnsweringArgumentHandler(ArgumentHandler):
    """
    QuestionAnsweringPipeline requires the user to provide multiple arguments (i.e. question & context) to be mapped
@@ -944,9 +941,9 @@ SUPPORTED_TASKS = {
        "tf": TFAutoModel if is_tf_available() else None,
        "pt": AutoModel if is_torch_available() else None,
        "default": {
-            "model": {"pt": "distilbert-base-cased", "tf": "distilbert-base-cased"},
+            "model": {"pt": "distilbert-base-uncased", "tf": "distilbert-base-uncased"},
            "config": None,
-            "tokenizer": "distilbert-base-cased",
+            "tokenizer": "distilbert-base-uncased",
        },
    },
    "sentiment-analysis": {
@@ -980,9 +977,12 @@ SUPPORTED_TASKS = {
        "tf": TFAutoModelForQuestionAnswering if is_tf_available() else None,
        "pt": AutoModelForQuestionAnswering if is_torch_available() else None,
        "default": {
-            "model": {"pt": "distilbert-base-cased-distilled-squad", "tf": "distilbert-base-cased-distilled-squad"},
+            "model": {
+                "pt": "distilbert-base-uncased-distilled-squad",
+                "tf": "distilbert-base-uncased-distilled-squad",
+            },
            "config": None,
-            "tokenizer": "distilbert-base-cased",
+            "tokenizer": "distilbert-base-uncased",
        },
    },
    "fill-mask": {
@@ -1004,7 +1004,6 @@ def pipeline(
    config: Optional[Union[str, PretrainedConfig]] = None,
    tokenizer: Optional[Union[str, PreTrainedTokenizer]] = None,
    modelcard: Optional[Union[str, ModelCard]] = None,
-    framework: Optional[str] = None,
    **kwargs
 ) -> Pipeline:
    """
@@ -1016,7 +1015,7 @@ def pipeline(

    Examples:
        pipeline('sentiment-analysis')
-        pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='bert-base-cased')
+        pipeline('question-answering', model='distilbert-base-uncased-distilled-squad', tokenizer='bert-base-cased')
        pipeline('ner', model=AutoModel.from_pretrained(...), tokenizer=AutoTokenizer.from_pretrained(...)
        pipeline('ner', model='dbmdz/bert-large-cased-finetuned-conll03-english', tokenizer='bert-base-cased')
        pipeline('ner', model='https://...pytorch-model.bin', config='https://...config.json', tokenizer='bert-base-cased')
@@ -1025,7 +1024,7 @@ def pipeline(
    if task not in SUPPORTED_TASKS:
        raise KeyError("Unknown task {}, available tasks are {}".format(task, list(SUPPORTED_TASKS.keys())))

-    framework = framework or get_framework(model)
+    framework = get_framework(model)

    targeted_task = SUPPORTED_TASKS[task]
    task, model_class = targeted_task["impl"], targeted_task[framework]
@@ -28,8 +28,6 @@ PRETRAINED_VOCAB_FILES_MAP = {
    "vocab_file": {
        "distilbert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt",
        "distilbert-base-uncased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt",
-        "distilbert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt",
-        "distilbert-base-cased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt",
        "distilbert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased-vocab.txt",
        "distilbert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt",
    }
@@ -38,8 +36,6 @@ PRETRAINED_VOCAB_FILES_MAP = {
 PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
    "distilbert-base-uncased": 512,
    "distilbert-base-uncased-distilled-squad": 512,
-    "distilbert-base-cased": 512,
-    "distilbert-base-cased-distilled-squad": 512,
    "distilbert-base-german-cased": 512,
    "distilbert-base-multilingual-cased": 512,
 }
@@ -48,8 +44,6 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
 PRETRAINED_INIT_CONFIGURATION = {
    "distilbert-base-uncased": {"do_lower_case": True},
    "distilbert-base-uncased-distilled-squad": {"do_lower_case": True},
-    "distilbert-base-cased": {"do_lower_case": False},
-    "distilbert-base-cased-distilled-squad": {"do_lower_case": False},
    "distilbert-base-german-cased": {"do_lower_case": False},
    "distilbert-base-multilingual-cased": {"do_lower_case": False},
 }
@@ -191,8 +191,15 @@ class GPT2Tokenizer(PreTrainedTokenizer):
        self.cache[token] = word
        return word

-    def _tokenize(self, text):
-        """ Tokenize a string. """
+    def _tokenize(self, text, add_prefix_space=False):
+        """ Tokenize a string.
+            Args:
+                - add_prefix_space (boolean, default False):
+                    Begin the sentence with at least one space to get invariance to word order in GPT-2 (and RoBERTa) tokenizers.
+        """
+        if add_prefix_space:
+            text = " " + text
+
        bpe_tokens = []
        for token in re.findall(self.pat, text):
            token = "".join(
@@ -241,11 +248,6 @@ class GPT2Tokenizer(PreTrainedTokenizer):

        return vocab_file, merge_file

-    def prepare_for_tokenization(self, text, **kwargs):
-        if "add_prefix_space" in kwargs and kwargs["add_prefix_space"]:
-            return " " + text
-        return text
-

 class GPT2TokenizerFast(PreTrainedTokenizerFast):
    vocab_files_names = VOCAB_FILES_NAMES
@@ -154,12 +154,3 @@ class RobertaTokenizer(GPT2Tokenizer):
        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
-
-    def prepare_for_tokenization(self, text, add_special_tokens=False, **kwargs):
-        if "add_prefix_space" in kwargs:
-            add_prefix_space = kwargs["add_prefix_space"]
-        else:
-            add_prefix_space = add_special_tokens
-        if add_prefix_space and not text[0].isspace():
-            text = " " + text
-        return text
@@ -159,8 +159,6 @@ class TransfoXLTokenizer(PreTrainedTokenizer):
        """Save the tokenizer vocabulary to a directory or file."""
        if os.path.isdir(vocab_path):
            vocab_file = os.path.join(vocab_path, VOCAB_FILES_NAMES["pretrained_vocab_file"])
-        else:
-            vocab_file = vocab_path
        torch.save(self.__dict__, vocab_file)
        return (vocab_file,)

@@ -662,12 +662,9 @@ class PreTrainedTokenizer(object):
            Take care of added tokens.

            text: The sequence to be encoded.
-            add_prefix_space: Only applies to GPT-2 and RoBERTa tokenizers. When `True`, this ensures that the sequence
-                begins with an empty space. False by default except for when using RoBERTa with `add_special_tokens=True`.
-            **kwargs: passed to the `prepare_for_tokenization` preprocessing method.
+            **kwargs: passed to the child `self.tokenize()` method
        """
        all_special_tokens = self.all_special_tokens
-        text = self.prepare_for_tokenization(text, **kwargs)

        def lowercase_text(t):
            # convert non-special tokens to lowercase
@@ -682,7 +679,7 @@ class PreTrainedTokenizer(object):
            result = []
            split_text = text.split(tok)
            for i, sub_text in enumerate(split_text):
-                sub_text = sub_text.rstrip()
+                sub_text = sub_text.strip()
                if i == 0 and not sub_text:
                    result += [tok]
                elif i == len(split_text) - 1:
@@ -700,7 +697,7 @@ class PreTrainedTokenizer(object):
            if not text.strip():
                return []
            if not tok_list:
-                return self._tokenize(text)
+                return self._tokenize(text, **kwargs)

            tokenized_text = []
            text_list = [text]
@@ -716,7 +713,7 @@ class PreTrainedTokenizer(object):
            return list(
                itertools.chain.from_iterable(
                    (
-                        self._tokenize(token) if token not in self.unique_added_tokens_encoder else [token]
+                        self._tokenize(token, **kwargs) if token not in self.unique_added_tokens_encoder else [token]
                        for token in tokenized_text
                    )
                )
@@ -805,8 +802,6 @@ class PreTrainedTokenizer(object):
                Defaults to False: no padding.
            return_tensors: (optional) can be set to 'tf' or 'pt' to return respectively TensorFlow tf.constant
                or PyTorch torch.Tensor instead of a list of python integers.
-            add_prefix_space: Only applies to GPT-2 and RoBERTa tokenizers. When `True`, this ensures that the sequence
-                begins with an empty space. False by default except for when using RoBERTa with `add_special_tokens=True`.
            **kwargs: passed to the `self.tokenize()` method
        """
        encoded_inputs = self.encode_plus(
@@ -870,8 +865,6 @@ class PreTrainedTokenizer(object):
                Defaults to False: no padding.
            return_tensors: (optional) can be set to 'tf' or 'pt' to return respectively TensorFlow tf.constant
                or PyTorch torch.Tensor instead of a list of python integers.
-            add_prefix_space: Only applies to GPT-2 and RoBERTa tokenizers. When `True`, this ensures that the sequence
-                begins with an empty space. False by default except for when using RoBERTa with `add_special_tokens=True`.
            return_token_type_ids: (optional) Set to False to avoid returning token_type_ids (default True).
            return_attention_mask: (optional) Set to False to avoid returning attention mask (default True)
            return_overflowing_tokens: (optional) Set to True to return overflowing token information (default False).
@@ -902,8 +895,7 @@ class PreTrainedTokenizer(object):

        def get_input_ids(text):
            if isinstance(text, str):
-                tokens = self.tokenize(text, add_special_tokens=add_special_tokens, **kwargs)
-                return self.convert_tokens_to_ids(tokens)
+                return self.convert_tokens_to_ids(self.tokenize(text, **kwargs))
            elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], str):
                return self.convert_tokens_to_ids(text)
            elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], int):
@@ -1223,10 +1215,6 @@ class PreTrainedTokenizer(object):

        return encoded_inputs

-    def prepare_for_tokenization(self, text, **kwargs):
-        """ Performs any necessary transformations before tokenization """
-        return text
-
    def truncate_sequences(
        self, ids, pair_ids=None, num_tokens_to_remove=0, truncation_strategy="longest_first", stride=0
    ):
@@ -76,10 +76,7 @@ def load_tf_weights_in_xxx(model, config, tf_checkpoint_path):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
-        if any(
-            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
-            for n in name
-        ):
+        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            logger.info("Skipping {}".format("/".join(name)))
            continue
        pointer = model
@@ -1,28 +0,0 @@
-import unittest
-
-from transformers import is_torch_available
-
-from .utils import require_torch
-
-
-if is_torch_available():
-    from transformers.activations import _gelu_python, get_activation, gelu_new
-    import torch
-
-
-@require_torch
-class TestActivations(unittest.TestCase):
-    def test_gelu_versions(self):
-        x = torch.Tensor([-100, -1, -0.1, 0, 0.1, 1.0, 100])
-        torch_builtin = get_activation("gelu")
-        self.assertTrue(torch.eq(_gelu_python(x), torch_builtin(x)).all().item())
-        self.assertFalse(torch.eq(_gelu_python(x), gelu_new(x)).all().item())
-
-    def test_get_activation(self):
-        get_activation("swish")
-        get_activation("relu")
-        get_activation("tanh")
-        with self.assertRaises(KeyError):
-            get_activation("bogus")
-        with self.assertRaises(KeyError):
-            get_activation(None)
@@ -438,34 +438,6 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
        config_and_inputs = self.model_tester.prepare_config_and_inputs_for_decoder()
        self.model_tester.create_and_check_bert_model_as_decoder(*config_and_inputs)

-    def test_bert_model_as_decoder_with_default_input_mask(self):
-        # This regression test was failing with PyTorch < 1.3
-        (
-            config,
-            input_ids,
-            token_type_ids,
-            input_mask,
-            sequence_labels,
-            token_labels,
-            choice_labels,
-            encoder_hidden_states,
-            encoder_attention_mask,
-        ) = self.model_tester.prepare_config_and_inputs_for_decoder()
-
-        input_mask = None
-
-        self.model_tester.create_and_check_bert_model_as_decoder(
-            config,
-            input_ids,
-            token_type_ids,
-            input_mask,
-            sequence_labels,
-            token_labels,
-            choice_labels,
-            encoder_hidden_states,
-            encoder_attention_mask,
-        )
-
    def test_for_masked_lm(self):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
        self.model_tester.create_and_check_bert_for_masked_lm(*config_and_inputs)
@@ -10,13 +10,13 @@ from .utils import require_tf, require_torch
 QA_FINETUNED_MODELS = {
    ("bert-base-uncased", "bert-large-uncased-whole-word-masking-finetuned-squad", None),
    ("bert-base-cased", "bert-large-cased-whole-word-masking-finetuned-squad", None),
-    ("bert-base-cased", "distilbert-base-cased-distilled-squad", None),
+    ("bert-base-uncased", "distilbert-base-uncased-distilled-squad", None),
 }

 TF_QA_FINETUNED_MODELS = {
    ("bert-base-uncased", "bert-large-uncased-whole-word-masking-finetuned-squad", None),
    ("bert-base-cased", "bert-large-cased-whole-word-masking-finetuned-squad", None),
-    ("bert-base-cased", "distilbert-base-cased-distilled-squad", None),
+    ("bert-base-uncased", "distilbert-base-uncased-distilled-squad", None),
 }

 TF_NER_FINETUNED_MODELS = {
@@ -38,13 +38,13 @@ NER_FINETUNED_MODELS = {
 FEATURE_EXTRACT_FINETUNED_MODELS = {
    ("bert-base-cased", "bert-base-cased", None),
    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
-    ("distilbert-base-cased", "distilbert-base-cased", None),
+    ("distilbert-base-uncased", "distilbert-base-uncased", None),
 }

 TF_FEATURE_EXTRACT_FINETUNED_MODELS = {
    ("bert-base-cased", "bert-base-cased", None),
    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
-    ("distilbert-base-cased", "distilbert-base-cased", None),
+    ("distilbert-base-uncased", "distilbert-base-uncased", None),
 }

 TF_TEXT_CLASSIF_FINETUNED_MODELS = {
@@ -94,7 +94,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
        for key in output_keys:
            self.assertIn(key, mono_result[0])

-        multi_result = [nlp(input) for input in valid_inputs]
+        multi_result = nlp(valid_inputs)
        self.assertIsInstance(multi_result, list)
        self.assertIsInstance(multi_result[0], (dict, list))

@@ -129,7 +129,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
        invalid_inputs = [None]
        for tokenizer, model, config in TF_NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer)
            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_torch
@@ -147,7 +147,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
        invalid_inputs = [None]
        for tokenizer, model, config in TF_TEXT_CLASSIF_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_torch
@@ -163,7 +163,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
        invalid_inputs = [None]
        for tokenizer, model, config in TF_FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})

    @require_torch
@@ -176,18 +176,14 @@ class MonoColumnInputTestCase(unittest.TestCase):
        invalid_inputs = [None]
        expected_multi_result = [
            [
-                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
-                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
+                {"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
+                {"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
            ],
            [
+                {"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
                {
-                    "sequence": "<s> The largest city in France is Paris</s>",
-                    "score": 0.3185044229030609,
-                    "token": 2201,
-                },
-                {
-                    "sequence": "<s> The largest city in France is Lyon</s>",
-                    "score": 0.21112334728240967,
+                    "score": 0.19764970242977142,
+                    "sequence": "<s>The largest city in France is Lyon</s>",
                    "token": 12790,
                },
            ],
@@ -213,24 +209,20 @@ class MonoColumnInputTestCase(unittest.TestCase):
        invalid_inputs = [None]
        expected_multi_result = [
            [
-                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
-                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
+                {"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
+                {"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
            ],
            [
+                {"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
                {
-                    "sequence": "<s> The largest city in France is Paris</s>",
-                    "score": 0.3185044229030609,
-                    "token": 2201,
-                },
-                {
-                    "sequence": "<s> The largest city in France is Lyon</s>",
-                    "score": 0.21112334728240967,
+                    "score": 0.19764970242977142,
+                    "sequence": "<s>The largest city in France is Lyon</s>",
                    "token": 12790,
                },
            ],
        ]
        for tokenizer, model, config in TF_FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, framework="tf", topk=2)
+            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
            self._test_mono_column_pipeline(
                nlp,
                valid_inputs,
@@ -301,5 +293,5 @@ class MultiColumnInputTestCase(unittest.TestCase):
        ]

        for tokenizer, model, config in TF_QA_FINETUNED_MODELS:
-            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer)
            self._test_multicolumn_pipeline(nlp, valid_samples, invalid_samples, mandatory_output_keys)
@@ -204,7 +204,7 @@ class TokenizerTesterMixin:
        encoded = tokenizer.encode(text, add_special_tokens=False)

        input_encoded = tokenizer.encode(input_text, add_special_tokens=False)
-        output_encoded = tokenizer.encode(" " + output_text, add_special_tokens=False)
+        output_encoded = tokenizer.encode(output_text, add_special_tokens=False)
        special_token_id = tokenizer.encode(special_token, add_special_tokens=False)
        assert encoded == input_encoded + special_token_id + output_encoded

@@ -264,7 +264,7 @@ class TokenizerTesterMixin:
        seq_1 = "With these inputs."

        sequences = tokenizer.encode(seq_0, seq_1, add_special_tokens=False)
-        attached_sequences = tokenizer.encode(seq_0, seq_1, add_special_tokens=True, add_prefix_space=False)
+        attached_sequences = tokenizer.encode(seq_0, seq_1, add_special_tokens=True)

        # Method is implemented (e.g. not GPT-2)
        if len(attached_sequences) != 2:
@@ -280,12 +280,7 @@ class TokenizerTesterMixin:
        num_added_tokens = tokenizer.num_added_tokens()
        total_length = len(sequence) + num_added_tokens
        information = tokenizer.encode_plus(
-            seq_0,
-            max_length=total_length - 2,
-            add_special_tokens=True,
-            stride=stride,
-            return_overflowing_tokens=True,
-            add_prefix_space=False,
+            seq_0, max_length=total_length - 2, add_special_tokens=True, stride=stride, return_overflowing_tokens=True,
        )

        truncated_sequence = information["input_ids"]
@@ -306,7 +301,7 @@ class TokenizerTesterMixin:
        sequence_0_no_special_tokens = tokenizer.encode(seq_0, add_special_tokens=False)
        sequence_1_no_special_tokens = tokenizer.encode(seq_1, add_special_tokens=False)

-        sequence = tokenizer.encode(seq_0, seq_1, add_special_tokens=True, add_prefix_space=False)
+        sequence = tokenizer.encode(seq_0, seq_1, add_special_tokens=True)
        truncated_second_sequence = tokenizer.build_inputs_with_special_tokens(
            tokenizer.encode(seq_0, add_special_tokens=False), tokenizer.encode(seq_1, add_special_tokens=False)[:-2],
        )
@@ -319,7 +314,6 @@ class TokenizerTesterMixin:
            stride=stride,
            truncation_strategy="only_second",
            return_overflowing_tokens=True,
-            add_prefix_space=False,
        )
        information_first_truncated = tokenizer.encode_plus(
            seq_0,
@@ -329,7 +323,6 @@ class TokenizerTesterMixin:
            stride=stride,
            truncation_strategy="only_first",
            return_overflowing_tokens=True,
-            add_prefix_space=False,
        )

        truncated_sequence = information["input_ids"]
@@ -349,39 +342,11 @@ class TokenizerTesterMixin:

        tokens = tokenizer.tokenize(sequence)
        input_ids = tokenizer.convert_tokens_to_ids(tokens)
-        formatted_input = tokenizer.encode(sequence, add_special_tokens=True, add_prefix_space=False)
+        formatted_input = tokenizer.encode(sequence, add_special_tokens=True)

        self.assertEqual(tokenizer.encode(tokens, add_special_tokens=True), formatted_input)
        self.assertEqual(tokenizer.encode(input_ids, add_special_tokens=True), formatted_input)

-    def test_swap_special_token(self):
-        tokenizer = self.get_tokenizer()
-
-        mask = "<mask>"
-        sequence = "Encode this sequence"
-        sequence_masked_0 = "Encode <mask> sequence"
-        sequence_masked_1 = "<mask> this sequence"
-
-        # Add tokens so that masked token isn't split
-        tokenizer.add_tokens(sequence.split())
-        tokenizer.add_special_tokens({"mask_token": mask})
-        mask_ind = tokenizer.convert_tokens_to_ids(mask)
-        encoded = tokenizer.encode(sequence, add_special_tokens=False)
-
-        # Test first masked sequence
-        encoded_masked = tokenizer.encode(sequence_masked_0, add_special_tokens=False)
-        mask_loc = encoded_masked.index(mask_ind)
-        encoded_masked[mask_loc] = encoded[mask_loc]
-
-        self.assertEqual(encoded_masked, encoded)
-
-        # Test second masked sequence
-        encoded_masked = tokenizer.encode(sequence_masked_1, add_special_tokens=False)
-        mask_loc = encoded_masked.index(mask_ind)
-        encoded_masked[mask_loc] = encoded[mask_loc]
-
-        self.assertEqual(encoded_masked, encoded)
-
    def test_special_tokens_mask(self):
        tokenizer = self.get_tokenizer()

@@ -391,7 +356,7 @@ class TokenizerTesterMixin:
        # Testing single inputs
        encoded_sequence = tokenizer.encode(sequence_0, add_special_tokens=False)
        encoded_sequence_dict = tokenizer.encode_plus(
-            sequence_0, add_special_tokens=True, return_special_tokens_mask=True, add_prefix_space=False
+            sequence_0, add_special_tokens=True, return_special_tokens_mask=True
        )
        encoded_sequence_w_special = encoded_sequence_dict["input_ids"]
        special_tokens_mask = encoded_sequence_dict["special_tokens_mask"]
@@ -404,10 +369,11 @@ class TokenizerTesterMixin:
        self.assertEqual(encoded_sequence, filtered_sequence)

        # Testing inputs pairs
-        encoded_sequence = tokenizer.encode(sequence_0, add_special_tokens=False)
-        encoded_sequence += tokenizer.encode(sequence_1, add_special_tokens=False)
+        encoded_sequence = tokenizer.encode(sequence_0, add_special_tokens=False) + tokenizer.encode(
+            sequence_1, add_special_tokens=False
+        )
        encoded_sequence_dict = tokenizer.encode_plus(
-            sequence_0, sequence_1, add_special_tokens=True, return_special_tokens_mask=True, add_prefix_space=False
+            sequence_0, sequence_1, add_special_tokens=True, return_special_tokens_mask=True
        )
        encoded_sequence_w_special = encoded_sequence_dict["input_ids"]
        special_tokens_mask = encoded_sequence_dict["special_tokens_mask"]
@@ -110,41 +110,3 @@ class RobertaTokenizationTest(TokenizerTesterMixin, unittest.TestCase):

        assert encoded_sentence == encoded_text_from_decode
        assert encoded_pair == encoded_pair_from_decode
-
-    def test_space_encoding(self):
-        tokenizer = self.get_tokenizer()
-
-        sequence = "Encode this sequence."
-        space_encoding = tokenizer.byte_encoder[" ".encode("utf-8")[0]]
-
-        # Testing encoder arguments
-        encoded = tokenizer.encode(sequence, add_special_tokens=False)
-        first_char = tokenizer.convert_ids_to_tokens(encoded[0])[0]
-        self.assertNotEqual(first_char, space_encoding)
-
-        encoded = tokenizer.encode(sequence, add_special_tokens=False, add_prefix_space=True)
-        first_char = tokenizer.convert_ids_to_tokens(encoded[0])[0]
-        self.assertEqual(first_char, space_encoding)
-
-        tokenizer.add_special_tokens({"bos_token": "<s>"})
-        encoded = tokenizer.encode(sequence, add_special_tokens=True)
-        first_char = tokenizer.convert_ids_to_tokens(encoded[1])[0]
-        self.assertEqual(first_char, space_encoding)
-
-        # Testing spaces after special tokenss
-        mask = "<mask>"
-        tokenizer.add_special_tokens({"mask_token": mask})
-        mask_ind = tokenizer.convert_tokens_to_ids(mask)
-
-        sequence = "Encode <mask> sequence"
-        sequence_nospace = "Encode <mask>sequence"
-
-        encoded = tokenizer.encode(sequence)
-        mask_loc = encoded.index(mask_ind)
-        first_char = tokenizer.convert_ids_to_tokens(encoded[mask_loc + 1])[0]
-        self.assertEqual(first_char, space_encoding)
-
-        encoded = tokenizer.encode(sequence_nospace)
-        mask_loc = encoded.index(mask_ind)
-        first_char = tokenizer.convert_ids_to_tokens(encoded[mask_loc + 1])[0]
-        self.assertNotEqual(first_char, space_encoding)
Author	SHA1	Message	Date
Lysandre	7ab2fe4471	RoBERTa TensorFlow Tests	2020-02-04 18:05:22 -05:00
Lysandre	23d03e4f60	Correct slow test	2020-02-04 18:05:22 -05:00
Lysandre	730d5e429b	Style	2020-02-04 18:05:22 -05:00
Lysandre	3d169f94fe	Flaubert PyTorch tests	2020-02-04 18:05:22 -05:00
Lysandre	28960e41fe	RoBERTa Pytorch tests	2020-02-04 18:05:22 -05:00