Remaining arguments

- mostly stylistic streamlining
- removed 'additional context' sections. They seem to be rarely used and might cause confusion. If more details are needed, users can add them to the 'details' section

.github/ISSUE_TEMPLATE/---new-benchmark.md

@@ -1,17 +1,17 @@
 ---
-name: "\U0001F5A5 New Benchmark"
-about: You benchmark a part of this library and would like to share your results
+name: "\U0001F5A5 New benchmark"
+about: Benchmark a part of this library and share your results
 title: "[Benchmark]"
 labels: ''
 assignees: ''
 
 ---
 
-# Benchmarking Transformers
+# 🖥 Benchmarking `transformers`
 
 ## Benchmark
 
-Which part of Transformers did you benchmark?
+Which part of `transformers` did you benchmark?
 
 ## Set-up
 

.github/ISSUE_TEMPLATE/--new-model-addition.md

@@ -1,5 +1,5 @@
 ---
-name: "\U0001F31FNew model addition"
+name: "\U0001F31F New model addition"
 about: Submit a proposal/request to implement a new Transformer-based model
 title: ''
 labels: ''
@@ -7,18 +7,14 @@ assignees: ''
 
 ---
 
-# 🌟New model addition
+# 🌟 New model addition
 
 ## Model description
 
 <!-- Important information -->
 
-## Open Source status
+## Open source status
 
 * [ ] the model implementation is available: (give details)
 * [ ] the model weights are available: (give details)
-* [ ] who are the authors: (mention them)
-
-## Additional context
-
-<!-- Add any other context about the problem here. -->
+* [ ] who are the authors: (mention them, if possible by @gh-username)

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,29 +1,29 @@
 ---
 name: "\U0001F41B Bug Report"
-about: Submit a bug report to help us improve PyTorch Transformers
+about: Submit a bug report to help us improve transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-## 🐛 Bug
+# 🐛 Bug
 
-<!-- Important information -->
+## Information
 
-Model I am using (Bert, XLNet....):
+Model I am using (Bert, XLNet ...):
 
-Language I am using the model on (English, Chinese....):
+Language I am using the model on (English, Chinese ...):
 
-The problem arise when using:
-* [ ] the official example scripts: (give details)
-* [ ] my own modified scripts: (give details)
+The problem arises when using:
+* [ ] the official example scripts: (give details below)
+* [ ] my own modified scripts: (give details below)
 
 The tasks I am working on is:
 * [ ] an official GLUE/SQUaD task: (give the name)
-* [ ] my own task or dataset: (give details)
+* [ ] my own task or dataset: (give details below)
 
-## To Reproduce
+## To reproduce
 
 Steps to reproduce the behavior:
 
@@ -31,22 +31,20 @@ Steps to reproduce the behavior:
 2.
 3.
 
-<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
+<!-- If you have code snippets, error messages, stack traces please provide them here as well.
+     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
+     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
 
 ## Expected behavior
 
-<!-- A clear and concise description of what you expected to happen. -->
+<!-- A clear and concise description of what you would expect to happen. -->
 
 ## Environment
 
 * OS:
 * Python version:
 * PyTorch version:
-* PyTorch Transformers version (or branch):
+* `transformers` version (or branch):
 * Using GPU ?
 * Distributed or parallel setup ?
 * Any other relevant information:
-
-## Additional context
-
-<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/feature-request.md

@@ -1,20 +1,25 @@
 ---
-name: "\U0001F680 Feature Request"
-about: Submit a proposal/request for a new PyTorch Transformers feature
+name: "\U0001F680 Feature request"
+about: Submit a proposal/request for a new transformers feature
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-## 🚀 Feature
+# 🚀 Feature request
 
-<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
+<!-- A clear and concise description of the feature proposal.
+     Please provide a link to the paper and code in case they exist. -->
 
 ## Motivation
 
-<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
+<!-- Please outline the motivation for the proposal. Is your feature request
+     related to a problem? e.g., I'm always frustrated when [...]. If this is related
+     to another GitHub issue, please link here too. -->
 
-## Additional context
+## Your contribution
 
-<!-- Add any other context or screenshots about the feature request here. -->
+<!-- Is there any way that you could help, e.g. by submitting a PR?
+     Make sure to read the CONTRIBUTING.MD readme:
+     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

.github/ISSUE_TEMPLATE/migration.md

@@ -1,47 +1,52 @@
 ---
-name: "\U0001F4DA Migration from PyTorch-pretrained-Bert"
-about: Report a problem when migrating from PyTorch-pretrained-Bert to Transformers
+name: "\U0001F4DA Migration from pytorch-pretrained-bert or pytorch-transformers"
+about: Report a problem when migrating from pytorch-pretrained-bert or pytorch-transformers to transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-## 📚 Migration
+# 📚 Migration
+
+## Information
 
 <!-- Important information -->
 
-Model I am using (Bert, XLNet....):
+Model I am using (Bert, XLNet ...):
 
-Language I am using the model on (English, Chinese....):
+Language I am using the model on (English, Chinese ...):
 
-The problem arise when using:
-* [ ] the official example scripts: (give details)
-* [ ] my own modified scripts: (give details)
+The problem arises when using:
+* [ ] the official example scripts: (give details below)
+* [ ] my own modified scripts: (give details below)
 
 The tasks I am working on is:
 * [ ] an official GLUE/SQUaD task: (give the name)
-* [ ] my own task or dataset: (give details)
+* [ ] my own task or dataset: (give details below)
 
-Details of the issue:
+## Details
 
-<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->
+<!-- A clear and concise description of the migration issue.
+    If you have code snippets, please provide it here as well.
+    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
+    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
+    -->
 
 ## Environment
 
 * OS:
 * Python version:
 * PyTorch version:
-* PyTorch Transformers version (or branch):
-* Using GPU ?
-* Distributed or parallel setup ?
+* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
+* `transformers` version (or branch):
+* Using GPU?
+* Distributed or parallel setup?
 * Any other relevant information:
 
 ## Checklist
 
 - [ ] I have read the migration guide in the readme.
+ ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
+  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
 - [ ] I checked if a related official extension example runs on my machine.
-
-## Additional context
-
-<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/question-help.md

@@ -1,12 +1,29 @@
 ---
-name: "❓Questions & Help"
-about: Start a general discussion related to PyTorch Transformers
+name: "❓ Questions & Help"
+about: Post your general questions on Stack Overflow tagged huggingface-transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-## ❓ Questions & Help
+# ❓ Questions & Help
 
-<!-- A clear and concise description of the question. -->
+<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
+     new models and benchmarks, and migration questions. For all other questions,
+     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
+     Tensorflow enthusiast can help you out. Make sure to tag your question with the
+     right deep learning framework as well as the huggingface-transformers tag: 
+     https://stackoverflow.com/questions/tagged/huggingface-transformers 
+     
+     If your question wasn't answered after a period of time on Stack Overflow, you
+     can always open a question on GitHub. You should then link to the SO question 
+     that you posted.
+     -->
+
+## Details
+<!-- Description of your issue -->
+
+<!-- You should first ask your question on SO, and only if
+     you didn't get an answer ask it here on GitHub. -->
+**A link to original question on Stack Overflow**: 

docs/source/model_doc/auto.rst

@@ -3,7 +3,7 @@ AutoModels
 
 In many cases, the architecture you want to use can be guessed from the name or the path of the pretrained model you are supplying to the ``from_pretrained`` method.
 
-AutoClasses are here to do this job for you so that you automatically retreive the relevant model given the name/path to the pretrained weights/config/vocabulary:
+AutoClasses are here to do this job for you so that you automatically retrieve the relevant model given the name/path to the pretrained weights/config/vocabulary:
 
 Instantiating one of ``AutoModel``, ``AutoConfig`` and ``AutoTokenizer`` will directly create a class of the relevant architecture (ex: ``model = AutoModel.from_pretrained('bert-base-cased')`` will create a instance of ``BertModel``).
 
@@ -15,6 +15,13 @@ Instantiating one of ``AutoModel``, ``AutoConfig`` and ``AutoTokenizer`` will di
     :members:
 
 
+``AutoTokenizer``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.AutoTokenizer
+    :members:
+
+
 ``AutoModel``
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -22,8 +29,37 @@ Instantiating one of ``AutoModel``, ``AutoConfig`` and ``AutoTokenizer`` will di
     :members:
 
 
-``AutoTokenizer``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``AutoModelForPreTraining``
+~~~~~~~~~~~~~~~~~~~~~
 
-.. autoclass:: transformers.AutoTokenizer
+.. autoclass:: transformers.AutoModelForPreTraining
     :members:
+
+
+``AutoModelWithLMHead``
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.AutoModelWithLMHead
+    :members:
+
+
+``AutoModelForSequenceClassification``
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.AutoModelForSequenceClassification
+    :members:
+
+
+``AutoModelForQuestionAnswering``
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.AutoModelForQuestionAnswering
+    :members:
+
+
+``AutoModelForTokenClassification``
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.AutoModelForTokenClassification
+    :members:
+

docs/source/pretrained_models.rst

@@ -88,6 +88,10 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``bert-base-finnish-uncased-v1``                           | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
 |                   |                                                            | | Trained on uncased Finnish text.                                                                                                    |
 |                   |                                                            | (see `details on turkunlp.org <http://turkunlp.org/FinBERT/>`__).                                                                     |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``bert-base-dutch-cased``                                  | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
+|                   |                                                            | | Trained on cased Dutch text.                                                                                                        |
+|                   |                                                            | (see `details on wietsedv repository <https://github.com/wietsedv/bertje/>`__).                                                       |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 | GPT               | ``openai-gpt``                                             | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
 |                   |                                                            | | OpenAI GPT English model                                                                                                            |

examples/README.md

@@ -135,21 +135,21 @@ Fine-tuning the library models for sequence classification on the GLUE benchmark
 Evaluation](https://gluebenchmark.com/). This script can fine-tune the following models: BERT, XLM, XLNet and RoBERTa. 
 
 GLUE is made up of a total of 9 different tasks. We get the following results on the dev set of the benchmark with an
-uncased  BERT base model (the checkpoint `bert-base-uncased`). All experiments ran on 8 V100 GPUs with a total train
-batch size of 24. Some of these tasks have a small dataset and training can lead to high variance in the results
+uncased  BERT base model (the checkpoint `bert-base-uncased`). All experiments ran single V100 GPUs with a total train
+batch sizes between 16 and 64. Some of these tasks have a small dataset and training can lead to high variance in the results
 between different runs. We report the median on 5 runs (with different seeds) for each of the metrics.
 
 | Task  | Metric                       | Result      |
 |-------|------------------------------|-------------|
-| CoLA  | Matthew's corr               | 48.87       |
-| SST-2 | Accuracy                     | 91.74       |
-| MRPC  | F1/Accuracy                  | 90.70/86.27 |
-| STS-B | Person/Spearman corr.        | 91.39/91.04 |
-| QQP   | Accuracy/F1                  | 90.79/87.66 |
-| MNLI  | Matched acc./Mismatched acc. | 83.70/84.83 |
-| QNLI  | Accuracy                     | 89.31       |
-| RTE   | Accuracy                     | 71.43       |
-| WNLI  | Accuracy                     | 43.66       |
+| CoLA  | Matthew's corr               | 49.23       |
+| SST-2 | Accuracy                     | 91.97       |
+| MRPC  | F1/Accuracy                  | 89.47/85.29 |
+| STS-B | Person/Spearman corr.        | 83.95/83.70 |
+| QQP   | Accuracy/F1                  | 88.40/84.31 |
+| MNLI  | Matched acc./Mismatched acc. | 80.61/81.08 |
+| QNLI  | Accuracy                     | 87.46       |
+| RTE   | Accuracy                     | 61.73       |
+| WNLI  | Accuracy                     | 45.07       |
 
 Some of these results are significantly different from the ones reported on the test set
 of GLUE benchmark on the website. For QQP and WNLI, please refer to [FAQ #12](https://gluebenchmark.com/faq) on the webite.

examples/distillation/README.md

@@ -2,23 +2,25 @@
 
 This folder contains the original code used to train Distil* as well as examples showcasing how to use DistilBERT, DistilRoBERTa and DistilGPT2.
 
-**December 6th, 2019 - Update** We release **DistilmBERT**: 92% of `bert-base-multilingual-cased` on XNLI. The model supports 104 different languages listed [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).
+**January 20, 2020 - Bug fixing** We have recently discovered and fixed [a bug](https://github.com/huggingface/transformers/commit/48cbf267c988b56c71a2380f748a3e6092ccaed3) in the evaluation of our `run_*.py` scripts that caused the reported metrics to be over-estimated on average. We have updated all the metrics with the latest runs.
 
-**November 19th, 2019 - Update** We release German **DistilBERT**: 98.8% of `bert-base-german-dbmdz-cased` on NER tasks.
+**December 6, 2019 - Update** We release **DistilmBERT**: 92% of `bert-base-multilingual-cased` on XNLI. The model supports 104 different languages listed [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).
 
-**October 23rd, 2019 - Update** We release **DistilRoBERTa**: 95% of `RoBERTa-base`'s performance on GLUE, twice as fast as RoBERTa while being 35% smaller.
+**November 19, 2019 - Update** We release German **DistilBERT**: 98.8% of `bert-base-german-dbmdz-cased` on NER tasks.
 
-**October 3rd, 2019 - Update** We release our [NeurIPS workshop paper](https://arxiv.org/abs/1910.01108) explaining our approach on **DistilBERT**. It includes updated results and further experiments. We applied the same method to GPT2 and release the weights of **DistilGPT2**. DistilGPT2 is two times faster and 33% smaller than GPT2. **The paper superseeds our [previous blogpost](https://medium.com/huggingface/distilbert-8cf3380435b5) with a different distillation loss and better performances. Please use the paper as a reference when comparing/reporting results on DistilBERT.**
+**October 23, 2019 - Update** We release **DistilRoBERTa**: 95% of `RoBERTa-base`'s performance on GLUE, twice as fast as RoBERTa while being 35% smaller.
 
-**September 19th, 2019 - Update:** We fixed bugs in the code and released an upadted version of the weights trained with a modification of the distillation loss. DistilBERT now reaches 97% of `BERT-base`'s performance on GLUE, and 86.9 F1 score on SQuAD v1.1 dev set (compared to 88.5 for `BERT-base`). We will publish a formal write-up of our approach in the near future!
+**October 3, 2019 - Update** We release our [NeurIPS workshop paper](https://arxiv.org/abs/1910.01108) explaining our approach on **DistilBERT**. It includes updated results and further experiments. We applied the same method to GPT2 and release the weights of **DistilGPT2**. DistilGPT2 is two times faster and 33% smaller than GPT2. **The paper superseeds our [previous blogpost](https://medium.com/huggingface/distilbert-8cf3380435b5) with a different distillation loss and better performances. Please use the paper as a reference when comparing/reporting results on DistilBERT.**
+
+**September 19, 2019 - Update:** We fixed bugs in the code and released an upadted version of the weights trained with a modification of the distillation loss. DistilBERT now reaches 99% of `BERT-base`'s performance on GLUE, and 86.9 F1 score on SQuAD v1.1 dev set (compared to 88.5 for `BERT-base`). We will publish a formal write-up of our approach in the near future!
 
 
 ## What is Distil*
 
-Distil* is a class of compressed models that started with DistilBERT. DistilBERT stands for Distillated-BERT. DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than `bert-base-uncased`, runs 60% faster while preserving 97% of BERT's performances as measured on the GLUE language understanding benchmark. DistilBERT is trained using knowledge distillation, a technique to compress a large model called the teacher into a smaller model called the student. By distillating Bert, we obtain a smaller Transformer model that bears a lot of similarities with the original BERT model while being lighter, smaller and faster to run. DistilBERT is thus an interesting option to put large-scaled trained Transformer model into production.
+Distil* is a class of compressed models that started with DistilBERT. DistilBERT stands for Distillated-BERT. DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than `bert-base-uncased`, runs 60% faster while preserving 99% of BERT's performances as measured on the GLUE language understanding benchmark. DistilBERT is trained using knowledge distillation, a technique to compress a large model called the teacher into a smaller model called the student. By distillating Bert, we obtain a smaller Transformer model that bears a lot of similarities with the original BERT model while being lighter, smaller and faster to run. DistilBERT is thus an interesting option to put large-scaled trained Transformer model into production.
 
 We have applied the same method to other Transformer architectures and released the weights:
-- GPT2: on the [WikiText-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark, GPT2 reaches a perplexity on the test set of 15.0 compared to 18.5 for **DistilGPT2** (after fine-tuning on the train set).
+- GPT2: on the [WikiText-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark, GPT2 reaches a perplexity on the test set of 16.3 compared to 21.1 for **DistilGPT2** (after fine-tuning on the train set).
 - RoBERTa: **DistilRoBERTa** reaches 95% of `RoBERTa-base`'s performance on GLUE while being twice faster and 35% smaller.
 - German BERT: **German DistilBERT** reaches 99% of `bert-base-german-dbmdz-cased`'s performance on German NER (CoNLL-2003).
 - Multilingual BERT: **DistilmBERT** reaches 92% of Multilingual BERT's performance on XNLI while being twice faster and 25% smaller. The model supports 104 languages listed [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).
@@ -29,11 +31,11 @@ Here are the results on the dev sets of GLUE:
 
 | Model                     | Macro-score                    | CoLA | MNLI | MRPC | QNLI | QQP  | RTE  | SST-2| STS-B| WNLI              |
 | :---:                     |    :---:                       | :---:| :---:| :---:| :---:| :---:| :---:| :---:| :---:| :---:             |
-| BERT-base                 |  **77.6**                      | 48.9 | 84.3 | 88.6 | 89.3 | 89.5 | 71.3 | 91.7 | 91.2 | 43.7              |
-| DistilBERT                |  **76.8**                      | 49.1 | 81.8 | 90.2 | 90.2 | 89.2 | 62.9 | 92.7 | 90.7 | 44.4              |
+| BERT-base-uncased         |  **77.6**                      | 49.2 | 80.8 | 87.4 | 87.5 | 86.4 | 61.7 | 92.0 | 83.8 | 45.1              |
+| DistilBERT-base-uncased   |  **76.8**                      | 43.6 | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7 | 81.2 | 56.3              |
 | ---                       |    ---                         |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  ---              |
 | RoBERTa-base (reported)   |  **83.2**/**86.4**<sup>2</sup> | 63.6 | 87.6 | 90.2 | 92.8 | 91.9 | 78.7 | 94.8 | 91.2 | 57.7<sup>3</sup>  |
-| DistilRoBERTa<sup>1</sup> |  **79.0**/**82.3**<sup>2</sup> | 59.4 | 83.9 | 86.6 | 90.8 | 89.4 | 67.9 | 92.5 | 88.3 | 52.1              |
+| DistilRoBERTa<sup>1</sup> |  **79.0**/**82.3**<sup>2</sup> | 59.3 | 84.0 | 86.6 | 90.8 | 89.4 | 67.9 | 92.5 | 88.3 | 52.1              |
 
 <sup>1</sup> We did not use the MNLI checkpoint for fine-tuning but directy perform transfer learning on the pre-trained DistilRoBERTa.
 

examples/run_glue.py

@@ -513,8 +513,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_lm_finetuning.py

@@ -202,8 +202,9 @@ def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> T
         tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
     ]
     probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
-    padding_mask = labels.eq(tokenizer.pad_token_id)
-    probability_matrix.masked_fill_(padding_mask, value=0.0)
+    if tokenizer._pad_token is not None:
+        padding_mask = labels.eq(tokenizer.pad_token_id)
+        probability_matrix.masked_fill_(padding_mask, value=0.0)
     masked_indices = torch.bernoulli(probability_matrix).bool()
     labels[~masked_indices] = -100  # We only compute loss on masked tokens
 
@@ -228,6 +229,8 @@ def train(args, train_dataset, model: PreTrainedModel, tokenizer: PreTrainedToke
     args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
 
     def collate(examples: List[torch.Tensor]):
+        if tokenizer._pad_token is None:
+            return pad_sequence(examples, batch_first=True)
         return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
 
     train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
@@ -421,6 +424,8 @@ def evaluate(args, model: PreTrainedModel, tokenizer: PreTrainedTokenizer, prefi
     # Note that DistributedSampler samples randomly
 
     def collate(examples: List[torch.Tensor]):
+        if tokenizer._pad_token is None:
+            return pad_sequence(examples, batch_first=True)
         return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
 
     eval_sampler = SequentialSampler(eval_dataset)
@@ -569,8 +574,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--save_total_limit",
         type=int,

examples/run_multiple_choice.py

@@ -478,8 +478,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_ner.py

@@ -485,8 +485,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_squad.py

@@ -636,8 +636,8 @@ def main():
         "A number of warnings are expected for a normal SQuAD evaluation.",
     )
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_tf_glue.py

@@ -1,3 +1,22 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
+
+
+import argparse
+import logging
 import os
 
 import tensorflow as tf
@@ -5,104 +24,309 @@ import tensorflow_datasets
 
 from transformers import (
     BertConfig,
-    BertForSequenceClassification,
     BertTokenizer,
+    DistilBertConfig,
+    DistilBertTokenizer,
+    RobertaConfig,
+    RobertaTokenizer,
     TFBertForSequenceClassification,
-    glue_convert_examples_to_features,
-    glue_processors,
+    TFDistilBertForSequenceClassification,
+    TFRobertaForSequenceClassification,
+    TFXLMForSequenceClassification,
+    TFXLNetForSequenceClassification,
+    XLMConfig,
+    XLMTokenizer,
+    XLNetConfig,
+    XLNetTokenizer,
+)
+from transformers import glue_convert_examples_to_features as convert_examples_to_features
+from transformers import glue_output_modes as output_modes
+from transformers import glue_processors as processors
+
+
+logger = logging.getLogger(__name__)
+
+ALL_MODELS = sum(
+    (
+        tuple(conf.pretrained_config_archive_map.keys())
+        for conf in (BertConfig, XLNetConfig, XLMConfig, RobertaConfig, DistilBertConfig)
+    ),
+    (),
 )
 
-
-# script parameters
-BATCH_SIZE = 32
-EVAL_BATCH_SIZE = BATCH_SIZE * 2
-USE_XLA = False
-USE_AMP = False
-EPOCHS = 3
-
-TASK = "mrpc"
-
-if TASK == "sst-2":
-    TFDS_TASK = "sst2"
-elif TASK == "sts-b":
-    TFDS_TASK = "stsb"
-else:
-    TFDS_TASK = TASK
-
-num_labels = len(glue_processors[TASK]().get_labels())
-print(num_labels)
-
-tf.config.optimizer.set_jit(USE_XLA)
-tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})
-
-# Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
-config = BertConfig.from_pretrained("bert-base-cased", num_labels=num_labels)
-tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
-model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
-
-# Load dataset via TensorFlow Datasets
-data, info = tensorflow_datasets.load(f"glue/{TFDS_TASK}", with_info=True)
-train_examples = info.splits["train"].num_examples
-
-# MNLI expects either validation_matched or validation_mismatched
-valid_examples = info.splits["validation"].num_examples
-
-# Prepare dataset for GLUE as a tf.data.Dataset instance
-train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, 128, TASK)
-
-# MNLI expects either validation_matched or validation_mismatched
-valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, 128, TASK)
-train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
-valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)
-
-# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
-opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
-if USE_AMP:
-    # loss scaling is currently required when using mixed precision
-    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
+MODEL_CLASSES = {
+    "bert": (BertConfig, TFBertForSequenceClassification, BertTokenizer),
+    "xlnet": (XLNetConfig, TFXLNetForSequenceClassification, XLNetTokenizer),
+    "xlm": (XLMConfig, TFXLMForSequenceClassification, XLMTokenizer),
+    "roberta": (RobertaConfig, TFRobertaForSequenceClassification, RobertaTokenizer),
+    "distilbert": (DistilBertConfig, TFDistilBertForSequenceClassification, DistilBertTokenizer),
+}
 
 
-if num_labels == 1:
-    loss = tf.keras.losses.MeanSquaredError()
-else:
-    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
+def load_and_cache_examples(args, data, task, tokenizer, split):
+    if task == "mnli" and split == "validation":
+        split = "validation_matched"
 
-metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
-model.compile(optimizer=opt, loss=loss, metrics=[metric])
+    features_output_dir = os.path.join(args.output_dir, "features")
+    cached_features_file = os.path.join(
+        features_output_dir,
+        "cached_{}_{}_{}_{}.tfrecord".format(
+            split, list(filter(None, args.model_name_or_path.split("/"))).pop(), str(args.max_seq_length), str(task)
+        ),
+    )
 
-# Train and evaluate using tf.keras.Model.fit()
-train_steps = train_examples // BATCH_SIZE
-valid_steps = valid_examples // EVAL_BATCH_SIZE
+    if not os.path.exists(cached_features_file) or args.overwrite_cache:
+        logger.info("Converting examples to features")
+        dataset = convert_examples_to_features(data[split], tokenizer, args.max_seq_length, task)
 
-history = model.fit(
-    train_dataset,
-    epochs=EPOCHS,
-    steps_per_epoch=train_steps,
-    validation_data=valid_dataset,
-    validation_steps=valid_steps,
-)
+        if not os.path.exists(features_output_dir):
+            os.makedirs(features_output_dir)
 
-# Save TF2 model
-os.makedirs("./save/", exist_ok=True)
-model.save_pretrained("./save/")
+        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
+            for feature in dataset:
+                example, label = feature
+                feature_key_value_pair = {
+                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
+                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
+                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
+                    "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
+                }
+                features = tf.train.Features(feature=feature_key_value_pair)
+                example = tf.train.Example(features=features)
 
-if TASK == "mrpc":
-    # Load the TensorFlow model in PyTorch for inspection
-    # This is to demo the interoperability between the two frameworks, you don't have to
-    # do this in real life (you can run the inference on the TF model).
-    pytorch_model = BertForSequenceClassification.from_pretrained("./save/", from_tf=True)
+                tfwriter.write(example.SerializeToString())
 
-    # Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
-    sentence_0 = "This research was consistent with his findings."
-    sentence_1 = "His findings were compatible with this research."
-    sentence_2 = "His findings were not compatible with this research."
-    inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors="pt")
-    inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors="pt")
+        logger.info("Features saved to cache")
 
-    del inputs_1["special_tokens_mask"]
-    del inputs_2["special_tokens_mask"]
+    features = {
+        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "label": tf.io.FixedLenFeature([], tf.int64),
+    }
 
-    pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
-    pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
-    print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
-    print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")
+    def select_data_from_record(record):
+        record = tf.io.parse_single_example(record, features)
+        x = {
+            "input_ids": record["input_ids"],
+            "attention_mask": record["attention_mask"],
+            "token_type_ids": record["token_type_ids"],
+        }
+        y = record["label"]
+        return (x, y)
+
+    dataset = tf.data.TFRecordDataset(cached_features_file)
+    dataset = dataset.map(select_data_from_record)
+
+    logger.info("Created dataset %s from TFRecord" % split)
+    return dataset
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        "--model_type",
+        default=None,
+        type=str,
+        required=True,
+        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        default=None,
+        type=str,
+        required=True,
+        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
+    )
+    parser.add_argument(
+        "--task_name",
+        default=None,
+        type=str,
+        required=True,
+        help="The name of the task to train selected in the list: " + ", ".join(processors.keys()),
+    )
+    parser.add_argument(
+        "--output_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The output directory where the model predictions and checkpoints will be written.",
+    )
+    parser.add_argument(
+        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
+    )
+
+    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
+    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
+    parser.add_argument(
+        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
+    )
+
+    parser.add_argument("--train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
+    parser.add_argument(
+        "--valid_batch_size", default=8, type=int, help="Batch size per GPU/CPU for validation during training."
+    )
+    parser.add_argument(
+        "--test_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation after training."
+    )
+    parser.add_argument(
+        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
+    )
+
+    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
+    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
+    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
+    parser.add_argument(
+        "--max_seq_length",
+        default=128,
+        type=int,
+        help="The maximum total input sequence length after tokenization. Sequences longer "
+        "than this will be truncated, sequences shorter will be padded.",
+    )
+    parser.add_argument(
+        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
+    )
+
+    parser.add_argument(
+        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
+    )
+    parser.add_argument(
+        "--tokenizer_name",
+        default="",
+        type=str,
+        help="Pretrained tokenizer name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--cache_dir",
+        default="",
+        type=str,
+        help="Where do you want to store the pre-trained models downloaded from s3",
+    )
+    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
+    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
+    parser.add_argument(
+        "--force_download",
+        action="store_true",
+        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
+    )
+    args = parser.parse_args()
+
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+
+    if os.path.exists(args.output_dir) and args.do_train:
+        if not args.overwrite_output_dir and bool(
+            [file for file in os.listdir(args.output_dir) if "features" not in file]
+        ):
+            raise ValueError(
+                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
+                    args.output_dir
+                )
+            )
+
+    args.model_type = args.model_type.lower()
+    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
+
+    TASK = args.task_name.lower()
+
+    if TASK not in processors:
+        raise ValueError("Task not found: %s" % (TASK))
+
+    if TASK == "sst-2":
+        TFDS_TASK = "sst2"
+    elif TASK == "sts-b":
+        TFDS_TASK = "stsb"
+    else:
+        TFDS_TASK = TASK
+
+    num_labels = len(processors[TASK]().get_labels())
+    print(num_labels)
+
+    tf.config.optimizer.set_jit(args.xla)
+    tf.config.optimizer.set_experimental_options({"auto_mixed_precision": args.amp})
+
+    # Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
+    config = config_class.from_pretrained(
+        args.config_name if args.config_name else args.model_name_or_path,
+        num_labels=num_labels,
+        finetuning_task=args.task_name,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+        force_download=args.force_download,
+    )
+
+    tokenizer = tokenizer_class.from_pretrained(
+        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
+        do_lower_case=args.do_lower_case,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+        force_download=args.force_download,
+    )
+
+    model = model_class.from_pretrained(
+        args.model_name_or_path,
+        config=config,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+        force_download=args.force_download,
+    )
+
+    # Load dataset via TensorFlow Datasets
+    data, info = tensorflow_datasets.load("glue/%s" % TFDS_TASK, with_info=True)
+
+    # Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
+    opt = tf.keras.optimizers.Adam(learning_rate=args.learning_rate, epsilon=args.adam_epsilon)
+
+    if args.amp:
+        # loss scaling is currently required when using mixed precision
+        opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
+
+    if num_labels == 1:
+        loss = tf.keras.losses.MeanSquaredError()
+    else:
+        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
+
+    metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
+    model.compile(optimizer=opt, loss=loss, metrics=[metric])
+
+    class save_model(tf.keras.callbacks.Callback):
+        def on_epoch_end(self, epoch, logs=None):
+            print("Saving model at epoch {}".format(epoch))
+            output_dir = os.path.join(args.output_dir, "checkpoint-epoch-{}".format(epoch))
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir)
+            self.model.save_pretrained(output_dir)
+
+    if args.do_train:
+        train_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="train")
+        train_dataset = train_dataset.batch(args.train_batch_size).repeat(args.num_train_epochs)
+        train_examples = info.splits["train"].num_examples / args.train_batch_size
+
+        validation_identifier = "validation_mismatched" if TASK == "mnli" else "validation"
+        valid_dataset = load_and_cache_examples(
+            args, data=data, task=TASK, tokenizer=tokenizer, split=validation_identifier
+        )
+        valid_dataset = valid_dataset.batch(args.valid_batch_size)
+        valid_examples = info.splits[validation_identifier].num_examples / args.valid_batch_size
+
+        history = model.fit(
+            train_dataset,
+            steps_per_epoch=train_examples,
+            epochs=args.num_train_epochs,
+            validation_data=valid_dataset if args.evaluate_during_training else None,
+            validation_steps=valid_examples if args.evaluate_during_training else None,
+            callbacks=[save_model()],
+        )
+
+    if args.do_eval:
+        test_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="test")
+        test_dataset = test_dataset.batch(args.test_batch_size)
+        test_examples = info.splits["test"].num_examples / args.test_batch_size
+
+        results = model.evaluate(test_dataset, steps=test_examples)
+
+
+if __name__ == "__main__":
+    main()

examples/run_tf_squad.py

@@ -0,0 +1,699 @@
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
+from __future__ import absolute_import, division, print_function
+
+import argparse
+import datetime
+import glob
+import logging
+import math
+import os
+import pickle
+
+import regex as re
+import tensorflow as tf
+from fastprogress import master_bar, progress_bar
+from seqeval import metrics
+from tqdm import tqdm, trange
+
+from transformers import (
+    TF2_WEIGHTS_NAME,
+    BertConfig,
+    BertTokenizer,
+    DistilBertConfig,
+    DistilBertTokenizer,
+    GradientAccumulator,
+    SquadV1Processor,
+    SquadV2Processor,
+    TFBertForQuestionAnswering,
+    TFDistilBertForQuestionAnswering,
+    TFXLMForQuestionAnsweringSimple,
+    TFXLNetForQuestionAnsweringSimple,
+    XLMConfig,
+    XLMTokenizer,
+    XLNetConfig,
+    XLNetTokenizer,
+    create_optimizer,
+    squad_convert_examples_to_features,
+)
+from transformers.data.metrics.squad_metrics import compute_predictions_logits, squad_evaluate
+from transformers.data.processors.squad import SquadResult
+
+
+logger = logging.getLogger(__name__)
+
+ALL_MODELS = sum(
+    (
+        tuple(conf.pretrained_config_archive_map.keys())
+        for conf in (BertConfig, XLNetConfig, XLMConfig, DistilBertConfig)
+    ),
+    (),
+)
+
+MODEL_CLASSES = {
+    "bert": (BertConfig, TFBertForQuestionAnswering, BertTokenizer),
+    "xlnet": (XLNetConfig, TFXLNetForQuestionAnsweringSimple, XLNetTokenizer),
+    "xlm": (XLMConfig, TFXLMForQuestionAnsweringSimple, XLMTokenizer),
+    "distilbert": (DistilBertConfig, TFDistilBertForQuestionAnswering, DistilBertTokenizer),
+}
+
+
+def train(args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size):
+    if args.max_steps > 0:
+        num_train_steps = args.max_steps * args.gradient_accumulation_steps
+        args.num_train_epochs = 1
+    else:
+        num_train_steps = (
+            math.ceil(num_train_examples / train_batch_size)
+            // args.gradient_accumulation_steps
+            * args.num_train_epochs
+        )
+
+    writer = tf.summary.create_file_writer("/tmp/mylogs")
+
+    with strategy.scope():
+        loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(
+            reduction=tf.keras.losses.Reduction.NONE, from_logits=True
+        )
+        optimizer = create_optimizer(args.learning_rate, num_train_steps, args.warmup_steps)
+
+        if args.xla:
+            tf.config.optimizer.set_jit(True)
+
+        if args.amp:
+            optimizer = tf.keras.mixed_precision.experimental.LossScaleOptimizer(optimizer, "dynamic")
+
+        loss_metric = tf.keras.metrics.Mean(name="loss", dtype=tf.float32)
+        gradient_accumulator = GradientAccumulator()
+
+    logging.info("***** Running training *****")
+    logging.info("  Num examples = %d", num_train_examples)
+    logging.info("  Num Epochs = %d", args.num_train_epochs)
+    logging.info("  Instantaneous batch size per device = %d", args.per_device_train_batch_size)
+    logging.info(
+        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
+        train_batch_size * args.gradient_accumulation_steps,
+    )
+    logging.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
+    logging.info("  Total training steps = %d", num_train_steps)
+
+    model.summary()
+
+    @tf.function
+    def apply_gradients():
+        grads_and_vars = []
+
+        for gradient, variable in zip(gradient_accumulator.gradients, model.trainable_variables):
+            if gradient is not None:
+                scaled_gradient = gradient / (args.n_device * args.gradient_accumulation_steps)
+                grads_and_vars.append((scaled_gradient, variable))
+            else:
+                grads_and_vars.append((gradient, variable))
+
+        optimizer.apply_gradients(grads_and_vars, args.max_grad_norm)
+        gradient_accumulator.reset()
+
+    @tf.function
+    def train_step(train_features, train_labels):
+        def step_fn(train_features, train_labels):
+            with tf.GradientTape() as tape:
+                start_logits, end_logits = model(train_features)
+                start_logits = tf.multiply(
+                    start_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32)
+                )
+                end_logits = tf.multiply(end_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32))
+                start_loss = loss_fct(train_labels["start_position"], start_logits)
+                end_loss = loss_fct(train_labels["end_position"], end_logits)
+                total_loss = (start_loss + end_loss) / 2
+
+                loss = tf.reduce_sum(total_loss) * (1.0 / train_batch_size)
+                grads = tape.gradient(loss, model.trainable_variables)
+
+                gradient_accumulator(grads)
+
+            return total_loss
+
+        per_example_losses = strategy.experimental_run_v2(step_fn, args=(train_features, train_labels))
+        mean_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_example_losses, axis=0)
+
+        return mean_loss
+
+    current_time = datetime.datetime.now()
+    train_iterator = master_bar(range(args.num_train_epochs))
+    global_step = 0
+    logging_loss = 0.0
+
+    for epoch in train_iterator:
+        epoch_iterator = progress_bar(
+            train_dataset,
+            total=num_train_steps / args.num_train_epochs,
+            parent=train_iterator,
+            display=args.n_device > 1,
+        )
+        step = 1
+
+        with strategy.scope():
+            for train_features, train_labels in tqdm(
+                epoch_iterator,
+                desc="Training, epoch {}".format(epoch),
+                total=int(num_train_steps / args.num_train_epochs),
+            ):
+                loss = train_step(train_features, train_labels)
+
+                if step % args.gradient_accumulation_steps == 0:
+                    strategy.experimental_run_v2(apply_gradients)
+
+                    loss_metric(loss)
+
+                    global_step += 1
+
+                    if args.logging_steps > 0 and global_step % args.logging_steps == 0:
+                        # Log metrics
+                        if args.n_device == 1 and args.evaluate_during_training:
+                            # Only evaluate when single GPU otherwise metrics may not average well
+                            results = evaluate(args, strategy, model, tokenizer, global_step)
+
+                            with writer.as_default():
+                                tf.summary.scalar("exact", results[0], global_step)
+                                tf.summary.scalar("f1", results[1], global_step)
+
+                        lr = optimizer.learning_rate
+                        learning_rate = lr(step)
+
+                        with writer.as_default():
+                            tf.summary.scalar("lr", learning_rate, global_step)
+                            tf.summary.scalar(
+                                "loss", (loss_metric.result() - logging_loss) / args.logging_steps, global_step
+                            )
+
+                        logging_loss = loss_metric.result()
+
+                    with writer.as_default():
+                        tf.summary.scalar("loss", loss_metric.result(), step=step)
+
+                    if args.save_steps > 0 and global_step % args.save_steps == 0:
+                        # Save model checkpoint
+                        output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
+
+                        if not os.path.exists(output_dir):
+                            os.makedirs(output_dir)
+
+                        model.save_pretrained(output_dir)
+                        logging.info("Saving model checkpoint to %s", output_dir)
+
+                train_iterator.child.comment = f"loss : {loss_metric.result()}"
+                step += 1
+
+        train_iterator.write(f"loss epoch {epoch + 1}: {loss_metric.result()}")
+
+        loss_metric.reset_states()
+
+    logging.info("  Training took time = {}".format(datetime.datetime.now() - current_time))
+
+
+def evaluate(args, strategy, model, tokenizer, prefix):
+    eval_batch_size = args.per_device_eval_batch_size * args.n_device
+    eval_dataset, size, examples, features = load_and_cache_examples(
+        args, tokenizer, evaluate=True, output_examples=True
+    )
+    eval_dataset = eval_dataset.batch(args.per_device_eval_batch_size)
+    eval_dataset = strategy.experimental_distribute_dataset(eval_dataset)
+    num_eval_steps = math.ceil(size / eval_batch_size)
+    master = master_bar(range(1))
+    eval_iterator = progress_bar(eval_dataset, total=num_eval_steps, parent=master, display=args.n_device > 1)
+
+    logging.info("***** Running evaluation *****")
+    logging.info("  Num examples = %d", size)
+    logging.info("  Batch size = %d", eval_batch_size)
+
+    all_results = []
+    for index, (eval_features, eval_labels) in tqdm(
+        enumerate(eval_iterator), total=size / args.per_device_eval_batch_size
+    ):
+        with strategy.scope():
+            start_logits, end_logits = model(eval_features)
+
+            for sample_index in range(args.per_device_eval_batch_size):
+                if index * args.per_device_eval_batch_size + sample_index < len(features):
+                    result = SquadResult(
+                        features[index * args.per_device_eval_batch_size + sample_index].unique_id,
+                        start_logits[sample_index],
+                        end_logits[sample_index],
+                    )
+                    all_results.append(result)
+
+    output_prediction_file = os.path.join(args.output_dir, "predictions_{}.json".format(prefix))
+    output_nbest_file = os.path.join(args.output_dir, "nbest_predictions_{}.json".format(prefix))
+
+    if args.version_2_with_negative:
+        output_null_log_odds_file = os.path.join(args.output_dir, "null_odds_{}.json".format(prefix))
+    else:
+        output_null_log_odds_file = None
+
+    predictions = compute_predictions_logits(
+        examples,
+        features,
+        all_results,
+        args.n_best_size,
+        args.max_answer_length,
+        args.do_lower_case,
+        output_prediction_file,
+        output_nbest_file,
+        output_null_log_odds_file,
+        args.verbose_logging,
+        args.version_2_with_negative,
+        args.null_score_diff_threshold,
+        tokenizer,
+    )
+
+    # Compute the F1 and exact scores.
+    results = squad_evaluate(examples, predictions)
+    return results
+
+
+def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
+    features_output_dir = os.path.join(args.output_dir, "features")
+    cached_features_file = os.path.join(
+        features_output_dir,
+        "cached_{}_{}_{}.tfrecord".format(
+            "dev" if evaluate else "train",
+            list(filter(None, args.model_name_or_path.split("/"))).pop(),
+            str(args.max_seq_length),
+        ),
+    )
+
+    if not os.path.exists(cached_features_file) or args.overwrite_cache:
+        if args.version_2_with_negative:
+            processor = SquadV2Processor()
+        else:
+            processor = SquadV1Processor()
+
+        if args.data_dir:
+            directory = args.directory
+            examples = processor.get_dev_examples(directory) if evaluate else processor.get_train_examples(directory)
+        else:
+            try:
+                import tensorflow_datasets as tfds
+            except ImportError:
+                raise ImportError("If not data_dir is specified, tensorflow_datasets needs to be installed.")
+
+            if args.version_2_with_negative:
+                logger.warning("tensorflow_datasets does not handle version 2 of SQuAD.")
+
+            examples = processor.get_examples_from_dataset(tfds.load("squad"), evaluate=evaluate)
+            examples = examples
+
+        logger.info("Converting examples to features")
+        features, dataset = squad_convert_examples_to_features(
+            examples,
+            tokenizer,
+            args.max_seq_length,
+            args.doc_stride,
+            args.max_query_length,
+            is_training=not evaluate,
+            return_dataset="tf",
+        )
+
+        if not os.path.exists(features_output_dir):
+            os.makedirs(features_output_dir)
+
+        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
+            for feature in tqdm(dataset, desc="Building tfrecord dataset", total=len(features)):
+                example, result = feature
+                feature_key_value_pair = {
+                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
+                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
+                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
+                    "start_position": tf.train.Feature(
+                        int64_list=tf.train.Int64List(value=[result["start_position"]])
+                    ),
+                    "end_position": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["end_position"]])),
+                    "cls_index": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["cls_index"]])),
+                    "p_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=result["p_mask"])),
+                }
+                feature_skeleton = tf.train.Features(feature=feature_key_value_pair)
+                example = tf.train.Example(features=feature_skeleton)
+
+                tfwriter.write(example.SerializeToString())
+
+            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "wb") as handle:
+                pickle.dump(examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
+
+            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "wb") as handle:
+                pickle.dump(features, handle, protocol=pickle.HIGHEST_PROTOCOL)
+
+        logger.info("Features saved to cache")
+
+    feature_skeleton = {
+        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+        "start_position": tf.io.FixedLenFeature([], tf.int64),
+        "end_position": tf.io.FixedLenFeature([], tf.int64),
+        "cls_index": tf.io.FixedLenFeature([], tf.int64),
+        "p_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
+    }
+
+    def select_data_from_record(record):
+        record = tf.io.parse_single_example(record, feature_skeleton)
+        x = {
+            "input_ids": record["input_ids"],
+            "attention_mask": record["attention_mask"],
+            "token_type_ids": record["token_type_ids"],
+        }
+        y = {
+            "start_position": record["start_position"],
+            "end_position": record["end_position"],
+            "cls_index": record["cls_index"],
+            "p_mask": record["p_mask"],
+        }
+        return x, y
+
+    dataset = tf.data.TFRecordDataset(cached_features_file)
+    dataset = dataset.map(select_data_from_record)
+
+    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "rb") as handle:
+        examples = pickle.load(handle)
+
+    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "rb") as handle:
+        features = pickle.load(handle)
+
+    logger.info("Created dataset %s from TFRecord" % "dev" if evaluate else "train")
+
+    if output_examples:
+        return dataset, len(list(dataset.__iter__())), examples, features
+    return dataset, len(list(dataset.__iter__()))
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        "--model_type",
+        default=None,
+        type=str,
+        required=True,
+        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        default=None,
+        type=str,
+        required=True,
+        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
+    )
+    parser.add_argument(
+        "--output_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The output directory where the model predictions and checkpoints will be written.",
+    )
+    parser.add_argument(
+        "--data_dir",
+        default=None,
+        type=str,
+        required=False,
+        help="The input data directory containing the .json files. If no data dir is specified, uses tensorflow_datasets to load the data."
+        + ", ".join(MODEL_CLASSES.keys()),
+    )
+    parser.add_argument(
+        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
+    )
+
+    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
+    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
+    parser.add_argument(
+        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
+    )
+
+    parser.add_argument(
+        "--per_device_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training."
+    )
+    parser.add_argument(
+        "--per_device_eval_batch_size",
+        default=8,
+        type=int,
+        help="Batch size per GPU/CPU for validation during training.",
+    )
+    parser.add_argument(
+        "--per_device_test_batch_size",
+        default=8,
+        type=int,
+        help="Batch size per GPU/CPU for evaluation after training.",
+    )
+    parser.add_argument(
+        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
+    )
+
+    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
+    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
+    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
+    parser.add_argument(
+        "--max_seq_length",
+        default=128,
+        type=int,
+        help="The maximum total input sequence length after tokenization. Sequences longer "
+        "than this will be truncated, sequences shorter will be padded.",
+    )
+    parser.add_argument(
+        "--doc_stride",
+        default=128,
+        type=int,
+        help="When splitting up a long document into chunks, how much stride to take between chunks.",
+    )
+    parser.add_argument(
+        "--max_query_length",
+        default=64,
+        type=int,
+        help="The maximum number of tokens for the question. Questions longer than this will "
+        "be truncated to this length.",
+    )
+    parser.add_argument(
+        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
+    )
+
+    parser.add_argument(
+        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
+    )
+    parser.add_argument(
+        "--tokenizer_name",
+        default="",
+        type=str,
+        help="Pretrained tokenizer name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--cache_dir",
+        default="",
+        type=str,
+        help="Where do you want to store the pre-trained models downloaded from s3",
+    )
+    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
+    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
+    parser.add_argument(
+        "--force_download",
+        action="store_true",
+        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
+    )
+
+    parser.add_argument(
+        "--tpu",
+        default=None,
+        help="The Cloud TPU to use for training. This should be either the name "
+        "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+        "url.",
+    )
+    parser.add_argument("--num_tpu_cores", default="8", help="Total number of TPU cores to use.")
+    parser.add_argument(
+        "--gpus",
+        default="0",
+        help="Comma separated list of gpus devices. If only one, switch to single gpu strategy, if None takes all the gpus available.",
+    )
+    parser.add_argument("--no_cuda", action="store_true", help="Whether not to use CUDA when available")
+    parser.add_argument(
+        "--version_2_with_negative",
+        action="store_true",
+        help="If true, the SQuAD examples contain some that do not have an answer.",
+    )
+    parser.add_argument(
+        "--max_steps",
+        default=-1,
+        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
+    )
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument(
+        "--warmup_steps", default=0, help="Linear warmup over warmup_steps.",
+    )
+    parser.add_argument("--max_grad_norm", default=1.0, help="Max gradient norm.")
+    parser.add_argument("--logging_steps", default=500, type=int, help="Log every X updates.")
+    parser.add_argument("--save_steps", default=500, type=int, help="Save checkpoint every X updates.")
+    parser.add_argument(
+        "--eval_all_checkpoints",
+        action="store_true",
+        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
+    )
+    parser.add_argument(
+        "--n_best_size",
+        default=20,
+        type=int,
+        help="The total number of n-best predictions to generate in the nbest_predictions.json output file.",
+    )
+    parser.add_argument(
+        "--max_answer_length",
+        default=30,
+        type=int,
+        help="The maximum length of an answer that can be generated. This is needed because the start "
+        "and end predictions are not conditioned on one another.",
+    )
+    parser.add_argument(
+        "--verbose_logging",
+        action="store_true",
+        help="If true, all of the warnings related to data processing will be printed. "
+        "A number of warnings are expected for a normal SQuAD evaluation.",
+    )
+    parser.add_argument(
+        "--null_score_diff_threshold",
+        type=float,
+        default=0.0,
+        help="If null_score - best_non_null is greater than the threshold predict null.",
+    )
+    args = parser.parse_args()
+
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+
+    if os.path.exists(args.output_dir) and args.do_train:
+        if not args.overwrite_output_dir and bool(
+            [file for file in os.listdir(args.output_dir) if "features" not in file]
+        ):
+            raise ValueError(
+                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
+                    args.output_dir
+                )
+            )
+
+        if args.amp:
+            tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})
+
+    if args.tpu:
+        resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=args.tpu)
+        tf.config.experimental_connect_to_cluster(resolver)
+        tf.tpu.experimental.initialize_tpu_system(resolver)
+        strategy = tf.distribute.experimental.TPUStrategy(resolver)
+        args.n_device = args.num_tpu_cores
+    elif len(args.gpus.split(",")) > 1:
+        args.n_device = len([f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
+        strategy = tf.distribute.MirroredStrategy(devices=[f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
+    elif args.no_cuda:
+        args.n_device = 1
+        strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
+    else:
+        args.n_device = len(args.gpus.split(","))
+        strategy = tf.distribute.OneDeviceStrategy(device="/gpu:" + args.gpus.split(",")[0])
+
+    logging.warning(
+        "n_device: %s, distributed training: %s, 16-bits training: %s",
+        args.n_device,
+        bool(args.n_device > 1),
+        args.amp,
+    )
+
+    args.model_type = args.model_type.lower()
+    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
+
+    config = config_class.from_pretrained(
+        args.config_name if args.config_name else args.model_name_or_path,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+    )
+
+    logging.info("Training/evaluation parameters %s", args)
+
+    if args.do_train:
+        tokenizer = tokenizer_class.from_pretrained(
+            args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
+            do_lower_case=args.do_lower_case,
+            cache_dir=args.cache_dir if args.cache_dir else None,
+        )
+
+        with strategy.scope():
+            model = model_class.from_pretrained(
+                args.model_name_or_path,
+                from_pt=bool(".bin" in args.model_name_or_path),
+                config=config,
+                cache_dir=args.cache_dir if args.cache_dir else None,
+            )
+            model.layers[-1].activation = tf.keras.activations.softmax
+
+        train_batch_size = args.per_device_train_batch_size * args.n_device
+        train_dataset, num_train_examples = load_and_cache_examples(args, tokenizer, evaluate=False)
+
+        train_dataset = train_dataset.batch(train_batch_size)
+        train_dataset = train_dataset.prefetch(buffer_size=train_batch_size)
+        train_dataset = strategy.experimental_distribute_dataset(train_dataset)
+
+        train(
+            args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size,
+        )
+
+        if not os.path.exists(args.output_dir):
+            os.makedirs(args.output_dir)
+
+        logging.info("Saving model to %s", args.output_dir)
+
+        model.save_pretrained(args.output_dir)
+        tokenizer.save_pretrained(args.output_dir)
+
+    if args.do_eval:
+        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+        checkpoints = []
+        results = []
+
+        if args.eval_all_checkpoints:
+            checkpoints = list(
+                os.path.dirname(c)
+                for c in sorted(
+                    glob.glob(args.output_dir + "/**/" + TF2_WEIGHTS_NAME, recursive=True),
+                    key=lambda f: int("".join(filter(str.isdigit, f)) or -1),
+                )
+            )
+
+        logging.info("Evaluate the following checkpoints: %s", checkpoints)
+
+        if len(checkpoints) == 0:
+            checkpoints.append(args.output_dir)
+
+        for checkpoint in checkpoints:
+            global_step = checkpoint.split("-")[-1] if re.match(".*checkpoint-[0-9]", checkpoint) else "final"
+
+            with strategy.scope():
+                model = model_class.from_pretrained(checkpoint)
+                results = evaluate(args, strategy, model, tokenizer, prefix=global_step)
+
+            result = dict((k + ("_{}".format(global_step) if global_step else ""), v) for k, v in results.items())
+            results.update(result)
+
+        logger.info("Results: {}".format(results))
+
+
+if __name__ == "__main__":
+    main()

examples/run_xnli.py

@@ -473,8 +473,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

setup.py

@@ -63,7 +63,7 @@ extras["sklearn"] = ["scikit-learn"]
 extras["tf"] = ["tensorflow"]
 extras["torch"] = ["torch"]
 
-extras["serving"] = ["pydantic", "uvicorn", "fastapi"]
+extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
 extras["all"] = extras["serving"] + ["tensorflow", "torch"]
 
 extras["testing"] = ["pytest", "pytest-xdist"]

src/transformers/__init__.py

@@ -133,6 +133,7 @@ if is_torch_available():
     from .modeling_utils import PreTrainedModel, prune_layer, Conv1D
     from .modeling_auto import (
         AutoModel,
+        AutoModelForPreTraining,
         AutoModelForSequenceClassification,
         AutoModelForQuestionAnswering,
         AutoModelWithLMHead,
@@ -267,6 +268,7 @@ if is_tf_available():
     from .modeling_tf_utils import TFPreTrainedModel, TFSharedEmbeddings, TFSequenceSummary, shape_list
     from .modeling_tf_auto import (
         TFAutoModel,
+        TFAutoModelForPreTraining,
         TFAutoModelForSequenceClassification,
         TFAutoModelForQuestionAnswering,
         TFAutoModelWithLMHead,

src/transformers/commands/serving.py

@@ -2,7 +2,6 @@ import logging
 from argparse import ArgumentParser, Namespace
 from typing import Any, List, Optional
 
-from starlette.responses import JSONResponse
 from transformers import Pipeline
 from transformers.commands import BaseTransformersCLICommand
 from transformers.pipelines import SUPPORTED_TASKS, pipeline
@@ -13,15 +12,16 @@ try:
     from fastapi import FastAPI, HTTPException, Body
     from fastapi.routing import APIRoute
     from pydantic import BaseModel
+    from starlette.responses import JSONResponse
 
-    _serve_dependancies_installed = True
+    _serve_dependencies_installed = True
 except (ImportError, AttributeError):
     BaseModel = object
 
     def Body(*x, **y):
         pass
 
-    _serve_dependancies_installed = False
+    _serve_dependencies_installed = False
 
 
 logger = logging.getLogger("transformers-cli/serving")
@@ -111,7 +111,7 @@ class ServeCommand(BaseTransformersCLICommand):
         self.port = port
         self.workers = workers
 
-        if not _serve_dependancies_installed:
+        if not _serve_dependencies_installed:
             raise RuntimeError(
                 "Using serve command requires FastAPI and unicorn. "
                 'Please install transformers with [serving]: pip install "transformers[serving]".'

src/transformers/configuration_bert.py

@@ -45,6 +45,7 @@ BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-config.json",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/config.json",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/config.json",
+    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/config.json",
 }
 
 

src/transformers/configuration_utils.py

@@ -40,12 +40,17 @@ class PretrainedConfig(object):
             - ``pretrained_config_archive_map``: a python ``dict`` with `shortcut names` (string) as keys and `url` (string) of associated pretrained model configurations as values.
             - ``model_type``: a string that identifies the model type, that we serialize into the JSON file, and that we use to recreate the correct object in :class:`~transformers.AutoConfig`.
 
-        Parameters:
-            ``finetuning_task``: string, default `None`. Name of the task used to fine-tune the model. This can be used when converting from an original (TensorFlow or PyTorch) checkpoint.
-            ``num_labels``: integer, default `2`. Number of classes to use when the model is a classification model (sequences/tokens)
-            ``output_attentions``: boolean, default `False`. Should the model returns attentions weights.
-            ``output_hidden_states``: string, default `False`. Should the model returns all hidden-states.
-            ``torchscript``: string, default `False`. Is the model used with Torchscript.
+        Args:
+            finetuning_task (:obj:`string` or :obj:`None`, `optional`, defaults to :obj:`None`):
+                Name of the task used to fine-tune the model. This can be used when converting from an original (TensorFlow or PyTorch) checkpoint.
+            num_labels (:obj:`int`, `optional`, defaults to `2`):
+                Number of classes to use when the model is a classification model (sequences/tokens)
+            output_attentions (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Should the model returns attentions weights.
+            output_hidden_states (:obj:`string`, `optional`, defaults to :obj:`False`):
+                Should the model returns all hidden-states.
+            torchscript (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Is the model used with Torchscript (for PyTorch models).
     """
     pretrained_config_archive_map = {}  # type: Dict[str, str]
     model_type = ""  # type: str
@@ -93,8 +98,13 @@ class PretrainedConfig(object):
                 raise err
 
     def save_pretrained(self, save_directory):
-        """ Save a configuration object to the directory `save_directory`, so that it
-            can be re-loaded using the :func:`~transformers.PretrainedConfig.from_pretrained` class method.
+        """
+        Save a configuration object to the directory `save_directory`, so that it
+        can be re-loaded using the :func:`~transformers.PretrainedConfig.from_pretrained` class method.
+
+        Args:
+            save_directory (:obj:`string`):
+                Directory where the configuration JSON file will be saved.
         """
         assert os.path.isdir(
             save_directory
@@ -107,40 +117,45 @@ class PretrainedConfig(object):
         logger.info("Configuration saved in {}".format(output_config_file))
 
     @classmethod
-    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
-        r""" Instantiate a :class:`~transformers.PretrainedConfig` (or a derived class) from a pre-trained model configuration.
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs) -> "PretrainedConfig":
+        r"""
 
-        Parameters:
-            pretrained_model_name_or_path: either:
+        Instantiate a :class:`~transformers.PretrainedConfig` (or a derived class) from a pre-trained model configuration.
 
-                - a string with the `shortcut name` of a pre-trained model configuration to load from cache or download, e.g.: ``bert-base-uncased``.
-                - a string with the `identifier name` of a pre-trained model configuration that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
-                - a path to a `directory` containing a configuration file saved using the :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.
-                - a path or url to a saved configuration JSON `file`, e.g.: ``./my_model_directory/configuration.json``.
-
-            cache_dir: (`optional`) string:
+        Args:
+            pretrained_model_name_or_path (:obj:`string`):
+                either:
+                  - a string with the `shortcut name` of a pre-trained model configuration to load from cache or
+                    download, e.g.: ``bert-base-uncased``.
+                  - a string with the `identifier name` of a pre-trained model configuration that was user-uploaded to
+                    our S3, e.g.: ``dbmdz/bert-base-german-cased``.
+                  - a path to a `directory` containing a configuration file saved using the
+                    :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.
+                  - a path or url to a saved configuration JSON `file`, e.g.:
+                    ``./my_model_directory/configuration.json``.
+            cache_dir (:obj:`string`, `optional`):
                 Path to a directory in which a downloaded pre-trained model
                 configuration should be cached if the standard cache should not be used.
-
-            kwargs: (`optional`) dict: key/value pairs with which to update the configuration object after loading.
-
-                - The values in kwargs of any keys which are configuration attributes will be used to override the loaded values.
-                - Behavior concerning key/value pairs whose keys are *not* configuration attributes is controlled by the `return_unused_kwargs` keyword parameter.
-
-            force_download: (`optional`) boolean, default False:
-                Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
-
-            resume_download: (`optional`) boolean, default False:
+            kwargs (:obj:`Dict[str, any]`, `optional`):
+                The values in kwargs of any keys which are configuration attributes will be used to override the loaded
+                values. Behavior concerning key/value pairs whose keys are *not* configuration attributes is
+                controlled by the `return_unused_kwargs` keyword parameter.
+            force_download (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Force to (re-)download the model weights and configuration files and override the cached versions if they exist.
+            resume_download (:obj:`bool`, `optional`, defaults to :obj:`False`):
                 Do not delete incompletely recieved file. Attempt to resume the download if such a file exists.
-
-            proxies: (`optional`) dict, default None:
-                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
+            proxies (:obj:`Dict`, `optional`):
+                A dictionary of proxy servers to use by protocol or endpoint, e.g.:
+                :obj:`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.`
                 The proxies are used on each request.
-
             return_unused_kwargs: (`optional`) bool:
+                If False, then this function returns just the final configuration object.
+                If True, then this functions returns a :obj:`Tuple(config, unused_kwargs)` where `unused_kwargs` is a
+                dictionary consisting of the key/value pairs whose keys are not configuration attributes: ie the part
+                of kwargs which has not been used to update `config` and is otherwise ignored.
 
-                - If False, then this function returns just the final configuration object.
-                - If True, then this functions returns a tuple `(config, unused_kwargs)` where `unused_kwargs` is a dictionary consisting of the key/value pairs whose keys are not configuration attributes: ie the part of kwargs which has not been used to update `config` and is otherwise ignored.
+        Returns:
+            :class:`PretrainedConfig`: An instance of a configuration object
 
         Examples::
 
@@ -169,9 +184,14 @@ class PretrainedConfig(object):
         for instantiating a Config using `from_dict`.
 
         Parameters:
-            pretrained_config_archive_map: (`optional`) Dict:
-                A map of `shortcut names` to `url`.
-                By default, will use the current class attribute.
+            pretrained_model_name_or_path (:obj:`string`):
+                The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
+            pretrained_config_archive_map: (:obj:`Dict[str, str]`, `optional`) Dict:
+                A map of `shortcut names` to `url`. By default, will use the current class attribute.
+
+        Returns:
+            :obj:`Tuple[Dict, Dict]`: The dictionary that will be used to instantiate the configuration object.
+
         """
         cache_dir = kwargs.pop("cache_dir", None)
         force_download = kwargs.pop("force_download", False)
@@ -235,8 +255,21 @@ class PretrainedConfig(object):
         return config_dict, kwargs
 
     @classmethod
-    def from_dict(cls, config_dict: Dict, **kwargs):
-        """Constructs a `Config` from a Python dictionary of parameters."""
+    def from_dict(cls, config_dict: Dict, **kwargs) -> "PretrainedConfig":
+        """
+        Constructs a `Config` from a Python dictionary of parameters.
+
+        Args:
+            config_dict (:obj:`Dict[str, any]`):
+                Dictionary that will be used to instantiate the configuration object. Such a dictionary can be retrieved
+                from a pre-trained checkpoint by leveraging the :func:`~transformers.PretrainedConfig.get_config_dict`
+                method.
+            kwargs (:obj:`Dict[str, any]`):
+                Additional parameters from which to initialize the configuration object.
+
+        Returns:
+            :class:`PretrainedConfig`: An instance of a configuration object
+        """
         return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)
 
         config = cls(**config_dict)
@@ -260,8 +293,18 @@ class PretrainedConfig(object):
             return config
 
     @classmethod
-    def from_json_file(cls, json_file: str):
-        """Constructs a `Config` from the path to a json file of parameters."""
+    def from_json_file(cls, json_file: str) -> "PretrainedConfig":
+        """
+        Constructs a `Config` from the path to a json file of parameters.
+
+        Args:
+            json_file (:obj:`string`):
+                Path to the JSON file containing the parameters.
+
+        Returns:
+            :class:`PretrainedConfig`: An instance of a configuration object
+
+        """
         config_dict = cls._dict_from_json_file(json_file)
         return cls(**config_dict)
 
@@ -278,17 +321,33 @@ class PretrainedConfig(object):
         return "{} {}".format(self.__class__.__name__, self.to_json_string())
 
     def to_dict(self):
-        """Serializes this instance to a Python dictionary."""
+        """
+        Serializes this instance to a Python dictionary.
+
+        Returns:
+            :obj:`Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
+        """
         output = copy.deepcopy(self.__dict__)
         if hasattr(self.__class__, "model_type"):
             output["model_type"] = self.__class__.model_type
         return output
 
     def to_json_string(self):
-        """Serializes this instance to a JSON string."""
+        """
+        Serializes this instance to a JSON string.
+
+        Returns:
+            :obj:`string`: String containing all the attributes that make up this configuration instance in JSON format.
+        """
         return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
 
     def to_json_file(self, json_file_path):
-        """ Save this instance to a json file."""
+        """
+        Save this instance to a json file.
+
+        Args:
+            json_file_path (:obj:`string`):
+                Path to the JSON file in which this configuration instance's parameters will be saved.
+        """
         with open(json_file_path, "w", encoding="utf-8") as writer:
             writer.write(self.to_json_string())

src/transformers/data/metrics/squad_metrics.py

@@ -555,10 +555,10 @@ def compute_predictions_logits(
         all_nbest_json[example.qas_id] = nbest_json
 
     with open(output_prediction_file, "w") as writer:
-        writer.write(json.dumps(all_predictions, indent=4) + "\n")
+        writer.write(json.dumps(str(all_predictions), indent=4) + "\n")
 
     with open(output_nbest_file, "w") as writer:
-        writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
+        writer.write(json.dumps(str(all_nbest_json), indent=4) + "\n")
 
     if version_2_with_negative:
         with open(output_null_log_odds_file, "w") as writer:

src/transformers/data/processors/squad.py

@@ -306,13 +306,15 @@ def squad_convert_examples_to_features(
             tqdm(
                 p.imap(annotate_, examples, chunksize=32),
                 total=len(examples),
-                desc="convert squad examples to features",
+                desc="Converting squad examples to features",
             )
         )
+
+    print("Converted {} examples into {} features".format(len(examples), len(features)))
     new_features = []
     unique_id = 1000000000
     example_index = 0
-    for example_features in tqdm(features, total=len(features), desc="add example index and unique id"):
+    for example_features in features:
         if not example_features:
             continue
         for example_feature in example_features:
@@ -376,31 +378,34 @@ def squad_convert_examples_to_features(
                     },
                 )
 
-        return tf.data.Dataset.from_generator(
-            gen,
-            (
-                {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
-                {
-                    "start_position": tf.int64,
-                    "end_position": tf.int64,
-                    "cls_index": tf.int64,
-                    "p_mask": tf.int32,
-                    "is_impossible": tf.int32,
-                },
-            ),
-            (
-                {
-                    "input_ids": tf.TensorShape([None]),
-                    "attention_mask": tf.TensorShape([None]),
-                    "token_type_ids": tf.TensorShape([None]),
-                },
-                {
-                    "start_position": tf.TensorShape([]),
-                    "end_position": tf.TensorShape([]),
-                    "cls_index": tf.TensorShape([]),
-                    "p_mask": tf.TensorShape([None]),
-                    "is_impossible": tf.TensorShape([]),
-                },
+        return (
+            features,
+            tf.data.Dataset.from_generator(
+                gen,
+                (
+                    {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
+                    {
+                        "start_position": tf.int64,
+                        "end_position": tf.int64,
+                        "cls_index": tf.int64,
+                        "p_mask": tf.int32,
+                        "is_impossible": tf.int32,
+                    },
+                ),
+                (
+                    {
+                        "input_ids": tf.TensorShape([None]),
+                        "attention_mask": tf.TensorShape([None]),
+                        "token_type_ids": tf.TensorShape([None]),
+                    },
+                    {
+                        "start_position": tf.TensorShape([]),
+                        "end_position": tf.TensorShape([]),
+                        "cls_index": tf.TensorShape([]),
+                        "p_mask": tf.TensorShape([None]),
+                        "is_impossible": tf.TensorShape([]),
+                    },
+                ),
             ),
         )
 

src/transformers/modeling_auto.py

@@ -45,6 +45,7 @@ from .modeling_albert import (
 from .modeling_bert import (
     BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     BertForMaskedLM,
+    BertForPreTraining,
     BertForQuestionAnswering,
     BertForSequenceClassification,
     BertForTokenClassification,
@@ -71,6 +72,7 @@ from .modeling_openai import OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP, OpenAIGPTL
 from .modeling_roberta import (
     ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
     RobertaForMaskedLM,
+    RobertaForQuestionAnswering,
     RobertaForSequenceClassification,
     RobertaForTokenClassification,
     RobertaModel,
@@ -142,6 +144,24 @@ MODEL_MAPPING = OrderedDict(
     ]
 )
 
+MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
+    [
+        (T5Config, T5WithLMHeadModel),
+        (DistilBertConfig, DistilBertForMaskedLM),
+        (AlbertConfig, AlbertForMaskedLM),
+        (CamembertConfig, CamembertForMaskedLM),
+        (XLMRobertaConfig, XLMRobertaForMaskedLM),
+        (RobertaConfig, RobertaForMaskedLM),
+        (BertConfig, BertForPreTraining),
+        (OpenAIGPTConfig, OpenAIGPTLMHeadModel),
+        (GPT2Config, GPT2LMHeadModel),
+        (TransfoXLConfig, TransfoXLLMHeadModel),
+        (XLNetConfig, XLNetLMHeadModel),
+        (XLMConfig, XLMWithLMHeadModel),
+        (CTRLConfig, CTRLLMHeadModel),
+    ]
+)
+
 MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
     [
         (T5Config, T5WithLMHeadModel),
@@ -177,6 +197,7 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
     [
         (DistilBertConfig, DistilBertForQuestionAnswering),
         (AlbertConfig, AlbertForQuestionAnswering),
+        (RobertaConfig, RobertaForQuestionAnswering),
         (BertConfig, BertForQuestionAnswering),
         (XLNetConfig, XLNetForQuestionAnswering),
         (XLMConfig, XLMForQuestionAnswering),
@@ -202,26 +223,6 @@ class AutoModel(object):
         when created with the `AutoModel.from_pretrained(pretrained_model_name_or_path)`
         or the `AutoModel.from_config(config)` class methods.
 
-        The `from_pretrained()` method takes care of returning the correct model class instance
-        based on the `model_type` property of the config object, or when it's missing,
-        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
-
-        The base model class to instantiate is selected as the first pattern matching
-        in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: T5Model (T5 model)
-            - contains `distilbert`: DistilBertModel (DistilBERT model)
-            - contains `albert`: AlbertModel (ALBERT model)
-            - contains `camembert`: CamembertModel (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaModel (XLM-RoBERTa model)
-            - contains `roberta`: RobertaModel (RoBERTa model)
-            - contains `bert`: BertModel (Bert model)
-            - contains `openai-gpt`: OpenAIGPTModel (OpenAI GPT model)
-            - contains `gpt2`: GPT2Model (OpenAI GPT-2 model)
-            - contains `transfo-xl`: TransfoXLModel (Transformer-XL model)
-            - contains `xlnet`: XLNetModel (XLNet model)
-            - contains `xlm`: XLMModel (XLM model)
-            - contains `ctrl`: CTRLModel (Salesforce CTRL  model)
-
         This class cannot be instantiated using `__init__()` (throws an error).
     """
 
@@ -237,17 +238,19 @@ class AutoModel(object):
         r""" Instantiates one of the base model classes of the library
         from a configuration.
 
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
                 The model class to instantiate is selected based on the configuration class:
-                    - isInstance of `distilbert` configuration class: DistilBertModel (DistilBERT model)
-                    - isInstance of `roberta` configuration class: RobertaModel (RoBERTa model)
-                    - isInstance of `bert` configuration class: BertModel (Bert model)
-                    - isInstance of `openai-gpt` configuration class: OpenAIGPTModel (OpenAI GPT model)
-                    - isInstance of `gpt2` configuration class: GPT2Model (OpenAI GPT-2 model)
-                    - isInstance of `ctrl` configuration class: CTRLModel (Salesforce CTRL  model)
-                    - isInstance of `transfo-xl` configuration class: TransfoXLModel (Transformer-XL model)
-                    - isInstance of `xlnet` configuration class: XLNetModel (XLNet model)
-                    - isInstance of `xlm` configuration class: XLMModel (XLM model)
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModel` (DistilBERT model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModel` (RoBERTa model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertModel` (Bert model)
+                - isInstance of `openai-gpt` configuration class: :class:`~transformers.OpenAIGPTModel` (OpenAI GPT model)
+                - isInstance of `gpt2` configuration class: :class:`~transformers.GPT2Model` (OpenAI GPT-2 model)
+                - isInstance of `ctrl` configuration class: :class:`~transformers.CTRLModel` (Salesforce CTRL  model)
+                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLModel` (Transformer-XL model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModel` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.XLMModel` (XLM model)
 
         Examples::
 
@@ -269,26 +272,30 @@ class AutoModel(object):
         r""" Instantiates one of the base model classes of the library
         from a pre-trained model configuration.
 
-        The model class to instantiate is selected as the first pattern matching
+        The `from_pretrained()` method takes care of returning the correct model class instance
+        based on the `model_type` property of the config object, or when it's missing,
+        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
+
+        The base model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: T5Model (T5 model)
-            - contains `distilbert`: DistilBertModel (DistilBERT model)
-            - contains `albert`: AlbertModel (ALBERT model)
-            - contains `camembert`: CamembertModel (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaModel (XLM-RoBERTa model)
-            - contains `roberta`: RobertaModel (RoBERTa model)
-            - contains `bert`: BertModel (Bert model)
-            - contains `openai-gpt`: OpenAIGPTModel (OpenAI GPT model)
-            - contains `gpt2`: GPT2Model (OpenAI GPT-2 model)
-            - contains `transfo-xl`: TransfoXLModel (Transformer-XL model)
-            - contains `xlnet`: XLNetModel (XLNet model)
-            - contains `xlm`: XLMModel (XLM model)
-            - contains `ctrl`: CTRLModel (Salesforce CTRL model)
+            - contains `t5`: :class:`~transformers.T5Model` (T5 model)
+            - contains `distilbert`: :class:`~transformers.DistilBertModel` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.AlbertModel` (ALBERT model)
+            - contains `camembert`: :class:`~transformers.CamembertModel` (CamemBERT model)
+            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaModel` (XLM-RoBERTa model)
+            - contains `roberta`: :class:`~transformers.RobertaModel` (RoBERTa model)
+            - contains `bert`: :class:`~transformers.BertModel` (Bert model)
+            - contains `openai-gpt`: :class:`~transformers.OpenAIGPTModel` (OpenAI GPT model)
+            - contains `gpt2`: :class:`~transformers.GPT2Model` (OpenAI GPT-2 model)
+            - contains `transfo-xl`: :class:`~transformers.TransfoXLModel` (Transformer-XL model)
+            - contains `xlnet`: :class:`~transformers.XLNetModel` (XLNet model)
+            - contains `xlm`: :class:`~transformers.XLMModel` (XLM model)
+            - contains `ctrl`: :class:`~transformers.CTRLModel` (Salesforce CTRL  model)
 
             The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
             To train the model, you should first set it back in training mode with `model.train()`
 
-        Params:
+        Args:
             pretrained_model_name_or_path: either:
 
                 - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
@@ -360,32 +367,162 @@ class AutoModel(object):
         )
 
 
-class AutoModelWithLMHead(object):
+class AutoModelForPreTraining(object):
     r"""
-        :class:`~transformers.AutoModelWithLMHead` is a generic model class
-        that will be instantiated as one of the language modeling model classes of the library
-        when created with the `AutoModelWithLMHead.from_pretrained(pretrained_model_name_or_path)`
+        :class:`~transformers.AutoModelForPreTraining` is a generic model class
+        that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
         class method.
 
+        This class cannot be instantiated using `__init__()` (throws an error).
+    """
+
+    def __init__(self):
+        raise EnvironmentError(
+            "AutoModelForPreTraining is designed to be instantiated "
+            "using the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
+            "`AutoModelForPreTraining.from_config(config)` methods."
+        )
+
+    @classmethod
+    def from_config(cls, config):
+        r""" Instantiates one of the base model classes of the library
+        from a configuration.
+
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
+                The model class to instantiate is selected based on the configuration class:
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForMaskedLM` (DistilBERT model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForMaskedLM` (RoBERTa model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertForPreTraining` (Bert model)
+                - isInstance of `openai-gpt` configuration class: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
+                - isInstance of `gpt2` configuration class: :class:`~transformers.GPT2ModelLMHeadModel` (OpenAI GPT-2 model)
+                - isInstance of `ctrl` configuration class: :class:`~transformers.CTRLModelLMHeadModel` (Salesforce CTRL  model)
+                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
+
+        Examples::
+
+            config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
+            model = AutoModelForPreTraining.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+        """
+        for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
+            if isinstance(config, config_class):
+                return model_class(config)
+        raise ValueError(
+            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
+            "Model type should be one of {}.".format(
+                config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
+            )
+        )
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
+        r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
+
         The `from_pretrained()` method takes care of returning the correct model class instance
         based on the `model_type` property of the config object, or when it's missing,
         falling back to using pattern matching on the `pretrained_model_name_or_path` string.
 
         The model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: T5ModelWithLMHead (T5 model)
-            - contains `distilbert`: DistilBertForMaskedLM (DistilBERT model)
-            - contains `albert`: AlbertForMaskedLM (ALBERT model)
-            - contains `camembert`: CamembertForMaskedLM (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaForMaskedLM (XLM-RoBERTa model)
-            - contains `roberta`: RobertaForMaskedLM (RoBERTa model)
-            - contains `bert`: BertForMaskedLM (Bert model)
-            - contains `openai-gpt`: OpenAIGPTLMHeadModel (OpenAI GPT model)
-            - contains `gpt2`: GPT2LMHeadModel (OpenAI GPT-2 model)
-            - contains `transfo-xl`: TransfoXLLMHeadModel (Transformer-XL model)
-            - contains `xlnet`: XLNetLMHeadModel (XLNet model)
-            - contains `xlm`: XLMWithLMHeadModel (XLM model)
-            - contains `ctrl`: CTRLLMHeadModel (Salesforce CTRL model)
+            - contains `t5`: :class:`~transformers.T5ModelWithLMHead` (T5 model)
+            - contains `distilbert`: :class:`~transformers.DistilBertForMaskedLM` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.AlbertForMaskedLM` (ALBERT model)
+            - contains `camembert`: :class:`~transformers.CamembertForMaskedLM` (CamemBERT model)
+            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaForMaskedLM` (XLM-RoBERTa model)
+            - contains `roberta`: :class:`~transformers.RobertaForMaskedLM` (RoBERTa model)
+            - contains `bert`: :class:`~transformers.BertForPreTraining` (Bert model)
+            - contains `openai-gpt`: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
+            - contains `gpt2`: :class:`~transformers.GPT2LMHeadModel` (OpenAI GPT-2 model)
+            - contains `transfo-xl`: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
+            - contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
+            - contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
+            - contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
+
+        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
+        To train the model, you should first set it back in training mode with `model.train()`
+
+        Args:
+            pretrained_model_name_or_path:
+                Either:
+
+                - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
+                - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
+                - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
+                - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
+            model_args: (`optional`) Sequence of positional arguments:
+                All remaning positional arguments will be passed to the underlying model's ``__init__`` method
+            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+                Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
+
+                - the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
+                - the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
+                - the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
+
+            state_dict: (`optional`) dict:
+                an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
+                This option can be used if you want to create a model from a pretrained configuration but load your own weights.
+                In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
+            cache_dir: (`optional`) string:
+                Path to a directory in which a downloaded pre-trained model
+                configuration should be cached if the standard cache should not be used.
+            force_download: (`optional`) boolean, default False:
+                Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
+            resume_download: (`optional`) boolean, default False:
+                Do not delete incompletely received file. Attempt to resume the download if such a file exists.
+            proxies: (`optional`) dict, default None:
+                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
+                The proxies are used on each request.
+            output_loading_info: (`optional`) boolean:
+                Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
+            kwargs: (`optional`) Remaining dictionary of keyword arguments:
+                Can be used to update the configuration object (after it being loaded) and initiate the model.
+                (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
+                automatically loaded:
+
+                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
+                  underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
+                  already been done)
+                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
+                  initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
+                  ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
+                  with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
+                  attribute will be passed to the underlying model's ``__init__`` function.
+
+        Examples::
+
+            model = AutoModelForPreTraining.from_pretrained('bert-base-uncased')    # Download model and configuration from S3 and cache.
+            model = AutoModelForPreTraining.from_pretrained('./test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+            model = AutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True)  # Update configuration during loading
+            assert model.config.output_attention == True
+            # Loading from a TF checkpoint file instead of a PyTorch model (slower)
+            config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
+            model = AutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
+
+        """
+        config = kwargs.pop("config", None)
+        if not isinstance(config, PretrainedConfig):
+            config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
+
+        for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
+            if isinstance(config, config_class):
+                return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
+        raise ValueError(
+            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
+            "Model type should be one of {}.".format(
+                config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
+            )
+        )
+
+
+class AutoModelWithLMHead(object):
+    r"""
+        :class:`~transformers.AutoModelWithLMHead` is a generic model class
+        that will be instantiated as one of the language modeling model classes of the library
+        when created with the `AutoModelWithLMHead.from_pretrained(pretrained_model_name_or_path)`
+        class method.
 
         This class cannot be instantiated using `__init__()` (throws an error).
     """
@@ -402,17 +539,19 @@ class AutoModelWithLMHead(object):
         r""" Instantiates one of the base model classes of the library
         from a configuration.
 
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
                 The model class to instantiate is selected based on the configuration class:
-                    - isInstance of `distilbert` configuration class: DistilBertModel (DistilBERT model)
-                    - isInstance of `roberta` configuration class: RobertaModel (RoBERTa model)
-                    - isInstance of `bert` configuration class: BertModel (Bert model)
-                    - isInstance of `openai-gpt` configuration class: OpenAIGPTModel (OpenAI GPT model)
-                    - isInstance of `gpt2` configuration class: GPT2Model (OpenAI GPT-2 model)
-                    - isInstance of `ctrl` configuration class: CTRLModel (Salesforce CTRL  model)
-                    - isInstance of `transfo-xl` configuration class: TransfoXLModel (Transformer-XL model)
-                    - isInstance of `xlnet` configuration class: XLNetModel (XLNet model)
-                    - isInstance of `xlm` configuration class: XLMModel (XLM model)
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForMaskedLM` (DistilBERT model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForMaskedLM` (RoBERTa model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertModelForMaskedLM` (Bert model)
+                - isInstance of `openai-gpt` configuration class: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
+                - isInstance of `gpt2` configuration class: :class:`~transformers.GPT2ModelLMHeadModel` (OpenAI GPT-2 model)
+                - isInstance of `ctrl` configuration class: :class:`~transformers.CTRLModelLMHeadModel` (Salesforce CTRL  model)
+                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
 
         Examples::
 
@@ -440,34 +579,33 @@ class AutoModelWithLMHead(object):
 
         The model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: T5ModelWithLMHead (T5 model)
-            - contains `distilbert`: DistilBertForMaskedLM (DistilBERT model)
-            - contains `albert`: AlbertForMaskedLM (ALBERT model)
-            - contains `camembert`: CamembertForMaskedLM (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaForMaskedLM (XLM-RoBERTa model)
-            - contains `roberta`: RobertaForMaskedLM (RoBERTa model)
-            - contains `bert`: BertForMaskedLM (Bert model)
-            - contains `openai-gpt`: OpenAIGPTLMHeadModel (OpenAI GPT model)
-            - contains `gpt2`: GPT2LMHeadModel (OpenAI GPT-2 model)
-            - contains `transfo-xl`: TransfoXLLMHeadModel (Transformer-XL model)
-            - contains `xlnet`: XLNetLMHeadModel (XLNet model)
-            - contains `xlm`: XLMWithLMHeadModel (XLM model)
-            - contains `ctrl`: CTRLLMHeadModel (Salesforce CTRL model)
+            - contains `t5`: :class:`~transformers.T5ModelWithLMHead` (T5 model)
+            - contains `distilbert`: :class:`~transformers.DistilBertForMaskedLM` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.AlbertForMaskedLM` (ALBERT model)
+            - contains `camembert`: :class:`~transformers.CamembertForMaskedLM` (CamemBERT model)
+            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaForMaskedLM` (XLM-RoBERTa model)
+            - contains `roberta`: :class:`~transformers.RobertaForMaskedLM` (RoBERTa model)
+            - contains `bert`: :class:`~transformers.BertForMaskedLM` (Bert model)
+            - contains `openai-gpt`: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
+            - contains `gpt2`: :class:`~transformers.GPT2LMHeadModel` (OpenAI GPT-2 model)
+            - contains `transfo-xl`: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
+            - contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
+            - contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
+            - contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
 
-        Params:
-            pretrained_model_name_or_path: either:
+        Args:
+            pretrained_model_name_or_path:
+                Either:
 
                 - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
                 - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
                 - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
                 - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
-
             model_args: (`optional`) Sequence of positional arguments:
                 All remaning positional arguments will be passed to the underlying model's ``__init__`` method
-
             config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
                 Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
 
@@ -479,28 +617,31 @@ class AutoModelWithLMHead(object):
                 an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
                 This option can be used if you want to create a model from a pretrained configuration but load your own weights.
                 In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
-
             cache_dir: (`optional`) string:
                 Path to a directory in which a downloaded pre-trained model
                 configuration should be cached if the standard cache should not be used.
-
             force_download: (`optional`) boolean, default False:
                 Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
             resume_download: (`optional`) boolean, default False:
-                Do not delete incompletely recieved file. Attempt to resume the download if such a file exists.
-
+                Do not delete incompletely received file. Attempt to resume the download if such a file exists.
             proxies: (`optional`) dict, default None:
                 A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
                 The proxies are used on each request.
-
             output_loading_info: (`optional`) boolean:
                 Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
-
             kwargs: (`optional`) Remaining dictionary of keyword arguments:
-                Can be used to update the configuration object (after it being loaded) and initiate the model. (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or automatically loaded:
+                Can be used to update the configuration object (after it being loaded) and initiate the model.
+                (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
+                automatically loaded:
 
-                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the underlying model's ``__init__`` method (we assume all relevant updates to the configuration have already been done)
-                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's ``__init__`` function.
+                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
+                  underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
+                  already been done)
+                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
+                  initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
+                  ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
+                  with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
+                  attribute will be passed to the underlying model's ``__init__`` function.
 
         Examples::
 
@@ -535,21 +676,6 @@ class AutoModelForSequenceClassification(object):
         when created with the `AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path)`
         class method.
 
-        The `from_pretrained()` method takes care of returning the correct model class instance
-        based on the `model_type` property of the config object, or when it's missing,
-        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
-
-        The model class to instantiate is selected as the first pattern matching
-        in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `distilbert`: DistilBertForSequenceClassification (DistilBERT model)
-            - contains `albert`: AlbertForSequenceClassification (ALBERT model)
-            - contains `camembert`: CamembertForSequenceClassification (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaForSequenceClassification (XLM-RoBERTa model)
-            - contains `roberta`: RobertaForSequenceClassification (RoBERTa model)
-            - contains `bert`: BertForSequenceClassification (Bert model)
-            - contains `xlnet`: XLNetForSequenceClassification (XLNet model)
-            - contains `xlm`: XLMForSequenceClassification (XLM model)
-
         This class cannot be instantiated using `__init__()` (throws an error).
     """
 
@@ -565,13 +691,19 @@ class AutoModelForSequenceClassification(object):
         r""" Instantiates one of the base model classes of the library
         from a configuration.
 
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
                 The model class to instantiate is selected based on the configuration class:
-                    - isInstance of `distilbert` configuration class: DistilBertModel (DistilBERT model)
-                    - isInstance of `roberta` configuration class: RobertaModel (RoBERTa model)
-                    - isInstance of `bert` configuration class: BertModel (Bert model)
-                    - isInstance of `xlnet` configuration class: XLNetModel (XLNet model)
-                    - isInstance of `xlm` configuration class: XLMModel (XLM model)
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForSequenceClassification` (DistilBERT model)
+                - isInstance of `albert` configuration class: :class:`~transformers.AlbertModelForSequenceClassification` (ALBERT model)
+                - isInstance of `camembert` configuration class: :class:`~transformers.CamembertModelForSequenceClassification` (CamemBERT model)
+                - isInstance of `xlm roberta` configuration class: :class:`~transformers.XLMRobertaModelForSequenceClassification` (XLM-RoBERTa model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForSequenceClassification` (RoBERTa model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertModelForSequenceClassification` (Bert model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForSequenceClassification` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForSequenceClassification` (XLM model)
+
 
         Examples::
 
@@ -601,19 +733,19 @@ class AutoModelForSequenceClassification(object):
 
         The model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `distilbert`: DistilBertForSequenceClassification (DistilBERT model)
-            - contains `albert`: AlbertForSequenceClassification (ALBERT model)
-            - contains `camembert`: CamembertForSequenceClassification (CamemBERT model)
-            - contains `xlm-roberta`: XLMRobertaForSequenceClassification (XLM-RoBERTa model)
-            - contains `roberta`: RobertaForSequenceClassification (RoBERTa model)
-            - contains `bert`: BertForSequenceClassification (Bert model)
-            - contains `xlnet`: XLNetForSequenceClassification (XLNet model)
-            - contains `xlm`: XLMForSequenceClassification (XLM model)
+            - contains `distilbert`: :class:`~transformers.DistilBertForSequenceClassification` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.AlbertForSequenceClassification` (ALBERT model)
+            - contains `camembert`: :class:`~transformers.CamembertForSequenceClassification` (CamemBERT model)
+            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaForSequenceClassification` (XLM-RoBERTa model)
+            - contains `roberta`: :class:`~transformers.RobertaForSequenceClassification` (RoBERTa model)
+            - contains `bert`: :class:`~transformers.BertForSequenceClassification` (Bert model)
+            - contains `xlnet`: :class:`~transformers.XLNetForSequenceClassification` (XLNet model)
+            - contains `xlm`: :class:`~transformers.XLMForSequenceClassification` (XLM model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
 
-        Params:
+        Args:
             pretrained_model_name_or_path: either:
 
                 - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
@@ -622,7 +754,7 @@ class AutoModelForSequenceClassification(object):
                 - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
 
             model_args: (`optional`) Sequence of positional arguments:
-                All remaning positional arguments will be passed to the underlying model's ``__init__`` method
+                All remaining positional arguments will be passed to the underlying model's ``__init__`` method
 
             config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
                 Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
@@ -694,18 +826,6 @@ class AutoModelForQuestionAnswering(object):
         when created with the `AutoModelForQuestionAnswering.from_pretrained(pretrained_model_name_or_path)`
         class method.
 
-        The `from_pretrained()` method takes care of returning the correct model class instance
-        based on the `model_type` property of the config object, or when it's missing,
-        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
-
-        The model class to instantiate is selected as the first pattern matching
-        in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `distilbert`: DistilBertForQuestionAnswering (DistilBERT model)
-            - contains `albert`: AlbertForQuestionAnswering (ALBERT model)
-            - contains `bert`: BertForQuestionAnswering (Bert model)
-            - contains `xlnet`: XLNetForQuestionAnswering (XLNet model)
-            - contains `xlm`: XLMForQuestionAnswering (XLM model)
-
         This class cannot be instantiated using `__init__()` (throws an error).
     """
 
@@ -721,12 +841,15 @@ class AutoModelForQuestionAnswering(object):
         r""" Instantiates one of the base model classes of the library
         from a configuration.
 
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
                 The model class to instantiate is selected based on the configuration class:
-                    - isInstance of `distilbert` configuration class: DistilBertModel (DistilBERT model)
-                    - isInstance of `bert` configuration class: BertModel (Bert model)
-                    - isInstance of `xlnet` configuration class: XLNetModel (XLNet model)
-                    - isInstance of `xlm` configuration class: XLMModel (XLM model)
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForQuestionAnswering` (DistilBERT model)
+                - isInstance of `albert` configuration class: :class:`~transformers.AlbertModelForQuestionAnswering` (ALBERT model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertModelForQuestionAnswering` (Bert model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForQuestionAnswering` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForQuestionAnswering` (XLM model)
 
         Examples::
 
@@ -757,16 +880,16 @@ class AutoModelForQuestionAnswering(object):
 
         The model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `distilbert`: DistilBertForQuestionAnswering (DistilBERT model)
-            - contains `albert`: AlbertForQuestionAnswering (ALBERT model)
-            - contains `bert`: BertForQuestionAnswering (Bert model)
-            - contains `xlnet`: XLNetForQuestionAnswering (XLNet model)
-            - contains `xlm`: XLMForQuestionAnswering (XLM model)
+            - contains `distilbert`: :class:`~transformers.DistilBertForQuestionAnswering` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.AlbertForQuestionAnswering` (ALBERT model)
+            - contains `bert`: :class:`~transformers.BertForQuestionAnswering` (Bert model)
+            - contains `xlnet`: :class:`~transformers.XLNetForQuestionAnswering` (XLNet model)
+            - contains `xlm`: :class:`~transformers.XLMForQuestionAnswering` (XLM model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
 
-        Params:
+        Args:
             pretrained_model_name_or_path: either:
 
                 - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
@@ -839,6 +962,15 @@ class AutoModelForQuestionAnswering(object):
 
 
 class AutoModelForTokenClassification:
+    r"""
+        :class:`~transformers.AutoModelForTokenClassification` is a generic model class
+        that will be instantiated as one of the token classification model classes of the library
+        when created with the `AutoModelForTokenClassification.from_pretrained(pretrained_model_name_or_path)`
+        class method.
+
+        This class cannot be instantiated using `__init__()` (throws an error).
+    """
+
     def __init__(self):
         raise EnvironmentError(
             "AutoModelForTokenClassification is designed to be instantiated "
@@ -851,13 +983,16 @@ class AutoModelForTokenClassification:
         r""" Instantiates one of the base model classes of the library
         from a configuration.
 
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
                 The model class to instantiate is selected based on the configuration class:
-                    - isInstance of `distilbert` configuration class: DistilBertModel (DistilBERT model)
-                    - isInstance of `bert` configuration class: BertModel (Bert model)
-                    - isInstance of `xlnet` configuration class: XLNetModel (XLNet model)
-                    - isInstance of `camembert` configuration class: CamembertModel (Camembert model)
-                    - isInstance of `roberta` configuration class: RobertaModel (Roberta model)
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForTokenClassification` (DistilBERT model)
+                - isInstance of `xlm roberta` configuration class: :class:`~transformers.XLMRobertaModelForTokenClassification` (XLMRoberta model)
+                - isInstance of `bert` configuration class: :class:`~transformers.BertModelForTokenClassification` (Bert model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForTokenClassification` (XLNet model)
+                - isInstance of `camembert` configuration class: :class:`~transformers.CamembertModelForTokenClassification` (Camembert model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForTokenClassification` (Roberta model)
 
         Examples::
 
@@ -888,17 +1023,19 @@ class AutoModelForTokenClassification:
 
         The model class to instantiate is selected as the first pattern matching
         in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `distilbert`: DistilBertForTokenClassification (DistilBERT model)
-            - contains `camembert`: CamembertForTokenClassification (Camembert model)
-            - contains `bert`: BertForTokenClassification (Bert model)
-            - contains `xlnet`: XLNetForTokenClassification (XLNet model)
-            - contains `roberta`: RobertaForTokenClassification (Roberta model)
+            - contains `distilbert`: :class:`~transformers.DistilBertForTokenClassification` (DistilBERT model)
+            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaForTokenClassification` (XLM-RoBERTa?Para model)
+            - contains `camembert`: :class:`~transformers.CamembertForTokenClassification` (Camembert model)
+            - contains `bert`: :class:`~transformers.BertForTokenClassification` (Bert model)
+            - contains `xlnet`: :class:`~transformers.XLNetForTokenClassification` (XLNet model)
+            - contains `roberta`: :class:`~transformers.RobertaForTokenClassification` (Roberta model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
 
-        Params:
-            pretrained_model_name_or_path: either:
+        Args:
+            pretrained_model_name_or_path:
+                Either:
 
                 - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
                 - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.

src/transformers/modeling_bert.py

@@ -53,6 +53,7 @@ BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-pytorch_model.bin",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/pytorch_model.bin",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/pytorch_model.bin",
+    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/pytorch_model.bin",
 }
 
 

src/transformers/modeling_t5.py

@@ -802,9 +802,9 @@ class T5WithLMHeadModel(T5PreTrainedModel):
     r"""
         **lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
-            Tokens with indices set to ``-1`` are ignored (masked), the loss is only computed for the tokens with labels
-            in ``[0, ..., config.vocab_size]``
+            Indices should either be in ``[0, ..., config.vocab_size]`` or -100 (see ``input_ids`` docstring).
+            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
+            in ``[0, ..., config.vocab_size]``.
 
     Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs:
         **loss**: (`optional`, returned when ``lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:

src/transformers/modeling_tf_auto.py

@@ -42,6 +42,7 @@ from .modeling_tf_albert import (
 from .modeling_tf_bert import (
     TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     TFBertForMaskedLM,
+    TFBertForPreTraining,
     TFBertForQuestionAnswering,
     TFBertForSequenceClassification,
     TFBertForTokenClassification,
@@ -125,6 +126,22 @@ TF_MODEL_MAPPING = OrderedDict(
     ]
 )
 
+TF_MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
+    [
+        (T5Config, TFT5WithLMHeadModel),
+        (DistilBertConfig, TFDistilBertForMaskedLM),
+        (AlbertConfig, TFAlbertForMaskedLM),
+        (RobertaConfig, TFRobertaForMaskedLM),
+        (BertConfig, TFBertForPreTraining),
+        (OpenAIGPTConfig, TFOpenAIGPTLMHeadModel),
+        (GPT2Config, TFGPT2LMHeadModel),
+        (TransfoXLConfig, TFTransfoXLLMHeadModel),
+        (XLNetConfig, TFXLNetLMHeadModel),
+        (XLMConfig, TFXLMWithLMHeadModel),
+        (CTRLConfig, TFCTRLLMHeadModel),
+    ]
+)
+
 TF_MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
     [
         (T5Config, TFT5WithLMHeadModel),
@@ -329,6 +346,154 @@ class TFAutoModel(object):
         )
 
 
+class TFAutoModelForPreTraining(object):
+    r"""
+        :class:`~transformers.TFAutoModelForPreTraining` is a generic model class
+        that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
+        class method.
+
+        This class cannot be instantiated using `__init__()` (throws an error).
+    """
+
+    def __init__(self):
+        raise EnvironmentError(
+            "TFAutoModelForPreTraining is designed to be instantiated "
+            "using the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
+            "`TFAutoModelForPreTraining.from_config(config)` methods."
+        )
+
+    @classmethod
+    def from_config(cls, config):
+        r""" Instantiates one of the base model classes of the library
+        from a configuration.
+
+        Args:
+            config (:class:`~transformers.PretrainedConfig`):
+                The model class to instantiate is selected based on the configuration class:
+
+                - isInstance of `distilbert` configuration class: :class:`~transformers.TFDistilBertModelForMaskedLM` (DistilBERT model)
+                - isInstance of `roberta` configuration class: :class:`~transformers.TFRobertaModelForMaskedLM` (RoBERTa model)
+                - isInstance of `bert` configuration class: :class:`~transformers.TFBertForPreTraining` (Bert model)
+                - isInstance of `openai-gpt` configuration class: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
+                - isInstance of `gpt2` configuration class: :class:`~transformers.TFGPT2ModelLMHeadModel` (OpenAI GPT-2 model)
+                - isInstance of `ctrl` configuration class: :class:`~transformers.TFCTRLModelLMHeadModel` (Salesforce CTRL  model)
+                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
+                - isInstance of `xlnet` configuration class: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
+                - isInstance of `xlm` configuration class: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
+
+        Examples::
+
+            config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
+            model = TFAutoModelForPreTraining.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+        """
+        for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
+            if isinstance(config, config_class):
+                return model_class(config)
+        raise ValueError(
+            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
+            "Model type should be one of {}.".format(
+                config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
+            )
+        )
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
+        r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
+
+        The `from_pretrained()` method takes care of returning the correct model class instance
+        based on the `model_type` property of the config object, or when it's missing,
+        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
+
+        The model class to instantiate is selected as the first pattern matching
+        in the `pretrained_model_name_or_path` string (in the following order):
+            - contains `t5`: :class:`~transformers.TFT5ModelWithLMHead` (T5 model)
+            - contains `distilbert`: :class:`~transformers.TFDistilBertForMaskedLM` (DistilBERT model)
+            - contains `albert`: :class:`~transformers.TFAlbertForMaskedLM` (ALBERT model)
+            - contains `roberta`: :class:`~transformers.TFRobertaForMaskedLM` (RoBERTa model)
+            - contains `bert`: :class:`~transformers.TFBertForPreTraining` (Bert model)
+            - contains `openai-gpt`: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
+            - contains `gpt2`: :class:`~transformers.TFGPT2LMHeadModel` (OpenAI GPT-2 model)
+            - contains `transfo-xl`: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
+            - contains `xlnet`: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
+            - contains `xlm`: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
+            - contains `ctrl`: :class:`~transformers.TFCTRLLMHeadModel` (Salesforce CTRL model)
+
+        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
+        To train the model, you should first set it back in training mode with `model.train()`
+
+        Args:
+            pretrained_model_name_or_path:
+                Either:
+
+                - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
+                - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
+                - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
+                - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
+            model_args: (`optional`) Sequence of positional arguments:
+                All remaning positional arguments will be passed to the underlying model's ``__init__`` method
+            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
+                Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
+
+                - the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
+                - the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
+                - the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
+
+            state_dict: (`optional`) dict:
+                an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
+                This option can be used if you want to create a model from a pretrained configuration but load your own weights.
+                In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
+            cache_dir: (`optional`) string:
+                Path to a directory in which a downloaded pre-trained model
+                configuration should be cached if the standard cache should not be used.
+            force_download: (`optional`) boolean, default False:
+                Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
+            resume_download: (`optional`) boolean, default False:
+                Do not delete incompletely received file. Attempt to resume the download if such a file exists.
+            proxies: (`optional`) dict, default None:
+                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
+                The proxies are used on each request.
+            output_loading_info: (`optional`) boolean:
+                Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
+            kwargs: (`optional`) Remaining dictionary of keyword arguments:
+                Can be used to update the configuration object (after it being loaded) and initiate the model.
+                (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
+                automatically loaded:
+
+                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
+                  underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
+                  already been done)
+                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
+                  initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
+                  ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
+                  with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
+                  attribute will be passed to the underlying model's ``__init__`` function.
+
+        Examples::
+
+            model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased')    # Download model and configuration from S3 and cache.
+            model = TFAutoModelForPreTraining.from_pretrained('./test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+            model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True)  # Update configuration during loading
+            assert model.config.output_attention == True
+            # Loading from a TF checkpoint file instead of a PyTorch model (slower)
+            config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
+            model = TFAutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
+
+        """
+        config = kwargs.pop("config", None)
+        if not isinstance(config, PretrainedConfig):
+            config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
+
+        for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
+            if isinstance(config, config_class):
+                return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
+        raise ValueError(
+            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
+            "Model type should be one of {}.".format(
+                config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
+            )
+        )
+
+
 class TFAutoModelWithLMHead(object):
     r"""
         :class:`~transformers.TFAutoModelWithLMHead` is a generic model class
@@ -383,7 +548,7 @@ class TFAutoModelWithLMHead(object):
         Examples::
 
             config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
-            model = AutoModelWithLMHead.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+            model = TFAutoModelWithLMHead.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
         """
         for config_class, model_class in TF_MODEL_WITH_LM_HEAD_MAPPING.items():
             if isinstance(config, config_class):

src/transformers/modeling_tf_bert.py

@@ -49,6 +49,7 @@ TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-tf_model.h5",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/tf_model.h5",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/tf_model.h5",
+    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/tf_model.h5",
 }
 
 

src/transformers/modeling_utils.py

@@ -512,7 +512,7 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
 
             # PyTorch's `_load_from_state_dict` does not copy parameters in a module's descendants
             # so we need to apply the function recursively.
-            def load(module, prefix=""):
+            def load(module: nn.Module, prefix=""):
                 local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
                 module._load_from_state_dict(
                     state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs

src/transformers/modeling_xlnet.py

@@ -756,7 +756,7 @@ class XLNetModel(XLNetPreTrainedModel):
             input_ids = input_ids.transpose(0, 1).contiguous()
             qlen, bsz = input_ids.shape[0], input_ids.shape[1]
         elif inputs_embeds is not None:
-            inputs_embeds.transpose(0, 1).contiguous()
+            inputs_embeds = inputs_embeds.transpose(0, 1).contiguous()
             qlen, bsz = inputs_embeds.shape[0], inputs_embeds.shape[1]
         else:
             raise ValueError("You have to specify either input_ids or inputs_embeds")

src/transformers/optimization_tf.py

@@ -56,7 +56,16 @@ class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
         }
 
 
-def create_optimizer(init_lr, num_train_steps, num_warmup_steps):
+def create_optimizer(
+    init_lr,
+    num_train_steps,
+    num_warmup_steps,
+    weight_decay=0.0,
+    adam_epsilon=1e-6,
+    beta_1=0.9,
+    beta_2=0.999,
+    exclude_from_weight_decay=("layer_norm", "bias"),
+):
     """Creates an optimizer with learning rate schedule."""
     # Implements linear decay of the learning rate.
     learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(
@@ -68,11 +77,11 @@ def create_optimizer(init_lr, num_train_steps, num_warmup_steps):
         )
     optimizer = AdamWeightDecay(
         learning_rate=learning_rate_fn,
-        weight_decay_rate=0.01,
-        beta_1=0.9,
-        beta_2=0.999,
-        epsilon=1e-6,
-        exclude_from_weight_decay=["layer_norm", "bias"],
+        weight_decay_rate=weight_decay,
+        beta_1=beta_1,
+        beta_2=beta_2,
+        epsilon=adam_epsilon,
+        exclude_from_weight_decay=exclude_from_weight_decay,
     )
     return optimizer
 

src/transformers/tokenization_bert.py

@@ -48,6 +48,7 @@ PRETRAINED_VOCAB_FILES_MAP = {
         "bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt",
         "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/vocab.txt",
         "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/vocab.txt",
+        "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/vocab.txt",
     }
 }
 
@@ -69,6 +70,7 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
     "bert-base-german-dbmdz-uncased": 512,
     "bert-base-finnish-cased-v1": 512,
     "bert-base-finnish-uncased-v1": 512,
+    "bert-base-dutch-cased": 512,
 }
 
 PRETRAINED_INIT_CONFIGURATION = {
@@ -89,6 +91,7 @@ PRETRAINED_INIT_CONFIGURATION = {
     "bert-base-german-dbmdz-uncased": {"do_lower_case": True},
     "bert-base-finnish-cased-v1": {"do_lower_case": False},
     "bert-base-finnish-uncased-v1": {"do_lower_case": True},
+    "bert-base-dutch-cased": {"do_lower_case": False},
 }
 
 

src/transformers/tokenization_camembert.py

@@ -169,6 +169,24 @@ class CamembertTokenizer(PreTrainedTokenizer):
             return self.fairseq_ids_to_tokens[index]
         return self.sp_model.IdToPiece(index - self.fairseq_offset)
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state["sp_model"] = None
+        return state
+
+    def __setstate__(self, d):
+        self.__dict__ = d
+        try:
+            import sentencepiece as spm
+        except ImportError:
+            logger.warning(
+                "You need to install SentencePiece to use AlbertTokenizer: https://github.com/google/sentencepiece"
+                "pip install sentencepiece"
+            )
+            raise
+        self.sp_model = spm.SentencePieceProcessor()
+        self.sp_model.Load(self.vocab_file)
+
     def convert_tokens_to_string(self, tokens):
         """Converts a sequence of tokens (strings for sub-words) in a single string."""
         out_string = "".join(tokens).replace(SPIECE_UNDERLINE, " ").strip()

src/transformers/tokenization_utils.py

@@ -998,7 +998,7 @@ class PreTrainedTokenizer(object):
             for key, value in batch_outputs.items():
 
                 padded_value = value
-                if key != "input_len":
+                if key != "input_len" and self._pad_token is not None:
                     # Padding handle
                     padded_value = [
                         v + [self.pad_token_id if key == "input_ids" else 1] * (max_seq_len - len(v))

src/transformers/tokenization_xlm_roberta.py

@@ -176,10 +176,7 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
     def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):
         """
         Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
-        A RoBERTa sequence pair mask has the following format:
-        0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
-        | first sequence    | second sequence
-
+        RoBERTa does not make use of token type ids, therefore a list of zeros is returned.
         if token_ids_1 is None, only returns the first portion of the mask (0's).
         """
         sep = [self.sep_token_id]
@@ -187,7 +184,7 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
 
         if token_ids_1 is None:
             return len(cls + token_ids_0 + sep) * [0]
-        return len(cls + token_ids_0 + sep + sep) * [0] + len(token_ids_1 + sep) * [1]
+        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
 
     @property
     def vocab_size(self):

tests/test_modeling_auto.py

@@ -28,6 +28,8 @@ if is_torch_available():
         BertConfig,
         AutoModel,
         BertModel,
+        AutoModelForPreTraining,
+        BertForPreTraining,
         AutoModelWithLMHead,
         BertForMaskedLM,
         RobertaForMaskedLM,
@@ -56,6 +58,21 @@ class AutoModelTest(unittest.TestCase):
             for value in loading_info.values():
                 self.assertEqual(len(value), 0)
 
+    @slow
+    def test_model_for_pretraining_from_pretrained(self):
+        logging.basicConfig(level=logging.INFO)
+        for model_name in list(BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+            config = AutoConfig.from_pretrained(model_name)
+            self.assertIsNotNone(config)
+            self.assertIsInstance(config, BertConfig)
+
+            model = AutoModelForPreTraining.from_pretrained(model_name)
+            model, loading_info = AutoModelForPreTraining.from_pretrained(model_name, output_loading_info=True)
+            self.assertIsNotNone(model)
+            self.assertIsInstance(model, BertForPreTraining)
+            for value in loading_info.values():
+                self.assertEqual(len(value), 0)
+
     @slow
     def test_lmhead_model_from_pretrained(self):
         logging.basicConfig(level=logging.INFO)

tests/test_modeling_tf_auto.py

@@ -28,6 +28,8 @@ if is_tf_available():
         BertConfig,
         TFAutoModel,
         TFBertModel,
+        TFAutoModelForPreTraining,
+        TFBertForPreTraining,
         TFAutoModelWithLMHead,
         TFBertForMaskedLM,
         TFRobertaForMaskedLM,
@@ -57,6 +59,23 @@ class TFAutoModelTest(unittest.TestCase):
             self.assertIsNotNone(model)
             self.assertIsInstance(model, TFBertModel)
 
+    @slow
+    def test_model_for_pretraining_from_pretrained(self):
+        import h5py
+
+        self.assertTrue(h5py.version.hdf5_version.startswith("1.10"))
+
+        logging.basicConfig(level=logging.INFO)
+        # for model_name in list(TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+        for model_name in ["bert-base-uncased"]:
+            config = AutoConfig.from_pretrained(model_name)
+            self.assertIsNotNone(config)
+            self.assertIsInstance(config, BertConfig)
+
+            model = TFAutoModelForPreTraining.from_pretrained(model_name)
+            self.assertIsNotNone(model)
+            self.assertIsInstance(model, TFBertForPreTraining)
+
     @slow
     def test_lmhead_model_from_pretrained(self):
         logging.basicConfig(level=logging.INFO)

tests/test_modeling_tf_common.py

@@ -112,8 +112,15 @@ class TFModelTesterMixin:
             tfo = tf_model(inputs_dict, training=False)
             tf_hidden_states = tfo[0].numpy()
             pt_hidden_states = pto[0].numpy()
-            tf_hidden_states[np.isnan(tf_hidden_states)] = 0
-            pt_hidden_states[np.isnan(pt_hidden_states)] = 0
+
+            tf_nans = np.copy(np.isnan(tf_hidden_states))
+            pt_nans = np.copy(np.isnan(pt_hidden_states))
+
+            pt_hidden_states[tf_nans] = 0
+            tf_hidden_states[tf_nans] = 0
+            pt_hidden_states[pt_nans] = 0
+            tf_hidden_states[pt_nans] = 0
+
             max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
             # Debug info (remove when fixed)
             if max_diff >= 2e-2:
@@ -144,8 +151,14 @@ class TFModelTesterMixin:
             tfo = tf_model(inputs_dict)
             tfo = tfo[0].numpy()
             pto = pto[0].numpy()
-            tfo[np.isnan(tfo)] = 0
-            pto[np.isnan(pto)] = 0
+            tf_nans = np.copy(np.isnan(tfo))
+            pt_nans = np.copy(np.isnan(pto))
+
+            pto[tf_nans] = 0
+            tfo[tf_nans] = 0
+            pto[pt_nans] = 0
+            tfo[pt_nans] = 0
+
             max_diff = np.amax(np.abs(tfo - pto))
             self.assertLessEqual(max_diff, 2e-2)
 

tests/test_modeling_tf_distilbert.py

@@ -219,5 +219,5 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
     # @slow
     # def test_model_from_pretrained(self):
     #     for model_name in list(DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
-    #         model = DistilBertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
+    #         model = DistilBertModesss.from_pretrained(model_name, cache_dir=CACHE_DIR)
     #         self.assertIsNotNone(model)