pin TF to 2.1 (#4297 )

* pin TF to 2.1 * Pin flake8 as well
[ci] Restrict GPU tests to actual code commits
2020-05-11 21:03:30 -04:00 · 2020-05-11 20:40:41 -04:00 · 2020-05-12 00:05:36 +00:00 · 2020-05-11 16:43:57 -04:00 · 2020-05-11 15:24:02 -04:00 · 2020-05-11 15:05:36 -04:00
102 changed files with 1807 additions and 348 deletions
@@ -1,9 +1,13 @@
 name: Self-hosted runner (push)

 on: 
-  # push:
-  #   branches:
-  #     - master
+  push:
+    branches:
+      - master
+    paths: 
+      - "src/**"
+      - "tests/**"
+      - ".github/**"
  # pull_request:
  repository_dispatch:

@@ -31,8 +35,8 @@ jobs:
    - name: Install dependencies
      run: |
        source .env/bin/activate
-        pip install .[sklearn,tf,torch,testing]
-        pip uninstall -y tensorflow
+        pip install torch==1.4.0
+        pip install .[sklearn,testing]

    - name: Are GPUs recognized by our DL frameworks
      run: |
@@ -164,8 +164,9 @@ At some point in the future, you'll be able to seamlessly move from pre-training
 17. **[ELECTRA](https://huggingface.co/transformers/model_doc/electra.html)** (from Google Research/Stanford University) released with the paper [ELECTRA: Pre-training text encoders as discriminators rather than generators](https://arxiv.org/abs/2003.10555) by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
 18. **[DialoGPT](https://huggingface.co/transformers/model_doc/dialogpt.html)** (from Microsoft Research) released with the paper [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://arxiv.org/abs/1911.00536) by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
 19. **[Reformer](https://huggingface.co/transformers/model_doc/reformer.html)** (from Google Research) released with the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.
-20. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
-21. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
+20. **[MarianMT](https://huggingface.co/transformers/model_doc/marian.html)** Machine translation models trained using [OPUS](http://opus.nlpl.eu/) data by Jörg Tiedemann. The [Marian Framework](https://marian-nmt.github.io/) is being developed by the Microsoft Translator Team.
+21. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
+22. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.

 These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).

@@ -414,7 +415,7 @@ Training with these hyper-parameters gave us the following results:
 This example code fine-tunes BERT on the SQuAD dataset using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD:

 ```bash
-python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
+python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
@@ -447,7 +448,7 @@ The generation script includes the [tricks](https://github.com/rusiaaman/XLNet-g
 Here is how to run the script with the small version of OpenAI GPT-2 model:

 ```shell
-python ./examples/run_generation.py \
+python ./examples/text-generation/run_generation.py \
    --model_type=gpt2 \
    --length=20 \
    --model_name_or_path=gpt2 \
@@ -455,7 +456,7 @@ python ./examples/run_generation.py \

 and from the Salesforce CTRL model:
 ```shell
-python ./examples/run_generation.py \
+python ./examples/text-generation/run_generation.py \
    --model_type=ctrl \
    --length=20 \
    --model_name_or_path=ctrl \
@@ -67,3 +67,131 @@ It should build the static app that will be available under `/docs/_build/html`

 Accepted files are reStructuredText (.rst) and Markdown (.md). Create a file with its extension and put it
 in the source directory. You can then link it to the toc-tree by putting the filename without the extension.
+
+## Writing Documentation - Specification
+
+The `huggingface/transformers` documentation follows the
+[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style. It is
+mostly written in ReStructuredText 
+([Sphinx simple documentation](https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html), 
+[Sourceforge complete documentation](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html))
+
+### Adding a new section
+
+A section is a page held in the `Notes` toc-tree on the documentation. Adding a new section is done in two steps:
+
+- Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md).
+- Link that file in `./source/index.rst` on the correct toc-tree.
+
+### Adding a new model
+
+When adding a new model:
+ 
+- Create a file `xxx.rst` under `./source/model_doc`. 
+- Link that file in `./source/index.rst` on the `model_doc` toc-tree.
+- Write a short overview of the model:
+    - Overview with paper & authors
+    - Paper abstract
+    - Tips and tricks and how to use it best
+- Add the classes that should be linked in the model. This generally includes the configuration, the tokenizer, and
+  every model of that class (the base model, alongside models with additional heads), both in PyTorch and TensorFlow.
+  The order is generally: 
+    - Configuration, 
+    - Tokenizer
+    - PyTorch base model
+    - PyTorch head models
+    - TensorFlow base model
+    - TensorFlow head models
+
+These classes should be added using the RST syntax. Usually as follows:
+```
+XXXConfig
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.XXXConfig
+    :members:
+```
+
+This will include every public method of the configuration. If for some reason you wish for a method not to be displayed
+in the documentation, you can do so by specifying which methods should be in the docs:
+
+```
+XXXTokenizer
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.XXXTokenizer
+    :members: build_inputs_with_special_tokens, get_special_tokens_mask,
+        create_token_type_ids_from_sequences, save_vocabulary
+
+```
+
+### Writing source documentation
+
+Values that should be put in `code` should either be surrounded by double backticks: \`\`like so\`\` or be written as an object
+using the :obj: syntax: :obj:\`like so\`.
+
+When mentionning a class, it is recommended to use the :class: syntax as the mentioned class will be automatically
+linked by Sphinx: :class:\`transformers.XXXClass\`
+
+When mentioning a function, it is recommended to use the :func: syntax as the mentioned method will be automatically
+linked by Sphinx: :func:\`transformers.XXXClass.method\`
+
+Links should be done as so (note the double underscore at the end): \`text for the link <./local-link-or-global-link#loc>\`__
+
+#### Defining arguments in a method
+
+Arguments should be defined with the `Args:` prefix, followed by a line return and an indentation. 
+The argument should be followed by its type, with its shape if it is a tensor, and a line return.
+Another indentation is necessary before writing the description of the argument.
+
+Here's an example showcasing everything so far:
+
+```
+    Args:
+        input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using :class:`transformers.AlbertTokenizer`.
+            See :func:`transformers.PreTrainedTokenizer.encode` and
+            :func:`transformers.PreTrainedTokenizer.encode_plus` for details.
+
+            `What are input IDs? <../glossary.html#input-ids>`__
+```
+
+#### Writing a multi-line code block 
+
+Multi-line code blocks can be useful for displaying examples. They are done like so:
+
+```
+Example::
+
+    # first line of code
+    # second line
+    # etc
+```
+
+The `Example` string at the beginning can be replaced by anything as long as there are two semicolons following it.
+
+#### Writing a return block
+
+Arguments should be defined with the `Args:` prefix, followed by a line return and an indentation. 
+The first line should be the type of the return, followed by a line return. No need to indent further for the elements
+building the return.
+
+Here's an example for tuple return, comprising several objects:
+
+```
+    Returns:
+        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
+        loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
+            Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss.
+        prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+```
+
+Here's an example for a single value return:
+
+```
+    Returns:
+        A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+```
@@ -15,4 +15,4 @@ In order to help this new field develop, we have included a few additional featu
 * accessing all the attention weights for each head of BERT/GPT/GPT-2,
 * retrieving heads output values and gradients to be able to compute head importance score and prune head as explained in https://arxiv.org/abs/1905.10650.

-To help you understand and use these features, we have added a specific example script: `bertology.py <https://github.com/huggingface/transformers/blob/master/examples/run_bertology.py>`_ while extract information and prune a model pre-trained on GLUE.
+To help you understand and use these features, we have added a specific example script: `bertology.py <https://github.com/huggingface/transformers/blob/master/examples/bertology/run_bertology.py>`_ while extract information and prune a model pre-trained on GLUE.
@@ -12,7 +12,7 @@ A command-line interface is provided to convert original Bert/GPT/GPT-2/Transfor
 BERT
 ^^^^

-You can convert any TensorFlow checkpoint for BERT (in particular `the pre-trained models released by Google <https://github.com/google-research/bert#pre-trained-models>`_\ ) in a PyTorch save file by using the `convert_tf_checkpoint_to_pytorch.py <https://github.com/huggingface/transformers/blob/master/transformers/convert_tf_checkpoint_to_pytorch.py>`_ script.
+You can convert any TensorFlow checkpoint for BERT (in particular `the pre-trained models released by Google <https://github.com/google-research/bert#pre-trained-models>`_\ ) in a PyTorch save file by using the `convert_bert_original_tf_checkpoint_to_pytorch.py <https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_ script.

 This CLI takes as input a TensorFlow checkpoint (three files starting with ``bert_model.ckpt``\ ) and the associated configuration file (\ ``bert_config.json``\ ), and creates a PyTorch model for this configuration, loads the weights from the TensorFlow checkpoint in the PyTorch model and saves the resulting model in a standard PyTorch save file that can be imported using ``torch.load()`` (see examples in `run_bert_extract_features.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_extract_features.py>`_\ , `run_bert_classifier.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_classifier.py>`_ and `run_bert_squad.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_squad.py>`_\ ).

@@ -33,6 +33,26 @@ Here is an example of the conversion process for a pre-trained ``BERT-Base Uncas

 You can download Google's pre-trained models for the conversion `here <https://github.com/google-research/bert#pre-trained-models>`__.

+ALBERT
+^^^^^^
+
+Convert TensorFlow model checkpoints of ALBERT to PyTorch using the `convert_albert_original_tf_checkpoint_to_pytorch.py <https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_ script.
+
+The CLI takes as input a TensorFlow checkpoint (three files starting with ``model.ckpt-best``\ ) and the accompanying configuration file (\ ``albert_config.json``\ ), then creates and saves a PyTorch model. To run this conversion you will need to have TensorFlow and PyTorch installed.
+
+Here is an example of the conversion process for the pre-trained ``ALBERT Base`` model:
+
+.. code-block:: shell
+
+   export ALBERT_BASE_DIR=/path/to/albert/albert_base
+
+   transformers-cli convert --model_type albert \
+     --tf_checkpoint $ALBERT_BASE_DIR/model.ckpt-best \
+     --config $ALBERT_BASE_DIR/albert_config.json \
+     --pytorch_dump_output $ALBERT_BASE_DIR/pytorch_model.bin
+
+You can download Google's pre-trained models for the conversion `here <https://github.com/google-research/albert#pre-trained-models>`__.
+
 OpenAI GPT
 ^^^^^^^^^^

@@ -23,13 +23,13 @@ pip install -r ./examples/requirements.txt
 | [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
 | [SQuAD](#squad) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
 | [Multiple Choice](#multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
-| [Named Entity Recognition](https://github.com/huggingface/transformers/tree/master/examples/ner) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
+| [Named Entity Recognition](https://github.com/huggingface/transformers/tree/master/examples/token-classification) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
 | [XNLI](#xnli) | Examples running BERT/XLM on the XNLI benchmark. |
 | [Adversarial evaluation of model performances](#adversarial-evaluation-of-model-performances) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |

 ## TensorFlow 2.0 Bert models on GLUE

-Based on the script [`run_tf_glue.py`](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py).
+Based on the script [`run_tf_glue.py`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_tf_glue.py).

 Fine-tuning the library TensorFlow 2.0 Bert model for sequence classification on the  MRPC task of the GLUE benchmark: [General Language Understanding Evaluation](https://gluebenchmark.com/).

@@ -93,7 +93,7 @@ python run_glue_tpu.py \

 ## Language model training

-Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).
+Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py).

 Fine-tuning (or training from scratch) the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT
 to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa
@@ -155,7 +155,7 @@ python run_language_modeling.py \

 ## Language generation

-Based on the script [`run_generation.py`](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py).
+Based on the script [`run_generation.py`](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py).

 Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL, XLNet, CTRL.
 A similar script is used for our official demo [Write With Transfomer](https://transformer.huggingface.co), where you
@@ -364,7 +364,7 @@ Download [swag](https://github.com/rowanz/swagaf/tree/master/data) data
 ```bash
 #training on 4 tesla V100(16GB) GPUS
 export SWAG_DIR=/path/to/swag_data_dir
-python ./examples/run_multiple_choice.py \
+python ./examples/multiple-choice/run_multiple_choice.py \
 --task_name swag \
 --model_name_or_path roberta-base \
 --do_train \
@@ -388,7 +388,7 @@ eval_loss = 0.44457291918821606

 ## SQuAD

-Based on the script [`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py).
+Based on the script [`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py).

 #### Fine-tuning BERT on SQuAD1.0

@@ -437,7 +437,7 @@ exact_match = 81.22
 Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.1:

 ```bash
-python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
+python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
@@ -548,7 +548,7 @@ Larger batch size may improve the performance while costing more memory.

 ## XNLI

-Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/run_xnli.py).
+Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py).

 [XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).

@@ -108,3 +108,4 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
    model_doc/electra
    model_doc/dialogpt
    model_doc/reformer
+    model_doc/marian
@@ -74,7 +74,7 @@ This library hosts the processor to load the XNLI data:
 Please note that since the gold labels are available on the test set, evaluation is performed on the test set.

 An example using these processors is given in the
-`run_xnli.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_xnli.py>`__ script.
+`run_xnli.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/text-classification/run_xnli.py>`__ script.


 SQuAD
@@ -150,4 +150,4 @@ Example::


 Another example using these processors is given in the
-`run_squad.py <https://github.com/huggingface/transformers/blob/master/examples/run_squad.py>`__ script.
+`run_squad.py <https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py>`__ script.
@@ -1,5 +1,18 @@
-# Migrating from pytorch-pretrained-bert
+# Migrating from previous packages

+## Migrating from pytorch-transformers to transformers
+
+Here is a quick summary of what you should take care of when migrating from `pytorch-transformers` to `transformers`.
+
+### Positional order of some models' keywords inputs (`attention_mask`, `token_type_ids`...) changed
+
+To be able to use Torchscript (see #1010, #1204 and #1195) the specific order of some models **keywords inputs** (`attention_mask`, `token_type_ids`...) has been changed.
+
+If you used to call the models with keyword names for keyword arguments, e.g. `model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)`, this should not cause any change.
+
+If you used to call the models with positional inputs for keyword arguments, e.g. `model(inputs_ids, attention_mask, token_type_ids)`, you may have to double check the exact order of input arguments.
+
+## Migrating from pytorch-pretrained-bert

 Here is a quick summary of what you should take care of when migrating from `pytorch-pretrained-bert` to `transformers`

@@ -1,6 +1,6 @@
 Bart
 ----------------------------------------------------
-**DISCLAIMER:** This model is still a work in progress, if you see something strange,
+**DISCLAIMER:** If you see something strange,
 file a `Github Issue <https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title>`__ and assign
@sshleifer

@@ -0,0 +1,43 @@
+MarianMTModel
+----------------------------------------------------
+**DISCLAIMER:** If you see something strange,
+file a `Github Issue <https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title>`__ and assign
+@sshleifer
+These models are for machine translation. The list of supported language pairs can be found `here <https://huggingface.co/Helsinki-NLP>`__.
+
+Opus Project
+~~~~~~~~~~~~
+The 1,000+ models were originally trained by `Jörg Tiedemann <https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann>`__ using the `Marian <https://marian-nmt.github.io/>`_ C++ library, which supports fast training and translation.
+All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented in a model card.
+
+Implementation Notes
+~~~~~~~~~~~~~~~~~~~~
+- each model is about 298 MB on disk, there are 1,000+ models.
+- Models are named with the following patter 'Helsinki-NLP/opus-mt-{src_langs}-{targ_langs}'. If there are multiple source or target languages they are joined by a '+' symbol.
+- the 80 opus models that require BPE preprocessing are not supported.
+- There is an outstanding issue w.r.t multilingual models and language codes.
+- The modeling code is the same as ``BartModel`` with a few minor modifications:
+    - static (sinusoid) positional embeddings (``MarianConfig.static_position_embeddings=True``)
+    - a new final_logits_bias (``MarianConfig.add_bias_logits=True``)
+    - no layernorm_embedding (``MarianConfig.normalize_embedding=False``)
+    - the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix. (Bart uses <s/>)
+- Code to bulk convert models can be found in ``convert_marian_to_pytorch.py``
+
+
+MarianMTModel
+~~~~~~~~~~~~~
+
+Pytorch version of marian-nmt's transformer.h (c++). Designed for the OPUS-NMT translation checkpoints.
+Model API is identical to BartForConditionalGeneration.
+Available models are listed at `Model List <https://huggingface.co/models?search=Helsinki-NLP>`__
+This class inherits all functionality from ``BartForConditionalGeneration``, see that page for method signatures.
+
+.. autoclass:: transformers.MarianMTModel
+    :members:
+
+
+MarianTokenizer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.MarianTokenizer
+    :members: prepare_translation_batch
@@ -29,7 +29,7 @@ Tips:
  XLNet is pretrained using only a sub-set of the output tokens as target which are selected
  with the `target_mapping` input.
 - To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
-  `target_mapping` inputs to control the attention span and outputs (see examples in `examples/run_generation.py`)
+  `target_mapping` inputs to control the attention span and outputs (see examples in `examples/text-generation/run_generation.py`)
 - XLNet is one of the few models that has no sequence length limit.

 The original code can be found `here <https://github.com/zihangdai/xlnet/>`_.
@@ -80,7 +80,7 @@ You can then feed it all as input to your model:
    outputs = model(input_ids, langs=langs)


-The example `run_generation.py <https://github.com/huggingface/transformers/blob/master/examples/run_generation.py>`__
+The example `run_generation.py <https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py>`__
 can generate text using the CLM checkpoints from XLM, using the language embeddings.

 XLM without Language Embeddings
@@ -275,7 +275,7 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   |                                                            | | FlauBERT large architecture                                                                                                         |
 |                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-| Bart              | ``bart-large``                                             | | 12-layer, 1024-hidden, 16-heads, 406M parameters                                                                                    |
+| Bart              | ``bart-large``                                             | | 24-layer, 1024-hidden, 16-heads, 406M parameters                                                                                    |
 |                   |                                                            | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/bart>`_)                                                       |
 |                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 |                   | ``bart-large-mnli``                                        | | Adds a 2 layer classification head with 1 million parameters                                                                        |
@@ -296,6 +296,12 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``DialoGPT-large``                                         | | 36-layer, 1280-hidden, 20-heads, 774M parameters                                                                                    |
 |                   |                                                            | | Trained on English text: 147M conversation-like exchanges extracted from Reddit.                                                    |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-| Reformer          | ``reformer-crime-and-punishment``                          | | 6-layer, 256-hidden, 2-heads, 3M parameters                                                                                         |
-|                   |                                                            | | Trained on English text: Crime and Punishment novel by Fyodor Dostoyevsky                                                           |
+| Reformer          | ``reformer-enwik8``                                        | | 12-layer, 1024-hidden, 8-heads, 149M parameters                                                                                     |
+|                   |                                                            | | Trained on English Wikipedia data - enwik8.                                                                                         |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``reformer-crime-and-punishment``                          | | 6-layer, 256-hidden, 2-heads, 3M parameters                                                                                         |
+|                   |                                                            | | Trained on English text: Crime and Punishment novel by Fyodor Dostoyevsky.                                                          |
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+| MarianMT          | ``Helsinki-NLP/opus-mt-{src}-{tgt}``                       | | 12-layer, 512-hidden, 8-heads, ~74M parameter Machine translation models. Parameter counts vary depending on vocab size.            |
+|                   |                                                            | | (see `model list <https://huggingface.co/Helsinki-NLP>`_)                                                                           |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
@@ -8,7 +8,7 @@ The library was designed with two strong goals in mind:

 - be as easy and fast to use as possible:

-  - we strongly limited the number of user-facing abstractions to learn, in fact there are almost no abstractions, just three standard classes required to use each model: configuration, models and tokenizer,
+  - we strongly limited the number of user-facing abstractions to learn, in fact, there are almost no abstractions, just three standard classes required to use each model: configuration, models and tokenizer,
  - all of these classes can be initialized in a simple and unified way from pretrained instances by using a common `from_pretrained()` instantiation method which will take care of downloading (if needed), caching and loading the related class from a pretrained instance supplied in the library or your own saved instance.
  - as a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to extend/build-upon the library, just use regular Python/PyTorch modules and inherit from the base classes of the library to reuse functionalities like model loading/saving.

@@ -31,27 +31,27 @@ A few other goals:

 ## Main concepts

-The library is build around three type of classes for each models:
+The library is build around three types of classes for each model:

- **model classes** which are PyTorch models (`torch.nn.Modules`) of the 8 models architectures currently provided in the library, e.g. `BertModel`
- **configuration classes** which store all the parameters required to build a model, e.g. `BertConfig`. You don't always need to instantiate these your-self, in particular if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model)
- **tokenizer classes** which store the vocabulary for each model and provide methods for encoding/decoding strings in list of token embeddings indices to be fed to a model, e.g. `BertTokenizer`
+- **model classes**  e.g., `BertModel` which are 20+ PyTorch models (`torch.nn.Modules`) that work with the pretrained weights provided in the library. In TF2, these are `tf.keras.Model`.
+- **configuration classes** which store all the parameters required to build a model, e.g., `BertConfig`. You don't always need to instantiate these your-self. In particular, if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model)
+- **tokenizer classes** which store the vocabulary for each model and provide methods for encoding/decoding strings in a list of token embeddings indices to be fed to a model, e.g., `BertTokenizer`

 All these classes can be instantiated from pretrained instances and saved locally using two methods:

 - `from_pretrained()` let you instantiate a model/configuration/tokenizer from a pretrained version either provided by the library itself (currently 27 models are provided as listed [here](https://huggingface.co/transformers/pretrained_models.html)) or stored locally (or on a server) by the user,
 - `save_pretrained()` let you save a model/configuration/tokenizer locally so that it can be reloaded using `from_pretrained()`.

-We'll finish this quickstart tour by going through a few simple quick-start examples to see how we can instantiate and use these classes. The rest of the documentation is organized in two parts:
+We'll finish this quickstart tour by going through a few simple quick-start examples to see how we can instantiate and use these classes. The rest of the documentation is organized into two parts:

 - the **MAIN CLASSES** section details the common functionalities/method/attributes of the three main type of classes (configuration, model, tokenizer) plus some optimization related classes provided as utilities for training,
- the **PACKAGE REFERENCE** section details all the variants of each class for each model architectures and in particular the input/output that you should expect when calling each of them.
+- the **PACKAGE REFERENCE** section details all the variants of each class for each model architectures and, in particular, the input/output that you should expect when calling each of them.

 ## Quick tour: Usage

 Here are two examples showcasing a few `Bert` and `GPT2` classes and pre-trained models.

-See full API reference for examples for each model class.
+See the full API reference for examples of each model class.

 ### BERT example

@@ -191,7 +191,7 @@ Examples for each model class of each model architecture (Bert, GPT, GPT-2, Tran

 #### Using the past

-GPT-2 as well as some other models (GPT, XLNet, Transfo-XL, CTRL) make use of a `past` or `mems` attribute which can be used to prevent re-computing the key/value pairs when using sequential decoding. It is useful when generating sequences as a big part of the attention mechanism benefits from previous computations.
+GPT-2, as well as some other models (GPT, XLNet, Transfo-XL, CTRL), make use of a `past` or `mems` attribute which can be used to prevent re-computing the key/value pairs when using sequential decoding. It is useful when generating sequences as a big part of the attention mechanism benefits from previous computations.

 Here is a fully-working example using the `past` with `GPT2LMHeadModel` and argmax decoding (which should only be used as an example, as argmax decoding introduces a lot of repetition):

@@ -45,7 +45,7 @@ Sequence classification is the task of classifying sequences according to a give
 of sequence classification is the GLUE dataset, which is entirely based on that task. If you would like to fine-tune
 a model on a GLUE sequence classification task, you may leverage the
 `run_glue.py <https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_glue.py>`_ or
-`run_tf_glue.py <https://github.com/huggingface/transformers/tree/master/examples/run_tf_glue.py>`_ scripts.
+`run_tf_glue.py <https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_tf_glue.py>`_ scripts.

 Here is an example using the pipelines do to sentiment analysis: identifying if a sequence is positive or negative.
 It leverages a fine-tuned model on sst2, which is a GLUE task.
@@ -1,10 +1,47 @@
 # Examples

-In this section a few examples are put together. All of these examples work for several models, making use of the very
-similar API between the different models.
+Version 2.9 of `transformers` introduces a new `Trainer` class for PyTorch, and its equivalent `TFTrainer` for TF 2.
+
+Here is the list of all our examples:
+- **grouped by task** (all official examples work for multiple models)
+- with information on whether they are **built on top of `Trainer`/`TFTrainer`** (if not, they still work, they might just lack some features),
+- whether they also include examples for **`pytorch-lightning`**, which is a great fully-featured, general-purpose training library for PyTorch,
+- links to **Colab notebooks** to walk through the scripts and run them easily,
+- links to **Cloud deployments** to be able to deploy large-scale trainings in the Cloud with little to no setup.
+
+This is still a work-in-progress – in particular documentation is still sparse – so please **contribute improvements/pull requests.**
+
+
+## Tasks built on Trainer
+
+| Task | Example datasets | Trainer support | TFTrainer support | pytorch-lightning | Colab | One-click Deploy to Azure (wip) | 
+|---|---|:---:|:---:|:---:|:---:|:---:|
+| [`language-modeling`](./language-modeling) | Raw text | ✅ | - | - | - | - |
+| [`text-classification`](./text-classification) | GLUE, XNLI | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb) | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-storage-account-create%2Fazuredeploy.json) |
+| [`token-classification`](./token-classification) | CoNLL NER | ✅ | ✅ | ✅ | - | - |
+| [`multiple-choice`](./multiple-choice) | SWAG, RACE, ARC | ✅ | - | - | - | - |
+
+
+
+## Other examples and how-to's
+
+| Section | Description |
+|---|---|
+| [TensorFlow 2.0 models on GLUE](./text-classification) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. |
+| [Running on TPUs](#running-on-tpus) | Examples on running fine-tuning tasks on Google TPUs to accelerate workloads. |
+| [Language Model training](./language-modeling) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
+| [Language Generation](./text-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
+| [GLUE](./text-classification) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
+| [SQuAD](./question-answering) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
+| [Multiple Choice](./multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
+| [Named Entity Recognition](./token-classification) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
+| [XNLI](./text-classification) | Examples running BERT/XLM on the XNLI benchmark. |
+| [Adversarial evaluation of model performances](./adversarial) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |
+
+## Important note

 **Important**
-To run the latest versions of the examples, you have to install from source and install some specific requirements for the examples.
+To make sure you can successfully run the latest versions of the example scripts, you have to install the library from source and install some example-specific requirements.
 Execute the following steps in a new virtual environment:

 ```bash
@@ -14,16 +51,30 @@ pip install .
 pip install -r ./examples/requirements.txt
 ```

-| Section                    | Description                                                                                                                                                |
-|----------------------------|-----------------------------------------------------
-| [TensorFlow 2.0 models on GLUE](#TensorFlow-2.0-Bert-models-on-GLUE) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. |
-| [Running on TPUs](#running-on-tpus) | Examples on running fine-tuning tasks on Google TPUs to accelerate workloads. |
-| [Language Model training](#language-model-training) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
-| [Language Generation](#language-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
-| [GLUE](#glue) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
-| [SQuAD](#squad) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
-| [Multiple Choice](#multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
-| [Named Entity Recognition](https://github.com/huggingface/transformers/tree/master/examples/ner) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
-| [XNLI](#xnli) | Examples running BERT/XLM on the XNLI benchmark. |
-| [Adversarial evaluation of model performances](#adversarial-evaluation-of-model-performances) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |
+## Running on TPUs

+When using Tensorflow, TPUs are supported out of the box as a `tf.distribute.Strategy`.
+
+When using PyTorch, we support TPUs thanks to `pytorch/xla`. For more context and information on how to setup your TPU environment refer to Google's documentation and to the
+very detailed [pytorch/xla README](https://github.com/pytorch/xla/blob/master/README.md).
+
+In this repo, we provide a very simple launcher script named [xla_spawn.py](./xla_spawn.py) that lets you run our example scripts on multiple TPU cores without any boilerplate.
+Just pass a `--num_cores` flag to this script, then your regular training script with its arguments (this is similar to the `torch.distributed.launch` helper for torch.distributed).
+
+For example for `run_glue`:
+
+```bash
+python examples/xla_spawn.py --num_cores 8 \
+	examples/text-classification/run_glue.py
+	--model_name_or_path bert-base-cased \
+	--task_name mnli \
+	--data_dir ./data/glue_data/MNLI \
+	--output_dir ./models/tpu \
+	--overwrite_output_dir \
+	--do_train \
+	--do_eval \
+	--num_train_epochs 1 \
+	--save_steps 20000
+```
+
+Feedback and more use cases and benchmarks involving TPUs are welcome, please share with the community.
@@ -404,7 +404,7 @@ def main():
    logger.info("Training/evaluation parameters %s", args)

    # Prepare dataset for the GLUE task
-    eval_dataset = GlueDataset(args, tokenizer=tokenizer, evaluate=True, local_rank=args.local_rank)
+    eval_dataset = GlueDataset(args, tokenizer=tokenizer, evaluate=True)
    if args.data_subset > 0:
        eval_dataset = Subset(eval_dataset, list(range(min(args.data_subset, len(eval_dataset)))))
    eval_sampler = SequentialSampler(eval_dataset) if args.local_rank == -1 else DistributedSampler(eval_dataset)
@@ -13,7 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" This is the exact same script as `examples/run_squad.py` (as of 2020, January 8th) with an additional and optional step of distillation."""
+""" This is the exact same script as `examples/question-answering/run_squad.py` (as of 2020, January 8th) with an additional and optional step of distillation."""

 import argparse
 import glob
@@ -1,7 +1,7 @@

 ## Language model training

-Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py).
+Based on the script [`run_language_modeling.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py).

 Fine-tuning (or training from scratch) the library models for language modeling on a text dataset for GPT, GPT-2, BERT and RoBERTa (DistilBERT
 to be added soon). GPT and GPT-2 are fine-tuned using a causal language modeling (CLM) loss while BERT and RoBERTa
@@ -280,5 +280,10 @@ def main():
    return results


+def _mp_fn(index):
+    # For xla_spawn (TPUs)
+    main()
+
+
 if __name__ == "__main__":
    main()
@@ -8,7 +8,7 @@ Download [swag](https://github.com/rowanz/swagaf/tree/master/data) data
 ```bash
 #training on 4 tesla V100(16GB) GPUS
 export SWAG_DIR=/path/to/swag_data_dir
-python ./examples/run_multiple_choice.py \
+python ./examples/multiple-choice/run_multiple_choice.py \
 --task_name swag \
 --model_name_or_path roberta-base \
 --do_train \
@@ -221,5 +221,10 @@ def main():
    return results


+def _mp_fn(index):
+    # For xla_spawn (TPUs)
+    main()
+
+
 if __name__ == "__main__":
    main()
@@ -2,7 +2,7 @@

 ## SQuAD

-Based on the script [`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py).
+Based on the script [`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py).

 #### Fine-tuning BERT on SQuAD1.0

@@ -51,7 +51,7 @@ exact_match = 81.22
 Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.1:

 ```bash
-python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
+python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
@@ -2,7 +2,7 @@

 # Run TensorFlow 2.0 version

-Based on the script [`run_tf_glue.py`](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py).
+Based on the script [`run_tf_glue.py`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_tf_glue.py).

 Fine-tuning the library TensorFlow 2.0 Bert model for sequence classification on the  MRPC task of the GLUE benchmark: [General Language Understanding Evaluation](https://gluebenchmark.com/).

@@ -85,10 +85,12 @@ CoLA, SST-2. The following section provides details on how to run half-precision
 said, there shouldn’t be any issues in running half-precision training with the remaining GLUE tasks as well,
 since the data processor for each task inherits from the base class DataProcessor.

-## Running on TPUs
+## Running on TPUs in PyTorch

-You can accelerate your workloads on Google's TPUs. For information on how to setup your TPU environment refer to this
-[README](https://github.com/pytorch/xla/blob/master/README.md).
+**Update**: read the more up-to-date [Running on TPUs](../README.md#running-on-tpus) in the main README.md instead.
+
+Even when running PyTorch, you can accelerate your workloads on Google's TPUs, using `pytorch/xla`. For information on how to setup your TPU environment refer to the
+[pytorch/xla README](https://github.com/pytorch/xla/blob/master/README.md).

 The following are some examples of running the `*_tpu.py` finetuning scripts on TPUs. All steps for data preparation are
 identical to your normal GPU + Huggingface setup.
@@ -101,7 +103,6 @@ export GLUE_DIR=/path/to/glue
 export TASK_NAME=MNLI

 python run_glue_tpu.py \
-  --model_type bert \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
@@ -115,8 +116,7 @@ python run_glue_tpu.py \
  --overwrite_output_dir \
  --logging_steps 50 \
  --save_steps 200 \
-  --num_cores=8 \
-  --only_log_master
+  --num_cores=8
 ```

 ### MRPC
@@ -256,7 +256,7 @@ TEST RESULTS {'val_loss': tensor(0.0707), 'precision': 0.852427800698191, 'recal

 # XNLI

-Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/run_xnli.py).
+Based on the script [`run_xnli.py`](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py).

 [XNLI](https://www.nyu.edu/projects/bowman/xnli/) is crowd-sourced dataset based on [MultiNLI](http://www.nyu.edu/projects/bowman/multinli/). It is an evaluation benchmark for cross-lingual text representations. Pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).

@@ -134,16 +134,8 @@ def main():
    )

    # Get datasets
-    train_dataset = (
-        GlueDataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank)
-        if training_args.do_train
-        else None
-    )
-    eval_dataset = (
-        GlueDataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank, evaluate=True)
-        if training_args.do_eval
-        else None
-    )
+    train_dataset = GlueDataset(data_args, tokenizer=tokenizer) if training_args.do_train else None
+    eval_dataset = GlueDataset(data_args, tokenizer=tokenizer, evaluate=True) if training_args.do_eval else None

    def compute_metrics(p: EvalPrediction) -> Dict:
        if output_mode == "classification":
@@ -181,9 +173,7 @@ def main():
        eval_datasets = [eval_dataset]
        if data_args.task_name == "mnli":
            mnli_mm_data_args = dataclasses.replace(data_args, task_name="mnli-mm")
-            eval_datasets.append(
-                GlueDataset(mnli_mm_data_args, tokenizer=tokenizer, local_rank=training_args.local_rank, evaluate=True)
-            )
+            eval_datasets.append(GlueDataset(mnli_mm_data_args, tokenizer=tokenizer, evaluate=True))

        for eval_dataset in eval_datasets:
            result = trainer.evaluate(eval_dataset=eval_dataset)
@@ -1,6 +1,6 @@
 ## Language generation

-Based on the script [`run_generation.py`](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py).
+Based on the script [`run_generation.py`](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py).

 Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL, XLNet, CTRL.
 A similar script is used for our official demo [Write With Transfomer](https://transformer.huggingface.co), where you
@@ -17,10 +17,10 @@

 """
 Example command with bag of words:
-python examples/run_pplm.py -B space --cond_text "The president" --length 100 --gamma 1.5 --num_iterations 3 --num_samples 10 --stepsize 0.01 --window_length 5 --kl_scale 0.01 --gm_scale 0.95
+python run_pplm.py -B space --cond_text "The president" --length 100 --gamma 1.5 --num_iterations 3 --num_samples 10 --stepsize 0.01 --window_length 5 --kl_scale 0.01 --gm_scale 0.95

 Example command with discriminator:
-python examples/run_pplm.py -D sentiment --class_label 3 --cond_text "The lake" --length 10 --gamma 1.0 --num_iterations 30 --num_samples 10 --stepsize 0.01 --kl_scale 0.01 --gm_scale 0.95
+python run_pplm.py -D sentiment --class_label 3 --cond_text "The lake" --length 10 --gamma 1.0 --num_iterations 30 --num_samples 10 --stepsize 0.01 --kl_scale 0.01 --gm_scale 0.95
 """

 import argparse
@@ -1,7 +1,7 @@
 ## Named Entity Recognition

-Based on the scripts [`run_ner.py`](https://github.com/huggingface/transformers/blob/master/examples/ner/run_ner.py) for Pytorch and
-[`run_tf_ner.py`](https://github.com/huggingface/transformers/blob/master/examples/ner/run_tf_ner.py) for Tensorflow 2.
+Based on the scripts [`run_ner.py`](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py) for Pytorch and
+[`run_tf_ner.py`](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_tf_ner.py) for Tensorflow 2.
 This example fine-tune Bert Multilingual on GermEval 2014 (German NER).
 Details and results for the fine-tuning provided by @stefan-it.

@@ -292,5 +292,10 @@ def main():
    return results


+def _mp_fn(index):
+    # For xla_spawn (TPUs)
+    main()
+
+
 if __name__ == "__main__":
    main()
@@ -12,17 +12,13 @@ Inspired by https://github.com/pytorch/pytorch/blob/master/torch/distributed/lau


 import importlib
-import os
 import sys
 from argparse import REMAINDER, ArgumentParser
+from pathlib import Path

 import torch_xla.distributed.xla_multiprocessing as xmp


-def trim_suffix(s: str, suffix: str):
-    return s if not s.endswith(suffix) or len(suffix) == 0 else s[: -len(suffix)]
-
-
 def parse_args():
    """
    Helper function parsing the command line options
@@ -44,7 +40,7 @@ def parse_args():
        "training_script",
        type=str,
        help=(
-            "The full module name to the single TPU training "
+            "The full path to the single TPU training "
            "program/script to be launched in parallel, "
            "followed by all the arguments for the "
            "training script"
@@ -61,7 +57,9 @@ def main():
    args = parse_args()

    # Import training_script as a module.
-    mod_name = trim_suffix(os.path.basename(args.training_script), ".py")
+    script_fpath = Path(args.training_script)
+    sys.path.append(str(script_fpath.parent.resolve()))
+    mod_name = script_fpath.stem
    mod = importlib.import_module(mod_name)

    # Patch sys.argv
@@ -59,56 +59,64 @@ tokenizer = GPT2Tokenizer.from_pretrained(
 ## Example using GPT2LMHeadModel

 ```python
-from transformers import GPT2Tokenizer, GPT2LMHeadModel
+from transformers import AutoTokenizer, AutoModelWithLMHead, pipeline, GPT2Tokenizer

-tokenizer = GPT2Tokenizer.from_pretrained('LorenzoDeMattei/GePpeTto')
-model = GPT2LMHeadModel.from_pretrained(
-    'LorenzoDeMattei/GePpeTto', pad_token_id = tokenizer.eos_token_id
+tokenizer = AutoTokenizer.from_pretrained("LorenzoDeMattei/GePpeTto")
+model = AutoModelWithLMHead.from_pretrained("LorenzoDeMattei/GePpeTto")
+
+text_generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
+prompts = [
+    "Wikipedia Geppetto",
+    "Maestro Ciliegia regala il pezzo di legno al suo amico Geppetto, il quale lo prende per fabbricarsi un burattino maraviglioso"]
+
+
+samples_outputs = text_generator(
+    prompts,
+    do_sample=True,
+    max_length=50,
+    top_k=50,
+    top_p=0.95,
+    num_return_sequences=3
 )

-input_ids = tokenizer.encode(
-    'Wikipedia Geppetto', return_tensors = 'pt'
-)
-sample_outputs = model.generate(
-    input_ids,
-    do_sample = True,
-    max_length = 50,
-    top_k = 50,
-    top_p = 0.95,
-    num_return_sequences = 3,
-)

-print('Output:\n' + 100 * '-')
-for i, sample_output in enumerate(sample_outputs):
-    print(
-        '{}: {}'.format(
-            i, tokenizer.decode(sample_output, skip_special_tokens = True)
-        )
-    )
+for i, sample_outputs in enumerate(samples_outputs):
+    print(100 * '-')
+    print("Prompt:", prompts[i])
+    for sample_output in sample_outputs:
+        print("Sample:", sample_output['generated_text'])
+        print()
+
 ```

 Output is,

-```text
-Output:
+```
 ----------------------------------------------------------------------------------------------------
-0: Wikipedia Geppetto
+Prompt: Wikipedia Geppetto
+Sample: Wikipedia Geppetto rosso (film 1920)

-Geppetto è una città degli Stati Uniti d'America, situata nello Stato dell'Iowa, nella Contea di Greene.
+Geppetto rosso ("The Smokes in the Black") è un film muto del 1920 diretto da Henry H. Leonard.

-Wikipedia The Sax
+Il film fu prodotto dalla Selig Poly

-The Sax è il primo album discografico
-2: Wikipedia Geppetto/Passione
+Sample: Wikipedia Geppetto

-Geppetto è il primo album in studio dei Saturday Night Live, pubblicato dalla Iron Maiden nel 1974.
+Geppetto ("Geppetto" in piemontese) è un comune italiano di 978 abitanti della provincia di Cuneo in Piemonte.

-L'album è un lavoro di debutto che lo porta a definire
-3: Wikipedia Geppetto
+L'abitato, che si trova nel versante valtellinese, si sviluppa nella

-Geppetto ("Fenëvëv" in calabrese) è un comune italiano di abitanti della regione Calabria.
+Sample: Wikipedia Geppetto di Natale (romanzo)

-Zona di particolare pregio storico-artistico, paesaggistico, storico-artistico,
+Geppetto di Natale è un romanzo di Mario Caiano, pubblicato nel 2012.
+
+----------------------------------------------------------------------------------------------------
+Prompt: Maestro Ciliegia regala il pezzo di legno al suo amico Geppetto, il quale lo prende per fabbricarsi un burattino maraviglioso
+Sample: Maestro Ciliegia regala il pezzo di legno al suo amico Geppetto, il quale lo prende per fabbricarsi un burattino maraviglioso. Il burattino riesce a scappare. Dopo aver trovato un prezioso sacchetto si reca
+
+Sample: Maestro Ciliegia regala il pezzo di legno al suo amico Geppetto, il quale lo prende per fabbricarsi un burattino maraviglioso, e l'unico che lo possiede, ma, di fronte a tutte queste prove
+
+Sample: Maestro Ciliegia regala il pezzo di legno al suo amico Geppetto, il quale lo prende per fabbricarsi un burattino maraviglioso: - A voi gli occhi, le guance! A voi il mio pezzo!
 ```

 ## Citation
@@ -0,0 +1,44 @@
+---
+language: polish
+---
+
+# HerBERT tokenizer
+
+**[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** tokenizer is a character level byte-pair encoding with
+vocabulary size of 50k tokens. The tokenizer was trained on [Wolne Lektury](https://wolnelektury.pl/) and a publicly available subset of
+[National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=0) with [fastBPE](https://github.com/glample/fastBPE) library.
+Tokenizer utilize `XLMTokenizer` implementation from [transformers](https://github.com/huggingface/transformers).
+
+## Tokenizer usage
+Herbert tokenizer should be used together with [HerBERT model](https://huggingface.co/allegro/herbert-klej-cased-v1):
+```python
+from transformers import XLMTokenizer, RobertaModel
+
+tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
+model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
+
+encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
+outputs = model(encoded_input)
+```
+
+## License
+CC BY-SA 4.0
+
+## Citation
+If you use this tokenizer, please cite the following paper:
+```
+@misc{rybak2020klej,
+    title={KLEJ: Comprehensive Benchmark for Polish Language Understanding},
+    author={Piotr Rybak and Robert Mroczkowski and Janusz Tracz and Ireneusz Gawlik},
+    year={2020},
+    eprint={2005.00630},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+Paper is accepted at ACL 2020, as soon as proceedings appear, we will update the BibTeX.
+
+## Authors
+Tokenizer was created by **Allegro Machine Learning Research** team.
+
+You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>
@@ -0,0 +1,85 @@
+---
+language: polish
+---
+
+# HerBERT 
+**[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** is a BERT-based Language Model trained on Polish Corpora
+using only MLM objective with dynamic masking of whole words. For more details, please refer to: 
+[KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://arxiv.org/abs/2005.00630).
+
+## Dataset
+**HerBERT** training dataset is a combination of several publicly available corpora for Polish language:
+
+| Corpus | Tokens | Texts |
+| :------ | ------: | ------: |
+| [OSCAR](https://traces1.inria.fr/oscar/)| 6710M  | 145M |
+| [Open Subtitles](http://opus.nlpl.eu/OpenSubtitles-v2018.php) | 1084M  | 1.1M |
+| [Wikipedia](https://dumps.wikimedia.org/) | 260M  | 1.5M |
+| [Wolne Lektury](https://wolnelektury.pl/) | 41M  | 5.5k |
+| [Allegro Articles](https://allegro.pl/artykuly) | 18M  | 33k |
+
+## Tokenizer
+The training dataset was tokenized into subwords using [HerBERT Tokenizer](https://huggingface.co/allegro/herbert-klej-cased-tokenizer-v1); a character level byte-pair encoding with
+a vocabulary size of 50k tokens. The tokenizer itself was trained on [Wolne Lektury](https://wolnelektury.pl/) and a publicly available subset of 
+[National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=0) with a [fastBPE](https://github.com/glample/fastBPE) library.
+
+Tokenizer utilizes `XLMTokenizer` implementation for that reason, one should load it as `allegro/herbert-klej-cased-tokenizer-v1`.
+
+## HerBERT models summary
+| Model | WWM | Cased | Tokenizer | Vocab Size  | Batch Size | Train Steps |
+| :------ | ------: | ------: | ------: | ------: | ------: | ------: |
+| herbert-klej-cased-v1 | YES | YES | BPE | 50K | 570 | 180k | 
+
+## Model evaluation
+HerBERT was evaluated on the [KLEJ](https://klejbenchmark.com/) benchmark, publicly available set of nine evaluation tasks for the Polish language understanding.
+It had the best average performance and obtained the best results for three of them.
+
+| Model | Average | NKJP-NER | CDSC-E | CDSC-R | CBD | PolEmo2.0-IN	|PolEmo2.0-OUT | DYK | PSC | AR	|
+| :------ | ------: | ------: | ------: | ------: | ------: | ------: | ------: |  ------: | ------: | ------: |
+| herbert-klej-cased-v1 | **80.5** | 92.7 | 92.5 | 91.9 | **50.3** | **89.2** |**76.3** |52.1 |95.3 | 84.5 |
+
+Full leaderboard is available [online](https://klejbenchmark.com/leaderboard). 
+
+
+## HerBERT usage
+Model training and experiments were conducted with [transformers](https://github.com/huggingface/transformers) in version 2.0.
+
+Example code:
+```python
+from transformers import XLMTokenizer, RobertaModel
+
+tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
+model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
+
+encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
+outputs = model(encoded_input)
+```
+
+HerBERT can also be loaded using `AutoTokenizer` and `AutoModel`:
+
+```python
+tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
+model = AutoModel.from_pretrained("allegro/herbert-klej-cased-v1")
+```
+
+## License
+CC BY-SA 4.0
+
+## Citation
+If you use this model, please cite the following paper:
+```
+@misc{rybak2020klej,
+    title={KLEJ: Comprehensive Benchmark for Polish Language Understanding},
+    author={Piotr Rybak and Robert Mroczkowski and Janusz Tracz and Ireneusz Gawlik},
+    year={2020},
+    eprint={2005.00630},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+Paper is accepted at ACL 2020, as soon as proceedings appear, we will update the BibTeX.
+
+## Authors
+Model was trained by **Allegro Machine Learning Research** team.
+
+You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>
@@ -11,7 +11,7 @@ A baseline model for question-answering in french ([CamemBERT](https://camembert
 ## Training hyperparameters

 ```shell
-python3 ./examples/run_squad.py \
+python3 ./examples/question-answering/run_squad.py \
 --model_type camembert \
 --model_name_or_path camembert-base \
 --do_train \
@@ -11,7 +11,7 @@ A baseline model for question-answering in french ([CamemBERT](https://camembert
 ## Training hyperparameters

 ```shell
-python3 ./examples/run_squad.py \
+python3 ./examples/question-answering/run_squad.py \
 --model_type camembert \
 --model_name_or_path camembert-base \
 --do_train \
@@ -11,7 +11,7 @@ A baseline model for question-answering in french ([flaubert](https://github.com
 ## Training hyperparameters

 ```shell
-python3 ./examples/run_squad.py \
+python3 ./examples/question-answering/run_squad.py \
 --model_type flaubert \
 --model_name_or_path flaubert-base-uncased \
 --do_train \
@@ -0,0 +1,57 @@
+## Reformer Language model on character level and trained on enwik8. 
+
+*enwik8* is a dataset based on Wikipedia and is often used to measure the model's ability to *compress* data, *e.g.* in 
+the scope of the *Hutter prize*: https://en.wikipedia.org/wiki/Hutter_Prize.
+
+`reformer-enwik8` was pretrained on the first 90M chars of *enwik8* whereas the text was chunked into batches of size 65536 chars (=2^16).
+The model's weights were taken from https://console.cloud.google.com/storage/browser/trax-ml/reformer/enwik8 and converted 
+to Hugging Face's PyTorch ReformerLM model `ReformerModelWithLMHead`.
+
+The model is a language model that operates on characters. 
+Therefore, this model does not need a tokenizer. The following function can instead be used for **encoding** and **decoding**:
+
+```python
+import torch
+
+# Encoding
+def encode(list_of_strings, pad_to_max_length=True, pad_token_id=0):
+    max_length = max([len(string) for string in list_of_strings])
+
+    # create emtpy tensors
+    attention_masks = torch.zeros((len(list_of_strings), max_length), dtype=torch.long)
+    input_ids = torch.full((len(list_of_strings), max_length), pad_token_id, dtype=torch.long)
+
+    for idx, string in enumerate(list_of_strings):
+        # make sure string is in byte format
+        if not isinstance(string, bytes):
+            string = str.encode(string)
+
+        input_ids[idx, :len(string)] = torch.tensor([x + 2 for x in string])
+        attention_masks[idx, :len(string)] = 1
+
+    return input_ids, attention_masks
+    
+# Decoding
+def decode(outputs_ids):
+    decoded_outputs = []
+    for output_ids in outputs_ids.tolist():
+        # transform id back to char IDs < 2 are simply transformed to ""
+        decoded_outputs.append("".join([chr(x - 2) if x > 1 else "" for x in output_ids]))
+    return decoded_outputs
+```
+
+Text can be generated as follows:
+
+```python
+from transformers import ReformerModelWithLMHead
+
+model = ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8")
+encoded, attention_masks = encode(["In 1965, Brooks left IBM to found the Department of"])
+decode(model.generate(encoded, do_sample=True, max_length=150))
+
+# gives:
+# In 1965, Brooks left IBM to found the Department of Journalism in 1968. IBM had jurisdiction himself in 1980, while Brooks resolved, nevertheless thro
+
+```
+
+***Note***: Language generation using `ReformerModelWithLMHead` is not optimized yet and is rather slow.
@@ -1,3 +1,4 @@
+
 # XLM-R + NER

 This model is a fine-tuned  [XLM-Roberta-base](https://arxiv.org/abs/1911.02116) over the 40 languages proposed in [XTREME]([https://github.com/google-research/xtreme](https://github.com/google-research/xtreme)) from [Wikiann](https://aclweb.org/anthology/P17-1178). This is still an on-going work and the results will be updated everytime an improvement is reached. 
@@ -12,6 +13,7 @@ O

 ## Metrics on evaluation set:
 ### Average over the 40 languages
+Number of documents: 262300
 ```
           precision    recall  f1-score   support

@@ -24,6 +26,7 @@ macro avg       0.86      0.87      0.87    333298
 ```

 ### Afrikaans
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -36,6 +39,7 @@ macro avg       0.87      0.91      0.89      1469
 ``` 

 ### Arabic
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -48,6 +52,7 @@ macro avg       0.87      0.88      0.88     10754
 ```

 ### Basque
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -60,6 +65,7 @@ macro avg       0.89      0.89      0.89     12954
 ```

 ### Bengali
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -72,6 +78,7 @@ macro avg       0.91      0.92      0.91      1095
 ```

 ### Bulgarian
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -84,6 +91,7 @@ macro avg       0.91      0.92      0.91     14116
 ```

 ### Burmese
+Number of documents: 100
 ```
           precision    recall  f1-score   support

@@ -96,6 +104,7 @@ macro avg       0.57      0.65      0.60       103
 ```

 ### Chinese
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -108,6 +117,7 @@ macro avg       0.76      0.78      0.77     11558
 ```

 ### Dutch
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -120,6 +130,7 @@ macro avg       0.91      0.92      0.91     13120
 ```

 ### English
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -132,6 +143,7 @@ macro avg       0.82      0.83      0.83     13973
 ```

 ### Estonian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -144,6 +156,7 @@ macro avg       0.90      0.91      0.90     13558
 ```

 ### Finnish
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -156,6 +169,7 @@ macro avg       0.89      0.89      0.89     13930
 ```

 ### French
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -168,6 +182,7 @@ macro avg       0.89      0.90      0.90     12933
 ```

 ### Georgian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -180,6 +195,7 @@ macro avg       0.84      0.86      0.85     12615
 ```

 ### German
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -192,6 +208,7 @@ macro avg       0.86      0.86      0.86     13638
 ```

 ### Greek
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -204,6 +221,7 @@ macro avg       0.88      0.90      0.89     12101
 ```

 ### Hebrew
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -216,6 +234,7 @@ macro avg       0.82      0.83      0.83     12934
 ```

 ### Hindi
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -228,6 +247,7 @@ macro avg       0.84      0.87      0.85      1211
 ```

 ### Hungarian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -240,6 +260,7 @@ macro avg       0.91      0.92      0.91     13879
 ```

 ### Indonesian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -252,6 +273,7 @@ macro avg       0.91      0.92      0.92     11376
 ```

 ### Italian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -264,6 +286,7 @@ macro avg       0.90      0.90      0.90     13412
 ```

 ### Japanese
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -276,6 +299,7 @@ macro avg       0.69      0.72      0.70     12277
 ```

 ### Javanese
+Number of documents: 100
 ```
           precision    recall  f1-score   support

@@ -288,6 +312,7 @@ macro avg       0.78      0.82      0.80       112
 ```

 ### Kazakh
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -300,6 +325,7 @@ macro avg       0.81      0.83      0.81      1135
 ```

 ### Korean
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -312,6 +338,7 @@ macro avg       0.83      0.83      0.83     13329
 ```

 ### Malay
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -324,6 +351,7 @@ macro avg       0.91      0.92      0.91      1088
 ```

 ### Malayalam
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -336,6 +364,7 @@ macro avg       0.78      0.80      0.79      1155
 ```

 ### Marathi
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -348,6 +377,7 @@ macro avg       0.85      0.86      0.85      1190
 ```

 ### Persian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -360,6 +390,7 @@ macro avg       0.92      0.92      0.92     10494
 ```

 ### Portuguese
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -372,6 +403,7 @@ macro avg       0.90      0.91      0.90     12673
 ```

 ### Russian
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -384,6 +416,7 @@ macro avg       0.87      0.88      0.88     12051
 ```

 ### Spanish
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -396,6 +429,7 @@ macro avg       0.90      0.91      0.90     12153
 ```

 ### Swahili
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -408,6 +442,7 @@ macro avg       0.88      0.89      0.88      1202
 ```

 ### Tagalog
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -420,6 +455,7 @@ macro avg       0.90      0.92      0.91      1027
 ```

 ### Tamil
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -432,6 +468,7 @@ macro avg       0.82      0.83      0.82      1183
 ```

 ### Telugu
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -444,6 +481,7 @@ macro avg       0.73      0.77      0.75      1193
 ```

 ### Thai
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -456,6 +494,7 @@ macro avg       0.68      0.74      0.71     14722
 ```

 ### Turkish
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -468,6 +507,7 @@ macro avg       0.91      0.92      0.91     13360
 ```

 ### Urdu
+Number of documents: 1000
 ```
           precision    recall  f1-score   support

@@ -480,6 +520,7 @@ macro avg       0.92      0.94      0.93      1011
 ```

 ### Vietnamese
+Number of documents: 10000
 ```
           precision    recall  f1-score   support

@@ -492,6 +533,7 @@ macro avg       0.89      0.90      0.90     11107
 ```

 ### Yoruba
+Number of documents: 100
 ```
           precision    recall  f1-score   support

@@ -504,7 +546,7 @@ macro avg       0.63      0.68      0.63       107
 ```

 ## Reproduce the results
-Download and prepare the dataset from the [[https://github.com/google-research/xtreme#download-the-data](https://github.com/google-research/xtreme#download-the-data)](XTREME repo). Next, from the root of the transformers repo run:
+Download and prepare the dataset from the [XTREME repo](https://github.com/google-research/xtreme#download-the-data). Next, from the root of the transformers repo run:
 ```
 cd examples/ner
 python run_tf_ner.py \
@@ -533,8 +575,9 @@ nlp_ner = pipeline(
    model="jplu/tf-xlm-r-ner-40-lang",
    tokenizer=(
        'jplu/tf-xlm-r-ner-40-lang',  
-        {"use_fast": True}
-))
+        {"use_fast": True}),
+    framework="tf"
+)

 text_fr = "Barack Obama est né à Hawaï."
 text_en = "Barack Obama was born in Hawaii."
@@ -553,4 +596,4 @@ nlp_ner(test_zh)
 nlp_ner(test_ar)
 #Output: [{'word': '▁با', 'score': 0.9903655648231506, 'entity': 'PER'}, {'word': 'راك', 'score': 0.9850614666938782, 'entity': 'PER'}, {'word': '▁أوباما', 'score': 0.9850308299064636, 'entity': 'PER'}, {'word': '▁ها', 'score': 0.9477543234825134, 'entity': 'LOC'}, {'word': 'وا', 'score': 0.9428229928016663, 'entity': 'LOC'}, {'word': 'ي', 'score': 0.9319471716880798, 'entity': 'LOC'}]

-```
+```
@@ -1,5 +1,5 @@
 ### Model
-**[`albert-xlarge-v2`](https://huggingface.co/albert-xlarge-v2)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+**[`albert-xlarge-v2`](https://huggingface.co/albert-xlarge-v2)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**

 ### Training Parameters
 Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
@@ -1,5 +1,5 @@
 ### Model
-**[`monologg/biobert_v1.1_pubmed`](https://huggingface.co/monologg/biobert_v1.1_pubmed)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+**[`monologg/biobert_v1.1_pubmed`](https://huggingface.co/monologg/biobert_v1.1_pubmed)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**

 This model is cased.

@@ -1,5 +1,5 @@
 ### Model
-**[`allenai/scibert_scivocab_uncased`](https://huggingface.co/allenai/scibert_scivocab_uncased)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
+**[`allenai/scibert_scivocab_uncased`](https://huggingface.co/allenai/scibert_scivocab_uncased)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)**

 ### Training Parameters
 Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
@@ -0,0 +1,60 @@
+---
+language: turkish
+---
+
+# bert-turkish-question-answering
+
+## Usage
+
+```python
+from transformers import pipeline
+nlp = pipeline('question-answering', model='lserinol/bert-turkish-question-answering', tokenizer='lserinol/bert-turkish-question-answering')
+nlp({
+    'question': "Ankara'da kaç ilçe vardır?",
+    'context': r"""Türkiye'nin başkenti Ankara'dır. Ülkenin en büyük idari birimleri illerdir ve 81 il vardır. Bu iller ilçelere ayrılmıştır, toplamda 973 ilçe mevcuttur."""
+})
+```
+
+```python
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering
+
+tokenizer = AutoTokenizer.from_pretrained("lserinol/bert-turkish-question-answering")
+model = AutoModelForQuestionAnswering.from_pretrained("lserinol/bert-turkish-question-answering")
+text = r"""
+Ankara'nın başkent ilan edilmesinin ardından (13 Ekim 1923) şehir hızla gelişmiş ve Türkiye'nin ikinci en kalabalık ili olmuştur.
+Türkiye Cumhuriyeti'nin ilk yıllarında ekonomisi tarım ve hayvancılığa dayanan ilin topraklarının yarısı hâlâ tarım amaçlı 
+kullanılmaktadır. Ekonomik etkinlik büyük oranda ticaret ve sanayiye dayalıdır. Tarım ve hayvancılığın ağırlığı ise giderek 
+azalmaktadır. Ankara ve civarındaki gerek kamu sektörü gerek özel sektör yatırımları, başka illerden büyük bir nüfus göçünü 
+teşvik etmiştir. Cumhuriyetin kuruluşundan günümüze, nüfusu ülke nüfusunun iki katı hızda artmıştır. Nüfusun yaklaşık dörtte 
+üçü hizmet sektörü olarak tanımlanabilecek memuriyet, ulaşım, haberleşme ve ticaret benzeri işlerde, dörtte biri sanayide, 
+%2'si ise tarım alanında çalışır. Sanayi, özellikle tekstil, gıda ve inşaat sektörlerinde yoğunlaşmıştır. Günümüzde ise en çok 
+savunma, metal ve motor sektörlerinde yatırım yapılmaktadır. Türkiye'nin en çok sayıda üniversiteye sahip ili olan Ankara'da 
+ayrıca, üniversite diplomalı kişi oranı ülke ortalamasının iki katıdır. Bu eğitimli nüfus, teknoloji ağırlıklı yatırımların 
+gereksinim duyduğu iş gücünü oluşturur. Ankara'dan otoyollar, demir yolu ve hava yoluyla Türkiye'nin diğer şehirlerine ulaşılır.
+Ankara aynı zamanda başkent olarak Türkiye Büyük Millet Meclisi (TBMM)'ye de ev sahipliği yapmaktadır.
+"""
+
+questions = [
+    "Ankara kaç yılında başkent oldu?",
+    "Ankara ne zaman başkent oldu?",
+    "Ankara'dan başka şehirlere nasıl ulaşılır?",
+    "TBMM neyin kısaltmasıdır?"
+]
+
+for question in questions:
+    inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="pt")
+    input_ids = inputs["input_ids"].tolist()[0]
+
+    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
+    answer_start_scores, answer_end_scores = model(**inputs)
+
+    answer_start = torch.argmax(
+        answer_start_scores
+    )  # Get the most likely beginning of answer with the argmax of the score
+    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score
+
+    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
+
+    print(f"Question: {question}")
+    print(f"Answer: {answer}\n")
+  ```
@@ -40,7 +40,7 @@ python run_language_modeling.py \

 ## Model in action / Example of usage ✒

-You can get the following script [here](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py)
+You can get the following script [here](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py)

 ```bash
 python run_generation.py \
@@ -37,7 +37,7 @@ python run_language_modeling.py \

 ## Model in action / Example of usage: ✒

-You can get the following script [here](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py)
+You can get the following script [here](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py)

 ```bash
 python run_generation.py \
@@ -19,7 +19,7 @@ I preprocessed the dataset and splitted it as train / dev (80/20)
 | Dev                    | 2.2 K |


- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py)
+- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

 - Labels covered:

@@ -29,7 +29,7 @@ The model was trained on a Tesla P100 GPU and 25GB of RAM with the following com

 ```bash
 export SQUAD_DIR=path/to/nl_squad
-python transformers/examples/run_squad.py \
+python transformers/examples/question-answering/run_squad.py \
  --model_type bert \
  --model_name_or_path dccuchile/bert-base-spanish-wwm-cased \
  --do_train \
@@ -29,7 +29,7 @@ The smaller BERT models are intended for environments with restricted computatio
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -29,7 +29,7 @@ The smaller BERT models are intended for environments with restricted computatio
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -29,7 +29,7 @@ The smaller BERT models are intended for environments with restricted computatio
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -11,7 +11,7 @@ thumbnail:

 - Dataset: [GitHub Typo Corpus](https://github.com/mhagiwara/github-typo-corpus) 📚

- [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py) 🏋️‍♂️
+- [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py) 🏋️‍♂️

 ## Metrics on test set 📋

@@ -19,7 +19,7 @@ I preprocessed the dataset and splitted it as train / dev (80/20)
 | Dev                    | 2.2 K |


- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py)
+- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

 - Labels covered:

@@ -11,7 +11,7 @@ This model is a fine-tuned version of the Spanish BERT [(BETO)](https://github.c

 - [Dataset: CONLL Corpora ES](https://www.kaggle.com/nltkdata/conll-corpora)

-#### [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py)
+#### [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

 #### 21 Syntax annotations (Labels) covered:

@@ -19,7 +19,7 @@ I preprocessed the dataset and splitted it as train / dev (80/20)
 | Dev                    | 50 K |


- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py)
+- [Fine-tune on NER script provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

 - **60** Labels covered:

@@ -29,7 +29,7 @@ The smaller BERT models are intended for environments with restricted computatio
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -0,0 +1,77 @@
+# *De Novo* Drug Design with MLM
+
+## What is it?
+
+An approximation to [Generative Recurrent Networks for De Novo Drug Design](https://onlinelibrary.wiley.com/doi/full/10.1002/minf.201700111) but training a MLM (RoBERTa like) from scratch.
+
+## Why?
+
+As mentioned in the paper:
+Generative artificial intelligence models present a fresh approach to chemogenomics and de novo drug design, as they provide researchers with the ability to narrow down their search of the chemical space and focus on regions of interest.
+They used a generative *recurrent neural network (RNN)* containing long short‐term memory (LSTM) cell to capture the syntax of molecular representations in terms of SMILES strings.
+The learned pattern probabilities can be used for de novo SMILES generation. This molecular design concept **eliminates the need for virtual compound library enumeration** and **enables virtual compound design without requiring secondary or external activity prediction**.
+
+
+## My Goal 🎯
+
+By training a MLM from scratch on 438552 (cleaned*) SMILES I wanted to build a model that learns this kind of molecular combinations so that given a partial SMILE it can generate plausible combinations so that it can be proposed as new drugs.
+By cleaned SMILES I mean that I used their [SMILES cleaning script](https://github.com/topazape/LSTM_Chem/blob/master/cleanup_smiles.py) to remove duplicates, salts, and stereochemical information.
+You can see the detailed process of gathering the data, preprocess it and train the LSTM in their [repo](https://github.com/topazape/LSTM_Chem).
+
+## Fast usage with ```pipelines``` 🧪
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+    "fill-mask",
+    model='/mrm8488/chEMBL_smiles_v1',
+    tokenizer='/mrm8488/chEMBL_smiles_v1'
+)
+
+# CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)cc1 Atazanavir
+smile1 = "CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)<mask>"
+
+fill_mask(smile1)
+
+# Output:
+'''
+[{'score': 0.6040295958518982,
+  'sequence': '<s> CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)nc</s>',
+  'token': 265},
+ {'score': 0.2185731679201126,
+  'sequence': '<s> CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)N</s>',
+  'token': 50},
+ {'score': 0.0642734169960022,
+  'sequence': '<s> CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)cc</s>',
+  'token': 261},
+ {'score': 0.01932266168296337,
+  'sequence': '<s> CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)CCCl</s>',
+  'token': 452},
+ {'score': 0.005068355705589056,
+  'sequence': '<s> CC(C)CN(CC(OP(=O)(O)O)C(Cc1ccccc1)NC(=O)OC1CCOC1)S(=O)(=O)c1ccc(N)C</s>',
+  'token': 39}]
+  '''
+  ```
+  ## More
+  I also created a [second version](https://huggingface.co/mrm8488/chEMBL26_smiles_v2) without applying the cleaning SMILES script mentioned above. You can use it in the same way as this one.
+  
+  ```python
+  fill_mask = pipeline(
+    "fill-mask",
+    model='/mrm8488/chEMBL26_smiles_v2',
+    tokenizer='/mrm8488/chEMBL26_smiles_v2'
+)
+```
+  
+ [Original paper](https://www.ncbi.nlm.nih.gov/pubmed/29095571) Authors:
+ <details>
+Swiss Federal Institute of Technology (ETH), Department of Chemistry and Applied Biosciences, Vladimir–Prelog–Weg 4, 8093, Zurich, Switzerland,
+Stanford University, Department of Computer Science, 450 Sierra Mall, Stanford, CA, 94305, USA,
+inSili.com GmbH, 8049, Zurich, Switzerland,
+Gisbert Schneider, Email: hc.zhte@trebsig.
+</details>
+  
+ > Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488) | [LinkedIn](https://www.linkedin.com/in/manuel-romero-cs/)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -11,7 +11,7 @@ thumbnail:

 - Dataset: [GitHub Typo Corpus](https://github.com/mhagiwara/github-typo-corpus) 📚 for 15 languages

- [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py) 🏋️‍♂️
+- [Fine-tune script on NER dataset provided by Huggingface](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py) 🏋️‍♂️

 ## Metrics on test set 📋

@@ -31,7 +31,7 @@ The model was fine-tuned on a Tesla P100 GPU and 25GB of RAM.
 The script is the following:

 ```python
-python transformers/examples/run_squad.py \
+python transformers/examples/question-answering/run_squad.py \
  --model_type distilbert \
  --model_name_or_path distilbert-base-multilingual-cased \
  --do_train \
@@ -0,0 +1,67 @@
+---
+language: spanish
+thumbnail: https://i.imgur.com/uxAvBfh.png
+
+
+---
+
+## ELECTRICIDAD: The Spanish Electra [Imgur](https://imgur.com/uxAvBfh)
+
+**ELECTRICIDAD** is a small Electra like model (discriminator in this case) trained on a + 20 GB of  the [OSCAR](https://oscar-corpus.com/) Spanish corpus.
+
+As mentioned in the original [paper](https://openreview.net/pdf?id=r1xMH1BtvB):
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer the paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+## Model details ⚙
+
+|Param| # Value|
+|-----|--------|
+|Layers|	12   |
+|Hidden |256 	|
+|Params| 14M|
+
+## Evaluation metrics (for discriminator) 🧾
+
+|Metric | # Score |
+|-------|---------|
+|Accuracy| 0.94|
+|Precision| 0.76|
+|AUC | 0.92|
+
+## Benchmarks 🔨
+
+WIP 🚧
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("mrm8488/electricidad-small-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("mrm8488/electricidad-small-discriminator")
+
+sentence = "El rápido zorro marrón salta sobre el perro perezoso"
+fake_sentence = "El rápido zorro marrón falsea sobre el perro perezoso"
+
+fake_tokens = tokenizer.tokenize(sentence)
+fake_inputs = tokenizer.encode(sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % prediction, end="") for prediction in predictions.tolist()]
+```
+
+## Acknowledgments
+
+I thank [🤗/transformers team](https://github.com/huggingface/transformers) for answering my doubts and Google for helping me with the [TensorFlow Research Cloud](https://www.tensorflow.org/tfrc) program.
+
+
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain
@@ -26,7 +26,7 @@ thumbnail:
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -23,7 +23,7 @@ thumbnail:
 ## Model training

 The model was trained on a Tesla P100 GPU and 25GB of RAM.
-The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)
+The script for fine tuning can be found [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

 ## Results:

@@ -0,0 +1,90 @@
+
+# For Turkish language, here is an easy-to-use NER application. 
+ ** Türkçe için kolay bir python  NER (Bert + Transfer Learning)  (İsim Varlık Tanıma) modeli... 
+
+
+Thanks to @stefan-it, I applied the followings for training
+
+
+cd tr-data
+
+for file in train.txt dev.txt test.txt labels.txt
+do
+  wget https://schweter.eu/storage/turkish-bert-wikiann/$file
+done
+
+cd ..
+It will download the pre-processed datasets with training, dev and test splits and put them in a tr-data folder.
+
+Run pre-training
+After downloading the dataset, pre-training can be started. Just set the following environment variables:
+```
+export MAX_LENGTH=128
+export BERT_MODEL=dbmdz/bert-base-turkish-cased 
+export OUTPUT_DIR=tr-new-model
+export BATCH_SIZE=32
+export NUM_EPOCHS=3
+export SAVE_STEPS=625
+export SEED=1
+```
+Then run pre-training:
+```
+python3 run_ner.py --data_dir ./tr-data3 \
+--model_type bert \
+--labels ./tr-data/labels.txt \
+--model_name_or_path $BERT_MODEL \
+--output_dir $OUTPUT_DIR-$SEED \
+--max_seq_length $MAX_LENGTH \
+--num_train_epochs $NUM_EPOCHS \
+--per_gpu_train_batch_size $BATCH_SIZE \
+--save_steps $SAVE_STEPS \
+--seed $SEED \
+--do_train \
+--do_eval \
+--do_predict \
+--fp16
+```
+
+
+# Usage
+
+```
+from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer
+model = AutoModelForTokenClassification.from_pretrained("savasy/bert-base-turkish-ner-cased")
+tokenizer = AutoTokenizer.from_pretrained("savasy/bert-base-turkish-ner-cased")
+ner=pipeline('ner', model=model, tokenizer=tokenizer)
+ner("Mustafa Kemal Atatürk 19 Mayıs 1919'da Samsun'a ayak bastı.")
+```
+# Some results
+Data1:  For the data above
+Eval Results:
+
+* precision = 0.916400580551524
+* recall = 0.9342309684101502
+* f1 = 0.9252298787412536
+* loss = 0.11335893666411284
+
+Test Results:
+* precision = 0.9192058759362955
+* recall = 0.9303010230367262
+* f1 = 0.9247201697271198
+* loss = 0.11182546521618497
+
+
+
+Data2:
+https://github.com/stefan-it/turkish-bert/files/4558187/nerdata.txt
+The performance for the data given by @kemalaraz is as follows
+
+savas@savas-lenova:~/Desktop/trans/tr-new-model-1$ cat eval_results.txt
+* precision = 0.9461980692049029
+* recall = 0.959309358847465
+* f1 = 0.9527086063783312
+* loss = 0.037054269206847804
+
+savas@savas-lenova:~/Desktop/trans/tr-new-model-1$ cat test_results.txt
+* precision = 0.9458370635631155
+* recall = 0.9588201928530913
+* f1 = 0.952284378344882
+* loss = 0.035431676572445225
+
@@ -0,0 +1,146 @@
+# Bert-base Turkish Sentiment Model
+
+https://huggingface.co/savasy/bert-base-turkish-sentiment-cased
+
+This model is used for Sentiment Analysis, which is based on BERTurk for Turkish Language https://huggingface.co/dbmdz/bert-base-turkish-cased
+
+
+# Dataset
+
+The dataset is taken from the studies [2] and [3] and merged.
+
+* The study [2] gathered movie and product reviews. The products are book, DVD, electronics, and kitchen.
+The movie dataset is taken from a cinema Web page (www.beyazperde.com) with
+5331 positive and 5331 negative sentences. Reviews in the Web page are marked in
+scale from 0 to 5 by the users who made the reviews. The study considered a review
+sentiment positive if the rating is equal to or bigger than 4, and negative if it is less
+or equal to 2. They also built Turkish product review dataset from an online retailer
+Web page. They constructed benchmark dataset consisting of reviews regarding some
+products (book, DVD, etc.). Likewise, reviews are marked in the range from 1 to 5,
+and majority class of reviews are 5. Each category has 700 positive and 700 negative
+reviews in which average rating of negative reviews is 2.27 and of positive reviews
+is 4.5. This dataset is also used the study [1]
+
+* The study[3] collected tweet dataset. They proposed a new approach for automatically classifying the sentiment of microblog messages. The proposed approach is based on utilizing robust feature representation and fusion. 
+
+*Merged Dataset* 
+
+| *size*   | *data* |
+|--------|----|
+|   8000 |dev.tsv|
+|   8262 |test.tsv|
+|  32000 |train.tsv|
+|  *48290* |*total*|
+
+
+The dataset is used by following papers
+ 
+* 1 Yildirim, Savaş. (2020). Comparing Deep Neural Networks to Traditional Models for Sentiment Analysis in Turkish Language. 10.1007/978-981-15-1216-2_12. 
+* 2 Demirtas, Erkin and Mykola Pechenizkiy. 2013. Cross-lingual polarity detection with machine translation. In Proceedings of the Second International Workshop on Issues of Sentiment
+Discovery and Opinion Mining (WISDOM ’13)
+* Hayran, A.,   Sert, M. (2017), "Sentiment Analysis on Microblog Data based on Word Embedding and Fusion Techniques", IEEE 25th Signal Processing and Communications Applications Conference (SIU 2017), Belek, Turkey
+
+# Training
+
+```
+export GLUE_DIR="./sst-2-newall"
+export TASK_NAME=SST-2
+ 
+
+python3 run_glue.py \
+  --model_type bert \
+  --model_name_or_path dbmdz/bert-base-turkish-uncased\
+  --task_name "SST-2" \
+  --do_train \
+  --do_eval \
+  --data_dir "./sst-2-newall" \
+  --max_seq_length 128 \
+  --per_gpu_train_batch_size 32 \
+  --learning_rate 2e-5 \
+  --num_train_epochs 3.0 \
+  --output_dir "./model"
+
+```
+
+
+
+
+# Results
+
+> 05/10/2020 17:00:43 - INFO - transformers.trainer -   ***** Running Evaluation *****
+
+> 05/10/2020 17:00:43 - INFO - transformers.trainer -     Num examples = 7999
+
+> 05/10/2020 17:00:43 - INFO - transformers.trainer -     Batch size = 8
+
+>Evaluation: 100% 1000/1000 [00:34<00:00, 29.04it/s]
+
+>05/10/2020 17:01:17 - INFO - __main__ -   ***** Eval results sst-2 *****
+
+>05/10/2020 17:01:17 - INFO - __main__ -     acc = 0.9539942492811602
+
+>05/10/2020 17:01:17 - INFO - __main__ -     loss = 0.16348013816401363
+
+
+Accuracy is about *%95.4*
+# Code Usage
+
+```
+from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
+model = AutoModelForSequenceClassification.from_pretrained("savasy/bert-base-turkish-sentiment-cased")
+tokenizer = AutoTokenizer.from_pretrained("savasy/bert-base-turkish-sentiment-cased")
+sa= pipeline("sentiment-analysis", tokenizer=tokenizer, model=model)
+
+p= sa("bu telefon modelleri çok kaliteli , her parçası çok özel bence")
+print(p)
+#[{'label': 'LABEL_1', 'score': 0.9871089}]
+print (p[0]['label']=='LABEL_1')
+#True
+
+
+p= sa("Film çok kötü ve çok sahteydi")
+print(p)
+#[{'label': 'LABEL_0', 'score': 0.9975505}]
+print (p[0]['label']=='LABEL_1')
+#False
+```
+
+# Test your data
+
+Suppose your file has lots of lines of comment and label (1 or 0) at the end  (tab seperated)
+
+> comment1 ... \t label
+
+> comment2 ... \t label
+ 
+> ...
+
+
+
+```
+from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
+
+f="/path/to/your/file/yourfile.tsv"
+model = AutoModelForSequenceClassification.from_pretrained(folder)
+tokenizer = AutoTokenizer.from_pretrained(folder)
+sa= pipeline("sentiment-analysis", tokenizer=tokenizer, model=model)
+
+i,crr=0,0
+for line in open(f):
+ lines=line.strip().split("\t")
+ if len(lines)==2:
+  i=i+1
+  if i%100==0:
+   print(i)
+  pred= sa(lines[0])
+  pred=pred[0]["label"].split("_")[1]
+  if pred== lines[1]:
+   crr=crr+1
+
+print(crr, i, crr/i)
+```
+
+
+
+
+
@@ -67,8 +67,8 @@ extras = {}

 extras["mecab"] = ["mecab-python3"]
 extras["sklearn"] = ["scikit-learn"]
-extras["tf"] = ["tensorflow"]
-extras["tf-cpu"] = ["tensorflow-cpu"]
+extras["tf"] = ["tensorflow <= 2.1"]
+extras["tf-cpu"] = ["tensorflow-cpu  <= 2.1"]
 extras["torch"] = ["torch"]

 extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
@@ -78,10 +78,10 @@ extras["testing"] = ["pytest", "pytest-xdist", "timeout-decorator"]
 extras["docs"] = ["recommonmark", "sphinx", "sphinx-markdown-tables", "sphinx-rtd-theme"]
 extras["quality"] = [
    "black",
-    "isort",
-    "flake8",
+    "isort @ git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort",
+    "flake8==3.7.9",
 ]
-extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow", "torch"]
+extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow <= 2.1", "torch"]

 setup(
    name="transformers",
@@ -248,7 +248,7 @@ if is_torch_available():
        BART_PRETRAINED_MODEL_ARCHIVE_MAP,
    )
    from .modeling_marian import MarianMTModel
-    from .tokenization_marian import MarianSentencePieceTokenizer
+    from .tokenization_marian import MarianTokenizer
    from .modeling_roberta import (
        RobertaForMaskedLM,
        RobertaModel,
@@ -287,6 +287,7 @@ if is_torch_available():
    from .modeling_albert import (
        AlbertPreTrainedModel,
        AlbertModel,
+        AlbertForPreTraining,
        AlbertForMaskedLM,
        AlbertForSequenceClassification,
        AlbertForQuestionAnswering,
@@ -490,6 +491,7 @@ if is_tf_available():
        TFAlbertPreTrainedModel,
        TFAlbertMainLayer,
        TFAlbertModel,
+        TFAlbertForPreTraining,
        TFAlbertForMaskedLM,
        TFAlbertForSequenceClassification,
        TFAlbertForQuestionAnswering,
@@ -26,7 +26,7 @@ def gelu_new(x):
    """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
        Also see https://arxiv.org/abs/1606.08415
    """
-    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
+    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))


 if torch.__version__ < "1.4.0":
@@ -36,7 +36,7 @@ else:


 def gelu_fast(x):
-    return 0.5 * x * (1 + torch.tanh(x * 0.7978845608 * (1 + 0.044715 * x * x)))
+    return 0.5 * x * (1.0 + torch.tanh(x * 0.7978845608 * (1.0 + 0.044715 * x * x)))


 ACT2FN = {
@@ -62,7 +62,21 @@ class ConvertCommand(BaseTransformersCLICommand):
        self._finetuning_task_name = finetuning_task_name

    def run(self):
-        if self._model_type == "bert":
+        if self._model_type == "albert":
+            try:
+                from transformers.convert_albert_original_tf_checkpoint_to_pytorch import (
+                    convert_tf_checkpoint_to_pytorch,
+                )
+            except ImportError:
+                msg = (
+                    "transformers can only be used from the commandline to convert TensorFlow models in PyTorch, "
+                    "In that case, it requires TensorFlow to be installed. Please see "
+                    "https://www.tensorflow.org/install/ for installation instructions."
+                )
+                raise ImportError(msg)
+
+            convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
+        elif self._model_type == "bert":
            try:
                from transformers.convert_bert_original_tf_checkpoint_to_pytorch import (
                    convert_tf_checkpoint_to_pytorch,
@@ -28,6 +28,7 @@ from .configuration_electra import ELECTRA_PRETRAINED_CONFIG_ARCHIVE_MAP, Electr
 from .configuration_encoder_decoder import EncoderDecoderConfig
 from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
 from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
+from .configuration_marian import MarianConfig
 from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
 from .configuration_reformer import ReformerConfig
 from .configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
@@ -73,6 +74,7 @@ CONFIG_MAPPING = OrderedDict(
        ("albert", AlbertConfig,),
        ("camembert", CamembertConfig,),
        ("xlm-roberta", XLMRobertaConfig,),
+        ("marian", MarianConfig,),
        ("bart", BartConfig,),
        ("reformer", ReformerConfig,),
        ("roberta", RobertaConfig,),
@@ -23,4 +23,5 @@ PRETRAINED_CONFIG_ARCHIVE_MAP = {


 class MarianConfig(BartConfig):
+    model_type = "marian"
    pretrained_config_archive_map = PRETRAINED_CONFIG_ARCHIVE_MAP
@@ -24,7 +24,8 @@ from .configuration_utils import PretrainedConfig
 logger = logging.getLogger(__name__)

 REFORMER_PRETRAINED_CONFIG_ARCHIVE_MAP = {
-    "google/reformer-crime-and-punishment": "https://cdn.huggingface.co/google/reformer-crime-and-punishment/config.json"
+    "google/reformer-crime-and-punishment": "https://cdn.huggingface.co/google/reformer-crime-and-punishment/config.json",
+    "google/reformer-enwik8": "https://cdn.huggingface.co/google/reformer-enwik8/config.json",
 }


@@ -20,7 +20,7 @@ import logging

 import torch

-from transformers import AlbertConfig, AlbertForMaskedLM, load_tf_weights_in_albert
+from transformers import AlbertConfig, AlbertForPreTraining, load_tf_weights_in_albert


 logging.basicConfig(level=logging.INFO)
@@ -30,7 +30,7 @@ def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, albert_config_file, pyt
    # Initialise PyTorch model
    config = AlbertConfig.from_json_file(albert_config_file)
    print("Building PyTorch model from configuration: {}".format(str(config)))
-    model = AlbertForMaskedLM(config)
+    model = AlbertForPreTraining(config)

    # Load weights from tf checkpoint
    load_tf_weights_in_albert(model, config, tf_checkpoint_path)
@@ -11,7 +11,8 @@ import numpy as np
 import torch
 from tqdm import tqdm

-from transformers import MarianConfig, MarianMTModel, MarianSentencePieceTokenizer
+from transformers import MarianConfig, MarianMTModel, MarianTokenizer
+from transformers.hf_api import HfApi


 def remove_prefix(text: str, prefix: str):
@@ -38,6 +39,19 @@ def load_layers_(layer_lst: torch.nn.ModuleList, opus_state: dict, converter, is
        layer.load_state_dict(sd, strict=True)


+def find_pretrained_model(src_lang: str, tgt_lang: str) -> List[str]:
+    """Find models that can accept src_lang as input and return tgt_lang as output."""
+    prefix = "Helsinki-NLP/opus-mt-"
+    api = HfApi()
+    model_list = api.model_list()
+    model_ids = [x.modelId for x in model_list if x.modelId.startswith("Helsinki-NLP")]
+    src_and_targ = [
+        remove_prefix(m, prefix).lower().split("-") for m in model_ids if "+" not in m
+    ]  # + cant be loaded.
+    matching = [f"{prefix}{a}-{b}" for (a, b) in src_and_targ if src_lang in a and tgt_lang in b]
+    return matching
+
+
 def add_emb_entries(wemb, final_bias, n_special_tokens=1):
    vsize, d_model = wemb.shape
    embs_to_add = np.zeros((n_special_tokens, d_model))
@@ -81,7 +95,12 @@ def find_model_file(dest_dir):  # this one better
    return model_file


-def parse_readmes(repo_path):
+def make_registry(repo_path="Opus-MT-train/models"):
+    if not (Path(repo_path) / "fr-en" / "README.md").exists():
+        raise ValueError(
+            f"repo_path:{repo_path} does not exist: "
+            "You must run: git clone git@github.com:Helsinki-NLP/Opus-MT-train.git before calling."
+        )
    results = {}
    for p in Path(repo_path).ls():
        n_dash = p.name.count("-")
@@ -90,22 +109,53 @@ def parse_readmes(repo_path):
        else:
            lns = list(open(p / "README.md").readlines())
            results[p.name] = _parse_readme(lns)
-    return results
+    return [(k, v["pre-processing"], v["download"]) for k, v in results.items()]


-def download_all_sentencepiece_models(repo_path="Opus-MT-train/models"):
+CH_GROUP = "cmn+cn+yue+ze_zh+zh_cn+zh_CN+zh_HK+zh_tw+zh_TW+zh_yue+zhs+zht+zh"
+
+
+def convert_all_sentencepiece_models(model_list=None, repo_path=None):
    """Requires 300GB"""
    save_dir = Path("marian_ckpt")
-    if not Path(repo_path).exists():
-        raise ValueError("You must run: git clone git@github.com:Helsinki-NLP/Opus-MT-train.git")
-    results: dict = parse_readmes(repo_path)
-    for k, v in tqdm(list(results.items())):
-        if os.path.exists(save_dir / k):
-            print(f"already have path {k}")
+    dest_dir = Path("marian_converted")
+    dest_dir.mkdir(exist_ok=True)
+    if model_list is None:
+        model_list: list = make_registry(repo_path=repo_path)
+    for k, prepro, download in tqdm(model_list):
+        if "SentencePiece" not in prepro:  # dont convert BPE models.
            continue
-        if "SentencePiece" not in v["pre-processing"]:
+        if not os.path.exists(save_dir / k / "pytorch_model.bin"):
+            download_and_unzip(download, save_dir / k)
+        pair_name = k.replace(CH_GROUP, "ch_group")
+        convert(save_dir / k, dest_dir / f"opus-mt-{pair_name}")
+
+
+def lmap(f, x) -> List:
+    return list(map(f, x))
+
+
+def fetch_test_set(readmes_raw, pair):
+    import wget
+
+    download_url = readmes_raw[pair]["download"]
+    test_set_url = download_url[:-4] + ".test.txt"
+    fname = wget.download(test_set_url, f"opus_test_{pair}.txt")
+    lns = Path(fname).open().readlines()
+    src = lmap(str.strip, lns[::4])
+    gold = lmap(str.strip, lns[1::4])
+    mar_model = lmap(str.strip, lns[2::4])
+    assert len(gold) == len(mar_model) == len(src)
+    os.remove(fname)
+    return src, mar_model, gold
+
+
+def convert_whole_dir(path=Path("marian_ckpt/")):
+    for subdir in tqdm(list(path.ls())):
+        dest_dir = f"marian_converted/{subdir.name}"
+        if (dest_dir / "pytorch_model.bin").exists():
            continue
-        download_and_unzip(v["download"], save_dir / k)
+        convert(source_dir, dest_dir)


 def _parse_readme(lns):
@@ -131,7 +181,7 @@ def _parse_readme(lns):
    return subres


-def write_metadata(dest_dir: Path):
+def save_tokenizer_config(dest_dir: Path):
    dname = dest_dir.name.split("-")
    dct = dict(target_lang=dname[-1], source_lang="-".join(dname[:-1]))
    save_json(dct, dest_dir / "tokenizer_config.json")
@@ -148,13 +198,17 @@ def add_to_vocab_(vocab: Dict[str, int], special_tokens: List[str]):
    return added


+def find_vocab_file(model_dir):
+    return list(model_dir.glob("*vocab.yml"))[0]
+
+
 def add_special_tokens_to_vocab(model_dir: Path) -> None:
-    vocab = load_yaml(model_dir / "opus.spm32k-spm32k.vocab.yml")
+    vocab = load_yaml(find_vocab_file(model_dir))
    vocab = {k: int(v) for k, v in vocab.items()}
    num_added = add_to_vocab_(vocab, ["<pad>"])
    print(f"added {num_added} tokens to vocab")
    save_json(vocab, model_dir / "vocab.json")
-    write_metadata(model_dir)
+    save_tokenizer_config(model_dir)


 def save_tokenizer(self, save_directory):
@@ -251,7 +305,6 @@ class OpusState:

        # Process decoder.yml
        decoder_yml = cast_marian_config(load_yaml(source_dir / "decoder.yml"))
-        # TODO: what are normalize and word-penalty?
        check_marian_cfg_assumptions(cfg)
        self.hf_config = MarianConfig(
            vocab_size=cfg["vocab_size"],
@@ -273,6 +326,9 @@ class OpusState:
            dropout=0.1,  # see opus-mt-train repo/transformer-dropout param.
            # default: add_final_layer_norm=False,
            num_beams=decoder_yml["beam-size"],
+            decoder_start_token_id=self.pad_token_id,
+            bad_words_ids=[[self.pad_token_id]],
+            max_length=512,
        )

    def _check_layer_entries(self):
@@ -349,12 +405,12 @@ def download_and_unzip(url, dest_dir):
    os.remove(filename)


-def main(source_dir, dest_dir):
+def convert(source_dir: Path, dest_dir):
    dest_dir = Path(dest_dir)
    dest_dir.mkdir(exist_ok=True)

    add_special_tokens_to_vocab(source_dir)
-    tokenizer = MarianSentencePieceTokenizer.from_pretrained(str(source_dir))
+    tokenizer = MarianTokenizer.from_pretrained(str(source_dir))
    save_tokenizer(tokenizer, dest_dir)

    opus_state = OpusState(source_dir)
@@ -377,7 +433,7 @@ if __name__ == "__main__":
    source_dir = Path(args.src)
    assert source_dir.exists()
    dest_dir = f"converted-{source_dir.name}" if args.dest is None else args.dest
-    main(source_dir, dest_dir)
+    convert(source_dir, dest_dir)


 def load_yaml(path):
@@ -46,7 +46,7 @@ from transformers import (
    OpenAIGPTConfig,
    RobertaConfig,
    T5Config,
-    TFAlbertForMaskedLM,
+    TFAlbertForPreTraining,
    TFBertForPreTraining,
    TFBertForQuestionAnswering,
    TFBertForSequenceClassification,
@@ -109,7 +109,7 @@ if is_torch_available():
        DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        CTRLLMHeadModel,
        CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
-        AlbertForMaskedLM,
+        AlbertForPreTraining,
        ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        T5ForConditionalGeneration,
        T5_PRETRAINED_MODEL_ARCHIVE_MAP,
@@ -148,7 +148,7 @@ else:
        DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        CTRLLMHeadModel,
        CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
-        AlbertForMaskedLM,
+        AlbertForPreTraining,
        ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        T5ForConditionalGeneration,
        T5_PRETRAINED_MODEL_ARCHIVE_MAP,
@@ -318,8 +318,8 @@ MODEL_CLASSES = {
    ),
    "albert": (
        AlbertConfig,
-        TFAlbertForMaskedLM,
-        AlbertForMaskedLM,
+        TFAlbertForPreTraining,
+        AlbertForPreTraining,
        ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
        ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
    ),
@@ -93,7 +93,7 @@ def set_block_weights_in_torch(weights, torch_block, hidden_size):
        set_layer_weights_in_torch_local(attn_weights, torch_block.attention, hidden_size)

    # intermediate weighs
-    intermediate_weights = weights[2][0][2][2]
+    intermediate_weights = weights[2][0][1][2]

    # Chunked Feed Forward
    if len(intermediate_weights) == 4:
@@ -145,19 +145,16 @@ def set_model_weights_in_torch(weights, torch_model, hidden_size):
            position_embeddings.weights[emb_idx] = torch.nn.Parameter(torch.tensor(emb_weights))

    trax_layer_weights = weights[5]
-    assert len(torch_model_reformer.encoder.layers) * 4 + 1 == len(
+    assert len(torch_model_reformer.encoder.layers) * 4 == len(
        trax_layer_weights
    ), "HF and trax model do not have the same number of layers"
    for layer_idx, layer in enumerate(torch_model_reformer.encoder.layers):
        block_weights = trax_layer_weights[4 * layer_idx : 4 * (layer_idx + 1)]
        set_block_weights_in_torch(block_weights, layer, hidden_size)

-    # output weights
-    out_weights = weights[6]
-
    # output layer norm
-    layer_norm_out_weight = np.asarray(out_weights[0][0])
-    layer_norm_out_bias = np.asarray(out_weights[0][1])
+    layer_norm_out_weight = np.asarray(weights[7][0])
+    layer_norm_out_bias = np.asarray(weights[7][1])
    set_param(
        torch_model_reformer.encoder.layer_norm,
        torch.tensor(layer_norm_out_weight),
@@ -165,8 +162,8 @@ def set_model_weights_in_torch(weights, torch_model, hidden_size):
    )

    # output embeddings
-    output_embed_weights = np.asarray(out_weights[2][0])
-    output_embed_bias = np.asarray(out_weights[2][1])
+    output_embed_weights = np.asarray(weights[9][0])
+    output_embed_bias = np.asarray(weights[9][1])
    set_param(
        torch_model.lm_head.decoder,
        torch.tensor(output_embed_weights).transpose(0, 1).contiguous(),
@@ -5,12 +5,12 @@ from dataclasses import dataclass, field
 from typing import List, Optional

 import torch
+from filelock import FileLock
 from torch.utils.data.dataset import Dataset

 from ...tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
 from ...tokenization_utils import PreTrainedTokenizer
 from ...tokenization_xlm_roberta import XLMRobertaTokenizer
-from ...trainer import torch_distributed_zero_first
 from ..processors.glue import glue_convert_examples_to_features, glue_output_modes, glue_processors
 from ..processors.utils import InputFeatures

@@ -63,7 +63,6 @@ class GlueDataset(Dataset):
        tokenizer: PreTrainedTokenizer,
        limit_length: Optional[int] = None,
        evaluate=False,
-        local_rank=-1,
    ):
        self.args = args
        processor = glue_processors[args.task_name]()
@@ -75,9 +74,11 @@ class GlueDataset(Dataset):
                "dev" if evaluate else "train", tokenizer.__class__.__name__, str(args.max_seq_length), args.task_name,
            ),
        )
-        with torch_distributed_zero_first(local_rank):
-            # Make sure only the first process in distributed training processes the dataset,
-            # and the others will use the cache.
+
+        # Make sure only the first process in distributed training processes the dataset,
+        # and the others will use the cache.
+        lock_path = cached_features_file + ".lock"
+        with FileLock(lock_path):

            if os.path.exists(cached_features_file) and not args.overwrite_cache:
                start = time.time()
@@ -109,13 +110,12 @@ class GlueDataset(Dataset):
                    label_list=label_list,
                    output_mode=self.output_mode,
                )
-                if local_rank in [-1, 0]:
-                    start = time.time()
-                    torch.save(self.features, cached_features_file)
-                    # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
-                    logger.info(
-                        f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
-                    )
+                start = time.time()
+                torch.save(self.features, cached_features_file)
+                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
+                logger.info(
+                    f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
+                )

    def __len__(self):
        return len(self.features)
@@ -15,6 +15,7 @@ import tempfile
 from contextlib import contextmanager
 from functools import partial, wraps
 from hashlib import sha256
+from pathlib import Path
 from typing import Optional
 from urllib.parse import urlparse
 from zipfile import ZipFile, is_zipfile
@@ -68,19 +69,10 @@ except ImportError:
    )
 default_cache_path = os.path.join(torch_cache_home, "transformers")

-try:
-    from pathlib import Path

-    PYTORCH_PRETRAINED_BERT_CACHE = Path(
-        os.getenv("PYTORCH_TRANSFORMERS_CACHE", os.getenv("PYTORCH_PRETRAINED_BERT_CACHE", default_cache_path))
-    )
-except (AttributeError, ImportError):
-    PYTORCH_PRETRAINED_BERT_CACHE = os.getenv(
-        "PYTORCH_TRANSFORMERS_CACHE", os.getenv("PYTORCH_PRETRAINED_BERT_CACHE", default_cache_path)
-    )
-
-PYTORCH_TRANSFORMERS_CACHE = PYTORCH_PRETRAINED_BERT_CACHE  # Kept for backward compatibility
-TRANSFORMERS_CACHE = PYTORCH_PRETRAINED_BERT_CACHE  # Kept for backward compatibility
+PYTORCH_PRETRAINED_BERT_CACHE = os.getenv("PYTORCH_PRETRAINED_BERT_CACHE", default_cache_path)
+PYTORCH_TRANSFORMERS_CACHE = os.getenv("PYTORCH_TRANSFORMERS_CACHE", PYTORCH_PRETRAINED_BERT_CACHE)
+TRANSFORMERS_CACHE = os.getenv("TRANSFORMERS_CACHE", PYTORCH_TRANSFORMERS_CACHE)

 WEIGHTS_NAME = "pytorch_model.bin"
 TF2_WEIGHTS_NAME = "tf_model.h5"
@@ -111,7 +111,8 @@ def load_tf_weights_in_albert(model, config, tf_checkpoint_path):

        # No ALBERT model currently handles the next sentence prediction task
        if "seq_relationship" in name:
-            continue
+            name = name.replace("seq_relationship/output_", "sop_classifier/classifier/")
+            name = name.replace("weights", "weight")

        name = name.split("/")

@@ -568,6 +569,115 @@ class AlbertModel(AlbertPreTrainedModel):
        return outputs


+@add_start_docstrings(
+    """Albert Model with two heads on top as done during the pre-training: a `masked language modeling` head and
+    a `sentence order prediction (classification)` head. """,
+    ALBERT_START_DOCSTRING,
+)
+class AlbertForPreTraining(AlbertPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.albert = AlbertModel(config)
+        self.predictions = AlbertMLMHead(config)
+        self.sop_classifier = AlbertSOPHead(config)
+
+        self.init_weights()
+        self.tie_weights()
+
+    def tie_weights(self):
+        self._tie_or_clone_weights(self.predictions.decoder, self.albert.embeddings.word_embeddings)
+
+    def get_output_embeddings(self):
+        return self.predictions.decoder
+
+    @add_start_docstrings_to_callable(ALBERT_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        token_type_ids=None,
+        position_ids=None,
+        head_mask=None,
+        inputs_embeds=None,
+        masked_lm_labels=None,
+        sentence_order_label=None,
+    ):
+        r"""
+        masked_lm_labels (``torch.LongTensor`` of shape ``(batch_size, sequence_length)``, `optional`, defaults to :obj:`None`):
+            Labels for computing the masked language modeling loss.
+            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
+            in ``[0, ..., config.vocab_size]``
+        sentence_order_label (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`, defaults to :obj:`None`):
+            Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see :obj:`input_ids` docstring)
+            Indices should be in ``[0, 1]``.
+            ``0`` indicates original order (sequence A, then sequence B),
+            ``1`` indicates switched order (sequence B, then sequence A).
+
+    Returns:
+        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
+        loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
+            Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss.
+        prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+        sop_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, 2)`):
+            Prediction scores of the next sequence prediction (classification) head (scores of True/False
+            continuation before SoftMax).
+        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`):
+            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+            heads.
+
+
+    Examples::
+
+        from transformers import AlbertTokenizer, AlbertForPreTraining
+        import torch
+
+        tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
+        model = AlbertForPreTraining.from_pretrained('albert-base-v2')
+
+        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
+        outputs = model(input_ids)
+
+        prediction_scores, sop_scores = outputs[:2]
+
+        """
+
+        outputs = self.albert(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+        )
+
+        sequence_output, pooled_output = outputs[:2]
+
+        prediction_scores = self.predictions(sequence_output)
+        sop_scores = self.sop_classifier(pooled_output)
+
+        outputs = (prediction_scores, sop_scores,) + outputs[2:]  # add hidden states and attention if they are here
+
+        if masked_lm_labels is not None and sentence_order_label is not None:
+            loss_fct = CrossEntropyLoss()
+            masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
+            sentence_order_loss = loss_fct(sop_scores.view(-1, 2), sentence_order_label.view(-1))
+            total_loss = masked_lm_loss + sentence_order_loss
+            outputs = (total_loss,) + outputs
+
+        return outputs  # (loss), prediction_scores, sop_scores, (hidden_states), (attentions)
+
+
 class AlbertMLMHead(nn.Module):
    def __init__(self, config):
        super().__init__()
@@ -592,6 +702,19 @@ class AlbertMLMHead(nn.Module):
        return prediction_scores


+class AlbertSOPHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+
+        self.dropout = nn.Dropout(config.classifier_dropout_prob)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+    def forward(self, pooled_output):
+        dropout_pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(dropout_pooled_output)
+        return logits
+
+
@add_start_docstrings(
    "Albert Model with a `language modeling` head on top.", ALBERT_START_DOCSTRING,
 )
@@ -932,7 +1055,7 @@ class AlbertForQuestionAnswering(AlbertPreTrainedModel):
    Examples::

        # The checkpoint albert-base-v2 is not fine-tuned for question answering. Please see the
-        # examples/run_squad.py example to see how to fine-tune a model to a question answering task.
+        # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task.

        from transformers import AlbertTokenizer, AlbertForQuestionAnswering
        import torch
@@ -39,10 +39,12 @@ from .configuration_auto import (
    XLMRobertaConfig,
    XLNetConfig,
 )
+from .configuration_marian import MarianConfig
 from .configuration_utils import PretrainedConfig
 from .modeling_albert import (
    ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
    AlbertForMaskedLM,
+    AlbertForPreTraining,
    AlbertForQuestionAnswering,
    AlbertForSequenceClassification,
    AlbertForTokenClassification,
@@ -97,6 +99,7 @@ from .modeling_flaubert import (
    FlaubertWithLMHeadModel,
 )
 from .modeling_gpt2 import GPT2_PRETRAINED_MODEL_ARCHIVE_MAP, GPT2LMHeadModel, GPT2Model
+from .modeling_marian import MarianMTModel
 from .modeling_openai import OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP, OpenAIGPTLMHeadModel, OpenAIGPTModel
 from .modeling_reformer import ReformerModel, ReformerModelWithLMHead
 from .modeling_roberta import (
@@ -189,7 +192,7 @@ MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
    [
        (T5Config, T5ForConditionalGeneration),
        (DistilBertConfig, DistilBertForMaskedLM),
-        (AlbertConfig, AlbertForMaskedLM),
+        (AlbertConfig, AlbertForPreTraining),
        (CamembertConfig, CamembertForMaskedLM),
        (XLMRobertaConfig, XLMRobertaForMaskedLM),
        (BartConfig, BartForConditionalGeneration),
@@ -213,6 +216,7 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
        (AlbertConfig, AlbertForMaskedLM),
        (CamembertConfig, CamembertForMaskedLM),
        (XLMRobertaConfig, XLMRobertaForMaskedLM),
+        (MarianConfig, MarianMTModel),
        (BartConfig, BartForConditionalGeneration),
        (RobertaConfig, RobertaForMaskedLM),
        (BertConfig, BertForMaskedLM),
@@ -902,7 +906,7 @@ class AutoModelForQuestionAnswering:
        Examples::

            config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
-            model = AutoModelForSequenceClassification.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+            model = AutoModelForQuestionAnswering.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
        """
        for config_class, model_class in MODEL_FOR_QUESTION_ANSWERING_MAPPING.items():
            if isinstance(config, config_class):
@@ -142,10 +142,10 @@ class Attention(nn.Module):
    def _attn(self, q, k, v, attention_mask=None, head_mask=None):
        w = torch.matmul(q, k)
        if self.scale:
-            w = w / (v.size(-1) ** 0.5)
+            w = w / (float(v.size(-1)) ** 0.5)
        nd, ns = w.size(-2), w.size(-1)
        mask = self.bias[:, :, ns - nd : ns, :ns]
-        w = torch.where(mask, w, self.masked_bias)
+        w = torch.where(mask.bool(), w, self.masked_bias.to(w.dtype))

        if attention_mask is not None:
            # Apply the attention mask
@@ -18,16 +18,30 @@
 from transformers.modeling_bart import BartForConditionalGeneration


-PRETRAINED_MODEL_ARCHIVE_MAP = {
-    "opus-mt-en-de": "https://cdn.huggingface.co/Helsinki-NLP/opus-mt-en-de/pytorch_model.bin",
-}
-
-
 class MarianMTModel(BartForConditionalGeneration):
-    """Pytorch version of marian-nmt's transformer.h (c++). Designed for the OPUS-NMT translation checkpoints.
-    Model API is identical to BartForConditionalGeneration"""
+    r"""
+    Pytorch version of marian-nmt's transformer.h (c++). Designed for the OPUS-NMT translation checkpoints.
+    Model API is identical to BartForConditionalGeneration.
+    Available models are listed at `Model List <https://huggingface.co/models?search=Helsinki-NLP>`__

-    pretrained_model_archive_map = PRETRAINED_MODEL_ARCHIVE_MAP
+    Examples::
+
+        from transformers import MarianTokenizer, MarianMTModel
+        from typing import List
+        src = 'fr'  # source language
+        trg = 'en'  # target language
+        sample_text = "où est l'arrêt de bus ?"
+        mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'   # `Model List`__
+
+        model = MarianMTModel.from_pretrained(mname)
+        tok = MarianTokenizer.from_pretrained(mname)
+        batch = tok.prepare_translation_batch(src_texts=[sample_text])  # don't need tgt_text for inference
+        gen = model.generate(**batch)  # for forward pass: model(**batch)
+        words: List[str] = tok.batch_decode(gen, skip_special_tokens=True)  # returns "Where is the the bus stop ?"
+
+    """
+
+    pretrained_model_archive_map = {}  # see https://huggingface.co/models?search=Helsinki-NLP

    def prepare_scores_for_generation(self, scores, cur_len, max_length):
        if cur_len == max_length - 1 and self.config.eos_token_id is not None:
@@ -36,7 +36,8 @@ from .modeling_utils import PreTrainedModel, apply_chunking_to_forward
 logger = logging.getLogger(__name__)

 REFORMER_PRETRAINED_MODEL_ARCHIVE_MAP = {
-    "google/reformer-crime-and-punishment": "https://cdn.huggingface.co/google/reformer-crime-and-punishment/pytorch_model.bin"
+    "google/reformer-crime-and-punishment": "https://cdn.huggingface.co/google/reformer-crime-and-punishment/pytorch_model.bin",
+    "google/reformer-enwik8": "https://cdn.huggingface.co/google/reformer-enwik8/pytorch_model.bin",
 }


@@ -561,8 +562,8 @@ class LSHSelfAttention(nn.Module, EfficientAttentionMixin):

        # get correct mask values depending on precision
        if query_key_dots.dtype == torch.float16:
-            self_mask_value = self.self_mask_value_float16
-            mask_value = self.mask_value_float16
+            self_mask_value = self.self_mask_value_float16.half()
+            mask_value = self.mask_value_float16.half()
        else:
            self_mask_value = self.self_mask_value_float32
            mask_value = self.mask_value_float32
@@ -833,7 +834,7 @@ class LocalSelfAttention(nn.Module, EfficientAttentionMixin):
        if mask is not None:
            # get mask tensor depending on half precision or not
            if query_key_dots.dtype == torch.float16:
-                mask_value = self.mask_value_float16
+                mask_value = self.mask_value_float16.half()
            else:
                mask_value = self.mask_value_float32

@@ -643,7 +643,7 @@ class RobertaForQuestionAnswering(BertPreTrainedModel):
    Examples::

        # The checkpoint roberta-large is not fine-tuned for question answering. Please see the
-        # examples/run_squad.py example to see how to fine-tune a model to a question answering task.
+        # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task.

        from transformers import RobertaTokenizer, RobertaForQuestionAnswering
        import torch
@@ -475,7 +475,6 @@ class TFAlbertMLMHead(tf.keras.layers.Layer):
        hidden_states = self.activation(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        hidden_states = self.decoder(hidden_states, mode="linear") + self.decoder_bias
-        hidden_states = hidden_states + self.bias
        return hidden_states


@@ -718,6 +717,73 @@ class TFAlbertModel(TFAlbertPreTrainedModel):
        return outputs


+@add_start_docstrings(
+    """Albert Model with two heads on top for pre-training:
+    a `masked language modeling` head and a `sentence order prediction` (classification) head. """,
+    ALBERT_START_DOCSTRING,
+)
+class TFAlbertForPreTraining(TFAlbertPreTrainedModel):
+    def __init__(self, config, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        self.num_labels = config.num_labels
+
+        self.albert = TFAlbertMainLayer(config, name="albert")
+        self.predictions = TFAlbertMLMHead(config, self.albert.embeddings, name="predictions")
+        self.sop_classifier = TFAlbertSOPHead(config, name="sop_classifier")
+
+    def get_output_embeddings(self):
+        return self.albert.embeddings
+
+    @add_start_docstrings_to_callable(ALBERT_INPUTS_DOCSTRING)
+    def call(self, inputs, **kwargs):
+        r"""
+    Return:
+        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
+        prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`):
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+        sop_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, 2)`):
+            Prediction scores of the sentence order prediction (classification) head (scores of True/False continuation before SoftMax).
+        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when :obj:`config.output_hidden_states=True`):
+            tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
+            tuple of :obj:`tf.Tensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`:
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
+    Examples::
+        import tensorflow as tf
+        from transformers import AlbertTokenizer, TFAlbertForPreTraining
+        tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
+        model = TFAlbertForPreTraining.from_pretrained('albert-base-v2')
+        input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]  # Batch size 1
+        outputs = model(input_ids)
+        prediction_scores, sop_scores = outputs[:2]
+        """
+
+        outputs = self.albert(inputs, **kwargs)
+        sequence_output, pooled_output = outputs[:2]
+        prediction_scores = self.predictions(sequence_output)
+        sop_scores = self.sop_classifier(pooled_output, training=kwargs.get("training", False))
+        outputs = (prediction_scores, sop_scores) + outputs[2:]
+        return outputs
+
+
+class TFAlbertSOPHead(tf.keras.layers.Layer):
+    def __init__(self, config, **kwargs):
+        super().__init__(**kwargs)
+
+        self.dropout = tf.keras.layers.Dropout(config.classifier_dropout_prob)
+        self.classifier = tf.keras.layers.Dense(
+            config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="classifier",
+        )
+
+    def call(self, pooled_output, training: bool):
+        dropout_pooled_output = self.dropout(pooled_output, training=training)
+        logits = self.classifier(dropout_pooled_output)
+        return logits
+
+
@add_start_docstrings("""Albert Model with a `language modeling` head on top. """, ALBERT_START_DOCSTRING)
 class TFAlbertForMaskedLM(TFAlbertPreTrainedModel):
    def __init__(self, config, *inputs, **kwargs):
@@ -865,7 +931,7 @@ class TFAlbertForQuestionAnswering(TFAlbertPreTrainedModel):
    Examples::

        # The checkpoint albert-base-v2 is not fine-tuned for question answering. Please see the
-        # examples/run_squad.py example to see how to fine-tune a model to a question answering task.
+        # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task.

        import tensorflow as tf
        from transformers import AlbertTokenizer, TFAlbertForQuestionAnswering
@@ -36,6 +36,7 @@ from .configuration_utils import PretrainedConfig
 from .modeling_tf_albert import (
    TF_ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
    TFAlbertForMaskedLM,
+    TFAlbertForPreTraining,
    TFAlbertForQuestionAnswering,
    TFAlbertForSequenceClassification,
    TFAlbertModel,
@@ -132,7 +133,7 @@ TF_MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
    [
        (T5Config, TFT5ForConditionalGeneration),
        (DistilBertConfig, TFDistilBertForMaskedLM),
-        (AlbertConfig, TFAlbertForMaskedLM),
+        (AlbertConfig, TFAlbertForPreTraining),
        (RobertaConfig, TFRobertaForMaskedLM),
        (BertConfig, TFBertForPreTraining),
        (OpenAIGPTConfig, TFOpenAIGPTLMHeadModel),
@@ -412,7 +413,7 @@ class TFAutoModelForPreTraining(object):
        in the `pretrained_model_name_or_path` string (in the following order):
            - contains `t5`: :class:`~transformers.TFT5ModelWithLMHead` (T5 model)
            - contains `distilbert`: :class:`~transformers.TFDistilBertForMaskedLM` (DistilBERT model)
-            - contains `albert`: :class:`~transformers.TFAlbertForMaskedLM` (ALBERT model)
+            - contains `albert`: :class:`~transformers.TFAlbertForPreTraining` (ALBERT model)
            - contains `roberta`: :class:`~transformers.TFRobertaForMaskedLM` (RoBERTa model)
            - contains `bert`: :class:`~transformers.TFBertForPreTraining` (Bert model)
            - contains `openai-gpt`: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
@@ -481,7 +481,7 @@ class TFRobertaForQuestionAnswering(TFRobertaPreTrainedModel):
    Examples::

        # The checkpoint roberta-base is not fine-tuned for question answering. Please see the
-        # examples/run_squad.py example to see how to fine-tune a model to a question answering task.
+        # examples/question-answering/run_squad.py example to see how to fine-tune a model to a question answering task.

        import tensorflow as tf
        from transformers import RobertaTokenizer, TFRobertaForQuestionAnswering
@@ -204,7 +204,10 @@ class GradientAccumulator(object):
        """Number of accumulated steps."""
        if self._accum_steps is None:
            self._accum_steps = tf.Variable(
-                tf.constant(0, dtype=tf.int64), trainable=False, synchronization=tf.VariableSynchronization.ON_READ,
+                tf.constant(0, dtype=tf.int64),
+                trainable=False,
+                synchronization=tf.VariableSynchronization.ON_READ,
+                aggregation=tf.VariableAggregation.ONLY_FIRST_REPLICA,
            )

        return self._accum_steps.value()
@@ -223,7 +226,10 @@ class GradientAccumulator(object):
            self._gradients.extend(
                [
                    tf.Variable(
-                        tf.zeros_like(gradient), trainable=False, synchronization=tf.VariableSynchronization.ON_READ,
+                        tf.zeros_like(gradient),
+                        trainable=False,
+                        synchronization=tf.VariableSynchronization.ON_READ,
+                        aggregation=tf.VariableAggregation.ONLY_FIRST_REPLICA,
                    )
                    for gradient in gradients
                ]
@@ -570,6 +570,7 @@ class TextGenerationPipeline(Pipeline):
    # Padding text to help Transformer-XL and XLNet with short prompts as proposed by Aman Rusia
    # in https://github.com/rusiaaman/XLNet-gen#methodology
    # and https://medium.com/@amanrusia/xlnet-speaks-comparison-to-gpt-2-ea1a4e9ba39e
+
    PADDING_TEXT = """In 1991, the remains of Russian Tsar Nicholas II and his family
    (except for Alexei and Maria) are discovered.
    The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the
@@ -581,9 +582,30 @@ class TextGenerationPipeline(Pipeline):
    the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous,
    with people, even a bishop, begging for his blessing. <eod> </s> <eos>"""

+    ALLOWED_MODELS = [
+        "XLNetLMHeadModel",
+        "TransfoXLLMHeadModel",
+        "ReformerModelWithLMHead",
+        "GPT2LMHeadModel",
+        "OpenAIGPTLMHeadModel",
+        "CTRLLMHeadModel",
+        "TFXLNetLMHeadModel",
+        "TFTransfoXLLMHeadModel",
+        "TFGPT2LMHeadModel",
+        "TFOpenAIGPTLMHeadModel",
+        "TFCTRLLMHeadModel",
+    ]
+
    def __call__(
        self, *args, return_tensors=False, return_text=True, clean_up_tokenization_spaces=False, **generate_kwargs
    ):
+        if self.model.__class__.__name__ not in self.ALLOWED_MODELS:
+            raise NotImplementedError(
+                "Generation is currently not supported for {}. Please select a model from {} for generation.".format(
+                    self.model.__class__.__name__, self.ALLOWED_MODELS
+                )
+            )
+
        text_inputs = self._args_parser(*args)

        results = []
@@ -614,7 +636,7 @@ class TextGenerationPipeline(Pipeline):

            result = []
            for generated_sequence in output_sequences:
-                generated_sequence = generated_sequence.tolist()
+                generated_sequence = generated_sequence.numpy().tolist()
                record = {}
                if return_tensors:
                    record["generated_token_ids"] = generated_sequence
@@ -1509,7 +1531,7 @@ SUPPORTED_TASKS = {
                "tf": "distilbert-base-uncased-finetuned-sst-2-english",
            },
            "config": "distilbert-base-uncased-finetuned-sst-2-english",
-            "tokenizer": "distilbert-base-cased",
+            "tokenizer": "distilbert-base-uncased",
        },
    },
    "ner": {
@@ -274,7 +274,7 @@ class AlbertTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """

        if already_has_special_tokens:
@@ -38,6 +38,7 @@ from .configuration_auto import (
    XLMRobertaConfig,
    XLNetConfig,
 )
+from .configuration_marian import MarianConfig
 from .configuration_utils import PretrainedConfig
 from .tokenization_albert import AlbertTokenizer
 from .tokenization_bart import BartTokenizer
@@ -49,6 +50,7 @@ from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFas
 from .tokenization_electra import ElectraTokenizer, ElectraTokenizerFast
 from .tokenization_flaubert import FlaubertTokenizer
 from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
+from .tokenization_marian import MarianTokenizer
 from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
 from .tokenization_reformer import ReformerTokenizer
 from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
@@ -69,6 +71,7 @@ TOKENIZER_MAPPING = OrderedDict(
        (AlbertConfig, (AlbertTokenizer, None)),
        (CamembertConfig, (CamembertTokenizer, None)),
        (XLMRobertaConfig, (XLMRobertaTokenizer, None)),
+        (MarianConfig, (MarianTokenizer, None)),
        (BartConfig, (BartTokenizer, None)),
        (RobertaConfig, (RobertaTokenizer, RobertaTokenizerFast)),
        (ReformerConfig, (ReformerTokenizer, None)),
@@ -269,7 +269,7 @@ class BertTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """

        if already_has_special_tokens:
@@ -102,6 +102,7 @@ class CamembertTokenizer(PreTrainedTokenizer):
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]

    def __init__(
        self,
@@ -181,7 +182,7 @@ class CamembertTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            if token_ids_1 is not None:
@@ -200,14 +201,7 @@ class CamembertTokenizer(PreTrainedTokenizer):
    ) -> List[int]:
        """
        Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
-        A CamemBERT sequence pair mask has the following format:
-
-        ::
-
-            0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
-            | first sequence  | | second sequence |
-
-        if token_ids_1 is None, only returns the first portion of the mask (0s).
+        CamemBERT, like RoBERTa, does not make use of token type ids, therefore a list of zeros is returned.

        Args:
            token_ids_0 (:obj:`List[int]`):
@@ -216,15 +210,15 @@ class CamembertTokenizer(PreTrainedTokenizer):
                Optional second list of IDs for sequence pairs.

        Returns:
-            :obj:`List[int]`: List of `token type IDs <../glossary.html#token-type-ids>`_ according to the given
-            sequence(s).
+            :obj:`List[int]`: List of zeros.
+
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
-        return len(cls + token_ids_0 + sep + sep) * [0] + len(token_ids_1 + sep) * [1]
+        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    @property
    def vocab_size(self):
@@ -22,7 +22,21 @@ PRETRAINED_VOCAB_FILES_MAP = {
 # Example URL https://s3.amazonaws.com/models.huggingface.co/bert/Helsinki-NLP/opus-mt-en-de/vocab.json


-class MarianSentencePieceTokenizer(PreTrainedTokenizer):
+class MarianTokenizer(PreTrainedTokenizer):
+    """Sentencepiece tokenizer for marian. Source and target languages have different SPM models.
+    The logic is use the relevant source_spm or target_spm to encode txt as pieces, then look up each piece in a vocab dictionary.
+
+    Examples::
+
+        from transformers import MarianTokenizer
+        tok = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
+        src_texts = [ "I am a small frog.", "Tom asked his teacher for advice."]
+        tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."]  # optional
+        batch_enc: BatchEncoding = tok.prepare_translation_batch(src_texts, tgt_texts=tgt_texts)
+        # keys  [input_ids, attention_mask, decoder_input_ids,  decoder_attention_mask].
+        # model(**batch) should work
+    """
+
    vocab_files_names = vocab_files_names
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = {m: 512 for m in MODEL_NAMES}
@@ -49,6 +63,8 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
            pad_token=pad_token,
        )
        self.encoder = load_json(vocab)
+        if self.unk_token not in self.encoder:
+            raise KeyError("<unk> token must be in vocab")
        assert self.pad_token in self.encoder
        self.decoder = {v: k for k, v in self.encoder.items()}

@@ -64,8 +80,11 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
        self.spm_target = sentencepiece.SentencePieceProcessor()
        self.spm_target.Load(target_spm)

-        # Note(SS): splitter would require lots of book-keeping.
-        # self.sentence_splitter = MosesSentenceSplitter(source_lang)
+        # Multilingual target side: default to using first supported language code.
+        self.supported_language_codes: list = [k for k in self.encoder if k.startswith(">>") and k.endswith("<<")]
+        self.tgt_lang_id = None  # will not be used unless it is set through prepare_translation_batch
+
+        # Note(SS): sentence_splitter would require lots of book-keeping.
        try:
            from mosestokenizer import MosesPunctuationNormalizer

@@ -75,11 +94,10 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
            self.punc_normalizer = lambda x: x

    def _convert_token_to_id(self, token):
-        return self.encoder[token]
+        return self.encoder.get(token, self.encoder[self.unk_token])

-    def _tokenize(self, text: str, src=True) -> List[str]:
-        spm = self.spm_source if src else self.spm_target
-        return spm.EncodeAsPieces(text)
+    def _tokenize(self, text: str) -> List[str]:
+        return self.current_spm.EncodeAsPieces(text)

    def _convert_id_to_token(self, index: int) -> str:
        """Converts an index (integer) in a token (str) using the encoder."""
@@ -89,10 +107,6 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
        """Uses target language sentencepiece model"""
        return self.spm_target.DecodePieces(tokens)

-    def _append_special_tokens_and_truncate(self, tokens: str, max_length: int,) -> List[int]:
-        ids: list = self.convert_tokens_to_ids(tokens)[:max_length]
-        return ids + [self.eos_token_id]
-
    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None) -> List[int]:
        """Build model inputs from a sequence by appending eos_token_id."""
        if token_ids_1 is None:
@@ -100,7 +114,7 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
        # We don't expect to process pairs, but leave the pair logic for API consistency
        return token_ids_0 + token_ids_1 + [self.eos_token_id]

-    def decode_batch(self, token_ids, **kwargs) -> List[str]:
+    def batch_decode(self, token_ids, **kwargs) -> List[str]:
        return [self.decode(ids, **kwargs) for ids in token_ids]

    def prepare_translation_batch(
@@ -114,40 +128,38 @@ class MarianSentencePieceTokenizer(PreTrainedTokenizer):
        """
        Arguments:
            src_texts: list of src language texts
-            src_lang: default en_XX (english)
            tgt_texts: list of tgt language texts
-            tgt_lang: default ro_RO (romanian)
            max_length: (None) defer to config (1024 for mbart-large-en-ro)
            pad_to_max_length: (bool)
+            return_tensors: (str) default "pt" returns pytorch tensors, pass None to return lists.

        Returns:
            BatchEncoding: with keys [input_ids, attention_mask, decoder_input_ids,  decoder_attention_mask]
-            all shaped bs, seq_len. (BatchEncoding is a dict of string -> tensor or lists)
-
-        Examples:
-            from transformers import MarianS
+            all shaped bs, seq_len. (BatchEncoding is a dict of string -> tensor or lists).
+            If no tgt_text is specified, the only keys will be input_ids and attention_mask.
        """
+        self.current_spm = self.spm_source
        model_inputs: BatchEncoding = self.batch_encode_plus(
            src_texts,
            add_special_tokens=True,
            return_tensors=return_tensors,
            max_length=max_length,
            pad_to_max_length=pad_to_max_length,
-            src=True,
        )
        if tgt_texts is None:
            return model_inputs

+        self.current_spm = self.spm_target
        decoder_inputs: BatchEncoding = self.batch_encode_plus(
            tgt_texts,
            add_special_tokens=True,
            return_tensors=return_tensors,
            max_length=max_length,
            pad_to_max_length=pad_to_max_length,
-            src=False,
        )
        for k, v in decoder_inputs.items():
            model_inputs[f"decoder_{k}"] = v
+        self.current_spm = self.spm_source
        return model_inputs

    @property
@@ -193,7 +193,7 @@ class RobertaTokenizer(GPT2Tokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            if token_ids_1 is not None:
@@ -893,7 +893,7 @@ class XLMTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """

        if already_has_special_tokens:
@@ -217,7 +217,7 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """

        if already_has_special_tokens:
@@ -278,7 +278,7 @@ class XLNetTokenizer(PreTrainedTokenizer):
                Set to True if the token list is already formatted with special tokens for the model

        Returns:
-            :obj:`List[int]`: A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
+            :obj:`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """

        if already_has_special_tokens:
@@ -6,7 +6,7 @@ import re
 import shutil
 from contextlib import contextmanager
 from pathlib import Path
-from typing import Callable, Dict, List, Optional, Tuple
+from typing import Callable, Dict, List, Optional, Tuple, Union

 import numpy as np
 import torch
@@ -195,10 +195,12 @@ class Trainer:
        if eval_dataset is None and self.eval_dataset is None:
            raise ValueError("Trainer: evaluation requires an eval_dataset.")

+        eval_dataset = eval_dataset if eval_dataset is not None else self.eval_dataset
+
        sampler = get_tpu_sampler(eval_dataset) if is_tpu_available() else None

        data_loader = DataLoader(
-            eval_dataset if eval_dataset is not None else self.eval_dataset,
+            eval_dataset,
            sampler=sampler,
            batch_size=self.args.eval_batch_size,
            shuffle=False,
@@ -267,6 +269,16 @@ class Trainer:
        # keep track of model topology and gradients
        wandb.watch(self.model)

+    def num_examples(self, dataloader: Union[DataLoader, "pl.PerDeviceLoader"]) -> int:
+        """
+        Helper to get num of examples from a DataLoader, by accessing its Dataset.
+        """
+        if is_tpu_available():
+            assert isinstance(dataloader, pl.PerDeviceLoader)
+            return len(dataloader._loader._loader.dataset)
+        else:
+            return len(dataloader.dataset)
+
    def train(self, model_path: Optional[str] = None):
        """
        Main training entry point.
@@ -326,17 +338,15 @@ class Trainer:

        # Train!
        if is_tpu_available():
-            num_examples = len(train_dataloader._loader._loader.dataset)
            total_train_batch_size = self.args.train_batch_size * xm.xrt_world_size()
        else:
-            num_examples = len(train_dataloader.dataset)
            total_train_batch_size = (
                self.args.train_batch_size
                * self.args.gradient_accumulation_steps
                * (torch.distributed.get_world_size() if self.args.local_rank != -1 else 1)
            )
        logger.info("***** Running training *****")
-        logger.info("  Num examples = %d", num_examples)
+        logger.info("  Num examples = %d", self.num_examples(train_dataloader))
        logger.info("  Num Epochs = %d", num_train_epochs)
        logger.info("  Instantaneous batch size per device = %d", self.args.per_gpu_train_batch_size)
        logger.info("  Total train batch size (w. parallel, distributed & accumulation) = %d", total_train_batch_size)
@@ -606,9 +616,13 @@ class Trainer:
            model = self.model
        model.to(self.args.device)

+        if is_tpu_available():
+            batch_size = dataloader._loader._loader.batch_size
+        else:
+            batch_size = dataloader.batch_size
        logger.info("***** Running %s *****", description)
-        logger.info("  Num examples = %d", len(dataloader.dataset))
-        logger.info("  Batch size = %d", dataloader.batch_size)
+        logger.info("  Num examples = %d", self.num_examples(dataloader))
+        logger.info("  Batch size = %d", batch_size)
        eval_losses: List[float] = []
        preds: np.ndarray = None
        label_ids: np.ndarray = None
@@ -56,9 +56,11 @@ class TFTrainingArguments(TrainingArguments):
                strategy = tf.distribute.experimental.TPUStrategy(tpu)
            elif len(gpus) == 0:
                strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
+            elif len(gpus) == 1:
+                strategy = tf.distribute.OneDeviceStrategy(device="/gpu:0")
            elif len(gpus) > 1:
                # If you only want to use a specific subset of GPUs use `CUDA_VISIBLE_DEVICES=0`
-                strategy = tf.distribute.MirroredStrategy(gpus)
+                strategy = tf.distribute.MirroredStrategy()
            else:
                raise ValueError("Cannot find the proper strategy please check your environment properties.")

@@ -27,6 +27,7 @@ if is_torch_available():
    from transformers import (
        AlbertConfig,
        AlbertModel,
+        AlbertForPreTraining,
        AlbertForMaskedLM,
        AlbertForSequenceClassification,
        AlbertForTokenClassification,
@@ -38,7 +39,7 @@ if is_torch_available():
@require_torch
 class AlbertModelTest(ModelTesterMixin, unittest.TestCase):

-    all_model_classes = (AlbertModel, AlbertForMaskedLM) if is_torch_available() else ()
+    all_model_classes = (AlbertModel, AlbertForPreTraining, AlbertForMaskedLM) if is_torch_available() else ()

    class AlbertModelTester(object):
        def __init__(
@@ -151,6 +152,30 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
            )
            self.parent.assertListEqual(list(result["pooled_output"].size()), [self.batch_size, self.hidden_size])

+        def create_and_check_albert_for_pretraining(
+            self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
+        ):
+            model = AlbertForPreTraining(config=config)
+            model.to(torch_device)
+            model.eval()
+            loss, prediction_scores, sop_scores = model(
+                input_ids,
+                attention_mask=input_mask,
+                token_type_ids=token_type_ids,
+                masked_lm_labels=token_labels,
+                sentence_order_label=sequence_labels,
+            )
+            result = {
+                "loss": loss,
+                "prediction_scores": prediction_scores,
+                "sop_scores": sop_scores,
+            }
+            self.parent.assertListEqual(
+                list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
+            )
+            self.parent.assertListEqual(list(result["sop_scores"].size()), [self.batch_size, config.num_labels])
+            self.check_loss_output(result)
+
        def create_and_check_albert_for_masked_lm(
            self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels, choice_labels
        ):
@@ -252,6 +277,10 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
        self.model_tester.create_and_check_albert_model(*config_and_inputs)

+    def test_for_pretraining(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_albert_for_pretraining(*config_and_inputs)
+
    def test_for_masked_lm(self):
        config_and_inputs = self.model_tester.prepare_config_and_inputs()
        self.model_tester.create_and_check_albert_for_masked_lm(*config_and_inputs)
@@ -18,35 +18,94 @@ import unittest

 from transformers import is_torch_available
 from transformers.file_utils import cached_property
+from transformers.hf_api import HfApi

 from .utils import require_torch, slow, torch_device


 if is_torch_available():
    import torch
-    from transformers import MarianMTModel, MarianSentencePieceTokenizer
+    from transformers import (
+        AutoTokenizer,
+        MarianConfig,
+        AutoConfig,
+        AutoModelWithLMHead,
+        MarianTokenizer,
+        MarianMTModel,
+    )
+
+
+class ModelManagementTests(unittest.TestCase):
+    @slow
+    def test_model_count(self):
+        model_list = HfApi().model_list()
+        expected_num_models = 1011
+        actual_num_models = len([x for x in model_list if x.modelId.startswith("Helsinki-NLP")])
+        self.assertEqual(expected_num_models, actual_num_models)


@require_torch
-class IntegrationTests(unittest.TestCase):
+class MarianIntegrationTest(unittest.TestCase):
+    src = "en"
+    tgt = "de"
+    src_text = [
+        "I am a small frog.",
+        "Now I can forget the 100 words of german that I know.",
+        "Tom asked his teacher for advice.",
+        "That's how I would do it.",
+        "Tom really admired Mary's courage.",
+        "Turn around and close your eyes.",
+    ]
+    expected_text = [
+        "Ich bin ein kleiner Frosch.",
+        "Jetzt kann ich die 100 Wörter des Deutschen vergessen, die ich kenne.",
+        "Tom bat seinen Lehrer um Rat.",
+        "So würde ich das machen.",
+        "Tom bewunderte Marias Mut wirklich.",
+        "Drehen Sie sich um und schließen Sie die Augen.",
+    ]
+    # ^^ actual C++ output differs slightly: (1) des Deutschen removed, (2) ""-> "O", (3) tun -> machen
+
    @classmethod
    def setUpClass(cls) -> None:
-        cls.model_name = "Helsinki-NLP/opus-mt-en-de"
-        cls.tokenizer = MarianSentencePieceTokenizer.from_pretrained(cls.model_name)
+        cls.model_name = f"Helsinki-NLP/opus-mt-{cls.src}-{cls.tgt}"
+        cls.tokenizer: MarianTokenizer = AutoTokenizer.from_pretrained(cls.model_name)
        cls.eos_token_id = cls.tokenizer.eos_token_id
        return cls

    @cached_property
    def model(self):
-        model = MarianMTModel.from_pretrained(self.model_name).to(torch_device)
+        model: MarianMTModel = AutoModelWithLMHead.from_pretrained(self.model_name).to(torch_device)
+        c = model.config
+        self.assertListEqual(c.bad_words_ids, [[c.pad_token_id]])
+        self.assertEqual(c.max_length, 512)
+        self.assertEqual(c.decoder_start_token_id, c.pad_token_id)
+
        if torch_device == "cuda":
            return model.half()
        else:
            return model

+    def _assert_generated_batch_equal_expected(self, **tokenizer_kwargs):
+        generated_words = self.translate_src_text(**tokenizer_kwargs)
+        self.assertListEqual(self.expected_text, generated_words)
+
+    def translate_src_text(self, **tokenizer_kwargs):
+        model_inputs: dict = self.tokenizer.prepare_translation_batch(src_texts=self.src_text, **tokenizer_kwargs).to(
+            torch_device
+        )
+        self.assertEqual(self.model.device, model_inputs["input_ids"].device)
+        generated_ids = self.model.generate(
+            model_inputs["input_ids"], attention_mask=model_inputs["attention_mask"], num_beams=2
+        )
+        generated_words = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+        return generated_words
+
+
+class TestMarian_EN_DE_More(MarianIntegrationTest):
    @slow
    def test_forward(self):
-        src, tgt = ["I am a small frog"], ["▁Ich ▁bin ▁ein ▁kleiner ▁Fro sch"]
+        src, tgt = ["I am a small frog"], ["Ich bin ein kleiner Frosch."]
        expected = [38, 121, 14, 697, 38848, 0]

        model_inputs: dict = self.tokenizer.prepare_translation_batch(src, tgt_texts=tgt).to(torch_device)
@@ -62,57 +121,112 @@ class IntegrationTests(unittest.TestCase):
        with torch.no_grad():
            logits, *enc_features = self.model(**model_inputs)
        max_indices = logits.argmax(-1)
-        self.tokenizer.decode_batch(max_indices)
+        self.tokenizer.batch_decode(max_indices)

-    @slow
-    def test_repl_generate_one(self):
-        src = ["I am a small frog.", "Hello"]
-        model_inputs: dict = self.tokenizer.prepare_translation_batch(src).to(torch_device)
-        self.assertEqual(self.model.device, model_inputs["input_ids"].device)
-        generated_ids = self.model.generate(model_inputs["input_ids"], num_beams=6,)
-        generated_words = self.tokenizer.decode_batch(generated_ids)[0]
-        expected_words = "Ich bin ein kleiner Frosch."
-        self.assertEqual(expected_words, generated_words)
-
-    @slow
-    def test_repl_generate_batch(self):
-        src = [
-            "I am a small frog.",
-            "Now I can forget the 100 words of german that I know.",
-            "O",
-            "Tom asked his teacher for advice.",
-            "That's how I would do it.",
-            "Tom really admired Mary's courage.",
-            "Turn around and close your eyes.",
-        ]
-        model_inputs: dict = self.tokenizer.prepare_translation_batch(src).to(torch_device)
-        self.assertEqual(self.model.device, model_inputs["input_ids"].device)
-        generated_ids = self.model.generate(
-            model_inputs["input_ids"],
-            length_penalty=1.0,
-            num_beams=2,  # 6 is the default
-            bad_words_ids=[[self.tokenizer.pad_token_id]],
-        )
-        expected = [
-            "Ich bin ein kleiner Frosch.",
-            "Jetzt kann ich die 100 Wörter des Deutschen vergessen, die ich kenne.",
-            "",
-            "Tom bat seinen Lehrer um Rat.",
-            "So würde ich das tun.",
-            "Tom bewunderte Marias Mut wirklich.",
-            "Umdrehen und die Augen schließen.",
-        ]
-        # actual C++ output differences: (1) des Deutschen removed, (2) ""-> "O", (3) tun -> machen
-        generated_words = self.tokenizer.decode_batch(generated_ids, skip_special_tokens=True)
-        self.assertListEqual(expected, generated_words)
-
-    def test_marian_equivalence(self):
+    def test_tokenizer_equivalence(self):
        batch = self.tokenizer.prepare_translation_batch(["I am a small frog"]).to(torch_device)
        input_ids = batch["input_ids"][0]
        expected = [38, 121, 14, 697, 38848, 0]
        self.assertListEqual(expected, input_ids.tolist())

+    def test_unk_support(self):
+        t = self.tokenizer
+        ids = t.prepare_translation_batch(["||"]).to(torch_device)["input_ids"][0].tolist()
+        expected = [t.unk_token_id, t.unk_token_id, t.eos_token_id]
+        self.assertEqual(expected, ids)
+
    def test_pad_not_split(self):
        input_ids_w_pad = self.tokenizer.prepare_translation_batch(["I am a small frog <pad>"])["input_ids"][0]
        expected_w_pad = [38, 121, 14, 697, 38848, self.tokenizer.pad_token_id, 0]  # pad
        self.assertListEqual(expected_w_pad, input_ids_w_pad.tolist())
+
+    @slow
+    def test_batch_generation_en_de(self):
+        self._assert_generated_batch_equal_expected()
+
+    def test_auto_config(self):
+        config = AutoConfig.from_pretrained(self.model_name)
+        self.assertIsInstance(config, MarianConfig)
+
+
+class TestMarian_EN_FR(MarianIntegrationTest):
+    src = "en"
+    tgt = "fr"
+    src_text = [
+        "I am a small frog.",
+        "Now I can forget the 100 words of german that I know.",
+    ]
+    expected_text = [
+        "Je suis une petite grenouille.",
+        "Maintenant, je peux oublier les 100 mots d'allemand que je connais.",
+    ]
+
+    @slow
+    def test_batch_generation_en_fr(self):
+        self._assert_generated_batch_equal_expected()
+
+
+class TestMarian_FR_EN(MarianIntegrationTest):
+    src = "fr"
+    tgt = "en"
+    src_text = [
+        "Donnez moi le micro.",
+        "Tom et Mary étaient assis à une table.",  # Accents
+    ]
+    expected_text = [
+        "Give me the microphone.",
+        "Tom and Mary were sitting at a table.",
+    ]
+
+    @slow
+    def test_batch_generation_fr_en(self):
+        self._assert_generated_batch_equal_expected()
+
+
+class TestMarian_RU_FR(MarianIntegrationTest):
+    src = "ru"
+    tgt = "fr"
+    src_text = ["Он показал мне рукопись своей новой пьесы."]
+    expected_text = ["Il me montre un manuscrit de sa nouvelle pièce."]
+
+    @slow
+    def test_batch_generation_ru_fr(self):
+        self._assert_generated_batch_equal_expected()
+
+
+class TestMarian_MT_EN(MarianIntegrationTest):
+    src = "mt"
+    tgt = "en"
+    src_text = ["Il - Babiloniżi b'mod żbaljat ikkonkludew li l - Alla l - veru kien dgħajjef."]
+    expected_text = ["The Babylonians wrongly concluded that the true God was weak."]
+
+    @unittest.skip("")  # Known Issue: This model generates a string of .... at the end of the translation.
+    def test_batch_generation_mt_en(self):
+        self._assert_generated_batch_equal_expected()
+
+
+class TestMarian_DE_Multi(MarianIntegrationTest):
+    src = "de"
+    tgt = "ch_group"
+    src_text = ["Er aber sprach: Das ist die Gottlosigkeit."]
+
+    @slow
+    def test_translation_de_multi_does_not_error(self):
+        self.translate_src_text()
+
+    @unittest.skip("")  # "Language codes are not yet supported."
+    def test_batch_generation_de_multi_tgt(self):
+        self._assert_generated_batch_equal_expected()
+
+    @unittest.skip("")  # "Language codes are not yet supported."
+    def test_lang_code(self):
+        t = "Er aber sprach"
+        zh_code = self.code
+        tok_fn = self.tokenizer.prepare_translation_batch
+        pass_code = tok_fn(src_texts=[t], tgt_lang_code=zh_code)["input_ids"][0]
+        preprocess_with_code = tok_fn(src_texts=[zh_code + "  " + t])["input_ids"][0]
+        self.assertListEqual(pass_code.tolist(), preprocess_with_code.tolist())
+        for code in self.tokenizer.supported_language_codes:
+            self.assertIn(code, self.tokenizer.encoder)
+        pass_only_code = tok_fn(src_texts=[""], tgt_lang_code=zh_code)["input_ids"][0].tolist()
+        self.assertListEqual(pass_only_code, [self.tokenizer.encoder[zh_code], self.tokenizer.eos_token_id])
Author	SHA1	Message	Date
Lysandre Debut	30e343862f	pin TF to 2.1 (#4297 ) * pin TF to 2.1 * Pin flake8 as well	2020-05-11 21:03:30 -04:00
Julien Chaumond	56e8ef632f	[ci] Restrict GPU tests to actual code commits	2020-05-11 20:40:41 -04:00
Julien Chaumond	ba6f6e44a8	[ci] Re-enable torch GPU tests	2020-05-12 00:05:36 +00:00
Lysandre Debut	9524956819	Documentation specification (#4294 )	2020-05-11 16:43:57 -04:00
Bram Vanroy	61d22f9cc7	Simplify cache vars and allow for TRANSFORMERS_CACHE env (#4226 ) * simplify cache vars and allow for TRANSFORMERS_CACHE env As it currently stands, "TRANSFORMERS_CACHE" is not an accepted variable. It seems that the these variables were not updated when moving from version pytorch_transformers to transformers. In addition, the fallback procedure could be improved. and simplified. Pathlib seems redundant here. * Update file_utils.py	2020-05-11 15:24:02 -04:00
Lysandre Debut	cd40cb8879	Fix special token doc (#4292 )	2020-05-11 15:05:36 -04:00
Tianlei Wu	82601f4c1a	Allow gpt2 to be exported to valid ONNX (#4244 ) * allow gpt2 to be exported to valid ONNX model * cast size from int to float explictly	2020-05-11 14:55:55 -04:00
Guo, Quan	39994051e4	Add migrating from `pytorch-transformers` (#4273 ) "Migrating from pytorch-transformers to transformers" is missing in the main document. It is available in the main `readme` thought. Just move it to the document.	2020-05-11 13:35:13 -04:00
Lysandre Debut	051dcb2a07	CamemBERT does not make use of Token Type IDs (#4289 )	2020-05-11 13:31:03 -04:00
fgaim	41e8291217	Add ALBERT to the Tensorflow to Pytorch model conversion cli (#3933 ) * Add ALBERT to convert command of transformers-cli * Document ALBERT tf to pytorch model conversion	2020-05-11 13:10:00 -04:00
Stefan Schweter	3f42eb979f	Documentation: fix links to NER examples (#4279 ) * docs: fix link to token classification (NER) example * examples: fix links to NER scripts	2020-05-11 12:48:21 -04:00
Funtowicz Morgan	8fdb7997c6	Align sentiment-analysis' tokenizer (currently uncased) to the model (uncased). (#4264 )	2020-05-11 12:45:53 -04:00
Sam Shleifer	4658896ee1	[Marian] Fix typo in docstring (#4284 )	2020-05-11 11:47:51 -04:00
Levent SerinolandJulien Chaumond	bf64b8cf09	Model card for bert-turkish-question-answering question-answering model (#4281 ) * Create README.md * Update model_cards/lserinol/bert-turkish-question-answering/README.md Co-authored-by: Julien Chaumond <chaumond@gmail.com>	2020-05-11 11:32:25 -04:00
Julien Plu	94b57bf796	[TF 2.2 compat] use tf.VariableAggregation.ONLY_FIRST_REPLICA (#4283 ) * Fix the issue to properly run the accumulator with TF 2.2 * Apply style * Fix training_args_tf for TF 2.2 * Fix the TF training args when only one GPU is available * Remove the fixed version of TF in setup.py	2020-05-11 11:28:37 -04:00
Savaş Yıldırım	cffbb3d8ed	Update README.md (#4276 )	2020-05-11 11:24:41 -04:00
Julien Plu	5f50d619dd	Fix XTREME link + add number of eval documents + fix usage code (#4280 )	2020-05-11 11:24:10 -04:00
theblackcat102	7751be7cee	fix reformer apex scaling issue (#4242 )	2020-05-11 16:53:42 +02:00
Patrick von Platen	ac7d5f67a2	[Reformer] Add Enwiki8 Reformer Model - Adapt convert script (#4282 ) * adapt convert script * update convert script * finish * fix marian pretrained docs	2020-05-11 16:38:07 +02:00
Patrick von Platen	336116d960	Reformer enwik8 - Model card (#4286 )	2020-05-11 16:22:08 +02:00
flozi00	b290c32e16	[docs] fix typo (#4249 )	2020-05-10 14:07:08 -04:00
Sam Shleifer	3487be75ef	[Marian] documentation and AutoModel support (#4152 ) - MarianSentencepieceTokenizer - > MarianTokenizer - Start using unk token. - add docs page - add better generation params to MarianConfig - more conversion utilities	2020-05-10 13:54:57 -04:00
Girishkumar	9d2f467bfb	[README] Corrected some grammatical mistakes (#4199 )	2020-05-10 09:02:36 -04:00
Julien Chaumond	7b75aa9fa5	[TPU] Doc, fix xla_spawn.py, only preprocess dataset once (#4223 ) * [TPU] Doc, fix xla_spawn.py, only preprocess dataset once * Update examples/README.md * [xla_spawn] Add `_mp_fn` to other Trainer scripts * [TPU] Fix: eval dataloader was None	2020-05-08 14:10:05 -04:00
Julien Chaumond	274d850d34	Fix #4098	2020-05-08 12:39:46 -04:00
Lorenzo De MatteiandJulien Chaumond	26dad0a9fa	example updated to use generation pipeline (#4230 ) * example updated to use generation pipeline * Update model_cards/LorenzoDeMattei/GePpeTto/README.md Co-authored-by: Julien Chaumond <chaumond@gmail.com>	2020-05-08 09:45:10 -04:00
rmroczkowski	9ebb5b2a54	Model card for allegro/herbert-klej-cased-tokenizer-v1 (#4184 )	2020-05-08 09:42:43 -04:00
rmroczkowski	9e54efd004	Model card for allegro/herbert-klej-cased-v1 (#4183 )	2020-05-08 09:42:28 -04:00
Manuel Romero	a8b798e6c4	Model card for spanish electra small (#4196 )	2020-05-08 09:30:15 -04:00
Savaş Yıldırım	242005d762	Create README.md (#4132 ) * Create README.md * Adding code fence around code block	2020-05-08 09:27:29 -04:00
Manuel Romero	5940c73bbb	Create README.md (#4179 ) model card for my De Novo Drug discovery model using MLM	2020-05-08 09:25:36 -04:00
Patrick von Platen	cf08830c28	[Pipeline, Generation] tf generation pipeline bug (#4217 ) * fix PR * move tests to correct place	2020-05-08 08:30:05 -04:00
Jared T NielsenandLysandre	8bf7312654	Add AlbertForPreTraining and TFAlbertForPreTraining models. (#4057 ) * Add AlbertForPreTraining and TFAlbertForPreTraining models. * PyTorch conversion * TensorFlow conversion * style Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>	2020-05-07 19:44:51 -04:00
Julien Chaumond	c99fe0386b	[doc] Fix broken links + remove crazy big notebook	2020-05-07 18:44:18 -04:00
Savaş Yıldırım	66113bd626	Create README.md (#4202 )	2020-05-07 18:31:22 -04:00
Julien Chaumond	6669915b65	[examples] Add column for pytorch-lightning support	2020-05-07 15:26:58 -04:00
Julien Chaumond	612fa1b10b	Examples readme.md (#4215 ) * README * Update README.md	2020-05-07 15:00:06 -04:00
Lysandre	2e57824374	Pin isort and tf <= 2.1.0	2020-05-07 14:42:00 -04:00