Bumping tokenizers dependency to 0.8.0.dev1 to check tests are passing.

Formatting.
Remove self._is_fast to avoid conflits with __getattr__()
2020-05-28 09:56:00 +02:00 · 2020-05-22 11:43:58 +02:00 · 2020-05-22 11:21:51 +02:00 · 2020-05-22 11:17:05 +02:00 · 2020-05-22 11:16:48 +02:00 · 2020-05-22 10:43:40 +02:00
39 changed files with 541 additions and 1263 deletions
@@ -35,7 +35,7 @@ jobs:
    - name: Install dependencies
      run: |
        source .env/bin/activate
-        pip install torch
+        pip install torch==1.4.0
        pip install .[sklearn,testing]

    - name: Are GPUs recognized by our DL frameworks
@@ -1,6 +1,6 @@
-## Examples
+# Examples

-Version 2.9 of `transformers` introduces a new [`Trainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py) class for PyTorch, and its equivalent [`TFTrainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py) for TF 2.
+Version 2.9 of `transformers` introduces a new `Trainer` class for PyTorch, and its equivalent `TFTrainer` for TF 2.

 Here is the list of all our examples:
 - **grouped by task** (all official examples work for multiple models)
@@ -12,24 +12,32 @@ Here is the list of all our examples:
 This is still a work-in-progress – in particular documentation is still sparse – so please **contribute improvements/pull requests.**


-# The Big Table of Tasks
+## Tasks built on Trainer

-| Task | Example datasets | Trainer support | TFTrainer support | pytorch-lightning | Colab
-|---|---|:---:|:---:|:---:|:---:|
-| [**`language-modeling`**](./language-modeling)       | Raw text        | ✅ | -  | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)
-| [**`text-classification`**](./text-classification)   | GLUE, XNLI      | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb)
-| [**`token-classification`**](./token-classification) | CoNLL NER       | ✅ | ✅ | ✅ | -
-| [**`multiple-choice`**](./multiple-choice)           | SWAG, RACE, ARC | ✅ | ✅ | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb)
-| [**`question-answering`**](./question-answering)     | SQuAD           | -  | ✅ | -  | -
-| [**`text-generation`**](./text-generation)     | -           | -  | - | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)
-| [**`distillation`**](./distillation)       | All               | -  | -  | -  | -
-| [**`summarization`**](./summarization)     | CNN/Daily Mail    | -  | -  | -  | -
-| [**`translation`**](./translation)         | WMT               | -  | -  | -  | -
-| [**`bertology`**](./bertology)             | -                 | -  | -  | -  | -
-| [**`adversarial`**](./adversarial)         | HANS              | -  | -  | -  | -
+| Task | Example datasets | Trainer support | TFTrainer support | pytorch-lightning | Colab | One-click Deploy to Azure (wip) |
+|---|---|:---:|:---:|:---:|:---:|:---:|
+| [`language-modeling`](./language-modeling) | Raw text | ✅ | - | - | - | - |
+| [`text-classification`](./text-classification) | GLUE, XNLI | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb) | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-storage-account-create%2Fazuredeploy.json) |
+| [`token-classification`](./token-classification) | CoNLL NER | ✅ | ✅ | ✅ | - | - |
+| [`multiple-choice`](./multiple-choice) | SWAG, RACE, ARC | ✅ | ✅ | - | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb) | - |
+| [`question-answering`](./question-answering) | SQuAD | - | ✅ | - | - | - |


-<br>
+
+## Other examples and how-to's
+
+| Section | Description |
+|---|---|
+| [TensorFlow 2.0 models on GLUE](./text-classification) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. |
+| [Running on TPUs](#running-on-tpus) | Examples on running fine-tuning tasks on Google TPUs to accelerate workloads. |
+| [Language Model training](./language-modeling) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
+| [Language Generation](./text-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
+| [GLUE](./text-classification) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
+| [SQuAD](./question-answering) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
+| [Multiple Choice](./multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
+| [Named Entity Recognition](./token-classification) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
+| [XNLI](./text-classification) | Examples running BERT/XLM on the XNLI benchmark. |
+| [Adversarial evaluation of model performances](./adversarial) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |

 ## Important note

@@ -44,12 +52,6 @@ pip install .
 pip install -r ./examples/requirements.txt
 ```

-## One-click Deploy to Cloud (wip)
-
-#### Azure
-
-[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-storage-account-create%2Fazuredeploy.json)
-
 ## Running on TPUs

 When using Tensorflow, TPUs are supported out of the box as a `tf.distribute.Strategy`.
@@ -478,7 +478,7 @@ def _compute_pytorch(
                            dictionary[model_name]["memory"][batch_size][slice_size] = "N/A"

                        if not no_speed:
-                            print_fn("Going through model with sequence of shape {}".format(sequence.shape))
+                            print_fn("Going through model with sequence of shape".format(sequence.shape))
                            runtimes = timeit.repeat(lambda: inference(sequence), repeat=average_over, number=3)
                            average_time = sum(runtimes) / float(len(runtimes)) / 3.0
                            dictionary[model_name]["time"][batch_size][slice_size] = average_time
@@ -80,7 +80,7 @@ class Distiller:

        self.mlm = params.mlm
        if self.mlm:
-            logger.info("Using MLM loss for LM step.")
+            logger.info(f"Using MLM loss for LM step.")
            self.mlm_mask_prop = params.mlm_mask_prop
            assert 0.0 <= self.mlm_mask_prop <= 1.0
            assert params.word_mask + params.word_keep + params.word_rand == 1.0
@@ -91,7 +91,7 @@ class Distiller:
                self.pred_probs = self.pred_probs.half()
                self.token_probs = self.token_probs.half()
        else:
-            logger.info("Using CLM loss for LM step.")
+            logger.info(f"Using CLM loss for LM step.")

        self.epoch = 0
        self.n_iter = 0
@@ -365,8 +365,8 @@ class Distiller:
            self.end_epoch()

        if self.is_master:
-            logger.info("Save very last checkpoint as `pytorch_model.bin`.")
-            self.save_checkpoint(checkpoint_name="pytorch_model.bin")
+            logger.info(f"Save very last checkpoint as `pytorch_model.bin`.")
+            self.save_checkpoint(checkpoint_name=f"pytorch_model.bin")
            logger.info("Training is finished")

    def step(self, input_ids: torch.tensor, attention_mask: torch.tensor, lm_labels: torch.tensor):
@@ -60,7 +60,7 @@ def main():
    with open(args.file_path, "r", encoding="utf8") as fp:
        data = fp.readlines()

-    logger.info("Start encoding")
+    logger.info(f"Start encoding")
    logger.info(f"{len(data)} examples to process.")

    rslt = []
@@ -93,7 +93,7 @@ if __name__ == "__main__":
    elif args.model_type == "gpt2":
        for w in ["weight", "bias"]:
            compressed_sd[f"{prefix}.ln_f.{w}"] = state_dict[f"{prefix}.ln_f.{w}"]
-        compressed_sd["lm_head.weight"] = state_dict["lm_head.weight"]
+        compressed_sd[f"lm_head.weight"] = state_dict[f"lm_head.weight"]

    print(f"N layers selected for distillation: {std_idx}")
    print(f"Number of params transfered for distillation: {len(compressed_sd.keys())}")
@@ -37,7 +37,7 @@ if __name__ == "__main__":
        model = BertForMaskedLM.from_pretrained(args.model_name)
        prefix = "bert"
    else:
-        raise ValueError('args.model_type should be "bert".')
+        raise ValueError(f'args.model_type should be "bert".')

    state_dict = model.state_dict()
    compressed_sd = {}
@@ -78,12 +78,12 @@ if __name__ == "__main__":
            ]
        std_idx += 1

-    compressed_sd["vocab_projector.weight"] = state_dict["cls.predictions.decoder.weight"]
-    compressed_sd["vocab_projector.bias"] = state_dict["cls.predictions.bias"]
+    compressed_sd[f"vocab_projector.weight"] = state_dict[f"cls.predictions.decoder.weight"]
+    compressed_sd[f"vocab_projector.bias"] = state_dict[f"cls.predictions.bias"]
    if args.vocab_transform:
        for w in ["weight", "bias"]:
-            compressed_sd[f"vocab_transform.{w}"] = state_dict["cls.predictions.transform.dense.{w}"]
-            compressed_sd[f"vocab_layer_norm.{w}"] = state_dict["cls.predictions.transform.LayerNorm.{w}"]
+            compressed_sd[f"vocab_transform.{w}"] = state_dict[f"cls.predictions.transform.dense.{w}"]
+            compressed_sd[f"vocab_layer_norm.{w}"] = state_dict[f"cls.predictions.transform.LayerNorm.{w}"]

    print(f"N layers selected for distillation: {std_idx}")
    print(f"Number of params transfered for distillation: {len(compressed_sd.keys())}")
@@ -273,7 +273,7 @@ def main():
        token_probs = None

    train_lm_seq_dataset = LmSeqsDataset(params=args, data=data)
-    logger.info("Data loader created.")
+    logger.info(f"Data loader created.")

    # STUDENT #
    logger.info(f"Loading student config from {args.student_config}")
@@ -288,7 +288,7 @@ def main():

    if args.n_gpu > 0:
        student.to(f"cuda:{args.local_rank}")
-    logger.info("Student loaded.")
+    logger.info(f"Student loaded.")

    # TEACHER #
    teacher = teacher_model_class.from_pretrained(args.teacher_name, output_hidden_states=True)
@@ -115,12 +115,16 @@ class DataTrainingArguments:
    )


-def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False):
+def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False, local_rank=-1):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    if args.line_by_line:
-        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
+        return LineByLineTextDataset(
+            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, local_rank=local_rank
+        )
    else:
-        return TextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
+        return TextDataset(
+            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, local_rank=local_rank,
+        )


 def main():
@@ -216,8 +220,16 @@ def main():
        data_args.block_size = min(data_args.block_size, tokenizer.max_len)

    # Get datasets
-    train_dataset = get_dataset(data_args, tokenizer=tokenizer) if training_args.do_train else None
-    eval_dataset = get_dataset(data_args, tokenizer=tokenizer, evaluate=True) if training_args.do_eval else None
+    train_dataset = (
+        get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank)
+        if training_args.do_train
+        else None
+    )
+    eval_dataset = (
+        get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank, evaluate=True)
+        if training_args.do_eval
+        else None
+    )
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=data_args.mlm, mlm_probability=data_args.mlm_probability
    )
@@ -159,6 +159,7 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.train,
+            local_rank=training_args.local_rank,
        )
        if training_args.do_train
        else None
@@ -171,6 +172,7 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.dev,
+            local_rank=training_args.local_rank,
        )
        if training_args.do_eval
        else None
@@ -26,7 +26,6 @@ from enum import Enum
 from typing import List, Optional

 import tqdm
-from filelock import FileLock

 from transformers import PreTrainedTokenizer, is_tf_available, is_torch_available

@@ -78,6 +77,7 @@ class Split(Enum):
 if is_torch_available():
    import torch
    from torch.utils.data.dataset import Dataset
+    from transformers import torch_distributed_zero_first

    class MultipleChoiceDataset(Dataset):
        """
@@ -95,6 +95,7 @@ if is_torch_available():
            max_seq_length: Optional[int] = None,
            overwrite_cache=False,
            mode: Split = Split.train,
+            local_rank=-1,
        ):
            processor = processors[task]()

@@ -102,11 +103,9 @@ if is_torch_available():
                data_dir,
                "cached_{}_{}_{}_{}".format(mode.value, tokenizer.__class__.__name__, str(max_seq_length), task,),
            )
-
-            # Make sure only the first process in distributed training processes the dataset,
-            # and the others will use the cache.
-            lock_path = cached_features_file + ".lock"
-            with FileLock(lock_path):
+            with torch_distributed_zero_first(local_rank):
+                # Make sure only the first process in distributed training processes the dataset,
+                # and the others will use the cache.

                if os.path.exists(cached_features_file) and not overwrite_cache:
                    logger.info(f"Loading features from cached file {cached_features_file}")
@@ -131,8 +130,9 @@ if is_torch_available():
                        pad_token=tokenizer.pad_token_id,
                        pad_token_segment_id=tokenizer.pad_token_type_id,
                    )
-                    logger.info("Saving features into cached file %s", cached_features_file)
-                    torch.save(self.features, cached_features_file)
+                    if local_rank in [-1, 0]:
+                        logger.info("Saving features into cached file %s", cached_features_file)
+                        torch.save(self.features, cached_features_file)

        def __len__(self):
            return len(self.features)
@@ -535,12 +535,7 @@ def convert_examples_to_features(
                text_b = example.question + " " + ending

            inputs = tokenizer.encode_plus(
-                text_a,
-                text_b,
-                add_special_tokens=True,
-                max_length=max_length,
-                pad_to_max_length=True,
-                return_overflowing_tokens=True,
+                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
@@ -171,6 +171,7 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.train,
+            local_rank=training_args.local_rank,
        )
        if training_args.do_train
        else None
@@ -184,6 +185,7 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.dev,
+            local_rank=training_args.local_rank,
        )
        if training_args.do_eval
        else None
@@ -259,6 +261,7 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.test,
+            local_rank=training_args.local_rank,
        )

        predictions, label_ids, metrics = trainer.predict(test_dataset)
@@ -22,8 +22,6 @@ from dataclasses import dataclass
 from enum import Enum
 from typing import List, Optional, Union

-from filelock import FileLock
-
 from transformers import PreTrainedTokenizer, is_tf_available, is_torch_available


@@ -70,6 +68,7 @@ if is_torch_available():
    import torch
    from torch import nn
    from torch.utils.data.dataset import Dataset
+    from transformers import torch_distributed_zero_first

    class NerDataset(Dataset):
        """
@@ -91,16 +90,16 @@ if is_torch_available():
            max_seq_length: Optional[int] = None,
            overwrite_cache=False,
            mode: Split = Split.train,
+            local_rank=-1,
        ):
            # Load data features from cache or dataset file
            cached_features_file = os.path.join(
                data_dir, "cached_{}_{}_{}".format(mode.value, tokenizer.__class__.__name__, str(max_seq_length)),
            )

-            # Make sure only the first process in distributed training processes the dataset,
-            # and the others will use the cache.
-            lock_path = cached_features_file + ".lock"
-            with FileLock(lock_path):
+            with torch_distributed_zero_first(local_rank):
+                # Make sure only the first process in distributed training processes the dataset,
+                # and the others will use the cache.

                if os.path.exists(cached_features_file) and not overwrite_cache:
                    logger.info(f"Loading features from cached file {cached_features_file}")
@@ -126,8 +125,9 @@ if is_torch_available():
                        pad_token_segment_id=tokenizer.pad_token_type_id,
                        pad_token_label_id=self.pad_token_label_id,
                    )
-                    logger.info(f"Saving features into cached file {cached_features_file}")
-                    torch.save(self.features, cached_features_file)
+                    if local_rank in [-1, 0]:
+                        logger.info(f"Saving features into cached file {cached_features_file}")
+                        torch.save(self.features, cached_features_file)

        def __len__(self):
            return len(self.features)
@@ -1,67 +0,0 @@
---
-language: turkish
---
-# Turkish SQuAD  Model : Question Answering
-
-I fine-tuned Turkish-Bert-Model for Question-Answering problem with Turkish version of SQuAD; TQuAD 
-* BERT-base: https://huggingface.co/dbmdz/bert-base-turkish-uncased
-* TQuAD dataset:  https://github.com/TQuad/turkish-nlp-qa-dataset
-
-
-# Training Code
-
-```
-!python3 run_squad.py \
-  --model_type bert \
-  --model_name_or_path dbmdz/bert-base-turkish-uncased\
-  --do_train \
-  --do_eval \
-  --train_file trainQ.json \
-  --predict_file dev1.json \
-  --per_gpu_train_batch_size 12 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 5.0 \
-  --max_seq_length 384 \
-  --doc_stride 128 \
-  --output_dir "./model"
-```
-
-
-# Example Usage
-
-> Load Model
-```
-from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
-import torch
-
-tokenizer = AutoTokenizer.from_pretrained("./model")
-model = AutoModelForQuestionAnswering.from_pretrained("./model")
-nlp=pipeline("question-answering", model=model, tokenizer=tokenizer)
-```
-
-> Apply the model
-```
-
-sait="ABASIYANIK, Sait Faik. Hikayeci (Adapazarı 23 Kasım 1906-İstanbul 11 Mayıs 1954). \
-İlk öğrenimine Adapazarı’nda Rehber-i Terakki Mektebi’nde başladı. İki yıl kadar Adapazarı İdadisi’nde okudu.\
-İstanbul Erkek Lisesi’nde devam ettiği orta öğrenimini Bursa Lisesi’nde tamamladı (1928). İstanbul Edebiyat \
-Fakültesi’ne iki yıl devam ettikten sonra babasının isteği üzerine iktisat öğrenimi için İsviçre’ye gitti. \
-Kısa süre sonra iktisat öğrenimini bırakarak Lozan’dan Grenoble’a geçti. Üç yıl başıboş bir edebiyat öğrenimi \
-gördükten sonra babası tarafından geri çağrıldı (1933). Bir müddet Halıcıoğlu Ermeni Yetim Mektebi'nde Türkçe \
-gurup dersleri öğretmenliği yaptı. Ticarete atıldıysa da tutunamadı. Bir ay Haber gazetesinde adliye muhabirliği\
-yaptı (1942). Babasının ölümü üzerine aileden kalan emlakin geliri ile avare bir hayata başladı. Evlenemedi.\
-Yazları Burgaz adasındaki köşklerinde, kışları Şişli’deki apartmanlarında annesi ile beraber geçen bu fazla \
-içkili bohem hayatı ömrünün sonuna kadar sürdü."
-
-print(nlp(question="Ne zaman avare bir hayata başladı?", context=sait))
-print(nlp(question="Sait Faik hangi Lisede orta öğrenimini tamamladı?", context=sait))
-
-```
-```
-# Ask your self ! type your question
-print(nlp(question="...?", context=sait))
-```
-
-
-Check My other Model
-https://huggingface.co/savasy
@@ -1,39 +0,0 @@
---
-tags:
- pytorch
- exbert
-license: apache-2.0
---
-# ouBioBERT-Base, Uncased
-Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) is a language model based on the BERT-Base (Devlin, et al., 2019) architecture. We pre-trained ouBioBERT on PubMed abstracts from the PubMed baseline (ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline) via our method.  
-
-The details of the pre-training procedure can be found in Wada, et al. (2020).  
-
-## Evaluation
-We evaluated the performance of ouBioBERT in terms of the biomedical language understanding evaluation (BLUE) benchmark (Peng, et al., 2019). The numbers are mean (standard deviation) on five different random seeds.  
-| Dataset         |  Task Type                |  Score       |
-|:----------------|:--------------------------|-------------:|
-| MedSTS          |  Sentence similarity      |  84.9 (0.6)  |
-| BIOSSES         |  Sentence similarity      |  92.3 (0.8)  |
-| BC5CDR-disease  |  Named-entity recognition |  87.4 (0.1)  |
-| BC5CDR-chemical |  Named-entity recognition |  93.7 (0.2)  |
-| ShARe/CLEFE     |  Named-entity recognition |  80.1 (0.4)  |
-| DDI             |  Relation extraction      |  81.1 (1.5)  |
-| ChemProt        |  Relation extraction      |  75.0 (0.3)  |
-| i2b2 2010       |  Relation extraction      |  74.0 (0.8)  |
-| HoC             |  Document classification  |  86.4 (0.5)  |
-| MedNLI          |  Inference                |  83.6 (0.7)  |
-| **Total**       |  -                        |**83.8 (0.3)**|
-
-## Code for Fine-tuning
-We made the source code for fine-tuning freely available at [our repository](https://github.com/sy-wada/blue_benchmark_with_transformers).
-
-## Citation
-If you use our work in your research, please kindly cite the following paper:  
-```bibtex
-now preparing...
-```
-
-<a href="https://huggingface.co/exbert/?model=seiya/oubiobert-base-uncased&sentence=Coronavirus%20disease%20(COVID-19)%20is%20caused%20by%20SARS-COV2%20and%20represents%20the%20causative%20agent%20of%20a%20potentially%20fatal%20disease%20that%20is%20of%20great%20global%20public%20health%20concern.">
-	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
-</a>
@@ -3,6 +3,7 @@
  {
   "cell_type": "markdown",
   "metadata": {
+    "collapsed": true,
    "pycharm": {
     "is_executing": false,
     "name": "#%% md\n"
@@ -76,7 +77,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": false,
@@ -84,7 +85,77 @@
    },
    "scrolled": true
   },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Requirement already satisfied: transformers in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (2.5.1)\n",
+      "Requirement already satisfied: filelock in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (3.0.12)\n",
+      "Requirement already satisfied: sentencepiece in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.1.83)\n",
+      "Requirement already satisfied: boto3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (1.12.0)\n",
+      "Requirement already satisfied: requests in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (2.22.0)\n",
+      "Requirement already satisfied: numpy in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (1.18.1)\n",
+      "Requirement already satisfied: sacremoses in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.0.35)\n",
+      "Requirement already satisfied: tokenizers==0.5.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.5.2)\n",
+      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (2020.1.8)\n",
+      "Requirement already satisfied: tqdm>=4.27 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (4.42.1)\n",
+      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (0.3.3)\n",
+      "Requirement already satisfied: botocore<1.16.0,>=1.15.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (1.15.0)\n",
+      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (0.9.4)\n",
+      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (2019.11.28)\n",
+      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (2.8)\n",
+      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (1.25.8)\n",
+      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (3.0.4)\n",
+      "Requirement already satisfied: joblib in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (0.14.0)\n",
+      "Requirement already satisfied: click in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (7.0)\n",
+      "Requirement already satisfied: six in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (1.14.0)\n",
+      "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.0->boto3->transformers) (0.15.2)\n",
+      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.0->boto3->transformers) (2.8.1)\n",
+      "Requirement already satisfied: tensorflow==2.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (2.1.0)\n",
+      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.1.0)\n",
+      "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.1.0)\n",
+      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (3.1.0)\n",
+      "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (3.11.4)\n",
+      "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.18.1)\n",
+      "Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (2.1.0)\n",
+      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.0.8)\n",
+      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.11.2)\n",
+      "Requirement already satisfied: six>=1.12.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.14.0)\n",
+      "Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (2.1.0)\n",
+      "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.4.1)\n",
+      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.1.8)\n",
+      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.34.2)\n",
+      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.16.1)\n",
+      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.9.0)\n",
+      "Requirement already satisfied: gast==0.2.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.2.2)\n",
+      "Requirement already satisfied: astor>=0.6.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.8.0)\n",
+      "Requirement already satisfied: setuptools in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from protobuf>=3.8.0->tensorflow==2.1.0) (45.2.0.post20200210)\n",
+      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.11.2)\n",
+      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.1)\n",
+      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.1)\n",
+      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.0)\n",
+      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.22.0)\n",
+      "Requirement already satisfied: h5py in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (2.10.0)\n",
+      "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.0)\n",
+      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.0.0)\n",
+      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)\n",
+      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.0)\n",
+      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.8)\n",
+      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2019.11.28)\n",
+      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.25.8)\r\n",
+      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)\r\n",
+      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\r\n"
+     ]
+    }
+   ],
   "source": [
    "!pip install transformers\n",
    "!pip install tensorflow==2.1.0"
@@ -103,7 +174,7 @@
    {
     "data": {
      "text/plain": [
-       "<torch.autograd.grad_mode.set_grad_enabled at 0x7f10b441e890>"
+       "<torch.autograd.grad_mode.set_grad_enabled at 0x102c0ce10>"
      ]
     },
     "execution_count": 2,
@@ -370,7 +441,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
@@ -387,22 +458,13 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "output differences: 1.6236e-05\n",
-      "pooled differences: -1.3039e-08\n"
-     ]
-    }
-   ],
+   "outputs": [],
   "source": [
    "# transformers generates a ready to use dictionary with all the required parameters for the specific framework.\n",
    "input_tf = tokenizer.encode_plus(\"This is a sample input\", return_tensors=\"tf\")\n",
@@ -414,7 +476,7 @@
    "# Models outputs 2 values (The value for each tokens, the pooled representation of the input sentence)\n",
    "# Here we compare the output differences between PyTorch and TensorFlow.\n",
    "for name, o_tf, o_pt in zip([\"output\", \"pooled\"], output_tf, output_pt):\n",
-    "    print(\"{} differences: {:.5}\".format(name, (o_tf.numpy() - o_pt.numpy()).sum()))"
+    "    print(\"{} differences: {}\".format(name, (o_tf.numpy() - o_pt.numpy()).sum()))"
   ]
  },
  {
@@ -442,24 +504,13 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "CPU times: user 232 ms, sys: 0 ns, total: 232 ms\n",
-      "Wall time: 21.1 ms\n",
-      "CPU times: user 511 ms, sys: 0 ns, total: 511 ms\n",
-      "Wall time: 43.9 ms\n"
-     ]
-    }
-   ],
+   "outputs": [],
   "source": [
    "from transformers import DistilBertModel\n",
    "\n",
@@ -490,25 +541,13 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Tokens (int)      : [102, 12272, 9355, 5746, 30881, 215, 261, 5945, 4118, 212, 2414, 153, 1942, 232, 3532, 566, 103]\n",
-      "Tokens (str)      : ['[CLS]', 'Hug', '##ging', 'Fac', '##e', 'ist', 'eine', 'französische', 'Firma', 'mit', 'Sitz', 'in', 'New', '-', 'York', '.', '[SEP]']\n",
-      "Tokens (attn_mask): [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
-      "\n",
-      "Token wise output: torch.Size([1, 7, 768]), Pooled output: torch.Size([1, 768])\n"
-     ]
-    }
-   ],
+   "outputs": [],
   "source": [
    "# Let's load German BERT from the Bavarian State Library\n",
    "de_bert = BertModel.from_pretrained(\"dbmdz/bert-base-german-cased\")\n",
@@ -518,14 +557,7 @@
    "    \"Hugging Face ist eine französische Firma mit Sitz in New-York.\",\n",
    "    return_tensors=\"pt\"\n",
    ")\n",
-    "print(\"Tokens (int)      : {}\".format(de_input['input_ids'].tolist()[0]))\n",
-    "print(\"Tokens (str)      : {}\".format([de_tokenizer.convert_ids_to_tokens(s) for s in de_input['input_ids'].tolist()[0]]))\n",
-    "print(\"Tokens (attn_mask): {}\".format(de_input['attention_mask'].tolist()[0]))\n",
-    "print()\n",
-    "\n",
-    "output_de, pooled_de = de_bert(**de_input)\n",
-    "\n",
-    "print(\"Token wise output: {}, Pooled output: {}\".format(outputs.shape, pooled.shape))"
+    "output_de, pooled_de = de_bert(**de_input)"
   ]
  }
 ],
@@ -545,7 +577,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
@@ -558,5 +590,5 @@
  }
 },
 "nbformat": 4,
- "nbformat_minor": 4
+ "nbformat_minor": 1
 }
@@ -10,10 +10,9 @@ Pull Request and we'll review it so it can be included here.
 ## Hugging Face's notebooks :hugs:

 | Notebook     |      Description      |   |
-|:----------|:-------------|------:|
+|:----------|:-------------:|------:|
 | [Getting Started Tokenizers](https://github.com/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb)  | How to train and use your very own tokenizer  |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb) |
 | [Getting Started Transformers](https://github.com/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb)   | How to easily start using transformers  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb) |
 | [How to use Pipelines](https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb)  | Simple and efficient way to use State-of-the-Art models on downstream tasks through transformers | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb) |
 | [How to train a language model](https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)| Highlight all the steps to effectively train Transformer model on custom data | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)|
 | [How to generate text](https://github.com/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)| How to use different decoding methods for language generation with transformers | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)|
-| [How to export model to ONNX](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) | Highlight how to export and run inference workloads through ONNX |
@@ -36,5 +36,5 @@ multi_line_output = 3
 use_parentheses = True

 [flake8]
-ignore = E203, E501, E741, W503
+ignore = E203, E501, W503
 max-line-length = 119
@@ -67,18 +67,8 @@ extras = {}

 extras["mecab"] = ["mecab-python3"]
 extras["sklearn"] = ["scikit-learn"]
-
-# keras2onnx and onnxconverter-common version is specific through a commit until 1.7.0 lands on pypi
-extras["tf"] = [
-    "tensorflow",
-    "onnxconverter-common @ git+git://github.com/microsoft/onnxconverter-common.git@f64ca15989b6dc95a1f3507ff6e4c395ba12dff5#egg=onnxconverter-common",
-    "keras2onnx @ git+git://github.com/onnx/keras-onnx.git@cbdc75cb950b16db7f0a67be96a278f8d2953b48#egg=keras2onnx"
-]
-extras["tf-cpu"] = [
-    "tensorflow-cpu",
-    "onnxconverter-common @ git+git://github.com/microsoft/onnxconverter-common.git@f64ca15989b6dc95a1f3507ff6e4c395ba12dff5#egg=onnxconverter-common",
-    "keras2onnx @ git+git://github.com/onnx/keras-onnx.git@cbdc75cb950b16db7f0a67be96a278f8d2953b48#egg=keras2onnx"
-]
+extras["tf"] = ["tensorflow"]
+extras["tf-cpu"] = ["tensorflow-cpu"]
 extras["torch"] = ["torch"]

 extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
@@ -89,7 +79,7 @@ extras["docs"] = ["recommonmark", "sphinx", "sphinx-markdown-tables", "sphinx-rt
 extras["quality"] = [
    "black",
    "isort @ git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort",
-    "flake8",
+    "flake8==3.7.9",
 ]
 extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow", "torch"]

@@ -108,7 +98,7 @@ setup(
    packages=find_packages("src"),
    install_requires=[
        "numpy",
-        "tokenizers == 0.7.0",
+        "tokenizers == 0.8.0.dev1",
        # dataclasses for Python versions that don't have it
        "dataclasses;python_version<'3.7'",
        # filesystem locks e.g. to prevent parallel downloads
@@ -26,8 +26,7 @@ def gelu_new(x):
    """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
        Also see https://arxiv.org/abs/1606.08415
    """
-    # math.sqrt(2.0 / math.pi) = 0.7978845608
-    return 0.5 * x * (1.0 + torch.tanh(0.7978845608 * (x + 0.044715 * torch.pow(x, 3.0))))
+    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))


 if torch.__version__ < "1.4.0":
@@ -54,4 +53,4 @@ def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:
-        raise KeyError("function {} not found in ACT2FN mapping {}".format(activation_string, list(ACT2FN.keys())))
+        raise KeyError("function {} not found in ACT2FN mapping {}".format(activation_string, list(ACT2FN.keys())))
@@ -68,6 +68,6 @@ class RobertaConfig(BertConfig):
    model_type = "roberta"

    def __init__(self, pad_token_id=1, bos_token_id=0, eos_token_id=2, **kwargs):
-        """Constructs RobertaConfig.
+        """Constructs FlaubertConfig.
        """
        super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
@@ -1,212 +0,0 @@
-from argparse import ArgumentParser
-from itertools import takewhile
-from os import listdir, makedirs
-from os.path import abspath, dirname, exists
-from typing import Dict, List, Optional, Tuple
-
-from transformers import is_tf_available, is_torch_available
-from transformers.pipelines import Pipeline, pipeline
-from transformers.tokenization_utils import BatchEncoding
-
-
-class OnnxConverterArgumentParser(ArgumentParser):
-    """
-    Wraps all the script arguments supported to export transformers models to ONNX IR
-    """
-
-    def __init__(self):
-        super(OnnxConverterArgumentParser, self).__init__("ONNX Converter")
-
-        self.add_argument("--model", type=str, required=True, help="Model's id or path (ex: bert-base-cased)")
-        self.add_argument("--tokenizer", type=str, help="Tokenizer's id or path (ex: bert-base-cased)")
-        self.add_argument("--framework", type=str, choices=["pt", "tf"], help="Framework for loading the model")
-        self.add_argument("--opset", type=int, default=11, help="ONNX opset to use")
-        self.add_argument("--check-loading", action="store_true", help="Check ONNX is able to load the model")
-        self.add_argument("output")
-
-
-def ensure_valid_input(model, tokens, input_names):
-    """
-    Ensure input are presented in the correct order, without any None
-    Args:
-        model: The model used to forward the input data
-        tokens: BatchEncoding holding the input data
-        input_names: The name of the inputs
-
-    Returns: Tuple
-
-    """
-    model_args_name = model.forward.__code__.co_varnames
-    model_args_pos = [(model_args_name.index(name) - 1, name) for name in input_names]
-    model_args = [None] * (max(map(lambda x: x[0], model_args_pos)) + 1)
-
-    for arg_pos, arg_name in model_args_pos:
-        model_args[arg_pos] = tokens[arg_name]
-
-    model_args = tuple(model_args)  # Need to be ordered
-    return tuple(takewhile(lambda arg: arg is not None, model_args))
-
-
-def infer_shapes(nlp: Pipeline, framework: str) -> Tuple[List[str], List[str], Dict, BatchEncoding]:
-    def build_shape_dict(tensor, is_input: bool, seq_len: int):
-        if isinstance(tensor, (tuple, list)):
-            return [build_shape_dict(t, is_input, seq_len) for t in tensor]
-
-        else:
-            # Let's assume batch is the first axis with only 1 element (~~ might not be always true ...)
-            axes = {[axis for axis, numel in enumerate(tensor.shape) if numel == 1][0]: "batch"}
-            if is_input:
-                if len(tensor.shape) == 2:
-                    axes[1] = "sequence"
-                else:
-                    raise ValueError("Unable to infer tensor axes ({})".format(len(tensor.shape)))
-            else:
-                seq_axes = [dim for dim, shape in enumerate(tensor.shape) if shape == seq_len]
-                axes.update({dim: "sequence" for dim in seq_axes})
-
-        return axes
-
-    tokens = nlp.tokenizer.encode_plus("This is a sample output", return_tensors=framework)
-    seq_len = tokens.input_ids.shape[-1]
-    outputs = nlp.model(**tokens) if framework == "pt" else nlp.model(tokens)
-
-    if not isinstance(outputs, (list, tuple)):
-        outputs = (outputs,)
-
-    # Generate input names & axes
-    input_vars = list(tokens.keys())
-    input_dynamic_axes = {k: build_shape_dict(v, True, seq_len) for k, v in tokens.items()}
-
-    # flatten potentially grouped outputs (past for gpt2, attentions)
-    outputs_flat = []
-    for output in outputs:
-        if isinstance(output, (tuple, list)):
-            outputs_flat.extend(output)
-        else:
-            outputs_flat.append(output)
-
-    # Generate output names & axes
-    output_names = ["output_{}".format(i) for i in range(len(outputs_flat))]
-    output_dynamic_axes = {k: build_shape_dict(v, False, seq_len) for k, v in zip(output_names, outputs_flat)}
-
-    # Create the aggregated axes representation
-    dynamic_axes = dict(input_dynamic_axes, **output_dynamic_axes)
-    return input_vars, output_names, dynamic_axes, tokens
-
-
-def load_graph_from_args(framework: str, model: str, tokenizer: Optional[str] = None) -> Pipeline:
-    # If no tokenizer provided
-    if tokenizer is None:
-        tokenizer = model
-
-    print("Loading pipeline (model: {}, tokenizer: {})".format(model, tokenizer))
-
-    # Allocate tokenizer and model
-    return pipeline("feature-extraction", model=model, framework=framework)
-
-
-def convert_pytorch(nlp: Pipeline, opset: int, output: str):
-    if not is_torch_available():
-        raise Exception("Cannot convert because PyTorch is not installed. Please install torch first.")
-
-    import torch
-    from torch.onnx import export
-
-    print("PyTorch: {}".format(torch.__version__))
-
-    with torch.no_grad():
-        input_names, output_names, dynamic_axes, tokens = infer_shapes(nlp, "pt")
-        model_args = ensure_valid_input(nlp.model, tokens, input_names)
-
-        export(
-            nlp.model,
-            model_args,
-            f=output,
-            input_names=input_names,
-            output_names=output_names,
-            dynamic_axes=dynamic_axes,
-            do_constant_folding=True,
-            use_external_data_format=True,
-            enable_onnx_checker=True,
-            opset_version=opset,
-        )
-
-
-def convert_tensorflow(nlp: Pipeline, opset: int, output: str):
-    if not is_tf_available():
-        raise Exception(
-            "Cannot convert {} because TF is not installed. Please install torch first.".format(args.model)
-        )
-
-    print("/!\\ Please note TensorFlow doesn't support exporting model > 2Gb /!\\")
-
-    try:
-        import tensorflow as tf
-        from keras2onnx import convert_keras, save_model, __version__ as k2ov
-
-        print("TensorFlow: {}, keras2onnx: {}".format(tf.version.VERSION, k2ov))
-
-        # Build
-        input_names, output_names, dynamic_axes, tokens = infer_shapes(nlp, "tf")
-
-        # Forward
-        nlp.model.predict(tokens.data)
-        onnx_model = convert_keras(nlp.model, nlp.model.name, target_opset=opset)
-        save_model(onnx_model, output)
-
-    except ImportError as e:
-        raise Exception(
-            "Cannot import {} required to convert TF model to ONNX. Please install {} first.".format(e.name, e.name)
-        )
-
-
-def convert(framework: str, model: str, output: str, opset: int, tokenizer: Optional[str] = None):
-    print("ONNX opset version set to: {}".format(opset))
-
-    # Load the pipeline
-    nlp = load_graph_from_args(framework, model, tokenizer)
-
-    parent = dirname(output)
-    if not exists(parent):
-        print("Creating folder {}".format(parent))
-        makedirs(parent)
-    elif len(listdir(parent)) > 0:
-        raise Exception("Folder {} is not empty, aborting conversion".format(parent))
-
-    # Export the graph
-    if framework == "pt":
-        convert_pytorch(nlp, opset, output)
-    else:
-        convert_tensorflow(nlp, opset, output)
-
-
-def verify(path: str):
-    from onnxruntime import InferenceSession, SessionOptions
-    from onnxruntime.capi.onnxruntime_pybind11_state import RuntimeException
-
-    print("Checking ONNX model loading from: {}".format(path))
-    try:
-        onnx_options = SessionOptions()
-        _ = InferenceSession(path, onnx_options, providers=["CPUExecutionProvider"])
-        print("Model correctly loaded")
-    except RuntimeException as re:
-        print("Error while loading the model: {}".format(re))
-
-
-if __name__ == "__main__":
-    parser = OnnxConverterArgumentParser()
-    args = parser.parse_args()
-
-    # Make sure output is absolute path
-    args.output = abspath(args.output)
-
-    try:
-        # Convert
-        convert(args.framework, args.model, args.output, args.opset, args.tokenizer)
-
-        # And verify
-        if args.check_loading:
-            verify(args.output)
-    except Exception as e:
-        print("Error while converting the model: {}".format(e))
-        exit(1)
@@ -226,7 +226,7 @@ def lmap(f, x) -> List:
 def fetch_test_set(test_set_url):
    import wget

-    fname = wget.download(test_set_url, "opus_test.txt")
+    fname = wget.download(test_set_url, f"opus_test.txt")
    lns = Path(fname).open().readlines()
    src = lmap(str.strip, lns[::4])
    gold = lmap(str.strip, lns[1::4])
@@ -114,7 +114,7 @@ class GlueDataset(Dataset):
                torch.save(self.features, cached_features_file)
                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
                logger.info(
-                    "Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
+                    f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
                )

    def __len__(self):
@@ -4,10 +4,10 @@ import pickle
 import time

 import torch
-from filelock import FileLock
 from torch.utils.data.dataset import Dataset

 from ...tokenization_utils import PreTrainedTokenizer
+from ...trainer import torch_distributed_zero_first


 logger = logging.getLogger(__name__)
@@ -20,7 +20,7 @@ class TextDataset(Dataset):
    """

    def __init__(
-        self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, overwrite_cache=False,
+        self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, overwrite_cache=False, local_rank=-1,
    ):
        assert os.path.isfile(file_path)

@@ -31,10 +31,9 @@ class TextDataset(Dataset):
            directory, "cached_lm_{}_{}_{}".format(tokenizer.__class__.__name__, str(block_size), filename,),
        )

-        # Make sure only the first process in distributed training processes the dataset,
-        # and the others will use the cache.
-        lock_path = cached_features_file + ".lock"
-        with FileLock(lock_path):
+        with torch_distributed_zero_first(local_rank):
+            # Make sure only the first process in distributed training processes the dataset,
+            # and the others will use the cache.

            if os.path.exists(cached_features_file) and not overwrite_cache:
                start = time.time()
@@ -65,7 +64,7 @@ class TextDataset(Dataset):
                with open(cached_features_file, "wb") as handle:
                    pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
                logger.info(
-                    "Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
+                    f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
                )

    def __len__(self):
@@ -81,7 +80,7 @@ class LineByLineTextDataset(Dataset):
    soon.
    """

-    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
+    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, local_rank=-1):
        assert os.path.isfile(file_path)
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
@@ -195,22 +195,18 @@ def squad_convert_example_to_features(example, max_seq_length, doc_stride, max_q
        cls_index = span["input_ids"].index(tokenizer.cls_token_id)

        # p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
-        # Original TF implem also keep the classification token (set to 0)
-        p_mask = np.ones_like(span["token_type_ids"])
+        # Original TF implem also keep the classification token (set to 0) (not sure why...)
+        p_mask = np.array(span["token_type_ids"])
+
+        p_mask = np.minimum(p_mask, 1)
+
        if tokenizer.padding_side == "right":
-            p_mask[len(truncated_query) + sequence_added_tokens :] = 0
-        else:
-            p_mask[-len(span["tokens"]) : -(len(truncated_query) + sequence_added_tokens)] = 0
+            # Limit positive values to one
+            p_mask = 1 - p_mask

-        pad_token_indices = np.where(span["input_ids"] == tokenizer.pad_token_id)
-        special_token_indices = np.asarray(
-            tokenizer.get_special_tokens_mask(span["input_ids"], already_has_special_tokens=True)
-        ).nonzero()
+        p_mask[np.where(np.array(span["input_ids"]) == tokenizer.sep_token_id)[0]] = 1

-        p_mask[pad_token_indices] = 1
-        p_mask[special_token_indices] = 1
-
-        # Set the cls index to 0: the CLS index can be used for impossible answers
+        # Set the CLS index to '0'
        p_mask[cls_index] = 0

        span_is_impossible = example.is_impossible
@@ -104,7 +104,7 @@ class Attention(nn.Module):
        # [switch nx => n_state from Block to Attention to keep identical to TF implem]
        assert n_state % config.n_head == 0
        self.register_buffer(
-            "bias", torch.tril(torch.ones((n_ctx, n_ctx), dtype=torch.bool)).view(1, 1, n_ctx, n_ctx)
+            "bias", torch.tril(torch.ones((n_ctx, n_ctx), dtype=torch.uint8)).view(1, 1, n_ctx, n_ctx)
        )
        self.register_buffer("masked_bias", torch.tensor(-1e4))
        self.n_head = config.n_head
@@ -143,7 +143,6 @@ class Attention(nn.Module):
        w = torch.matmul(q, k)
        if self.scale:
            w = w / (float(v.size(-1)) ** 0.5)
-
        nd, ns = w.size(-2), w.size(-1)
        mask = self.bias[:, :, ns - nd : ns, :ns]
        w = torch.where(mask.bool(), w, self.masked_bias.to(w.dtype))
@@ -152,22 +151,22 @@ class Attention(nn.Module):
            # Apply the attention mask
            w = w + attention_mask

-        w = torch.softmax(w, dim=-1)
+        w = nn.Softmax(dim=-1)(w)
        w = self.attn_dropout(w)

        # Mask heads if we want to
        if head_mask is not None:
            w = w * head_mask

-        wv = torch.matmul(w, v)
+        outputs = [torch.matmul(w, v)]
        if self.output_attentions:
-            return wv, w
-        else:
-            return (wv, )
+            outputs.append(w)
+        return outputs

    def merge_heads(self, x):
-        x = x.transpose(2, 1).contiguous()
-        return x.flatten(2)  # in Tensorflow implem: fct merge_states
+        x = x.permute(0, 2, 1, 3).contiguous()
+        new_x_shape = x.size()[:-2] + (x.size(-2) * x.size(-1),)
+        return x.view(*new_x_shape)  # in Tensorflow implem: fct merge_states

    def split_heads(self, x, k=False):
        new_x_shape = x.size()[:-1] + (self.n_head, x.size(-1) // self.n_head)
@@ -175,11 +174,11 @@ class Attention(nn.Module):
        if k:
            return x.permute(0, 2, 3, 1)  # (batch, head, head_features, seq_length)
        else:
-            return x.transpose(2, 1)  # (batch, head, seq_length, head_features)
+            return x.permute(0, 2, 1, 3)  # (batch, head, seq_length, head_features)

    def forward(self, x, layer_past=None, attention_mask=None, head_mask=None, use_cache=False):
        x = self.c_attn(x)
-        query, key, value = x.chunk(3, dim=2)
+        query, key, value = x.split(self.split_size, dim=2)
        query = self.split_heads(query)
        key = self.split_heads(key, k=True)
        value = self.split_heads(value)
@@ -200,7 +199,8 @@ class Attention(nn.Module):
        a = self.c_proj(a)
        a = self.resid_dropout(a)

-        return a, present, attn_outputs[1:]
+        outputs = [a, present] + attn_outputs[1:]
+        return outputs  # a, present, (attentions)


 class MLP(nn.Module):
@@ -241,7 +241,8 @@ class Block(nn.Module):
        m = self.mlp(self.ln_2(x))
        x = x + m

-        return x, output_attn[1:]
+        outputs = [x] + output_attn[1:]
+        return outputs  # x, present, (attentions)


 class GPT2PreTrainedModel(PreTrainedModel):
@@ -424,17 +425,17 @@ class GPT2Model(GPT2PreTrainedModel):
        elif input_ids is not None:
            input_shape = input_ids.size()
            input_ids = input_ids.view(-1, input_shape[-1])
-            batch_size = input_ids.size(0)
+            batch_size = input_ids.shape[0]
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.size()[:-1]
-            batch_size = inputs_embeds.size(0)
+            batch_size = inputs_embeds.shape[0]
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds")

        if token_type_ids is not None:
-            token_type_ids = token_type_ids.view_as(input_ids)
+            token_type_ids = token_type_ids.view(-1, input_shape[-1])
        if position_ids is not None:
-            position_ids = position_ids.view_as(input_ids)
+            position_ids = position_ids.view(-1, input_shape[-1])

        if past is None:
            past_length = 0
@@ -449,25 +450,22 @@ class GPT2Model(GPT2PreTrainedModel):
        # Attention mask.
        if attention_mask is not None:
            assert batch_size > 0, "batch_size has to be defined and > 0"
+            attention_mask = attention_mask.view(batch_size, -1)
+            # We create a 3D attention mask from a 2D tensor mask.
+            # Sizes are [batch_size, 1, 1, to_seq_length]
+            # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
+            # this attention mask is more simple than the triangular masking of causal attention
+            # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
+            attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)

            # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
            # masked positions, this operation will create a tensor which is 0.0 for
            # positions we want to attend and -10000.0 for masked positions.
            # Since we are adding it to the raw scores before the softmax, this is
            # effectively the same as removing these entirely.
-
-            # Computation is done over contiguous tensor (no view / unsqueeze) so probably faster
-            # Do the shaping ops afterwards
-            attention_mask = attention_mask.to(self.dtype)  # fp16 compatibility
+            attention_mask = attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
            attention_mask = (1.0 - attention_mask) * -10000.0

-            # We create a 3D attention mask from a 2D tensor mask.
-            # Sizes are [batch_size, 1, 1, to_seq_length]
-            # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
-            # this attention mask is more simple than the triangular masking of causal attention
-            # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
-            attention_mask = attention_mask.view(batch_size, 1, 1, -1)
-
        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
@@ -480,7 +478,7 @@ class GPT2Model(GPT2PreTrainedModel):
        if token_type_ids is not None:
            token_type_embeds = self.wte(token_type_ids)
        else:
-            token_type_embeds = 0.
+            token_type_embeds = 0
        hidden_states = inputs_embeds + position_embeds + token_type_embeds
        hidden_states = self.drop(hidden_states)

@@ -523,7 +521,7 @@ class GPT2Model(GPT2PreTrainedModel):
        if self.output_attentions:
            # let the number of heads free (-1) so we can extract attention even after head pruning
            attention_output_shape = input_shape[:-1] + (-1,) + all_attentions[0].shape[-2:]
-            all_attentions = tuple([t.view(*attention_output_shape) for t in all_attentions])
+            all_attentions = tuple(t.view(*attention_output_shape) for t in all_attentions)
            outputs = outputs + (all_attentions,)
        return outputs  # last hidden state, (presents), (all hidden_states), (attentions)

@@ -754,7 +752,6 @@ class GPT2DoubleHeadsModel(GPT2PreTrainedModel):
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)), mc_labels.view(-1))
            outputs = (loss,) + outputs
-
        if lm_labels is not None:
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
@@ -1708,7 +1708,10 @@ class Conv1D(nn.Module):
        self.bias = nn.Parameter(torch.zeros(nf))

    def forward(self, x):
-        return torch.baddbmm(self.bias[None, None], x, self.weight[None])
+        size_out = x.size()[:-1] + (self.nf,)
+        x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
+        x = x.view(*size_out)
+        return x


 class PoolerStartLogits(nn.Module):
@@ -152,8 +152,8 @@ class AdamW(Optimizer):

                # Decay the first and second moment running average coefficient
                # In-place operations to update the averages at the same time
-                exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
-                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1.0 - beta2)
+                exp_avg.mul_(beta1).add_(1.0 - beta1, grad)
+                exp_avg_sq.mul_(beta2).addcmul_(1.0 - beta2, grad, grad)
                denom = exp_avg_sq.sqrt().add_(group["eps"])

                step_size = group["lr"]
@@ -173,6 +173,6 @@ class AdamW(Optimizer):
                # of the weights to the loss with plain (non-momentum) SGD.
                # Add weight decay at the end (fixed version)
                if group["weight_decay"] > 0.0:
-                    p.data.add_(p.data, alpha=-group["lr"] * group["weight_decay"])
+                    p.data.add_(-group["lr"] * group["weight_decay"], p.data)

        return loss
@@ -217,7 +217,7 @@ class GradientAccumulator(object):
        """The accumulated gradients on the current replica."""
        if not self._gradients:
            raise ValueError("The accumulator should be called first to initialize the gradients")
-        return list(gradient.value() if gradient is not None else gradient for gradient in self._gradients)
+        return list(gradient.value() for gradient in self._gradients)

    def __call__(self, gradients):
        """Accumulates :obj:`gradients` on the current replica."""
@@ -231,8 +231,6 @@ class GradientAccumulator(object):
                        synchronization=tf.VariableSynchronization.ON_READ,
                        aggregation=tf.VariableAggregation.ONLY_FIRST_REPLICA,
                    )
-                    if gradient is not None
-                    else gradient
                    for gradient in gradients
                ]
            )
@@ -240,8 +238,7 @@ class GradientAccumulator(object):
            raise ValueError("Expected %s gradients, but got %d" % (len(self._gradients), len(gradients)))

        for accum_gradient, gradient in zip(self._gradients, gradients):
-            if accum_gradient is not None and gradient is not None:
-                accum_gradient.assign_add(gradient)
+            accum_gradient.assign_add(gradient)

        self._accum_steps.assign_add(1)

@@ -251,5 +248,4 @@ class GradientAccumulator(object):
            return
        self._accum_steps.assign(0)
        for gradient in self._gradients:
-            if gradient is not None:
-                gradient.assign(tf.zeros_like(gradient))
+            gradient.assign(tf.zeros_like(gradient))
@@ -24,7 +24,7 @@ from abc import ABC, abstractmethod
 from contextlib import contextmanager
 from itertools import chain
 from os.path import abspath, exists
-from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union
+from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union

 import numpy as np

@@ -58,10 +58,6 @@ if is_torch_available():
        AutoModelWithLMHead,
    )

-if TYPE_CHECKING:
-    from .modeling_utils import PreTrainedModel
-    from .modeling_tf_utils import TFPreTrainedModel
-

 logger = logging.getLogger(__name__)

@@ -1513,7 +1509,7 @@ class TranslationPipeline(Pipeline):
            return results


-# Register all the supported tasks here
+# Register all the supported task here
 SUPPORTED_TASKS = {
    "feature-extraction": {
        "impl": FeatureExtractionPipeline,
@@ -1576,9 +1572,9 @@ SUPPORTED_TASKS = {
        "tf": TFAutoModelWithLMHead if is_tf_available() else None,
        "pt": AutoModelWithLMHead if is_torch_available() else None,
        "default": {
-            "model": {"pt": "bart-large-cnn", "tf": "t5-small"},
+            "model": {"pt": "bart-large-cnn", "tf": None},
            "config": None,
-            "tokenizer": {"pt": ("bart-large-cnn", {"use_fast": False}), "tf": "t5-small"},
+            "tokenizer": ("bart-large-cnn", {"use_fast": False}),
        },
    },
    "translation_en_to_fr": {
@@ -124,6 +124,9 @@ class MarianTokenizer(PreTrainedTokenizer):
        # We don't expect to process pairs, but leave the pair logic for API consistency
        return token_ids_0 + token_ids_1 + [self.eos_token_id]

+    def batch_decode(self, token_ids, **kwargs) -> List[str]:
+        return [self.decode(ids, **kwargs) for ids in token_ids]
+
    def prepare_translation_batch(
        self,
        src_texts: List[str],
@@ -185,6 +185,15 @@ class BatchEncoding(UserDict):

        self._encodings = encoding

+    @property
+    def is_fast(self):
+        """
+        Indicate if this BatchEncoding was generated from the result of a PreTrainedTokenizerFast
+        Returns: True if generated from subclasses of PreTrainedTokenizerFast, else otherwise
+
+        """
+        return self._encodings is not None
+
    def __getitem__(self, item: Union[int, str]) -> EncodingFast:
        """ If the key is a string, get the value of the dict associated to `key` ('input_ids', 'attention_mask'...)
            If the key is an integer, get the EncodingFast for batch item with index `key`
@@ -202,6 +211,16 @@ class BatchEncoding(UserDict):
    def __getattr__(self, item: str):
        return self.data[item]

+    def __getstate__(self):
+        return {"data": self.data, "encodings": self._encodings}
+
+    def __setstate__(self, state):
+        if "data" in state:
+            self.data = state["data"]
+
+        if "encodings" in state:
+            self._encodings = state["encodings"]
+
    def keys(self):
        return self.data.keys()

@@ -224,7 +243,7 @@ class BatchEncoding(UserDict):
        """
        return self._encodings

-    def tokens(self, batch_index: int = 0) -> List[int]:
+    def tokens(self, batch_index: int = 0) -> List[str]:
        if not self._encodings:
            raise ValueError("tokens() is not available when using Python based tokenizers")
        return self._encodings[batch_index].tokens
@@ -2183,9 +2202,6 @@ class PreTrainedTokenizer(SpecialTokensMixin):
        else:
            return text

-    def batch_decode(self, sequences: List[List[int]], **kwargs) -> List[str]:
-        return [self.decode(seq, **kwargs) for seq in sequences]
-
    @staticmethod
    def clean_up_tokenization(out_string: str) -> str:
        """ Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.
@@ -679,7 +679,7 @@ class Trainer:
        model.eval()

        for inputs in tqdm(dataloader, desc=description):
-            has_labels = any(inputs.get(k) is not None for k in ["labels", "lm_labels", "masked_lm_labels"])
+            has_labels = any(inputs.get(k) is not None for k in ["labels", "masked_lm_labels"])

            for k, v in inputs.items():
                inputs[k] = v.to(self.args.device)
@@ -703,7 +703,7 @@ class Trainer:
                    else:
                        label_ids = np.append(label_ids, inputs["labels"].detach().cpu().numpy(), axis=0)

-        if is_tpu_available() and preds is not None and label_ids is not None:
+        if is_tpu_available():
            # tpu-comment: Get all predictions and labels from all worker shards of eval dataset
            preds = xm.mesh_reduce("eval_preds", preds, np.concatenate)
            label_ids = xm.mesh_reduce("eval_out_label_ids", label_ids, np.concatenate)
@@ -1,116 +0,0 @@
-import unittest
-from os import sep
-from os.path import dirname, exists
-from shutil import rmtree
-
-from tests.utils import require_tf, require_torch
-from transformers import BertConfig, BertTokenizerFast, FeatureExtractionPipeline
-from transformers.convert_graph_to_onnx import convert, ensure_valid_input, infer_shapes
-
-
-class FuncContiguousArgs:
-    def forward(self, input_ids, token_type_ids, attention_mask):
-        return None
-
-
-class FuncNonContiguousArgs:
-    def forward(self, input_ids, some_other_args, token_type_ids, attention_mask):
-        return None
-
-
-class OnnxExportTestCase(unittest.TestCase):
-    MODEL_TO_TEST = ["bert-base-cased", "gpt2", "roberta-base"]
-
-    @require_tf
-    def test_export_tensorflow(self):
-        for model in OnnxExportTestCase.MODEL_TO_TEST:
-            self._test_export(model, "tf", 11)
-
-    @require_torch
-    def test_export_pytorch(self):
-        for model in OnnxExportTestCase.MODEL_TO_TEST:
-            self._test_export(model, "pt", 11)
-
-    def _test_export(self, model, framework, opset):
-        try:
-            # Compute path
-            path = "onnx" + sep + model + ".onnx"
-
-            # Remove folder if exists
-            if exists(dirname(path)):
-                rmtree(dirname(path))
-
-            # Export
-            convert(framework, model, path, opset)
-        except Exception as e:
-            self.fail(e)
-
-    @require_torch
-    def test_infer_dynamic_axis_pytorch(self):
-        """
-        Validate the dynamic axis generated for each parameters are correct
-        """
-        from transformers import BertModel
-
-        model = BertModel(BertConfig.from_pretrained("bert-base-cased"))
-        tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased")
-        self._test_infer_dynamic_axis(model, tokenizer, "pt")
-
-    @require_tf
-    def test_infer_dynamic_axis_tf(self):
-        """
-        Validate the dynamic axis generated for each parameters are correct
-        """
-        from transformers import TFBertModel
-
-        model = TFBertModel(BertConfig.from_pretrained("bert-base-cased"))
-        tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased")
-        self._test_infer_dynamic_axis(model, tokenizer, "tf")
-
-    def _test_infer_dynamic_axis(self, model, tokenizer, framework):
-        nlp = FeatureExtractionPipeline(model, tokenizer)
-
-        variable_names = ["input_ids", "token_type_ids", "attention_mask", "output_0", "output_1"]
-        input_vars, output_vars, shapes, tokens = infer_shapes(nlp, framework)
-
-        # Assert all variables are present
-        self.assertEqual(len(shapes), len(variable_names))
-        self.assertTrue(all([var_name in shapes for var_name in variable_names]))
-        self.assertSequenceEqual(variable_names[:3], input_vars)
-        self.assertSequenceEqual(variable_names[3:], output_vars)
-
-        # Assert inputs are {0: batch, 1: sequence}
-        for var_name in ["input_ids", "token_type_ids", "attention_mask"]:
-            self.assertDictEqual(shapes[var_name], {0: "batch", 1: "sequence"})
-
-        # Assert outputs are {0: batch, 1: sequence} and {0: batch}
-        self.assertDictEqual(shapes["output_0"], {0: "batch", 1: "sequence"})
-        self.assertDictEqual(shapes["output_1"], {0: "batch"})
-
-    def test_ensure_valid_input(self):
-        """
-        Validate parameters are correctly exported
-        GPT2 has "past" parameter in the middle of input_ids, token_type_ids and attention_mask.
-        ONNX doesn't support export with a dictionary, only a tuple. Thus we need to ensure we remove
-        token_type_ids and attention_mask for now to not having a None tensor in the middle
-        """
-        # All generated args are valid
-        input_names = ["input_ids", "attention_mask", "token_type_ids"]
-        tokens = {"input_ids": [1, 2, 3, 4], "attention_mask": [0, 0, 0, 0], "token_type_ids": [1, 1, 1, 1]}
-        inputs_args = ensure_valid_input(FuncContiguousArgs(), tokens, input_names)
-
-        # Should have exactly the same number of args (all are valid)
-        self.assertEqual(len(inputs_args), 3)
-
-        # Parameter should be reordered according to their respective place in the function:
-        # (input_ids, token_type_ids, attention_mask)
-        self.assertEqual(inputs_args, (tokens["input_ids"], tokens["token_type_ids"], tokens["attention_mask"]))
-
-        # Generated args are interleaved with another args (for instance parameter "past" in GPT2)
-        inputs_args = ensure_valid_input(FuncNonContiguousArgs(), tokens, input_names)
-
-        # Should have exactly the one arg (all before the one not provided "some_other_args")
-        self.assertEqual(len(inputs_args), 1)
-
-        # Should have only "input_ids"
-        self.assertEqual(inputs_args[0], tokens["input_ids"])
@@ -2,41 +2,94 @@ import unittest
 from typing import Iterable, List, Optional

 from transformers import pipeline
-from transformers.pipelines import SUPPORTED_TASKS, DefaultArgumentHandler, Pipeline
+from transformers.pipelines import DefaultArgumentHandler, Pipeline

 from .utils import require_tf, require_torch, slow


-NER_FINETUNED_MODELS = ["sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english"]
-
-# xlnet-base-cased disabled for now, since it crashes TF2
-FEATURE_EXTRACT_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-cased"]
-TEXT_CLASSIF_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english"]
-TEXT_GENERATION_FINETUNED_MODELS = ["sshleifer/tiny-ctrl"]
-
-FILL_MASK_FINETUNED_MODELS = ["sshleifer/tiny-distilroberta-base"]
-LARGE_FILL_MASK_FINETUNED_MODELS = ["distilroberta-base"]  # @slow
-
-SUMMARIZATION_FINETUNED_MODELS = ["sshleifer/bart-tiny-random", "patrickvonplaten/t5-tiny-random"]
-TF_SUMMARIZATION_FINETUNED_MODELS = ["patrickvonplaten/t5-tiny-random"]
-
-TRANSLATION_FINETUNED_MODELS = [
-    ("patrickvonplaten/t5-tiny-random", "translation_en_to_de"),
-    ("patrickvonplaten/t5-tiny-random", "translation_en_to_ro"),
+QA_FINETUNED_MODELS = [
+    (("bert-base-uncased", {"use_fast": False}), "bert-large-uncased-whole-word-masking-finetuned-squad", None),
+    (("distilbert-base-cased-distilled-squad", {"use_fast": False}), "distilbert-base-cased-distilled-squad", None),
 ]
-TF_TRANSLATION_FINETUNED_MODELS = [("patrickvonplaten/t5-tiny-random", "translation_en_to_fr")]

-expected_fill_mask_result = [
-    [
-        {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
-        {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
-    ],
-    [
-        {"sequence": "<s> The largest city in France is Paris</s>", "score": 0.3185044229030609, "token": 2201},
-        {"sequence": "<s> The largest city in France is Lyon</s>", "score": 0.21112334728240967, "token": 12790},
-    ],
+TF_QA_FINETUNED_MODELS = [
+    (("bert-base-uncased", {"use_fast": False}), "bert-large-uncased-whole-word-masking-finetuned-squad", None),
+    (("distilbert-base-cased-distilled-squad", {"use_fast": False}), "distilbert-base-cased-distilled-squad", None),
 ]

+TF_NER_FINETUNED_MODELS = {
+    (
+        "bert-base-cased",
+        "dbmdz/bert-large-cased-finetuned-conll03-english",
+        "dbmdz/bert-large-cased-finetuned-conll03-english",
+    )
+}
+
+NER_FINETUNED_MODELS = {
+    (
+        "bert-base-cased",
+        "dbmdz/bert-large-cased-finetuned-conll03-english",
+        "dbmdz/bert-large-cased-finetuned-conll03-english",
+    )
+}
+
+FEATURE_EXTRACT_FINETUNED_MODELS = {
+    ("bert-base-cased", "bert-base-cased", None),
+    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
+    ("distilbert-base-cased", "distilbert-base-cased", None),
+}
+
+TF_FEATURE_EXTRACT_FINETUNED_MODELS = {
+    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
+    ("distilbert-base-cased", "distilbert-base-cased", None),
+}
+
+TF_TEXT_CLASSIF_FINETUNED_MODELS = {
+    (
+        "bert-base-uncased",
+        "distilbert-base-uncased-finetuned-sst-2-english",
+        "distilbert-base-uncased-finetuned-sst-2-english",
+    )
+}
+
+TEXT_CLASSIF_FINETUNED_MODELS = {
+    (
+        "distilbert-base-cased",
+        "distilbert-base-uncased-finetuned-sst-2-english",
+        "distilbert-base-uncased-finetuned-sst-2-english",
+    )
+}
+
+TEXT_GENERATION_FINETUNED_MODELS = {
+    ("gpt2", "gpt2"),
+    ("xlnet-base-cased", "xlnet-base-cased"),
+}
+
+TF_TEXT_GENERATION_FINETUNED_MODELS = {
+    ("gpt2", "gpt2"),
+    ("xlnet-base-cased", "xlnet-base-cased"),
+}
+
+FILL_MASK_FINETUNED_MODELS = [
+    (("distilroberta-base", {"use_fast": False}), "distilroberta-base", None),
+]
+
+TF_FILL_MASK_FINETUNED_MODELS = [
+    (("distilroberta-base", {"use_fast": False}), "distilroberta-base", None),
+]
+
+SUMMARIZATION_FINETUNED_MODELS = {
+    ("sshleifer/bart-tiny-random", "bart-large-cnn"),
+    ("patrickvonplaten/t5-tiny-random", "t5-small"),
+}
+TF_SUMMARIZATION_FINETUNED_MODELS = {("patrickvonplaten/t5-tiny-random", "t5-small")}
+
+TRANSLATION_FINETUNED_MODELS = {
+    ("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_de"),
+    ("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_ro"),
+}
+TF_TRANSLATION_FINETUNED_MODELS = {("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_fr")}
+

 class DefaultArgumentHandlerTestCase(unittest.TestCase):
    def setUp(self) -> None:
@@ -115,8 +168,8 @@ class MonoColumnInputTestCase(unittest.TestCase):
        self,
        nlp: Pipeline,
        valid_inputs: List,
+        invalid_inputs: List,
        output_keys: Iterable[str],
-        invalid_inputs: List = [None],
        expected_multi_result: Optional[List] = None,
        expected_check_keys: Optional[List[str]] = None,
    ):
@@ -153,61 +206,93 @@ class MonoColumnInputTestCase(unittest.TestCase):
        self.assertRaises(Exception, nlp, invalid_inputs)

    @require_torch
-    def test_torch_ner(self):
+    def test_ner(self):
        mandatory_keys = {"entity", "word", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)
+        invalid_inputs = [None]
+        for tokenizer, model, config in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer)
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_tf
    def test_tf_ner(self):
        mandatory_keys = {"entity", "word", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)
+        invalid_inputs = [None]
+        for tokenizer, model, config in TF_NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_torch
-    def test_torch_sentiment_analysis(self):
+    def test_sentiment_analysis(self):
        mandatory_keys = {"label", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in TEXT_CLASSIF_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model_name, tokenizer=model_name)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)
+        invalid_inputs = [None]
+        for tokenizer, model, config in TEXT_CLASSIF_FINETUNED_MODELS:
+            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_tf
    def test_tf_sentiment_analysis(self):
        mandatory_keys = {"label", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in TEXT_CLASSIF_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)
+        invalid_inputs = [None]
+        for tokenizer, model, config in TF_TEXT_CLASSIF_FINETUNED_MODELS:
+            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)

    @require_torch
-    def test_torch_feature_extraction(self):
+    def test_feature_extraction(self):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="feature-extraction", model=model_name, tokenizer=model_name)
-            self._test_mono_column_pipeline(nlp, valid_inputs, {})
+        invalid_inputs = [None]
+        for tokenizer, model, config in FEATURE_EXTRACT_FINETUNED_MODELS:
+            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})

    @require_tf
    def test_tf_feature_extraction(self):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        for model_name in FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="feature-extraction", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, {})
+        invalid_inputs = [None]
+        for tokenizer, model, config in TF_FEATURE_EXTRACT_FINETUNED_MODELS:
+            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})

    @require_torch
-    def test_torch_fill_mask(self):
+    def test_fill_mask(self):
        mandatory_keys = {"sequence", "score", "token"}
        valid_inputs = [
            "My name is <mask>",
            "The largest city in France is <mask>",
        ]
-        for model_name in FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="pt", topk=2,)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, expected_check_keys=["sequence"])
+        invalid_inputs = [None]
+        expected_multi_result = [
+            [
+                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
+                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
+            ],
+            [
+                {
+                    "sequence": "<s> The largest city in France is Paris</s>",
+                    "score": 0.3185044229030609,
+                    "token": 2201,
+                },
+                {
+                    "sequence": "<s> The largest city in France is Lyon</s>",
+                    "score": 0.21112334728240967,
+                    "token": 12790,
+                },
+            ],
+        ]
+        for tokenizer, model, config in FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
+            self._test_mono_column_pipeline(
+                nlp,
+                valid_inputs,
+                invalid_inputs,
+                mandatory_keys,
+                expected_multi_result=expected_multi_result,
+                expected_check_keys=["sequence"],
+            )

    @require_tf
    def test_tf_fill_mask(self):
@@ -216,117 +301,103 @@ class MonoColumnInputTestCase(unittest.TestCase):
            "My name is <mask>",
            "The largest city in France is <mask>",
        ]
-        for model_name in FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="tf", topk=2,)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, expected_check_keys=["sequence"])
-
-    @require_torch
-    @slow
-    def test_torch_fill_mask_results(self):
-        mandatory_keys = {"sequence", "score", "token"}
-        valid_inputs = [
-            "My name is <mask>",
-            "The largest city in France is <mask>",
+        invalid_inputs = [None]
+        expected_multi_result = [
+            [
+                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
+                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
+            ],
+            [
+                {
+                    "sequence": "<s> The largest city in France is Paris</s>",
+                    "score": 0.3185044229030609,
+                    "token": 2201,
+                },
+                {
+                    "sequence": "<s> The largest city in France is Lyon</s>",
+                    "score": 0.21112334728240967,
+                    "token": 12790,
+                },
+            ],
        ]
-        for model_name in LARGE_FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="pt", topk=2,)
+        for tokenizer, model, config in TF_FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, framework="tf", topk=2)
            self._test_mono_column_pipeline(
                nlp,
                valid_inputs,
+                invalid_inputs,
                mandatory_keys,
-                expected_multi_result=expected_fill_mask_result,
-                expected_check_keys=["sequence"],
-            )
-
-    @require_tf
-    @slow
-    def test_tf_fill_mask_results(self):
-        mandatory_keys = {"sequence", "score", "token"}
-        valid_inputs = [
-            "My name is <mask>",
-            "The largest city in France is <mask>",
-        ]
-        for model_name in LARGE_FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="tf", topk=2)
-            self._test_mono_column_pipeline(
-                nlp,
-                valid_inputs,
-                mandatory_keys,
-                expected_multi_result=expected_fill_mask_result,
+                expected_multi_result=expected_multi_result,
                expected_check_keys=["sequence"],
            )

    @require_torch
-    def test_torch_summarization(self):
+    def test_summarization(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["summary_text"]
-        for model in SUMMARIZATION_FINETUNED_MODELS:
-            nlp = pipeline(task="summarization", model=model, tokenizer=model)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)
+        for model, tokenizer in SUMMARIZATION_FINETUNED_MODELS:
+            nlp = pipeline(task="summarization", model=model, tokenizer=tokenizer)
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, mandatory_keys,
+            )

    @require_tf
    def test_tf_summarization(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["summary_text"]
-        for model_name in TF_SUMMARIZATION_FINETUNED_MODELS:
-            nlp = pipeline(task="summarization", model=model_name, tokenizer=model_name, framework="tf",)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)
+        for model, tokenizer in TF_SUMMARIZATION_FINETUNED_MODELS:
+            nlp = pipeline(task="summarization", model=model, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, mandatory_keys,
+            )

    @require_torch
-    def test_torch_translation(self):
+    def test_translation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["translation_text"]
-        for model_name, task in TRANSLATION_FINETUNED_MODELS:
-            nlp = pipeline(task=task, model=model_name, tokenizer=model_name)
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs)
+        for model, tokenizer, task in TRANSLATION_FINETUNED_MODELS:
+            nlp = pipeline(task=task, model=model, tokenizer=tokenizer)
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, mandatory_keys,
+            )

    @require_tf
-    @slow
    def test_tf_translation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["translation_text"]
-        for model, task in TF_TRANSLATION_FINETUNED_MODELS:
-            nlp = pipeline(task=task, model=model, tokenizer=model, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)
+        for model, tokenizer, task in TF_TRANSLATION_FINETUNED_MODELS:
+            nlp = pipeline(task=task, model=model, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, mandatory_keys,
+            )

    @require_torch
-    def test_torch_text_generation(self):
+    def test_text_generation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
-        for model_name in TEXT_GENERATION_FINETUNED_MODELS:
-            nlp = pipeline(task="text-generation", model=model_name, tokenizer=model_name, framework="pt")
-            self._test_mono_column_pipeline(nlp, valid_inputs, {})
+        invalid_inputs = [None]
+        for model, tokenizer in TEXT_GENERATION_FINETUNED_MODELS:
+            nlp = pipeline(task="text-generation", model=model, tokenizer=tokenizer, framework="pt")
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, {},
+            )

    @require_tf
    def test_tf_text_generation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
-        for model_name in TEXT_GENERATION_FINETUNED_MODELS:
-            nlp = pipeline(task="text-generation", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, {})
+        invalid_inputs = [None]
+        for model, tokenizer in TF_TEXT_GENERATION_FINETUNED_MODELS:
+            nlp = pipeline(task="text-generation", model=model, tokenizer=tokenizer, framework="tf")
+            self._test_mono_column_pipeline(
+                nlp, valid_inputs, invalid_inputs, {},
+            )


-QA_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-cased-distilled-squad"]
-
-
-class QAPipelineTests(unittest.TestCase):
-    def _test_qa_pipeline(self, nlp):
-        output_keys = {"score", "answer", "start", "end"}
-        valid_inputs = [
-            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
-            {
-                "question": "In what field is HuggingFace working ?",
-                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
-            },
-        ]
-        invalid_inputs = [
-            {"question": "", "context": "This is a test to try empty question edge case"},
-            {"question": None, "context": "This is a test to try empty question edge case"},
-            {"question": "What is does with empty context ?", "context": ""},
-            {"question": "What is does with empty context ?", "context": None},
-        ]
+class MultiColumnInputTestCase(unittest.TestCase):
+    def _test_multicolumn_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):
        self.assertIsNotNone(nlp)

        mono_result = nlp(valid_inputs[0])
@@ -342,33 +413,75 @@ class QAPipelineTests(unittest.TestCase):
        for result in multi_result:
            for key in output_keys:
                self.assertIn(key, result)
-        for bad_input in invalid_inputs:
-            self.assertRaises(Exception, nlp, bad_input)
+
+        self.assertRaises(Exception, nlp, invalid_inputs[0])
        self.assertRaises(Exception, nlp, invalid_inputs)

    @require_torch
-    def test_torch_question_answering(self):
-        for model_name in QA_FINETUNED_MODELS:
-            nlp = pipeline(task="question-answering", model=model_name, tokenizer=model_name)
-            self._test_qa_pipeline(nlp)
+    def test_question_answering(self):
+        mandatory_output_keys = {"score", "answer", "start", "end"}
+        valid_samples = [
+            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
+            {
+                "question": "In what field is HuggingFace working ?",
+                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
+            },
+        ]
+        invalid_samples = [
+            {"question": "", "context": "This is a test to try empty question edge case"},
+            {"question": None, "context": "This is a test to try empty question edge case"},
+            {"question": "What is does with empty context ?", "context": ""},
+            {"question": "What is does with empty context ?", "context": None},
+        ]
+
+        for tokenizer, model, config in QA_FINETUNED_MODELS:
+            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer)
+            self._test_multicolumn_pipeline(nlp, valid_samples, invalid_samples, mandatory_output_keys)

    @require_tf
+    @slow
    def test_tf_question_answering(self):
-        for model_name in QA_FINETUNED_MODELS:
-            nlp = pipeline(task="question-answering", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_qa_pipeline(nlp)
+        mandatory_output_keys = {"score", "answer", "start", "end"}
+        valid_samples = [
+            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
+            {
+                "question": "In what field is HuggingFace working ?",
+                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
+            },
+        ]
+        invalid_samples = [
+            {"question": "", "context": "This is a test to try empty question edge case"},
+            {"question": None, "context": "This is a test to try empty question edge case"},
+            {"question": "What is does with empty context ?", "context": ""},
+            {"question": "What is does with empty context ?", "context": None},
+        ]
+
+        for tokenizer, model, config in TF_QA_FINETUNED_MODELS:
+            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer, framework="tf")
+            self._test_multicolumn_pipeline(nlp, valid_samples, invalid_samples, mandatory_output_keys)


 class PipelineCommonTests(unittest.TestCase):

-    pipelines = SUPPORTED_TASKS.keys()
+    pipelines = (
+        "ner",
+        "feature-extraction",
+        "question-answering",
+        "fill-mask",
+        "summarization",
+        "sentiment-analysis",
+        "translation_en_to_fr",
+        "translation_en_to_de",
+        "translation_en_to_ro",
+        "text-generation",
+    )

    @slow
    @require_tf
    def test_tf_defaults(self):
        # Test that pipelines can be correctly loaded without any argument
        for task in self.pipelines:
-            with self.subTest(msg="Testing TF defaults with TF and {}".format(task)):
+            with self.subTest(msg="Testing Torch defaults with PyTorch and {}".format(task)):
                pipeline(task, framework="tf")

    @slow
@@ -19,21 +19,11 @@ import pickle
 import shutil
 import tempfile
 from collections import OrderedDict
-from typing import TYPE_CHECKING, Dict, Tuple, Union
+from typing import Dict, Tuple, Union

 from tests.utils import require_tf, require_torch


-if TYPE_CHECKING:
-    from transformers import (
-        PretrainedConfig,
-        PreTrainedTokenizer,
-        PreTrainedTokenizerFast,
-        PreTrainedModel,
-        TFPreTrainedModel,
-    )
-
-
 def merge_model_tokenizer_mappings(
    model_mapping: Dict["PretrainedConfig", Union["PreTrainedModel", "TFPreTrainedModel"]],
    tokenizer_mapping: Dict["PretrainedConfig", Tuple["PreTrainedTokenizer", "PreTrainedTokenizerFast"]],
@@ -16,7 +16,7 @@

 import unittest

-from transformers import PreTrainedTokenizer
+from transformers import BertTokenizer, BertTokenizerFast, PreTrainedTokenizer
 from transformers.tokenization_gpt2 import GPT2Tokenizer

 from .utils import slow
@@ -39,3 +39,34 @@ class TokenizerUtilsTest(unittest.TestCase):
    @slow
    def test_pretrained_tokenizers(self):
        self.check_tokenizer_from_pretrained(GPT2Tokenizer)
+
+    def test_batch_encoding_pickle(self):
+        from pickle import loads, dumps
+
+        # Get a slow & a fast tokenizer
+        tok_slow = BertTokenizer.from_pretrained("bert-base-cased")
+        tok_fast = BertTokenizerFast.from_pretrained("bert-base-cased")
+
+        # Encode a sentence
+        be_slow = tok_slow.encode_plus("This is a dummy input sentence")
+        be_fast = tok_fast.encode_plus("This is a dummy input sentence")
+
+        # Make sure both are pickable
+        be_slow_data = dumps(be_slow)
+        be_fast_data = dumps(be_fast)
+
+        # Try to restore
+        be_slow_pickled = loads(be_slow_data)
+        be_fast_pickled = loads(be_fast_data)
+
+        # Ensure pickled objects keeps the is_fast attribute
+        self.assertFalse(be_slow_pickled.is_fast)
+        self.assertTrue(be_fast_pickled.is_fast)
+
+        # Ensure .data match
+        self.assertDictEqual(be_slow_pickled.data, be_slow.data)
+        self.assertDictEqual(be_fast_pickled.data, be_fast.data)
+
+        # Ensure .encodings match
+        self.assertIsNone(be_slow_pickled.encodings)
+        self.assertEqual(len(be_fast_pickled.encodings), len(be_fast.encodings))
Author	SHA1	Message	Date
Morgan Funtowicz	6cdaff706e	Bumping tokenizers dependency to 0.8.0.dev1 to check tests are passing.	2020-05-28 09:56:00 +02:00
Morgan Funtowicz	904119b9e0	Formatting.	2020-05-22 11:43:58 +02:00
Morgan Funtowicz	353deb4f6a	Remove self._is_fast to avoid conflits with __getattr__()	2020-05-22 11:21:51 +02:00
Morgan Funtowicz	6580dbdda4	Added pickle unittest for BatchEncoding	2020-05-22 11:17:05 +02:00
Morgan Funtowicz	70947826c5	Change property is_fast on BatchEncoding to be evaluated lazily to avoid lookup recursion through the dict	2020-05-22 11:16:48 +02:00
Morgan Funtowicz	b1530b5145	Added pickle support for Encoding & BatchEncoding	2020-05-22 10:43:40 +02:00
Morgan Funtowicz	664a04e90d	Correct tokens() return type to List[str]	2020-05-22 10:40:53 +02:00
Morgan Funtowicz	626727bdee	Add flag to indicate if a BatchEncoding was generated from a fast tokenizer.	2020-05-20 11:04:25 +02:00