Using chunk instead of split to avoid accessing instance variable.

Prefer using tuples instead of list for return types.
Use constant value instead of math.sqrt(2.0 / pi) in gelu
2020-05-16 18:20:57 +02:00 · 2020-05-16 16:49:27 +02:00 · 2020-05-16 15:50:50 +02:00 · 2020-05-16 15:47:27 +02:00 · 2020-05-16 15:47:27 +02:00 · 2020-05-16 15:47:27 +02:00
39 changed files with 1263 additions and 541 deletions
@@ -35,7 +35,7 @@ jobs:
    - name: Install dependencies
      run: |
        source .env/bin/activate
-        pip install torch==1.4.0
+        pip install torch
        pip install .[sklearn,testing]

    - name: Are GPUs recognized by our DL frameworks
@@ -1,6 +1,6 @@
-# Examples
+## Examples

-Version 2.9 of `transformers` introduces a new `Trainer` class for PyTorch, and its equivalent `TFTrainer` for TF 2.
+Version 2.9 of `transformers` introduces a new [`Trainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py) class for PyTorch, and its equivalent [`TFTrainer`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py) for TF 2.

 Here is the list of all our examples:
 - **grouped by task** (all official examples work for multiple models)
@@ -12,32 +12,24 @@ Here is the list of all our examples:
 This is still a work-in-progress – in particular documentation is still sparse – so please **contribute improvements/pull requests.**


-## Tasks built on Trainer
+# The Big Table of Tasks

-| Task | Example datasets | Trainer support | TFTrainer support | pytorch-lightning | Colab | One-click Deploy to Azure (wip) |
-|---|---|:---:|:---:|:---:|:---:|:---:|
-| [`language-modeling`](./language-modeling) | Raw text | ✅ | - | - | - | - |
-| [`text-classification`](./text-classification) | GLUE, XNLI | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb) | [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-storage-account-create%2Fazuredeploy.json) |
-| [`token-classification`](./token-classification) | CoNLL NER | ✅ | ✅ | ✅ | - | - |
-| [`multiple-choice`](./multiple-choice) | SWAG, RACE, ARC | ✅ | ✅ | - | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb) | - |
-| [`question-answering`](./question-answering) | SQuAD | - | ✅ | - | - | - |
+| Task | Example datasets | Trainer support | TFTrainer support | pytorch-lightning | Colab
+|---|---|:---:|:---:|:---:|:---:|
+| [**`language-modeling`**](./language-modeling)       | Raw text        | ✅ | -  | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)
+| [**`text-classification`**](./text-classification)   | GLUE, XNLI      | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb)
+| [**`token-classification`**](./token-classification) | CoNLL NER       | ✅ | ✅ | ✅ | -
+| [**`multiple-choice`**](./multiple-choice)           | SWAG, RACE, ARC | ✅ | ✅ | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb)
+| [**`question-answering`**](./question-answering)     | SQuAD           | -  | ✅ | -  | -
+| [**`text-generation`**](./text-generation)     | -           | -  | - | -  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)
+| [**`distillation`**](./distillation)       | All               | -  | -  | -  | -
+| [**`summarization`**](./summarization)     | CNN/Daily Mail    | -  | -  | -  | -
+| [**`translation`**](./translation)         | WMT               | -  | -  | -  | -
+| [**`bertology`**](./bertology)             | -                 | -  | -  | -  | -
+| [**`adversarial`**](./adversarial)         | HANS              | -  | -  | -  | -


-
-## Other examples and how-to's
-
-| Section | Description |
-|---|---|
-| [TensorFlow 2.0 models on GLUE](./text-classification) | Examples running BERT TensorFlow 2.0 model on the GLUE tasks. |
-| [Running on TPUs](#running-on-tpus) | Examples on running fine-tuning tasks on Google TPUs to accelerate workloads. |
-| [Language Model training](./language-modeling) | Fine-tuning (or training from scratch) the library models for language modeling on a text dataset. Causal language modeling for GPT/GPT-2, masked language modeling for BERT/RoBERTa. |
-| [Language Generation](./text-generation) | Conditional text generation using the auto-regressive models of the library: GPT, GPT-2, Transformer-XL and XLNet. |
-| [GLUE](./text-classification) | Examples running BERT/XLM/XLNet/RoBERTa on the 9 GLUE tasks. Examples feature distributed training as well as half-precision. |
-| [SQuAD](./question-answering) | Using BERT/RoBERTa/XLNet/XLM for question answering, examples with distributed training. |
-| [Multiple Choice](./multiple-choice) | Examples running BERT/XLNet/RoBERTa on the SWAG/RACE/ARC tasks. |
-| [Named Entity Recognition](./token-classification) | Using BERT for Named Entity Recognition (NER) on the CoNLL 2003 dataset, examples with distributed training. |
-| [XNLI](./text-classification) | Examples running BERT/XLM on the XNLI benchmark. |
-| [Adversarial evaluation of model performances](./adversarial) | Testing a model with adversarial evaluation of natural language inference on the Heuristic Analysis for NLI Systems (HANS) dataset (McCoy et al., 2019.) |
+<br>

 ## Important note

@@ -52,6 +44,12 @@ pip install .
 pip install -r ./examples/requirements.txt
 ```

+## One-click Deploy to Cloud (wip)
+
+#### Azure
+
+[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F101-storage-account-create%2Fazuredeploy.json)
+
 ## Running on TPUs

 When using Tensorflow, TPUs are supported out of the box as a `tf.distribute.Strategy`.
@@ -478,7 +478,7 @@ def _compute_pytorch(
                            dictionary[model_name]["memory"][batch_size][slice_size] = "N/A"

                        if not no_speed:
-                            print_fn("Going through model with sequence of shape".format(sequence.shape))
+                            print_fn("Going through model with sequence of shape {}".format(sequence.shape))
                            runtimes = timeit.repeat(lambda: inference(sequence), repeat=average_over, number=3)
                            average_time = sum(runtimes) / float(len(runtimes)) / 3.0
                            dictionary[model_name]["time"][batch_size][slice_size] = average_time
@@ -80,7 +80,7 @@ class Distiller:

        self.mlm = params.mlm
        if self.mlm:
-            logger.info(f"Using MLM loss for LM step.")
+            logger.info("Using MLM loss for LM step.")
            self.mlm_mask_prop = params.mlm_mask_prop
            assert 0.0 <= self.mlm_mask_prop <= 1.0
            assert params.word_mask + params.word_keep + params.word_rand == 1.0
@@ -91,7 +91,7 @@ class Distiller:
                self.pred_probs = self.pred_probs.half()
                self.token_probs = self.token_probs.half()
        else:
-            logger.info(f"Using CLM loss for LM step.")
+            logger.info("Using CLM loss for LM step.")

        self.epoch = 0
        self.n_iter = 0
@@ -365,8 +365,8 @@ class Distiller:
            self.end_epoch()

        if self.is_master:
-            logger.info(f"Save very last checkpoint as `pytorch_model.bin`.")
-            self.save_checkpoint(checkpoint_name=f"pytorch_model.bin")
+            logger.info("Save very last checkpoint as `pytorch_model.bin`.")
+            self.save_checkpoint(checkpoint_name="pytorch_model.bin")
            logger.info("Training is finished")

    def step(self, input_ids: torch.tensor, attention_mask: torch.tensor, lm_labels: torch.tensor):
@@ -60,7 +60,7 @@ def main():
    with open(args.file_path, "r", encoding="utf8") as fp:
        data = fp.readlines()

-    logger.info(f"Start encoding")
+    logger.info("Start encoding")
    logger.info(f"{len(data)} examples to process.")

    rslt = []
@@ -93,7 +93,7 @@ if __name__ == "__main__":
    elif args.model_type == "gpt2":
        for w in ["weight", "bias"]:
            compressed_sd[f"{prefix}.ln_f.{w}"] = state_dict[f"{prefix}.ln_f.{w}"]
-        compressed_sd[f"lm_head.weight"] = state_dict[f"lm_head.weight"]
+        compressed_sd["lm_head.weight"] = state_dict["lm_head.weight"]

    print(f"N layers selected for distillation: {std_idx}")
    print(f"Number of params transfered for distillation: {len(compressed_sd.keys())}")
@@ -37,7 +37,7 @@ if __name__ == "__main__":
        model = BertForMaskedLM.from_pretrained(args.model_name)
        prefix = "bert"
    else:
-        raise ValueError(f'args.model_type should be "bert".')
+        raise ValueError('args.model_type should be "bert".')

    state_dict = model.state_dict()
    compressed_sd = {}
@@ -78,12 +78,12 @@ if __name__ == "__main__":
            ]
        std_idx += 1

-    compressed_sd[f"vocab_projector.weight"] = state_dict[f"cls.predictions.decoder.weight"]
-    compressed_sd[f"vocab_projector.bias"] = state_dict[f"cls.predictions.bias"]
+    compressed_sd["vocab_projector.weight"] = state_dict["cls.predictions.decoder.weight"]
+    compressed_sd["vocab_projector.bias"] = state_dict["cls.predictions.bias"]
    if args.vocab_transform:
        for w in ["weight", "bias"]:
-            compressed_sd[f"vocab_transform.{w}"] = state_dict[f"cls.predictions.transform.dense.{w}"]
-            compressed_sd[f"vocab_layer_norm.{w}"] = state_dict[f"cls.predictions.transform.LayerNorm.{w}"]
+            compressed_sd[f"vocab_transform.{w}"] = state_dict["cls.predictions.transform.dense.{w}"]
+            compressed_sd[f"vocab_layer_norm.{w}"] = state_dict["cls.predictions.transform.LayerNorm.{w}"]

    print(f"N layers selected for distillation: {std_idx}")
    print(f"Number of params transfered for distillation: {len(compressed_sd.keys())}")
@@ -273,7 +273,7 @@ def main():
        token_probs = None

    train_lm_seq_dataset = LmSeqsDataset(params=args, data=data)
-    logger.info(f"Data loader created.")
+    logger.info("Data loader created.")

    # STUDENT #
    logger.info(f"Loading student config from {args.student_config}")
@@ -288,7 +288,7 @@ def main():

    if args.n_gpu > 0:
        student.to(f"cuda:{args.local_rank}")
-    logger.info(f"Student loaded.")
+    logger.info("Student loaded.")

    # TEACHER #
    teacher = teacher_model_class.from_pretrained(args.teacher_name, output_hidden_states=True)
@@ -115,16 +115,12 @@ class DataTrainingArguments:
    )


-def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False, local_rank=-1):
+def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    if args.line_by_line:
-        return LineByLineTextDataset(
-            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, local_rank=local_rank
-        )
+        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
    else:
-        return TextDataset(
-            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, local_rank=local_rank,
-        )
+        return TextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)


 def main():
@@ -220,16 +216,8 @@ def main():
        data_args.block_size = min(data_args.block_size, tokenizer.max_len)

    # Get datasets
-    train_dataset = (
-        get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank)
-        if training_args.do_train
-        else None
-    )
-    eval_dataset = (
-        get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank, evaluate=True)
-        if training_args.do_eval
-        else None
-    )
+    train_dataset = get_dataset(data_args, tokenizer=tokenizer) if training_args.do_train else None
+    eval_dataset = get_dataset(data_args, tokenizer=tokenizer, evaluate=True) if training_args.do_eval else None
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=data_args.mlm, mlm_probability=data_args.mlm_probability
    )
@@ -159,7 +159,6 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.train,
-            local_rank=training_args.local_rank,
        )
        if training_args.do_train
        else None
@@ -172,7 +171,6 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.dev,
-            local_rank=training_args.local_rank,
        )
        if training_args.do_eval
        else None
@@ -26,6 +26,7 @@ from enum import Enum
 from typing import List, Optional

 import tqdm
+from filelock import FileLock

 from transformers import PreTrainedTokenizer, is_tf_available, is_torch_available

@@ -77,7 +78,6 @@ class Split(Enum):
 if is_torch_available():
    import torch
    from torch.utils.data.dataset import Dataset
-    from transformers import torch_distributed_zero_first

    class MultipleChoiceDataset(Dataset):
        """
@@ -95,7 +95,6 @@ if is_torch_available():
            max_seq_length: Optional[int] = None,
            overwrite_cache=False,
            mode: Split = Split.train,
-            local_rank=-1,
        ):
            processor = processors[task]()

@@ -103,9 +102,11 @@ if is_torch_available():
                data_dir,
                "cached_{}_{}_{}_{}".format(mode.value, tokenizer.__class__.__name__, str(max_seq_length), task,),
            )
-            with torch_distributed_zero_first(local_rank):
-                # Make sure only the first process in distributed training processes the dataset,
-                # and the others will use the cache.
+
+            # Make sure only the first process in distributed training processes the dataset,
+            # and the others will use the cache.
+            lock_path = cached_features_file + ".lock"
+            with FileLock(lock_path):

                if os.path.exists(cached_features_file) and not overwrite_cache:
                    logger.info(f"Loading features from cached file {cached_features_file}")
@@ -130,9 +131,8 @@ if is_torch_available():
                        pad_token=tokenizer.pad_token_id,
                        pad_token_segment_id=tokenizer.pad_token_type_id,
                    )
-                    if local_rank in [-1, 0]:
-                        logger.info("Saving features into cached file %s", cached_features_file)
-                        torch.save(self.features, cached_features_file)
+                    logger.info("Saving features into cached file %s", cached_features_file)
+                    torch.save(self.features, cached_features_file)

        def __len__(self):
            return len(self.features)
@@ -535,7 +535,12 @@ def convert_examples_to_features(
                text_b = example.question + " " + ending

            inputs = tokenizer.encode_plus(
-                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
+                text_a,
+                text_b,
+                add_special_tokens=True,
+                max_length=max_length,
+                pad_to_max_length=True,
+                return_overflowing_tokens=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
@@ -171,7 +171,6 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.train,
-            local_rank=training_args.local_rank,
        )
        if training_args.do_train
        else None
@@ -185,7 +184,6 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.dev,
-            local_rank=training_args.local_rank,
        )
        if training_args.do_eval
        else None
@@ -261,7 +259,6 @@ def main():
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.test,
-            local_rank=training_args.local_rank,
        )

        predictions, label_ids, metrics = trainer.predict(test_dataset)
@@ -22,6 +22,8 @@ from dataclasses import dataclass
 from enum import Enum
 from typing import List, Optional, Union

+from filelock import FileLock
+
 from transformers import PreTrainedTokenizer, is_tf_available, is_torch_available


@@ -68,7 +70,6 @@ if is_torch_available():
    import torch
    from torch import nn
    from torch.utils.data.dataset import Dataset
-    from transformers import torch_distributed_zero_first

    class NerDataset(Dataset):
        """
@@ -90,16 +91,16 @@ if is_torch_available():
            max_seq_length: Optional[int] = None,
            overwrite_cache=False,
            mode: Split = Split.train,
-            local_rank=-1,
        ):
            # Load data features from cache or dataset file
            cached_features_file = os.path.join(
                data_dir, "cached_{}_{}_{}".format(mode.value, tokenizer.__class__.__name__, str(max_seq_length)),
            )

-            with torch_distributed_zero_first(local_rank):
-                # Make sure only the first process in distributed training processes the dataset,
-                # and the others will use the cache.
+            # Make sure only the first process in distributed training processes the dataset,
+            # and the others will use the cache.
+            lock_path = cached_features_file + ".lock"
+            with FileLock(lock_path):

                if os.path.exists(cached_features_file) and not overwrite_cache:
                    logger.info(f"Loading features from cached file {cached_features_file}")
@@ -125,9 +126,8 @@ if is_torch_available():
                        pad_token_segment_id=tokenizer.pad_token_type_id,
                        pad_token_label_id=self.pad_token_label_id,
                    )
-                    if local_rank in [-1, 0]:
-                        logger.info(f"Saving features into cached file {cached_features_file}")
-                        torch.save(self.features, cached_features_file)
+                    logger.info(f"Saving features into cached file {cached_features_file}")
+                    torch.save(self.features, cached_features_file)

        def __len__(self):
            return len(self.features)
@@ -0,0 +1,67 @@
+---
+language: turkish
+---
+# Turkish SQuAD  Model : Question Answering
+
+I fine-tuned Turkish-Bert-Model for Question-Answering problem with Turkish version of SQuAD; TQuAD 
+* BERT-base: https://huggingface.co/dbmdz/bert-base-turkish-uncased
+* TQuAD dataset:  https://github.com/TQuad/turkish-nlp-qa-dataset
+
+
+# Training Code
+
+```
+!python3 run_squad.py \
+  --model_type bert \
+  --model_name_or_path dbmdz/bert-base-turkish-uncased\
+  --do_train \
+  --do_eval \
+  --train_file trainQ.json \
+  --predict_file dev1.json \
+  --per_gpu_train_batch_size 12 \
+  --learning_rate 3e-5 \
+  --num_train_epochs 5.0 \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --output_dir "./model"
+```
+
+
+# Example Usage
+
+> Load Model
+```
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("./model")
+model = AutoModelForQuestionAnswering.from_pretrained("./model")
+nlp=pipeline("question-answering", model=model, tokenizer=tokenizer)
+```
+
+> Apply the model
+```
+
+sait="ABASIYANIK, Sait Faik. Hikayeci (Adapazarı 23 Kasım 1906-İstanbul 11 Mayıs 1954). \
+İlk öğrenimine Adapazarı’nda Rehber-i Terakki Mektebi’nde başladı. İki yıl kadar Adapazarı İdadisi’nde okudu.\
+İstanbul Erkek Lisesi’nde devam ettiği orta öğrenimini Bursa Lisesi’nde tamamladı (1928). İstanbul Edebiyat \
+Fakültesi’ne iki yıl devam ettikten sonra babasının isteği üzerine iktisat öğrenimi için İsviçre’ye gitti. \
+Kısa süre sonra iktisat öğrenimini bırakarak Lozan’dan Grenoble’a geçti. Üç yıl başıboş bir edebiyat öğrenimi \
+gördükten sonra babası tarafından geri çağrıldı (1933). Bir müddet Halıcıoğlu Ermeni Yetim Mektebi'nde Türkçe \
+gurup dersleri öğretmenliği yaptı. Ticarete atıldıysa da tutunamadı. Bir ay Haber gazetesinde adliye muhabirliği\
+yaptı (1942). Babasının ölümü üzerine aileden kalan emlakin geliri ile avare bir hayata başladı. Evlenemedi.\
+Yazları Burgaz adasındaki köşklerinde, kışları Şişli’deki apartmanlarında annesi ile beraber geçen bu fazla \
+içkili bohem hayatı ömrünün sonuna kadar sürdü."
+
+print(nlp(question="Ne zaman avare bir hayata başladı?", context=sait))
+print(nlp(question="Sait Faik hangi Lisede orta öğrenimini tamamladı?", context=sait))
+
+```
+```
+# Ask your self ! type your question
+print(nlp(question="...?", context=sait))
+```
+
+
+Check My other Model
+https://huggingface.co/savasy
@@ -0,0 +1,39 @@
+---
+tags:
+- pytorch
+- exbert
+license: apache-2.0
+---
+# ouBioBERT-Base, Uncased
+Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) is a language model based on the BERT-Base (Devlin, et al., 2019) architecture. We pre-trained ouBioBERT on PubMed abstracts from the PubMed baseline (ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline) via our method.  
+
+The details of the pre-training procedure can be found in Wada, et al. (2020).  
+
+## Evaluation
+We evaluated the performance of ouBioBERT in terms of the biomedical language understanding evaluation (BLUE) benchmark (Peng, et al., 2019). The numbers are mean (standard deviation) on five different random seeds.  
+| Dataset         |  Task Type                |  Score       |
+|:----------------|:--------------------------|-------------:|
+| MedSTS          |  Sentence similarity      |  84.9 (0.6)  |
+| BIOSSES         |  Sentence similarity      |  92.3 (0.8)  |
+| BC5CDR-disease  |  Named-entity recognition |  87.4 (0.1)  |
+| BC5CDR-chemical |  Named-entity recognition |  93.7 (0.2)  |
+| ShARe/CLEFE     |  Named-entity recognition |  80.1 (0.4)  |
+| DDI             |  Relation extraction      |  81.1 (1.5)  |
+| ChemProt        |  Relation extraction      |  75.0 (0.3)  |
+| i2b2 2010       |  Relation extraction      |  74.0 (0.8)  |
+| HoC             |  Document classification  |  86.4 (0.5)  |
+| MedNLI          |  Inference                |  83.6 (0.7)  |
+| **Total**       |  -                        |**83.8 (0.3)**|
+
+## Code for Fine-tuning
+We made the source code for fine-tuning freely available at [our repository](https://github.com/sy-wada/blue_benchmark_with_transformers).
+
+## Citation
+If you use our work in your research, please kindly cite the following paper:  
+```bibtex
+now preparing...
+```
+
+<a href="https://huggingface.co/exbert/?model=seiya/oubiobert-base-uncased&sentence=Coronavirus%20disease%20(COVID-19)%20is%20caused%20by%20SARS-COV2%20and%20represents%20the%20causative%20agent%20of%20a%20potentially%20fatal%20disease%20that%20is%20of%20great%20global%20public%20health%20concern.">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>
@@ -3,7 +3,6 @@
  {
   "cell_type": "markdown",
   "metadata": {
-    "collapsed": true,
    "pycharm": {
     "is_executing": false,
     "name": "#%% md\n"
@@ -77,7 +76,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
@@ -85,77 +84,7 @@
    },
    "scrolled": true
   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Requirement already satisfied: transformers in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (2.5.1)\n",
-      "Requirement already satisfied: filelock in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (3.0.12)\n",
-      "Requirement already satisfied: sentencepiece in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.1.83)\n",
-      "Requirement already satisfied: boto3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (1.12.0)\n",
-      "Requirement already satisfied: requests in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (2.22.0)\n",
-      "Requirement already satisfied: numpy in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (1.18.1)\n",
-      "Requirement already satisfied: sacremoses in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.0.35)\n",
-      "Requirement already satisfied: tokenizers==0.5.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (0.5.2)\n",
-      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (2020.1.8)\n",
-      "Requirement already satisfied: tqdm>=4.27 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from transformers) (4.42.1)\n",
-      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (0.3.3)\n",
-      "Requirement already satisfied: botocore<1.16.0,>=1.15.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (1.15.0)\n",
-      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from boto3->transformers) (0.9.4)\n",
-      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (2019.11.28)\n",
-      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (2.8)\n",
-      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (1.25.8)\n",
-      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests->transformers) (3.0.4)\n",
-      "Requirement already satisfied: joblib in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (0.14.0)\n",
-      "Requirement already satisfied: click in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (7.0)\n",
-      "Requirement already satisfied: six in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from sacremoses->transformers) (1.14.0)\n",
-      "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.0->boto3->transformers) (0.15.2)\n",
-      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.0->boto3->transformers) (2.8.1)\n",
-      "Requirement already satisfied: tensorflow==2.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (2.1.0)\n",
-      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.1.0)\n",
-      "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.1.0)\n",
-      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (3.1.0)\n",
-      "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (3.11.4)\n",
-      "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.18.1)\n",
-      "Requirement already satisfied: tensorboard<2.2.0,>=2.1.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (2.1.0)\n",
-      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.0.8)\n",
-      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.11.2)\n",
-      "Requirement already satisfied: six>=1.12.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.14.0)\n",
-      "Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (2.1.0)\n",
-      "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.4.1)\n",
-      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.1.8)\n",
-      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.34.2)\n",
-      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (1.16.1)\n",
-      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.9.0)\n",
-      "Requirement already satisfied: gast==0.2.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.2.2)\n",
-      "Requirement already satisfied: astor>=0.6.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorflow==2.1.0) (0.8.0)\n",
-      "Requirement already satisfied: setuptools in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from protobuf>=3.8.0->tensorflow==2.1.0) (45.2.0.post20200210)\n",
-      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.11.2)\n",
-      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.1)\n",
-      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.1)\n",
-      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.0)\n",
-      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.22.0)\n",
-      "Requirement already satisfied: h5py in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (2.10.0)\n",
-      "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.0)\n",
-      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.0.0)\n",
-      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)\n",
-      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.0)\n",
-      "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.8)\n",
-      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2019.11.28)\n",
-      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.25.8)\r\n",
-      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)\r\n",
-      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/Caskroom/miniconda/base/envs/huggingface/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\r\n"
-     ]
-    }
-   ],
+   "outputs": [],
   "source": [
    "!pip install transformers\n",
    "!pip install tensorflow==2.1.0"
@@ -174,7 +103,7 @@
    {
     "data": {
      "text/plain": [
-       "<torch.autograd.grad_mode.set_grad_enabled at 0x102c0ce10>"
+       "<torch.autograd.grad_mode.set_grad_enabled at 0x7f10b441e890>"
      ]
     },
     "execution_count": 2,
@@ -441,7 +370,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "is_executing": false
@@ -458,13 +387,22 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "output differences: 1.6236e-05\n",
+      "pooled differences: -1.3039e-08\n"
+     ]
+    }
+   ],
   "source": [
    "# transformers generates a ready to use dictionary with all the required parameters for the specific framework.\n",
    "input_tf = tokenizer.encode_plus(\"This is a sample input\", return_tensors=\"tf\")\n",
@@ -476,7 +414,7 @@
    "# Models outputs 2 values (The value for each tokens, the pooled representation of the input sentence)\n",
    "# Here we compare the output differences between PyTorch and TensorFlow.\n",
    "for name, o_tf, o_pt in zip([\"output\", \"pooled\"], output_tf, output_pt):\n",
-    "    print(\"{} differences: {}\".format(name, (o_tf.numpy() - o_pt.numpy()).sum()))"
+    "    print(\"{} differences: {:.5}\".format(name, (o_tf.numpy() - o_pt.numpy()).sum()))"
   ]
  },
  {
@@ -504,13 +442,24 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 232 ms, sys: 0 ns, total: 232 ms\n",
+      "Wall time: 21.1 ms\n",
+      "CPU times: user 511 ms, sys: 0 ns, total: 511 ms\n",
+      "Wall time: 43.9 ms\n"
+     ]
+    }
+   ],
   "source": [
    "from transformers import DistilBertModel\n",
    "\n",
@@ -541,13 +490,25 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tokens (int)      : [102, 12272, 9355, 5746, 30881, 215, 261, 5945, 4118, 212, 2414, 153, 1942, 232, 3532, 566, 103]\n",
+      "Tokens (str)      : ['[CLS]', 'Hug', '##ging', 'Fac', '##e', 'ist', 'eine', 'französische', 'Firma', 'mit', 'Sitz', 'in', 'New', '-', 'York', '.', '[SEP]']\n",
+      "Tokens (attn_mask): [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
+      "\n",
+      "Token wise output: torch.Size([1, 7, 768]), Pooled output: torch.Size([1, 768])\n"
+     ]
+    }
+   ],
   "source": [
    "# Let's load German BERT from the Bavarian State Library\n",
    "de_bert = BertModel.from_pretrained(\"dbmdz/bert-base-german-cased\")\n",
@@ -557,7 +518,14 @@
    "    \"Hugging Face ist eine französische Firma mit Sitz in New-York.\",\n",
    "    return_tensors=\"pt\"\n",
    ")\n",
-    "output_de, pooled_de = de_bert(**de_input)"
+    "print(\"Tokens (int)      : {}\".format(de_input['input_ids'].tolist()[0]))\n",
+    "print(\"Tokens (str)      : {}\".format([de_tokenizer.convert_ids_to_tokens(s) for s in de_input['input_ids'].tolist()[0]]))\n",
+    "print(\"Tokens (attn_mask): {}\".format(de_input['attention_mask'].tolist()[0]))\n",
+    "print()\n",
+    "\n",
+    "output_de, pooled_de = de_bert(**de_input)\n",
+    "\n",
+    "print(\"Token wise output: {}, Pooled output: {}\".format(outputs.shape, pooled.shape))"
   ]
  }
 ],
@@ -577,7 +545,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.7.6"
+   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
@@ -590,5 +558,5 @@
  }
 },
 "nbformat": 4,
- "nbformat_minor": 1
+ "nbformat_minor": 4
 }
@@ -0,0 +1,462 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "jBasof3bv1LB"
+   },
+   "source": [
+    "<h1><center>How to export 🤗 Transformers Models to ONNX ?<h1><center>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[ONNX](http://onnx.ai/) is open format for machine learning models. It allows to save your neural network's computation graph in a framework agnostic way, which might be particulary helpful when deploying deep learning models.\n",
+    "\n",
+    "Indeed, businesses might have other requirements _(languages, hardware, ...)_ for which the training framework might not be the best suited in inference scenarios. In that context, having a representation of the actual computation graph that can be shared accross various business units and logics across an organization might be a desirable component.\n",
+    "\n",
+    "Along with the serialization format, ONNX also provides a runtime library which allows efficient and hardware specific execution of the ONNX graph. This is done through the [onnxruntime](https://microsoft.github.io/onnxruntime/) project and already includes collaborations with many hardware vendors to seamlessly deploy models on various platforms.\n",
+    "\n",
+    "Through this notebook we'll walk you through the process to convert a PyTorch or TensorFlow transformers model to the [ONNX](http://onnx.ai/) and leverage [onnxruntime](https://microsoft.github.io/onnxruntime/) to run inference tasks on models from  🤗 __transformers__"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "yNnbrSg-5e1s"
+   },
+   "source": [
+    "## Exporting 🤗 transformers model to ONNX\n",
+    "\n",
+    "---\n",
+    "\n",
+    "Exporting models _(either PyTorch or TensorFlow)_ is easily achieved through the conversion tool provided as part of 🤗 __transformers__ repository. \n",
+    "\n",
+    "Under the hood the process is sensibly the following: \n",
+    "\n",
+    "1. Allocate the model from transformers (**PyTorch or TensorFlow**)\n",
+    "2. Forward dummy inputs through the model this way **ONNX** can record the set of operations executed\n",
+    "3. Optionally define dynamic axes on input and output tensors\n",
+    "4. Save the graph along with the network parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "!pip install --upgrade git+https://github.com/huggingface/transformers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {},
+    "colab_type": "code",
+    "id": "PwAaOchY4N2-"
+   },
+   "outputs": [],
+   "source": [
+    "!rm -rf onnx/\n",
+    "from transformers.convert_graph_to_onnx import convert\n",
+    "\n",
+    "# Handles all the above steps for you\n",
+    "convert(framework=\"pt\", model=\"bert-base-cased\", output=\"onnx/bert-base-cased.onnx\", opset=11)\n",
+    "\n",
+    "# Tensorflow \n",
+    "# convert(framework=\"tf\", model=\"bert-base-cased\", output=\"onnx/bert-base-cased.onnx\", opset=11)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## How to leverage runtime for inference over an ONNX graph\n",
+    "\n",
+    "---\n",
+    "\n",
+    "As mentionned in the introduction, **ONNX** is a serialization format and many side projects can load the saved graph and run the actual computations from it. Here, we'll focus on the official [onnxruntime](https://microsoft.github.io/onnxruntime/). The runtime is implemented in C++ for performance reasons and provides API/Bindings for C++, C, C#, Java and Python.\n",
+    "\n",
+    "In the case of this notebook, we will use the Python API to highlight how to load a serialized **ONNX** graph and run inference workload on various backends through **onnxruntime**.\n",
+    "\n",
+    "**onnxruntime** is available on pypi:\n",
+    "\n",
+    "- onnxruntime: ONNX + MLAS (Microsoft Linear Algebra Subprograms)\n",
+    "- onnxruntime-gpu: ONNX + MLAS + CUDA\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!pip install transformers onnxruntime-gpu onnx psutil matplotlib"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "-gP08tHfBvgY"
+   },
+   "source": [
+    "## Preparing for an Inference Session\n",
+    "\n",
+    "---\n",
+    "\n",
+    "Inference is done using a specific backend definition which turns on hardware specific optimizations of the graph. \n",
+    "\n",
+    "Optimizations are basically of three kinds: \n",
+    "\n",
+    "- **Constant Folding**: Convert static variables to constants in the graph \n",
+    "- **Deadcode Elimination**: Remove nodes never accessed in the graph\n",
+    "- **Operator Fusing**: Merge multiple instruction into one (Linear -> ReLU can be fused to be LinearReLU)\n",
+    "\n",
+    "All of this is done on **onnxruntime** by settings specific `SessionOptions`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from os import environ\n",
+    "from psutil import cpu_count\n",
+    "\n",
+    "# Constants from the performance optimization available in onnxruntime\n",
+    "# It needs to be done before importing onnxruntime\n",
+    "environ[\"OMP_NUM_THREADS\"] = str(cpu_count(logical=True))\n",
+    "environ[\"OMP_WAIT_POLICY\"] = 'ACTIVE'\n",
+    "\n",
+    "from onnxruntime import InferenceSession, SessionOptions, get_all_providers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "colab": {},
+    "colab_type": "code",
+    "id": "2k-jHLfdcTFS"
+   },
+   "outputs": [],
+   "source": [
+    "def create_model_for_provider(model_path: str, provider: str) -> InferenceSession: \n",
+    "  \n",
+    "  assert provider in get_all_providers(), f\"provider {provider} not found, {get_all_providers()}\"\n",
+    "\n",
+    "  # Few properties than might have an impact on performances (provided by MS)\n",
+    "  options = SessionOptions()\n",
+    "  options.intra_op_num_threads = 1\n",
+    "\n",
+    "  # Load the model as a graph and prepare the CPU backend \n",
+    "  return InferenceSession(model_path, options, providers=[provider])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "teJdG3amE-hR"
+   },
+   "source": [
+    "## Forwarding through our optimized ONNX model running on CPU\n",
+    "\n",
+    "---\n",
+    "\n",
+    "When the model is loaded for inference over a specific provider, for instance **CPUExecutionProvider** as above, an optimized graph can be saved. This graph will might include various optimizations, and you might be able to see some **higher-level** operations in the graph _(through [Netron](https://github.com/lutzroeder/Netron) for instance)_ such as:\n",
+    "- **EmbedLayerNormalization**\n",
+    "- **Attention**\n",
+    "- **FastGeLU**\n",
+    "\n",
+    "These operations are an example of the kind of optimization **onnxruntime** is doing, for instance here gathering multiple operations into bigger one _(Operator Fusing)_."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 34
+    },
+    "colab_type": "code",
+    "id": "dmC22kJfVGYe",
+    "outputId": "f3aba5dc-15c0-4f82-b38c-1bbae1bf112e"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Sequence output: (1, 6, 768), Pooled output: (1, 768)\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import BertTokenizerFast\n",
+    "\n",
+    "tokenizer = BertTokenizerFast.from_pretrained(\"bert-base-cased\")\n",
+    "cpu_model = create_model_for_provider(\"onnx/bert-base-cased.onnx\", \"CPUExecutionProvider\")\n",
+    "\n",
+    "# Inputs are provided through numpy array\n",
+    "model_inputs = tokenizer.encode_plus(\"My name is Bert\", return_tensors=\"pt\")\n",
+    "inputs_onnx = {k: v.cpu().detach().numpy() for k, v in model_inputs.items()}\n",
+    "\n",
+    "# Run the model (None = get all the outputs)\n",
+    "sequence, pooled = cpu_model.run(None, inputs_onnx)\n",
+    "\n",
+    "# Print information about outputs\n",
+    "\n",
+    "print(f\"Sequence output: {sequence.shape}, Pooled output: {pooled.shape}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "Kda1e7TkEqNR"
+   },
+   "source": [
+    "## Benchmarking different CPU & GPU providers\n",
+    "\n",
+    "_**Disclamer: results may vary from the actual hardware used to run the model**_"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 170
+    },
+    "colab_type": "code",
+    "id": "WcdFZCvImVig",
+    "outputId": "bfd779a1-0bc7-42db-8587-e52a485ec5e3"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Doing GPU inference on TITAN RTX\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Warming up: 100%|██████████| 10/10 [00:00<00:00, 333.82it/s]\n",
+      "Tracking inference time on CUDAExecutionProvider: 100%|██████████| 100/100 [00:00<00:00, 521.76it/s]\n",
+      "Warming up: 100%|██████████| 10/10 [00:00<00:00, 62.95it/s]\n",
+      "Tracking inference time on CPUExecutionProvider: 100%|██████████| 100/100 [00:01<00:00, 68.65it/s]\n",
+      "Warming up: 100%|██████████| 10/10 [00:00<00:00, 69.72it/s]\n",
+      "Tracking inference time on TensorrtExecutionProvider: 100%|██████████| 100/100 [00:01<00:00, 71.31it/s]\n",
+      "Warming up: 100%|██████████| 10/10 [00:00<00:00, 66.28it/s]\n",
+      "Tracking inference time on DnnlExecutionProvider: 100%|██████████| 100/100 [00:01<00:00, 72.03it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "from torch.cuda import get_device_name\n",
+    "from contextlib import contextmanager\n",
+    "from dataclasses import dataclass\n",
+    "from time import time\n",
+    "from tqdm import trange\n",
+    "\n",
+    "print(f\"Doing GPU inference on {get_device_name(0)}\", flush=True)\n",
+    "\n",
+    "@contextmanager\n",
+    "def track_infer_time(buffer: [int]):\n",
+    "    start = time()\n",
+    "    yield\n",
+    "    end = time()\n",
+    "\n",
+    "    buffer.append(end - start)\n",
+    "\n",
+    "\n",
+    "@dataclass\n",
+    "class OnnxInferenceResult:\n",
+    "  model_inference_time: [int]  \n",
+    "  optimized_model_path: str\n",
+    "\n",
+    "\n",
+    "# All the providers we'll be using in the test\n",
+    "results = {}\n",
+    "providers = [\n",
+    "  \"CUDAExecutionProvider\",\n",
+    "  \"CPUExecutionProvider\",            \n",
+    "  \"TensorrtExecutionProvider\",\n",
+    "  \"DnnlExecutionProvider\",          \n",
+    "]\n",
+    "\n",
+    "# Iterate over all the providers\n",
+    "for provider in providers:\n",
+    "\n",
+    "  # Create the model with the specified provider\n",
+    "  model = create_model_for_provider(\"onnx/bert-base-cased.onnx\", provider)\n",
+    "\n",
+    "  # Keep track of the inference time\n",
+    "  time_buffer = []\n",
+    "\n",
+    "  # Warm up the model\n",
+    "  for _ in trange(10, desc=\"Warming up\"):\n",
+    "    model.run(None, inputs_onnx)\n",
+    "\n",
+    "  # Compute \n",
+    "  for _ in trange(100, desc=f\"Tracking inference time on {provider}\"):\n",
+    "    with track_infer_time(time_buffer):\n",
+    "      model.run(None, inputs_onnx)\n",
+    "\n",
+    "  # Store the result\n",
+    "  results[provider] = OnnxInferenceResult(\n",
+    "      time_buffer,\n",
+    "      model.get_session_options().optimized_model_filepath\n",
+    "  )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 51
+    },
+    "colab_type": "code",
+    "id": "PS_49goe197g",
+    "outputId": "0ef0f70c-f5a7-46a0-949a-1a93f231d193"
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Warming up: 100%|██████████| 10/10 [00:00<00:00, 18.04it/s]\n",
+      "Tracking inference time on PyTorch: 100%|██████████| 100/100 [00:05<00:00, 18.88it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import BertModel\n",
+    "\n",
+    "# Add PyTorch to the providers\n",
+    "model_pt = BertModel.from_pretrained(\"bert-base-cased\")\n",
+    "for _ in trange(10, desc=\"Warming up\"):\n",
+    "  model_pt(**model_inputs)\n",
+    "\n",
+    "# Compute \n",
+    "time_buffer = []\n",
+    "for _ in trange(100, desc=f\"Tracking inference time on PyTorch\"):\n",
+    "  with track_infer_time(time_buffer):\n",
+    "    model_pt(**model_inputs)\n",
+    "\n",
+    "# Store the result\n",
+    "results[\"Pytorch\"] = OnnxInferenceResult(\n",
+    "    time_buffer, \n",
+    "    model.get_session_options().optimized_model_filepath\n",
+    ") "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Show the inference performance of each providers \n",
+    "\n",
+    "_Note: PyTorch model benchmark is run on CPU_"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 676
+    },
+    "colab_type": "code",
+    "id": "dj-rS8AcqRZQ",
+    "outputId": "b4bf07d1-a7b4-4eff-e6bd-d5d424fd17fb"
+   },
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAABRoAAAPeCAYAAABjjKazAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdd5SU5d34/88sZZe2gIBABAHBgjWICYIKqFhBJKKAGoOFkMRYwBgjMRGJKIpGUaOiMQ+xrAELEDSPgkY0FuyxRKOiAeVBBUQBqSI73z/87fwYdqnX6oJ5vc7Zc3avuWfua+4pR97eJZPNZrMBAAAAAJCgoKonAAAAAABs+4RGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgBgm9K6des49dRTt/j+V111Vey0005RrVq1+O53v1t5E/sWefzxxyOTycTjjz9e1VPJs3Tp0th+++2jpKTkG13vhRdeGJ06ddqs+3zb32dl75H77ruvqqdSaf785z9HJpOJ2bNnb3TZ1O8hAPi2EhoBYCt00003RSaT2ey4wYZNmzYtLrjggjjggANi3Lhxcfnll1f1lKrUTTfdFH/+85+rehqb7Lrrrot69erFgAEDvtH1DhkyJF599dWYMmXKJi3vfQYA/LeqXtUTAADKKykpidatW8fzzz8f7777brRr166qp7TVePvtt6OgYMv+X+ljjz0WBQUF8ac//Slq1qxZyTPb9tx0003RuHHjcntmde3aNVasWLFVbaPVq1fHddddF0OHDo1q1ap9o+tu1qxZHHvssXH11VdH7969N7q899m26ZRTTokBAwZEYWFhVU8FALZZ9mgEgK3MrFmz4plnnolrrrkmmjRp8o0fJhoRUVpaGitXrvzG17spCgsLo0aNGlt03/nz50etWrUqNf4sX7680h5ra1FQUBBFRUVbHHS/Dg8++GAsWLAg+vXrVyXr79evXzz11FPxn//8Z6PLVvb7LJvNxooVKyrlsb4Nvq7vp2rVqkVRUVFkMplKf+wN+fLLL+OLL774RtcJAF+Xree/HgGAiPhqb8aGDRtGz5494/jjj88LjatXr47tttsuTjvttHL3W7JkSRQVFcX555+fG1u1alUMHz482rVrF4WFhdGyZcu44IILYtWqVXn3zWQycdZZZ0VJSUnsscceUVhYGA8//HBERFx99dXRpUuXaNSoUdSqVSs6duxY4XnZVqxYEeecc040btw46tWrF7179465c+dGJpOJSy65JG/ZuXPnxumnnx5NmzaNwsLC2GOPPeJ//ud/Nmn7rHtutLLzqj399NNx3nnnRZMmTaJOnTrxgx/8IBYsWJD3HMeNGxfLli2LTCYTmUwm77Dhu+66Kzp27Bi1atWK7bbbLgYMGBBz5szJW3f37t1jzz33jJdeeim6du0atWvXjl//+tdbtK0nT54ce+65Z+75l23vdbfTGWecEd/5zneisLAw2rRpEz/72c/yosSiRYtiyJAh0bJlyygsLIx27drFlVdeGaWlpRvdjm+88UY88cQTue3RvXv3iKj4HI1lz/21116Lbt26Re3ataNdu3a598ITTzwRnTp1ilq1asWuu+4ajz76aIXPZ0tf98mTJ0fr1q2jbdu2eeOnnnpq1K1bNz744IPo1atX1K1bN3bYYYe48cYbIyLi9ddfj0MOOSTq1KkTrVq1irvvvjvv/qtXr44RI0bEzjvvHEVFRdGoUaM48MAD45FHHslbrkePHhER8de//nWD89zQ++zLL7+MSy+9NNq2bRuFhYXRunXr+PWvf13uPdK6devo1atXTJ06Nfbbb7+oVatW3HLLLRtc73PPPRdHHnlk1K9fP2rXrh3dunWLp59+Om+Z999/P84888zYddddo1atWtGoUaM44YQTKjwn4aJFi2Lo0KHRunXrKCwsjBYtWsSPfvSj+OSTT/KWKy0tjcsuuyxatGgRRUVFceihh8a77767wblGRFxyySWRyWTirbfein79+kVxcXE0atQozj333HIRcUPfT//85z/jqKOOiuLi4qhbt24ceuih8eyzz+bu++KLL0Ymk4nbb7+93BymTp0amUwmHnzwwYio+ByN2Ww2Ro4cGS1atIjatWvHwQcfHG+88UaFz2lTPouzZ8+OTCYTV199dYwZMyb3XnjzzTc3us0AYFvg0GkA2MqUlJTEcccdFzVr1owTTzwxbr755njhhRfie9/7XtSoUSN+8IMfxMSJE+OWW27J22Nq8uTJsWrVqtz560pLS6N3797x1FNPxeDBg6N9+/bx+uuvx7XXXhvvvPNOTJ48OW+9jz32WNxzzz1x1llnRePGjaN169YR8dV58Xr37h0nn3xyfPHFFzF+/Pg44YQT4sEHH4yePXvm7n/qqafGPffcE6ecckrsv//+8cQTT+TdXmbevHmx//775+JBkyZN4qGHHoozzjgjlixZEkOGDNmi7Xb22WdHw4YNY/jw4TF79uwYM2ZMnHXWWTFhwoSIiLjzzjvj1ltvjeeffz5uu+22iIjo0qVLRERcdtll8dvf/jb69esXgwYNigULFsQNN9wQXbt2jX/+85/RoEGD3HoWLlwYRx11VAwYMCB++MMfRtOmTTd7Wz/11FMxceLEOPPMM6NevXpx/fXXR9++feODDz6IRo0aRUTEhx9+GN///vdj0aJFMXjw4Nhtt91i7ty5cd9998Xy5cujZs2asXz58ujWrVvMnTs3fvKTn8SOO+4YzzzzTAwbNiw++uijGDNmzHq315gxY+Lss8+OunXrxkUXXRQREU2bNt3gNv7ss8+iV69eMWDAgDjhhBPi5ptvjgEDBkRJSUkMGTIkfvrTn8ZJJ50UV111VRx//PExZ86cqFevXkSkv+7PPPNM7LvvvhXetmbNmjjqqKOia9euMXr06CgpKYmzzjor6tSpExdddFGcfPLJcdxxx8XYsWPjRz/6UXTu3DnatGkTEV8Fr1GjRsWgQYPi+9//fixZsiRefPHFePnll+Owww7LraN+/frRtm3bePrpp2Po0KHrneeG3meDBg2K22+/PY4//vj4xS9+Ec8991yMGjUq/v3vf8ekSZPyHuftt9+OE088MX7yk5/Ej3/849h1113Xu87HHnssjjrqqOjYsWMMHz48CgoKYty4cXHIIYfEk08+Gd///vcjIuKFF16IZ555JgYMGBAtWrSI2bNnx8033xzdu3ePN998M2rXrh0RX11056CDDop///vfcfrpp8e+++4bn3zySUyZMiX+7//+Lxo3bpxb9xVXXBEFBQVx/vnnx+LFi2P06NFx8sknx3PPPbfe+a6tX79+0bp16xg1alQ8++yzcf3118dnn30Wd9xxR7nnuO730xtvvBEHHXRQFBcXxwUXXBA1atSIW265Jbp3754L3/vtt1/stNNOcc8998TAgQPzHnPChAnRsGHDOOKII9Y7v4svvjhGjhwZRx99dBx99NHx8ssvx+GHH15uD8TN/SyOGzcuVq5cGYMHD47CwsLYbrvtNml7AcBWLwsAbDVefPHFbERkH3nkkWw2m82WlpZmW7RokT333HNzy0ydOjUbEdkHHngg775HH310dqeddsr9feedd2YLCgqyTz75ZN5yY8eOzUZE9umnn86NRUS2oKAg+8Ybb5Sb0/Lly/P+/uKLL7J77rln9pBDDsmNvfTSS9mIyA4ZMiRv2VNPPTUbEdnhw4fnxs4444xs8+bNs5988knesgMGDMjWr1+/3PrW1apVq+zAgQNzf48bNy4bEdkePXpkS0tLc+NDhw7NVqtWLbto0aLc2MCBA7N16tTJe7zZs2dnq1Wrlr3sssvyxl9//fVs9erV88a7deuWjYjs2LFj85bd3G1ds2bN7Lvvvpsbe/XVV7MRkb3hhhtyYz/60Y+yBQUF2RdeeKHcNih7npdeemm2Tp062XfeeSfv9gsvvDBbrVq17AcffFDuvmvbY489st26dSs3Pn369GxEZKdPn17uud999925sbfeeiv33nn22Wdz42Xv0XHjxuXGUl731atXZzOZTPYXv/hFudsGDhyYjYjs5Zdfnhv77LPPsrVq1cpmMpns+PHjy8137ffjPvvsk+3Zs+d61722ww8/PNu+ffuNLlfR++yVV17JRkR20KBBeePnn39+NiKyjz32WG6sVatW2YjIPvzwwxtdV2lpaXbnnXfOHnHEEXnv/+XLl2fbtGmTPeyww/LG1jVjxoxsRGTvuOOO3NjFF1+cjYjsxIkTK1xfNvv/v0fat2+fXbVqVe726667LhsR2ddff32D8x4+fHg2IrK9e/fOGz/zzDOzEZF99dVXc2Pr+37q06dPtmbNmtn33nsvN/bhhx9m69Wrl+3atWtubNiwYdkaNWpkP/3009zYqlWrsg0aNMiefvrpubGy75JZs2Zls9lsdv78+dmaNWtme/bsmbdtf/3rX2cjIu97aFM/i7NmzcpGRLa4uDg7f/78DW4jANgWOXQaALYiJSUl0bRp0zj44IMj4qtDBvv37x/jx4+PNWvWRETEIYccEo0bN87tqRfx1Z5mjzzySPTv3z83du+990b79u1jt912i08++ST3c8ghh0RExPTp0/PW3a1bt9h9993LzalWrVp561m8eHEcdNBB8fLLL+fGyw5jPPPMM/Pue/bZZ+f9nc1m4/77749jjjkmstls3ryOOOKIWLx4cd7jbo7BgwfnnVvtoIMOijVr1sT777+/wftNnDgxSktLo1+/fnnzadasWey8887ltlNhYWG5Q9c3d1v36NEj7xDgvffeO4qLi3Pn/ystLY3JkyfHMcccE/vtt1+5OZc9z3vvvTcOOuigaNiwYd56e/ToEWvWrIl//OMfG9tsm6Vu3bp5V3zeddddo0GDBtG+ffu8K6SX/V72fFJf908//TSLine truncated
+      "text/plain": [
+       "<Figure size 1600x1200 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "import matplotlib\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "import os\n",
+    "\n",
+    "# Compute average inference time + std\n",
+    "time_results = {k: np.mean(v.model_inference_time) * 1e3 for k, v in results.items()}\n",
+    "time_results_std = np.std([v.model_inference_time for v in results.values()]) * 1000\n",
+    "\n",
+    "plt.rcdefaults()\n",
+    "fig, ax = plt.subplots(figsize=(16, 12))\n",
+    "ax.set_ylabel(\"Avg Inference time (ms)\")\n",
+    "ax.set_title(\"Average inference time (ms) for each provider\")\n",
+    "ax.bar(time_results.keys(), time_results.values(), yerr=time_results_std)\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "collapsed_sections": [],
+   "name": "ONNX Overview",
+   "provenance": [],
+   "toc_visible": true
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
@@ -10,9 +10,10 @@ Pull Request and we'll review it so it can be included here.
 ## Hugging Face's notebooks :hugs:

 | Notebook     |      Description      |   |
-|:----------|:-------------:|------:|
+|:----------|:-------------|------:|
 | [Getting Started Tokenizers](https://github.com/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb)  | How to train and use your very own tokenizer  |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb) |
 | [Getting Started Transformers](https://github.com/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb)   | How to easily start using transformers  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb) |
 | [How to use Pipelines](https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb)  | Simple and efficient way to use State-of-the-Art models on downstream tasks through transformers | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb) |
 | [How to train a language model](https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)| Highlight all the steps to effectively train Transformer model on custom data | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)|
 | [How to generate text](https://github.com/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)| How to use different decoding methods for language generation with transformers | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb)|
+| [How to export model to ONNX](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) | Highlight how to export and run inference workloads through ONNX |
@@ -36,5 +36,5 @@ multi_line_output = 3
 use_parentheses = True

 [flake8]
-ignore = E203, E501, W503
+ignore = E203, E501, E741, W503
 max-line-length = 119
@@ -67,8 +67,18 @@ extras = {}

 extras["mecab"] = ["mecab-python3"]
 extras["sklearn"] = ["scikit-learn"]
-extras["tf"] = ["tensorflow"]
-extras["tf-cpu"] = ["tensorflow-cpu"]
+
+# keras2onnx and onnxconverter-common version is specific through a commit until 1.7.0 lands on pypi
+extras["tf"] = [
+    "tensorflow",
+    "onnxconverter-common @ git+git://github.com/microsoft/onnxconverter-common.git@f64ca15989b6dc95a1f3507ff6e4c395ba12dff5#egg=onnxconverter-common",
+    "keras2onnx @ git+git://github.com/onnx/keras-onnx.git@cbdc75cb950b16db7f0a67be96a278f8d2953b48#egg=keras2onnx"
+]
+extras["tf-cpu"] = [
+    "tensorflow-cpu",
+    "onnxconverter-common @ git+git://github.com/microsoft/onnxconverter-common.git@f64ca15989b6dc95a1f3507ff6e4c395ba12dff5#egg=onnxconverter-common",
+    "keras2onnx @ git+git://github.com/onnx/keras-onnx.git@cbdc75cb950b16db7f0a67be96a278f8d2953b48#egg=keras2onnx"
+]
 extras["torch"] = ["torch"]

 extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
@@ -79,7 +89,7 @@ extras["docs"] = ["recommonmark", "sphinx", "sphinx-markdown-tables", "sphinx-rt
 extras["quality"] = [
    "black",
    "isort @ git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort",
-    "flake8==3.7.9",
+    "flake8",
 ]
 extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow", "torch"]

@@ -98,7 +108,7 @@ setup(
    packages=find_packages("src"),
    install_requires=[
        "numpy",
-        "tokenizers == 0.8.0.dev1",
+        "tokenizers == 0.7.0",
        # dataclasses for Python versions that don't have it
        "dataclasses;python_version<'3.7'",
        # filesystem locks e.g. to prevent parallel downloads
@@ -26,7 +26,8 @@ def gelu_new(x):
    """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
        Also see https://arxiv.org/abs/1606.08415
    """
-    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
+    # math.sqrt(2.0 / math.pi) = 0.7978845608
+    return 0.5 * x * (1.0 + torch.tanh(0.7978845608 * (x + 0.044715 * torch.pow(x, 3.0))))


 if torch.__version__ < "1.4.0":
@@ -53,4 +54,4 @@ def get_activation(activation_string):
    if activation_string in ACT2FN:
        return ACT2FN[activation_string]
    else:
-        raise KeyError("function {} not found in ACT2FN mapping {}".format(activation_string, list(ACT2FN.keys())))
+        raise KeyError("function {} not found in ACT2FN mapping {}".format(activation_string, list(ACT2FN.keys())))
@@ -68,6 +68,6 @@ class RobertaConfig(BertConfig):
    model_type = "roberta"

    def __init__(self, pad_token_id=1, bos_token_id=0, eos_token_id=2, **kwargs):
-        """Constructs FlaubertConfig.
+        """Constructs RobertaConfig.
        """
        super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
@@ -0,0 +1,212 @@
+from argparse import ArgumentParser
+from itertools import takewhile
+from os import listdir, makedirs
+from os.path import abspath, dirname, exists
+from typing import Dict, List, Optional, Tuple
+
+from transformers import is_tf_available, is_torch_available
+from transformers.pipelines import Pipeline, pipeline
+from transformers.tokenization_utils import BatchEncoding
+
+
+class OnnxConverterArgumentParser(ArgumentParser):
+    """
+    Wraps all the script arguments supported to export transformers models to ONNX IR
+    """
+
+    def __init__(self):
+        super(OnnxConverterArgumentParser, self).__init__("ONNX Converter")
+
+        self.add_argument("--model", type=str, required=True, help="Model's id or path (ex: bert-base-cased)")
+        self.add_argument("--tokenizer", type=str, help="Tokenizer's id or path (ex: bert-base-cased)")
+        self.add_argument("--framework", type=str, choices=["pt", "tf"], help="Framework for loading the model")
+        self.add_argument("--opset", type=int, default=11, help="ONNX opset to use")
+        self.add_argument("--check-loading", action="store_true", help="Check ONNX is able to load the model")
+        self.add_argument("output")
+
+
+def ensure_valid_input(model, tokens, input_names):
+    """
+    Ensure input are presented in the correct order, without any None
+    Args:
+        model: The model used to forward the input data
+        tokens: BatchEncoding holding the input data
+        input_names: The name of the inputs
+
+    Returns: Tuple
+
+    """
+    model_args_name = model.forward.__code__.co_varnames
+    model_args_pos = [(model_args_name.index(name) - 1, name) for name in input_names]
+    model_args = [None] * (max(map(lambda x: x[0], model_args_pos)) + 1)
+
+    for arg_pos, arg_name in model_args_pos:
+        model_args[arg_pos] = tokens[arg_name]
+
+    model_args = tuple(model_args)  # Need to be ordered
+    return tuple(takewhile(lambda arg: arg is not None, model_args))
+
+
+def infer_shapes(nlp: Pipeline, framework: str) -> Tuple[List[str], List[str], Dict, BatchEncoding]:
+    def build_shape_dict(tensor, is_input: bool, seq_len: int):
+        if isinstance(tensor, (tuple, list)):
+            return [build_shape_dict(t, is_input, seq_len) for t in tensor]
+
+        else:
+            # Let's assume batch is the first axis with only 1 element (~~ might not be always true ...)
+            axes = {[axis for axis, numel in enumerate(tensor.shape) if numel == 1][0]: "batch"}
+            if is_input:
+                if len(tensor.shape) == 2:
+                    axes[1] = "sequence"
+                else:
+                    raise ValueError("Unable to infer tensor axes ({})".format(len(tensor.shape)))
+            else:
+                seq_axes = [dim for dim, shape in enumerate(tensor.shape) if shape == seq_len]
+                axes.update({dim: "sequence" for dim in seq_axes})
+
+        return axes
+
+    tokens = nlp.tokenizer.encode_plus("This is a sample output", return_tensors=framework)
+    seq_len = tokens.input_ids.shape[-1]
+    outputs = nlp.model(**tokens) if framework == "pt" else nlp.model(tokens)
+
+    if not isinstance(outputs, (list, tuple)):
+        outputs = (outputs,)
+
+    # Generate input names & axes
+    input_vars = list(tokens.keys())
+    input_dynamic_axes = {k: build_shape_dict(v, True, seq_len) for k, v in tokens.items()}
+
+    # flatten potentially grouped outputs (past for gpt2, attentions)
+    outputs_flat = []
+    for output in outputs:
+        if isinstance(output, (tuple, list)):
+            outputs_flat.extend(output)
+        else:
+            outputs_flat.append(output)
+
+    # Generate output names & axes
+    output_names = ["output_{}".format(i) for i in range(len(outputs_flat))]
+    output_dynamic_axes = {k: build_shape_dict(v, False, seq_len) for k, v in zip(output_names, outputs_flat)}
+
+    # Create the aggregated axes representation
+    dynamic_axes = dict(input_dynamic_axes, **output_dynamic_axes)
+    return input_vars, output_names, dynamic_axes, tokens
+
+
+def load_graph_from_args(framework: str, model: str, tokenizer: Optional[str] = None) -> Pipeline:
+    # If no tokenizer provided
+    if tokenizer is None:
+        tokenizer = model
+
+    print("Loading pipeline (model: {}, tokenizer: {})".format(model, tokenizer))
+
+    # Allocate tokenizer and model
+    return pipeline("feature-extraction", model=model, framework=framework)
+
+
+def convert_pytorch(nlp: Pipeline, opset: int, output: str):
+    if not is_torch_available():
+        raise Exception("Cannot convert because PyTorch is not installed. Please install torch first.")
+
+    import torch
+    from torch.onnx import export
+
+    print("PyTorch: {}".format(torch.__version__))
+
+    with torch.no_grad():
+        input_names, output_names, dynamic_axes, tokens = infer_shapes(nlp, "pt")
+        model_args = ensure_valid_input(nlp.model, tokens, input_names)
+
+        export(
+            nlp.model,
+            model_args,
+            f=output,
+            input_names=input_names,
+            output_names=output_names,
+            dynamic_axes=dynamic_axes,
+            do_constant_folding=True,
+            use_external_data_format=True,
+            enable_onnx_checker=True,
+            opset_version=opset,
+        )
+
+
+def convert_tensorflow(nlp: Pipeline, opset: int, output: str):
+    if not is_tf_available():
+        raise Exception(
+            "Cannot convert {} because TF is not installed. Please install torch first.".format(args.model)
+        )
+
+    print("/!\\ Please note TensorFlow doesn't support exporting model > 2Gb /!\\")
+
+    try:
+        import tensorflow as tf
+        from keras2onnx import convert_keras, save_model, __version__ as k2ov
+
+        print("TensorFlow: {}, keras2onnx: {}".format(tf.version.VERSION, k2ov))
+
+        # Build
+        input_names, output_names, dynamic_axes, tokens = infer_shapes(nlp, "tf")
+
+        # Forward
+        nlp.model.predict(tokens.data)
+        onnx_model = convert_keras(nlp.model, nlp.model.name, target_opset=opset)
+        save_model(onnx_model, output)
+
+    except ImportError as e:
+        raise Exception(
+            "Cannot import {} required to convert TF model to ONNX. Please install {} first.".format(e.name, e.name)
+        )
+
+
+def convert(framework: str, model: str, output: str, opset: int, tokenizer: Optional[str] = None):
+    print("ONNX opset version set to: {}".format(opset))
+
+    # Load the pipeline
+    nlp = load_graph_from_args(framework, model, tokenizer)
+
+    parent = dirname(output)
+    if not exists(parent):
+        print("Creating folder {}".format(parent))
+        makedirs(parent)
+    elif len(listdir(parent)) > 0:
+        raise Exception("Folder {} is not empty, aborting conversion".format(parent))
+
+    # Export the graph
+    if framework == "pt":
+        convert_pytorch(nlp, opset, output)
+    else:
+        convert_tensorflow(nlp, opset, output)
+
+
+def verify(path: str):
+    from onnxruntime import InferenceSession, SessionOptions
+    from onnxruntime.capi.onnxruntime_pybind11_state import RuntimeException
+
+    print("Checking ONNX model loading from: {}".format(path))
+    try:
+        onnx_options = SessionOptions()
+        _ = InferenceSession(path, onnx_options, providers=["CPUExecutionProvider"])
+        print("Model correctly loaded")
+    except RuntimeException as re:
+        print("Error while loading the model: {}".format(re))
+
+
+if __name__ == "__main__":
+    parser = OnnxConverterArgumentParser()
+    args = parser.parse_args()
+
+    # Make sure output is absolute path
+    args.output = abspath(args.output)
+
+    try:
+        # Convert
+        convert(args.framework, args.model, args.output, args.opset, args.tokenizer)
+
+        # And verify
+        if args.check_loading:
+            verify(args.output)
+    except Exception as e:
+        print("Error while converting the model: {}".format(e))
+        exit(1)
@@ -226,7 +226,7 @@ def lmap(f, x) -> List:
 def fetch_test_set(test_set_url):
    import wget

-    fname = wget.download(test_set_url, f"opus_test.txt")
+    fname = wget.download(test_set_url, "opus_test.txt")
    lns = Path(fname).open().readlines()
    src = lmap(str.strip, lns[::4])
    gold = lmap(str.strip, lns[1::4])
@@ -114,7 +114,7 @@ class GlueDataset(Dataset):
                torch.save(self.features, cached_features_file)
                # ^ This seems to take a lot of time so I want to investigate why and how we can improve.
                logger.info(
-                    f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
+                    "Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
                )

    def __len__(self):
@@ -4,10 +4,10 @@ import pickle
 import time

 import torch
+from filelock import FileLock
 from torch.utils.data.dataset import Dataset

 from ...tokenization_utils import PreTrainedTokenizer
-from ...trainer import torch_distributed_zero_first


 logger = logging.getLogger(__name__)
@@ -20,7 +20,7 @@ class TextDataset(Dataset):
    """

    def __init__(
-        self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, overwrite_cache=False, local_rank=-1,
+        self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, overwrite_cache=False,
    ):
        assert os.path.isfile(file_path)

@@ -31,9 +31,10 @@ class TextDataset(Dataset):
            directory, "cached_lm_{}_{}_{}".format(tokenizer.__class__.__name__, str(block_size), filename,),
        )

-        with torch_distributed_zero_first(local_rank):
-            # Make sure only the first process in distributed training processes the dataset,
-            # and the others will use the cache.
+        # Make sure only the first process in distributed training processes the dataset,
+        # and the others will use the cache.
+        lock_path = cached_features_file + ".lock"
+        with FileLock(lock_path):

            if os.path.exists(cached_features_file) and not overwrite_cache:
                start = time.time()
@@ -64,7 +65,7 @@ class TextDataset(Dataset):
                with open(cached_features_file, "wb") as handle:
                    pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
                logger.info(
-                    f"Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
+                    "Saving features into cached file %s [took %.3f s]", cached_features_file, time.time() - start
                )

    def __len__(self):
@@ -80,7 +81,7 @@ class LineByLineTextDataset(Dataset):
    soon.
    """

-    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int, local_rank=-1):
+    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
        assert os.path.isfile(file_path)
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
@@ -195,18 +195,22 @@ def squad_convert_example_to_features(example, max_seq_length, doc_stride, max_q
        cls_index = span["input_ids"].index(tokenizer.cls_token_id)

        # p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
-        # Original TF implem also keep the classification token (set to 0) (not sure why...)
-        p_mask = np.array(span["token_type_ids"])
-
-        p_mask = np.minimum(p_mask, 1)
-
+        # Original TF implem also keep the classification token (set to 0)
+        p_mask = np.ones_like(span["token_type_ids"])
        if tokenizer.padding_side == "right":
-            # Limit positive values to one
-            p_mask = 1 - p_mask
+            p_mask[len(truncated_query) + sequence_added_tokens :] = 0
+        else:
+            p_mask[-len(span["tokens"]) : -(len(truncated_query) + sequence_added_tokens)] = 0

-        p_mask[np.where(np.array(span["input_ids"]) == tokenizer.sep_token_id)[0]] = 1
+        pad_token_indices = np.where(span["input_ids"] == tokenizer.pad_token_id)
+        special_token_indices = np.asarray(
+            tokenizer.get_special_tokens_mask(span["input_ids"], already_has_special_tokens=True)
+        ).nonzero()

-        # Set the CLS index to '0'
+        p_mask[pad_token_indices] = 1
+        p_mask[special_token_indices] = 1
+
+        # Set the cls index to 0: the CLS index can be used for impossible answers
        p_mask[cls_index] = 0

        span_is_impossible = example.is_impossible
@@ -104,7 +104,7 @@ class Attention(nn.Module):
        # [switch nx => n_state from Block to Attention to keep identical to TF implem]
        assert n_state % config.n_head == 0
        self.register_buffer(
-            "bias", torch.tril(torch.ones((n_ctx, n_ctx), dtype=torch.uint8)).view(1, 1, n_ctx, n_ctx)
+            "bias", torch.tril(torch.ones((n_ctx, n_ctx), dtype=torch.bool)).view(1, 1, n_ctx, n_ctx)
        )
        self.register_buffer("masked_bias", torch.tensor(-1e4))
        self.n_head = config.n_head
@@ -143,6 +143,7 @@ class Attention(nn.Module):
        w = torch.matmul(q, k)
        if self.scale:
            w = w / (float(v.size(-1)) ** 0.5)
+
        nd, ns = w.size(-2), w.size(-1)
        mask = self.bias[:, :, ns - nd : ns, :ns]
        w = torch.where(mask.bool(), w, self.masked_bias.to(w.dtype))
@@ -151,22 +152,22 @@ class Attention(nn.Module):
            # Apply the attention mask
            w = w + attention_mask

-        w = nn.Softmax(dim=-1)(w)
+        w = torch.softmax(w, dim=-1)
        w = self.attn_dropout(w)

        # Mask heads if we want to
        if head_mask is not None:
            w = w * head_mask

-        outputs = [torch.matmul(w, v)]
+        wv = torch.matmul(w, v)
        if self.output_attentions:
-            outputs.append(w)
-        return outputs
+            return wv, w
+        else:
+            return (wv, )

    def merge_heads(self, x):
-        x = x.permute(0, 2, 1, 3).contiguous()
-        new_x_shape = x.size()[:-2] + (x.size(-2) * x.size(-1),)
-        return x.view(*new_x_shape)  # in Tensorflow implem: fct merge_states
+        x = x.transpose(2, 1).contiguous()
+        return x.flatten(2)  # in Tensorflow implem: fct merge_states

    def split_heads(self, x, k=False):
        new_x_shape = x.size()[:-1] + (self.n_head, x.size(-1) // self.n_head)
@@ -174,11 +175,11 @@ class Attention(nn.Module):
        if k:
            return x.permute(0, 2, 3, 1)  # (batch, head, head_features, seq_length)
        else:
-            return x.permute(0, 2, 1, 3)  # (batch, head, seq_length, head_features)
+            return x.transpose(2, 1)  # (batch, head, seq_length, head_features)

    def forward(self, x, layer_past=None, attention_mask=None, head_mask=None, use_cache=False):
        x = self.c_attn(x)
-        query, key, value = x.split(self.split_size, dim=2)
+        query, key, value = x.chunk(3, dim=2)
        query = self.split_heads(query)
        key = self.split_heads(key, k=True)
        value = self.split_heads(value)
@@ -199,8 +200,7 @@ class Attention(nn.Module):
        a = self.c_proj(a)
        a = self.resid_dropout(a)

-        outputs = [a, present] + attn_outputs[1:]
-        return outputs  # a, present, (attentions)
+        return a, present, attn_outputs[1:]


 class MLP(nn.Module):
@@ -241,8 +241,7 @@ class Block(nn.Module):
        m = self.mlp(self.ln_2(x))
        x = x + m

-        outputs = [x] + output_attn[1:]
-        return outputs  # x, present, (attentions)
+        return x, output_attn[1:]


 class GPT2PreTrainedModel(PreTrainedModel):
@@ -425,17 +424,17 @@ class GPT2Model(GPT2PreTrainedModel):
        elif input_ids is not None:
            input_shape = input_ids.size()
            input_ids = input_ids.view(-1, input_shape[-1])
-            batch_size = input_ids.shape[0]
+            batch_size = input_ids.size(0)
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.size()[:-1]
-            batch_size = inputs_embeds.shape[0]
+            batch_size = inputs_embeds.size(0)
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds")

        if token_type_ids is not None:
-            token_type_ids = token_type_ids.view(-1, input_shape[-1])
+            token_type_ids = token_type_ids.view_as(input_ids)
        if position_ids is not None:
-            position_ids = position_ids.view(-1, input_shape[-1])
+            position_ids = position_ids.view_as(input_ids)

        if past is None:
            past_length = 0
@@ -450,22 +449,25 @@ class GPT2Model(GPT2PreTrainedModel):
        # Attention mask.
        if attention_mask is not None:
            assert batch_size > 0, "batch_size has to be defined and > 0"
-            attention_mask = attention_mask.view(batch_size, -1)
-            # We create a 3D attention mask from a 2D tensor mask.
-            # Sizes are [batch_size, 1, 1, to_seq_length]
-            # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
-            # this attention mask is more simple than the triangular masking of causal attention
-            # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
-            attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)

            # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
            # masked positions, this operation will create a tensor which is 0.0 for
            # positions we want to attend and -10000.0 for masked positions.
            # Since we are adding it to the raw scores before the softmax, this is
            # effectively the same as removing these entirely.
-            attention_mask = attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
+
+            # Computation is done over contiguous tensor (no view / unsqueeze) so probably faster
+            # Do the shaping ops afterwards
+            attention_mask = attention_mask.to(self.dtype)  # fp16 compatibility
            attention_mask = (1.0 - attention_mask) * -10000.0

+            # We create a 3D attention mask from a 2D tensor mask.
+            # Sizes are [batch_size, 1, 1, to_seq_length]
+            # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
+            # this attention mask is more simple than the triangular masking of causal attention
+            # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
+            attention_mask = attention_mask.view(batch_size, 1, 1, -1)
+
        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
@@ -478,7 +480,7 @@ class GPT2Model(GPT2PreTrainedModel):
        if token_type_ids is not None:
            token_type_embeds = self.wte(token_type_ids)
        else:
-            token_type_embeds = 0
+            token_type_embeds = 0.
        hidden_states = inputs_embeds + position_embeds + token_type_embeds
        hidden_states = self.drop(hidden_states)

@@ -521,7 +523,7 @@ class GPT2Model(GPT2PreTrainedModel):
        if self.output_attentions:
            # let the number of heads free (-1) so we can extract attention even after head pruning
            attention_output_shape = input_shape[:-1] + (-1,) + all_attentions[0].shape[-2:]
-            all_attentions = tuple(t.view(*attention_output_shape) for t in all_attentions)
+            all_attentions = tuple([t.view(*attention_output_shape) for t in all_attentions])
            outputs = outputs + (all_attentions,)
        return outputs  # last hidden state, (presents), (all hidden_states), (attentions)

@@ -752,6 +754,7 @@ class GPT2DoubleHeadsModel(GPT2PreTrainedModel):
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)), mc_labels.view(-1))
            outputs = (loss,) + outputs
+
        if lm_labels is not None:
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
@@ -1708,10 +1708,7 @@ class Conv1D(nn.Module):
        self.bias = nn.Parameter(torch.zeros(nf))

    def forward(self, x):
-        size_out = x.size()[:-1] + (self.nf,)
-        x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
-        x = x.view(*size_out)
-        return x
+        return torch.baddbmm(self.bias[None, None], x, self.weight[None])


 class PoolerStartLogits(nn.Module):
@@ -152,8 +152,8 @@ class AdamW(Optimizer):

                # Decay the first and second moment running average coefficient
                # In-place operations to update the averages at the same time
-                exp_avg.mul_(beta1).add_(1.0 - beta1, grad)
-                exp_avg_sq.mul_(beta2).addcmul_(1.0 - beta2, grad, grad)
+                exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
+                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1.0 - beta2)
                denom = exp_avg_sq.sqrt().add_(group["eps"])

                step_size = group["lr"]
@@ -173,6 +173,6 @@ class AdamW(Optimizer):
                # of the weights to the loss with plain (non-momentum) SGD.
                # Add weight decay at the end (fixed version)
                if group["weight_decay"] > 0.0:
-                    p.data.add_(-group["lr"] * group["weight_decay"], p.data)
+                    p.data.add_(p.data, alpha=-group["lr"] * group["weight_decay"])

        return loss
@@ -217,7 +217,7 @@ class GradientAccumulator(object):
        """The accumulated gradients on the current replica."""
        if not self._gradients:
            raise ValueError("The accumulator should be called first to initialize the gradients")
-        return list(gradient.value() for gradient in self._gradients)
+        return list(gradient.value() if gradient is not None else gradient for gradient in self._gradients)

    def __call__(self, gradients):
        """Accumulates :obj:`gradients` on the current replica."""
@@ -231,6 +231,8 @@ class GradientAccumulator(object):
                        synchronization=tf.VariableSynchronization.ON_READ,
                        aggregation=tf.VariableAggregation.ONLY_FIRST_REPLICA,
                    )
+                    if gradient is not None
+                    else gradient
                    for gradient in gradients
                ]
            )
@@ -238,7 +240,8 @@ class GradientAccumulator(object):
            raise ValueError("Expected %s gradients, but got %d" % (len(self._gradients), len(gradients)))

        for accum_gradient, gradient in zip(self._gradients, gradients):
-            accum_gradient.assign_add(gradient)
+            if accum_gradient is not None and gradient is not None:
+                accum_gradient.assign_add(gradient)

        self._accum_steps.assign_add(1)

@@ -248,4 +251,5 @@ class GradientAccumulator(object):
            return
        self._accum_steps.assign(0)
        for gradient in self._gradients:
-            gradient.assign(tf.zeros_like(gradient))
+            if gradient is not None:
+                gradient.assign(tf.zeros_like(gradient))
@@ -24,7 +24,7 @@ from abc import ABC, abstractmethod
 from contextlib import contextmanager
 from itertools import chain
 from os.path import abspath, exists
-from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Sequence, Tuple, Union

 import numpy as np

@@ -58,6 +58,10 @@ if is_torch_available():
        AutoModelWithLMHead,
    )

+if TYPE_CHECKING:
+    from .modeling_utils import PreTrainedModel
+    from .modeling_tf_utils import TFPreTrainedModel
+

 logger = logging.getLogger(__name__)

@@ -1509,7 +1513,7 @@ class TranslationPipeline(Pipeline):
            return results


-# Register all the supported task here
+# Register all the supported tasks here
 SUPPORTED_TASKS = {
    "feature-extraction": {
        "impl": FeatureExtractionPipeline,
@@ -1572,9 +1576,9 @@ SUPPORTED_TASKS = {
        "tf": TFAutoModelWithLMHead if is_tf_available() else None,
        "pt": AutoModelWithLMHead if is_torch_available() else None,
        "default": {
-            "model": {"pt": "bart-large-cnn", "tf": None},
+            "model": {"pt": "bart-large-cnn", "tf": "t5-small"},
            "config": None,
-            "tokenizer": ("bart-large-cnn", {"use_fast": False}),
+            "tokenizer": {"pt": ("bart-large-cnn", {"use_fast": False}), "tf": "t5-small"},
        },
    },
    "translation_en_to_fr": {
@@ -124,9 +124,6 @@ class MarianTokenizer(PreTrainedTokenizer):
        # We don't expect to process pairs, but leave the pair logic for API consistency
        return token_ids_0 + token_ids_1 + [self.eos_token_id]

-    def batch_decode(self, token_ids, **kwargs) -> List[str]:
-        return [self.decode(ids, **kwargs) for ids in token_ids]
-
    def prepare_translation_batch(
        self,
        src_texts: List[str],
@@ -185,15 +185,6 @@ class BatchEncoding(UserDict):

        self._encodings = encoding

-    @property
-    def is_fast(self):
-        """
-        Indicate if this BatchEncoding was generated from the result of a PreTrainedTokenizerFast
-        Returns: True if generated from subclasses of PreTrainedTokenizerFast, else otherwise
-
-        """
-        return self._encodings is not None
-
    def __getitem__(self, item: Union[int, str]) -> EncodingFast:
        """ If the key is a string, get the value of the dict associated to `key` ('input_ids', 'attention_mask'...)
            If the key is an integer, get the EncodingFast for batch item with index `key`
@@ -211,16 +202,6 @@ class BatchEncoding(UserDict):
    def __getattr__(self, item: str):
        return self.data[item]

-    def __getstate__(self):
-        return {"data": self.data, "encodings": self._encodings}
-
-    def __setstate__(self, state):
-        if "data" in state:
-            self.data = state["data"]
-
-        if "encodings" in state:
-            self._encodings = state["encodings"]
-
    def keys(self):
        return self.data.keys()

@@ -243,7 +224,7 @@ class BatchEncoding(UserDict):
        """
        return self._encodings

-    def tokens(self, batch_index: int = 0) -> List[str]:
+    def tokens(self, batch_index: int = 0) -> List[int]:
        if not self._encodings:
            raise ValueError("tokens() is not available when using Python based tokenizers")
        return self._encodings[batch_index].tokens
@@ -2202,6 +2183,9 @@ class PreTrainedTokenizer(SpecialTokensMixin):
        else:
            return text

+    def batch_decode(self, sequences: List[List[int]], **kwargs) -> List[str]:
+        return [self.decode(seq, **kwargs) for seq in sequences]
+
    @staticmethod
    def clean_up_tokenization(out_string: str) -> str:
        """ Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.
@@ -679,7 +679,7 @@ class Trainer:
        model.eval()

        for inputs in tqdm(dataloader, desc=description):
-            has_labels = any(inputs.get(k) is not None for k in ["labels", "masked_lm_labels"])
+            has_labels = any(inputs.get(k) is not None for k in ["labels", "lm_labels", "masked_lm_labels"])

            for k, v in inputs.items():
                inputs[k] = v.to(self.args.device)
@@ -703,7 +703,7 @@ class Trainer:
                    else:
                        label_ids = np.append(label_ids, inputs["labels"].detach().cpu().numpy(), axis=0)

-        if is_tpu_available():
+        if is_tpu_available() and preds is not None and label_ids is not None:
            # tpu-comment: Get all predictions and labels from all worker shards of eval dataset
            preds = xm.mesh_reduce("eval_preds", preds, np.concatenate)
            label_ids = xm.mesh_reduce("eval_out_label_ids", label_ids, np.concatenate)
@@ -0,0 +1,116 @@
+import unittest
+from os import sep
+from os.path import dirname, exists
+from shutil import rmtree
+
+from tests.utils import require_tf, require_torch
+from transformers import BertConfig, BertTokenizerFast, FeatureExtractionPipeline
+from transformers.convert_graph_to_onnx import convert, ensure_valid_input, infer_shapes
+
+
+class FuncContiguousArgs:
+    def forward(self, input_ids, token_type_ids, attention_mask):
+        return None
+
+
+class FuncNonContiguousArgs:
+    def forward(self, input_ids, some_other_args, token_type_ids, attention_mask):
+        return None
+
+
+class OnnxExportTestCase(unittest.TestCase):
+    MODEL_TO_TEST = ["bert-base-cased", "gpt2", "roberta-base"]
+
+    @require_tf
+    def test_export_tensorflow(self):
+        for model in OnnxExportTestCase.MODEL_TO_TEST:
+            self._test_export(model, "tf", 11)
+
+    @require_torch
+    def test_export_pytorch(self):
+        for model in OnnxExportTestCase.MODEL_TO_TEST:
+            self._test_export(model, "pt", 11)
+
+    def _test_export(self, model, framework, opset):
+        try:
+            # Compute path
+            path = "onnx" + sep + model + ".onnx"
+
+            # Remove folder if exists
+            if exists(dirname(path)):
+                rmtree(dirname(path))
+
+            # Export
+            convert(framework, model, path, opset)
+        except Exception as e:
+            self.fail(e)
+
+    @require_torch
+    def test_infer_dynamic_axis_pytorch(self):
+        """
+        Validate the dynamic axis generated for each parameters are correct
+        """
+        from transformers import BertModel
+
+        model = BertModel(BertConfig.from_pretrained("bert-base-cased"))
+        tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased")
+        self._test_infer_dynamic_axis(model, tokenizer, "pt")
+
+    @require_tf
+    def test_infer_dynamic_axis_tf(self):
+        """
+        Validate the dynamic axis generated for each parameters are correct
+        """
+        from transformers import TFBertModel
+
+        model = TFBertModel(BertConfig.from_pretrained("bert-base-cased"))
+        tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased")
+        self._test_infer_dynamic_axis(model, tokenizer, "tf")
+
+    def _test_infer_dynamic_axis(self, model, tokenizer, framework):
+        nlp = FeatureExtractionPipeline(model, tokenizer)
+
+        variable_names = ["input_ids", "token_type_ids", "attention_mask", "output_0", "output_1"]
+        input_vars, output_vars, shapes, tokens = infer_shapes(nlp, framework)
+
+        # Assert all variables are present
+        self.assertEqual(len(shapes), len(variable_names))
+        self.assertTrue(all([var_name in shapes for var_name in variable_names]))
+        self.assertSequenceEqual(variable_names[:3], input_vars)
+        self.assertSequenceEqual(variable_names[3:], output_vars)
+
+        # Assert inputs are {0: batch, 1: sequence}
+        for var_name in ["input_ids", "token_type_ids", "attention_mask"]:
+            self.assertDictEqual(shapes[var_name], {0: "batch", 1: "sequence"})
+
+        # Assert outputs are {0: batch, 1: sequence} and {0: batch}
+        self.assertDictEqual(shapes["output_0"], {0: "batch", 1: "sequence"})
+        self.assertDictEqual(shapes["output_1"], {0: "batch"})
+
+    def test_ensure_valid_input(self):
+        """
+        Validate parameters are correctly exported
+        GPT2 has "past" parameter in the middle of input_ids, token_type_ids and attention_mask.
+        ONNX doesn't support export with a dictionary, only a tuple. Thus we need to ensure we remove
+        token_type_ids and attention_mask for now to not having a None tensor in the middle
+        """
+        # All generated args are valid
+        input_names = ["input_ids", "attention_mask", "token_type_ids"]
+        tokens = {"input_ids": [1, 2, 3, 4], "attention_mask": [0, 0, 0, 0], "token_type_ids": [1, 1, 1, 1]}
+        inputs_args = ensure_valid_input(FuncContiguousArgs(), tokens, input_names)
+
+        # Should have exactly the same number of args (all are valid)
+        self.assertEqual(len(inputs_args), 3)
+
+        # Parameter should be reordered according to their respective place in the function:
+        # (input_ids, token_type_ids, attention_mask)
+        self.assertEqual(inputs_args, (tokens["input_ids"], tokens["token_type_ids"], tokens["attention_mask"]))
+
+        # Generated args are interleaved with another args (for instance parameter "past" in GPT2)
+        inputs_args = ensure_valid_input(FuncNonContiguousArgs(), tokens, input_names)
+
+        # Should have exactly the one arg (all before the one not provided "some_other_args")
+        self.assertEqual(len(inputs_args), 1)
+
+        # Should have only "input_ids"
+        self.assertEqual(inputs_args[0], tokens["input_ids"])
@@ -2,94 +2,41 @@ import unittest
 from typing import Iterable, List, Optional

 from transformers import pipeline
-from transformers.pipelines import DefaultArgumentHandler, Pipeline
+from transformers.pipelines import SUPPORTED_TASKS, DefaultArgumentHandler, Pipeline

 from .utils import require_tf, require_torch, slow


-QA_FINETUNED_MODELS = [
-    (("bert-base-uncased", {"use_fast": False}), "bert-large-uncased-whole-word-masking-finetuned-squad", None),
-    (("distilbert-base-cased-distilled-squad", {"use_fast": False}), "distilbert-base-cased-distilled-squad", None),
+NER_FINETUNED_MODELS = ["sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english"]
+
+# xlnet-base-cased disabled for now, since it crashes TF2
+FEATURE_EXTRACT_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-cased"]
+TEXT_CLASSIF_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english"]
+TEXT_GENERATION_FINETUNED_MODELS = ["sshleifer/tiny-ctrl"]
+
+FILL_MASK_FINETUNED_MODELS = ["sshleifer/tiny-distilroberta-base"]
+LARGE_FILL_MASK_FINETUNED_MODELS = ["distilroberta-base"]  # @slow
+
+SUMMARIZATION_FINETUNED_MODELS = ["sshleifer/bart-tiny-random", "patrickvonplaten/t5-tiny-random"]
+TF_SUMMARIZATION_FINETUNED_MODELS = ["patrickvonplaten/t5-tiny-random"]
+
+TRANSLATION_FINETUNED_MODELS = [
+    ("patrickvonplaten/t5-tiny-random", "translation_en_to_de"),
+    ("patrickvonplaten/t5-tiny-random", "translation_en_to_ro"),
 ]
+TF_TRANSLATION_FINETUNED_MODELS = [("patrickvonplaten/t5-tiny-random", "translation_en_to_fr")]

-TF_QA_FINETUNED_MODELS = [
-    (("bert-base-uncased", {"use_fast": False}), "bert-large-uncased-whole-word-masking-finetuned-squad", None),
-    (("distilbert-base-cased-distilled-squad", {"use_fast": False}), "distilbert-base-cased-distilled-squad", None),
+expected_fill_mask_result = [
+    [
+        {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
+        {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
+    ],
+    [
+        {"sequence": "<s> The largest city in France is Paris</s>", "score": 0.3185044229030609, "token": 2201},
+        {"sequence": "<s> The largest city in France is Lyon</s>", "score": 0.21112334728240967, "token": 12790},
+    ],
 ]

-TF_NER_FINETUNED_MODELS = {
-    (
-        "bert-base-cased",
-        "dbmdz/bert-large-cased-finetuned-conll03-english",
-        "dbmdz/bert-large-cased-finetuned-conll03-english",
-    )
-}
-
-NER_FINETUNED_MODELS = {
-    (
-        "bert-base-cased",
-        "dbmdz/bert-large-cased-finetuned-conll03-english",
-        "dbmdz/bert-large-cased-finetuned-conll03-english",
-    )
-}
-
-FEATURE_EXTRACT_FINETUNED_MODELS = {
-    ("bert-base-cased", "bert-base-cased", None),
-    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
-    ("distilbert-base-cased", "distilbert-base-cased", None),
-}
-
-TF_FEATURE_EXTRACT_FINETUNED_MODELS = {
-    # ('xlnet-base-cased', 'xlnet-base-cased', None), # Disabled for now as it crash for TF2
-    ("distilbert-base-cased", "distilbert-base-cased", None),
-}
-
-TF_TEXT_CLASSIF_FINETUNED_MODELS = {
-    (
-        "bert-base-uncased",
-        "distilbert-base-uncased-finetuned-sst-2-english",
-        "distilbert-base-uncased-finetuned-sst-2-english",
-    )
-}
-
-TEXT_CLASSIF_FINETUNED_MODELS = {
-    (
-        "distilbert-base-cased",
-        "distilbert-base-uncased-finetuned-sst-2-english",
-        "distilbert-base-uncased-finetuned-sst-2-english",
-    )
-}
-
-TEXT_GENERATION_FINETUNED_MODELS = {
-    ("gpt2", "gpt2"),
-    ("xlnet-base-cased", "xlnet-base-cased"),
-}
-
-TF_TEXT_GENERATION_FINETUNED_MODELS = {
-    ("gpt2", "gpt2"),
-    ("xlnet-base-cased", "xlnet-base-cased"),
-}
-
-FILL_MASK_FINETUNED_MODELS = [
-    (("distilroberta-base", {"use_fast": False}), "distilroberta-base", None),
-]
-
-TF_FILL_MASK_FINETUNED_MODELS = [
-    (("distilroberta-base", {"use_fast": False}), "distilroberta-base", None),
-]
-
-SUMMARIZATION_FINETUNED_MODELS = {
-    ("sshleifer/bart-tiny-random", "bart-large-cnn"),
-    ("patrickvonplaten/t5-tiny-random", "t5-small"),
-}
-TF_SUMMARIZATION_FINETUNED_MODELS = {("patrickvonplaten/t5-tiny-random", "t5-small")}
-
-TRANSLATION_FINETUNED_MODELS = {
-    ("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_de"),
-    ("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_ro"),
-}
-TF_TRANSLATION_FINETUNED_MODELS = {("patrickvonplaten/t5-tiny-random", "t5-small", "translation_en_to_fr")}
-

 class DefaultArgumentHandlerTestCase(unittest.TestCase):
    def setUp(self) -> None:
@@ -168,8 +115,8 @@ class MonoColumnInputTestCase(unittest.TestCase):
        self,
        nlp: Pipeline,
        valid_inputs: List,
-        invalid_inputs: List,
        output_keys: Iterable[str],
+        invalid_inputs: List = [None],
        expected_multi_result: Optional[List] = None,
        expected_check_keys: Optional[List[str]] = None,
    ):
@@ -206,93 +153,61 @@ class MonoColumnInputTestCase(unittest.TestCase):
        self.assertRaises(Exception, nlp, invalid_inputs)

    @require_torch
-    def test_ner(self):
+    def test_torch_ner(self):
        mandatory_keys = {"entity", "word", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer)
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)

    @require_tf
    def test_tf_ner(self):
        mandatory_keys = {"entity", "word", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in TF_NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model, config=config, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)

    @require_torch
-    def test_sentiment_analysis(self):
+    def test_torch_sentiment_analysis(self):
        mandatory_keys = {"label", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in TEXT_CLASSIF_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
+        for model_name in TEXT_CLASSIF_FINETUNED_MODELS:
+            nlp = pipeline(task="sentiment-analysis", model=model_name, tokenizer=model_name)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)

    @require_tf
    def test_tf_sentiment_analysis(self):
        mandatory_keys = {"label", "score"}
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in TF_TEXT_CLASSIF_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
+        for model_name in TEXT_CLASSIF_FINETUNED_MODELS:
+            nlp = pipeline(task="sentiment-analysis", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys)

    @require_torch
-    def test_feature_extraction(self):
+    def test_torch_feature_extraction(self):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
+        for model_name in FEATURE_EXTRACT_FINETUNED_MODELS:
+            nlp = pipeline(task="feature-extraction", model=model_name, tokenizer=model_name)
+            self._test_mono_column_pipeline(nlp, valid_inputs, {})

    @require_tf
    def test_tf_feature_extraction(self):
        valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
-        invalid_inputs = [None]
-        for tokenizer, model, config in TF_FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
+        for model_name in FEATURE_EXTRACT_FINETUNED_MODELS:
+            nlp = pipeline(task="feature-extraction", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, {})

    @require_torch
-    def test_fill_mask(self):
+    def test_torch_fill_mask(self):
        mandatory_keys = {"sequence", "score", "token"}
        valid_inputs = [
            "My name is <mask>",
            "The largest city in France is <mask>",
        ]
-        invalid_inputs = [None]
-        expected_multi_result = [
-            [
-                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
-                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
-            ],
-            [
-                {
-                    "sequence": "<s> The largest city in France is Paris</s>",
-                    "score": 0.3185044229030609,
-                    "token": 2201,
-                },
-                {
-                    "sequence": "<s> The largest city in France is Lyon</s>",
-                    "score": 0.21112334728240967,
-                    "token": 12790,
-                },
-            ],
-        ]
-        for tokenizer, model, config in FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
-            self._test_mono_column_pipeline(
-                nlp,
-                valid_inputs,
-                invalid_inputs,
-                mandatory_keys,
-                expected_multi_result=expected_multi_result,
-                expected_check_keys=["sequence"],
-            )
+        for model_name in FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="pt", topk=2,)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, expected_check_keys=["sequence"])

    @require_tf
    def test_tf_fill_mask(self):
@@ -301,103 +216,117 @@ class MonoColumnInputTestCase(unittest.TestCase):
            "My name is <mask>",
            "The largest city in France is <mask>",
        ]
-        invalid_inputs = [None]
-        expected_multi_result = [
-            [
-                {"sequence": "<s> My name is:</s>", "score": 0.009954338893294334, "token": 35},
-                {"sequence": "<s> My name is John</s>", "score": 0.0080940006300807, "token": 610},
-            ],
-            [
-                {
-                    "sequence": "<s> The largest city in France is Paris</s>",
-                    "score": 0.3185044229030609,
-                    "token": 2201,
-                },
-                {
-                    "sequence": "<s> The largest city in France is Lyon</s>",
-                    "score": 0.21112334728240967,
-                    "token": 12790,
-                },
-            ],
+        for model_name in FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="tf", topk=2,)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, expected_check_keys=["sequence"])
+
+    @require_torch
+    @slow
+    def test_torch_fill_mask_results(self):
+        mandatory_keys = {"sequence", "score", "token"}
+        valid_inputs = [
+            "My name is <mask>",
+            "The largest city in France is <mask>",
        ]
-        for tokenizer, model, config in TF_FILL_MASK_FINETUNED_MODELS:
-            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, framework="tf", topk=2)
+        for model_name in LARGE_FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="pt", topk=2,)
            self._test_mono_column_pipeline(
                nlp,
                valid_inputs,
-                invalid_inputs,
                mandatory_keys,
-                expected_multi_result=expected_multi_result,
+                expected_multi_result=expected_fill_mask_result,
+                expected_check_keys=["sequence"],
+            )
+
+    @require_tf
+    @slow
+    def test_tf_fill_mask_results(self):
+        mandatory_keys = {"sequence", "score", "token"}
+        valid_inputs = [
+            "My name is <mask>",
+            "The largest city in France is <mask>",
+        ]
+        for model_name in LARGE_FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model_name, tokenizer=model_name, framework="tf", topk=2)
+            self._test_mono_column_pipeline(
+                nlp,
+                valid_inputs,
+                mandatory_keys,
+                expected_multi_result=expected_fill_mask_result,
                expected_check_keys=["sequence"],
            )

    @require_torch
-    def test_summarization(self):
+    def test_torch_summarization(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["summary_text"]
-        for model, tokenizer in SUMMARIZATION_FINETUNED_MODELS:
-            nlp = pipeline(task="summarization", model=model, tokenizer=tokenizer)
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, mandatory_keys,
-            )
+        for model in SUMMARIZATION_FINETUNED_MODELS:
+            nlp = pipeline(task="summarization", model=model, tokenizer=model)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)

    @require_tf
    def test_tf_summarization(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["summary_text"]
-        for model, tokenizer in TF_SUMMARIZATION_FINETUNED_MODELS:
-            nlp = pipeline(task="summarization", model=model, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, mandatory_keys,
-            )
+        for model_name in TF_SUMMARIZATION_FINETUNED_MODELS:
+            nlp = pipeline(task="summarization", model=model_name, tokenizer=model_name, framework="tf",)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)

    @require_torch
-    def test_translation(self):
+    def test_torch_translation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["translation_text"]
-        for model, tokenizer, task in TRANSLATION_FINETUNED_MODELS:
-            nlp = pipeline(task=task, model=model, tokenizer=tokenizer)
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, mandatory_keys,
-            )
+        for model_name, task in TRANSLATION_FINETUNED_MODELS:
+            nlp = pipeline(task=task, model=model_name, tokenizer=model_name)
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs)

    @require_tf
+    @slow
    def test_tf_translation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
        invalid_inputs = [4, "<mask>"]
        mandatory_keys = ["translation_text"]
-        for model, tokenizer, task in TF_TRANSLATION_FINETUNED_MODELS:
-            nlp = pipeline(task=task, model=model, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, mandatory_keys,
-            )
+        for model, task in TF_TRANSLATION_FINETUNED_MODELS:
+            nlp = pipeline(task=task, model=model, tokenizer=model, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, mandatory_keys, invalid_inputs=invalid_inputs)

    @require_torch
-    def test_text_generation(self):
+    def test_torch_text_generation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
-        invalid_inputs = [None]
-        for model, tokenizer in TEXT_GENERATION_FINETUNED_MODELS:
-            nlp = pipeline(task="text-generation", model=model, tokenizer=tokenizer, framework="pt")
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, {},
-            )
+        for model_name in TEXT_GENERATION_FINETUNED_MODELS:
+            nlp = pipeline(task="text-generation", model=model_name, tokenizer=model_name, framework="pt")
+            self._test_mono_column_pipeline(nlp, valid_inputs, {})

    @require_tf
    def test_tf_text_generation(self):
        valid_inputs = ["A string like this", ["list of strings entry 1", "list of strings v2"]]
-        invalid_inputs = [None]
-        for model, tokenizer in TF_TEXT_GENERATION_FINETUNED_MODELS:
-            nlp = pipeline(task="text-generation", model=model, tokenizer=tokenizer, framework="tf")
-            self._test_mono_column_pipeline(
-                nlp, valid_inputs, invalid_inputs, {},
-            )
+        for model_name in TEXT_GENERATION_FINETUNED_MODELS:
+            nlp = pipeline(task="text-generation", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_mono_column_pipeline(nlp, valid_inputs, {})


-class MultiColumnInputTestCase(unittest.TestCase):
-    def _test_multicolumn_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):
+QA_FINETUNED_MODELS = ["sshleifer/tiny-distilbert-base-cased-distilled-squad"]
+
+
+class QAPipelineTests(unittest.TestCase):
+    def _test_qa_pipeline(self, nlp):
+        output_keys = {"score", "answer", "start", "end"}
+        valid_inputs = [
+            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
+            {
+                "question": "In what field is HuggingFace working ?",
+                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
+            },
+        ]
+        invalid_inputs = [
+            {"question": "", "context": "This is a test to try empty question edge case"},
+            {"question": None, "context": "This is a test to try empty question edge case"},
+            {"question": "What is does with empty context ?", "context": ""},
+            {"question": "What is does with empty context ?", "context": None},
+        ]
        self.assertIsNotNone(nlp)

        mono_result = nlp(valid_inputs[0])
@@ -413,75 +342,33 @@ class MultiColumnInputTestCase(unittest.TestCase):
        for result in multi_result:
            for key in output_keys:
                self.assertIn(key, result)
-
-        self.assertRaises(Exception, nlp, invalid_inputs[0])
+        for bad_input in invalid_inputs:
+            self.assertRaises(Exception, nlp, bad_input)
        self.assertRaises(Exception, nlp, invalid_inputs)

    @require_torch
-    def test_question_answering(self):
-        mandatory_output_keys = {"score", "answer", "start", "end"}
-        valid_samples = [
-            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
-            {
-                "question": "In what field is HuggingFace working ?",
-                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
-            },
-        ]
-        invalid_samples = [
-            {"question": "", "context": "This is a test to try empty question edge case"},
-            {"question": None, "context": "This is a test to try empty question edge case"},
-            {"question": "What is does with empty context ?", "context": ""},
-            {"question": "What is does with empty context ?", "context": None},
-        ]
-
-        for tokenizer, model, config in QA_FINETUNED_MODELS:
-            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer)
-            self._test_multicolumn_pipeline(nlp, valid_samples, invalid_samples, mandatory_output_keys)
+    def test_torch_question_answering(self):
+        for model_name in QA_FINETUNED_MODELS:
+            nlp = pipeline(task="question-answering", model=model_name, tokenizer=model_name)
+            self._test_qa_pipeline(nlp)

    @require_tf
-    @slow
    def test_tf_question_answering(self):
-        mandatory_output_keys = {"score", "answer", "start", "end"}
-        valid_samples = [
-            {"question": "Where was HuggingFace founded ?", "context": "HuggingFace was founded in Paris."},
-            {
-                "question": "In what field is HuggingFace working ?",
-                "context": "HuggingFace is a startup based in New-York founded in Paris which is trying to solve NLP.",
-            },
-        ]
-        invalid_samples = [
-            {"question": "", "context": "This is a test to try empty question edge case"},
-            {"question": None, "context": "This is a test to try empty question edge case"},
-            {"question": "What is does with empty context ?", "context": ""},
-            {"question": "What is does with empty context ?", "context": None},
-        ]
-
-        for tokenizer, model, config in TF_QA_FINETUNED_MODELS:
-            nlp = pipeline(task="question-answering", model=model, config=config, tokenizer=tokenizer, framework="tf")
-            self._test_multicolumn_pipeline(nlp, valid_samples, invalid_samples, mandatory_output_keys)
+        for model_name in QA_FINETUNED_MODELS:
+            nlp = pipeline(task="question-answering", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_qa_pipeline(nlp)


 class PipelineCommonTests(unittest.TestCase):

-    pipelines = (
-        "ner",
-        "feature-extraction",
-        "question-answering",
-        "fill-mask",
-        "summarization",
-        "sentiment-analysis",
-        "translation_en_to_fr",
-        "translation_en_to_de",
-        "translation_en_to_ro",
-        "text-generation",
-    )
+    pipelines = SUPPORTED_TASKS.keys()

    @slow
    @require_tf
    def test_tf_defaults(self):
        # Test that pipelines can be correctly loaded without any argument
        for task in self.pipelines:
-            with self.subTest(msg="Testing Torch defaults with PyTorch and {}".format(task)):
+            with self.subTest(msg="Testing TF defaults with TF and {}".format(task)):
                pipeline(task, framework="tf")

    @slow
@@ -19,11 +19,21 @@ import pickle
 import shutil
 import tempfile
 from collections import OrderedDict
-from typing import Dict, Tuple, Union
+from typing import TYPE_CHECKING, Dict, Tuple, Union

 from tests.utils import require_tf, require_torch


+if TYPE_CHECKING:
+    from transformers import (
+        PretrainedConfig,
+        PreTrainedTokenizer,
+        PreTrainedTokenizerFast,
+        PreTrainedModel,
+        TFPreTrainedModel,
+    )
+
+
 def merge_model_tokenizer_mappings(
    model_mapping: Dict["PretrainedConfig", Union["PreTrainedModel", "TFPreTrainedModel"]],
    tokenizer_mapping: Dict["PretrainedConfig", Tuple["PreTrainedTokenizer", "PreTrainedTokenizerFast"]],
@@ -16,7 +16,7 @@

 import unittest

-from transformers import BertTokenizer, BertTokenizerFast, PreTrainedTokenizer
+from transformers import PreTrainedTokenizer
 from transformers.tokenization_gpt2 import GPT2Tokenizer

 from .utils import slow
@@ -39,34 +39,3 @@ class TokenizerUtilsTest(unittest.TestCase):
    @slow
    def test_pretrained_tokenizers(self):
        self.check_tokenizer_from_pretrained(GPT2Tokenizer)
-
-    def test_batch_encoding_pickle(self):
-        from pickle import loads, dumps
-
-        # Get a slow & a fast tokenizer
-        tok_slow = BertTokenizer.from_pretrained("bert-base-cased")
-        tok_fast = BertTokenizerFast.from_pretrained("bert-base-cased")
-
-        # Encode a sentence
-        be_slow = tok_slow.encode_plus("This is a dummy input sentence")
-        be_fast = tok_fast.encode_plus("This is a dummy input sentence")
-
-        # Make sure both are pickable
-        be_slow_data = dumps(be_slow)
-        be_fast_data = dumps(be_fast)
-
-        # Try to restore
-        be_slow_pickled = loads(be_slow_data)
-        be_fast_pickled = loads(be_fast_data)
-
-        # Ensure pickled objects keeps the is_fast attribute
-        self.assertFalse(be_slow_pickled.is_fast)
-        self.assertTrue(be_fast_pickled.is_fast)
-
-        # Ensure .data match
-        self.assertDictEqual(be_slow_pickled.data, be_slow.data)
-        self.assertDictEqual(be_fast_pickled.data, be_fast.data)
-
-        # Ensure .encodings match
-        self.assertIsNone(be_slow_pickled.encodings)
-        self.assertEqual(len(be_fast_pickled.encodings), len(be_fast.encodings))
Author	SHA1	Message	Date
Morgan Funtowicz	b55bc33ba0	Using chunk instead of split to avoid accessing instance variable.	2020-05-16 18:20:57 +02:00
Morgan Funtowicz	f604cbd0e2	Prefer using tuples instead of list for return types.	2020-05-16 16:49:27 +02:00
Morgan Funtowicz	30ef0ad019	Use constant value instead of math.sqrt(2.0 / pi) in gelu	2020-05-16 15:50:50 +02:00
Morgan Funtowicz	5d9a2273bb	Use positive indexing for flatten so it can be exported to ONNX.	2020-05-16 15:47:27 +02:00
Morgan Funtowicz	630adb2b6a	Avoid accessing shape and slices by using flatten in merge_heads	2020-05-16 15:47:27 +02:00
Morgan Funtowicz	23bb7b9449	Avoid reshaping operations in Conv1D by using baddbmm	2020-05-16 15:47:27 +02:00
Morgan Funtowicz	2e36f6d3d7	Make sure gelu_new component are not represented as integers.	2020-05-16 15:47:24 +02:00
Morgan Funtowicz	3ceb683378	Revert "Use torch/tf rsqrt to scale attentions instead of div and math" This reverts commit 54d46729	2020-05-16 15:45:46 +02:00
Morgan Funtowicz	5b01621675	Use torch/tf rsqrt to scale attentions instead of div and math	2020-05-16 15:44:39 +02:00
Morgan Funtowicz	dd132be1e3	Permuting only 2 axis should use transpose, as it always returns a view.	2020-05-16 15:42:16 +02:00
Morgan Funtowicz	3989328ce0	Do not allocate Softmax every forward call, use functional.	2020-05-16 15:42:15 +02:00
Morgan Funtowicz	252f635e11	Enable model export through ONNX runtime which some adjustments.	2020-05-16 15:42:09 +02:00
Julien Chaumond	3e0f062106	Fix addcmul_	2020-05-15 17:44:17 -04:00
Julien Chaumond	fc2a4c88ce	Fix: one more try	2020-05-15 17:38:48 -04:00
Julien Chaumond	55bda52555	Same fix for `addcmul_`	2020-05-15 17:23:48 -04:00
Julien Chaumond	ad02c961c6	Fix UserWarning: This overload of add_ is deprecated in pytorch==1.5.0	2020-05-15 17:09:11 -04:00
Julien Chaumond	15550ce0d1	[skip ci] remove local rank	2020-05-15 17:08:38 -04:00
Nikita	62427d0815	rerun notebook 02-transformers (#4341 )	2020-05-15 10:33:08 -04:00
Jared T Nielsen	34706ba050	Allow for None gradients in GradientAccumulator. (#4372 )	2020-05-15 09:52:00 -04:00
Lysandre Debut	edf9ac11d4	Should return overflowing information for the log (#4385 )	2020-05-15 09:49:11 -04:00
Funtowicz Morgan	b908f2e9dd	Attempt to unpin torch version for Github Action. (#4384 )	2020-05-15 15:47:15 +02:00
Julien Chaumond	af2e6bf87c	[examples] Streamline doc	2020-05-14 20:34:31 -04:00
Lysandre Debut	7defc6670f	p_mask in SQuAD pre-processing (#4049 ) * Better p_mask building * Adressing @mfuntowicz comments	2020-05-14 17:07:52 -04:00
Morgan Funtowicz	84894974bd	Updated ONNX notebook link in README.	2020-05-14 22:40:59 +02:00
Funtowicz Morgan	db0076a9df	Conversion script to export transformers models to ONNX IR. (#4253 ) * Added generic ONNX conversion script for PyTorch model. * WIP initial TF support. * TensorFlow/Keras ONNX export working. * Print framework version info * Add possibility to check the model is correctly loading on ONNX runtime. * Remove quantization option. * Specify ONNX opset version when exporting. * Formatting. * Remove unused imports. * Make functions more generally reusable from other part of the code. * isort happy. * flake happy * Export only feature-extraction for now * Correctly check inputs order / filter before export. * Removed task variable * Fix invalid args call in load_graph_from_args. * Fix invalid args call in convert. * Fix invalid args call in infer_shapes. * Raise exception and catch in caller function instead of exit. * Add 04-onnx-export.ipynb notebook * More WIP on the notebook * Remove unused imports * Simplify & remove unused constants. * Export with constant_folding in PyTorch * Let's try to put function args in the right order this time ... * Disable external_data_format temporary * ONNX notebook draft ready. * Updated notebooks charts + wording * Correct error while exporting last chart in notebook. * Adressing @LysandreJik comment. * Set ONNX opset to 11 as default value. * Set opset param mandatory * Added ONNX export unittests * Quality. * flake8 happy * Add keras2onnx dependency on extras["tf"] * Pin keras2onnx on github master to v1.6.5 * Second attempt. * Third attempt. * Use the right repo URL this time ... * Do the same for onnxconverter-common * Added keras2onnx and onnxconveter-common to 1.7.0 to supports TF2.2 * Correct commit hash. * Addressing PR review: Optimization are enabled by default. * Addressing PR review: small changes in the notebook * setup.py comment about keras2onnx versioning.	2020-05-14 16:35:52 -04:00
Suraj Patil	2d05480174	Fix trainer evaluation (#4363 ) * fix loss calculation in evaluation * fix evaluation on TPU when prediction_loss_only is True	2020-05-14 14:39:44 -04:00
Savaş YıldırımandJulien Chaumond	035678efdb	Create README.md (#4359 ) * Create README.md * Update model_cards/savasy/bert-base-turkish-squad/README.md Co-authored-by: Julien Chaumond <chaumond@gmail.com>	2020-05-14 14:07:32 -04:00
sy-wada	b9c9e05381	Create README.md (#4357 )	2020-05-14 14:06:10 -04:00
Sam Shleifer	9535bf1977	Tokenizer.batch_decode convenience method (#4159 )	2020-05-14 13:50:47 -04:00
Sam Shleifer	7822cd38a0	[tests] make pipelines tests faster with smaller models (#4238 ) covers torch and tf. Also fixes a failing @slow test	2020-05-14 13:36:02 -04:00
Julien Chaumond	448c467256	Fix: unpin flake8 and fix cs errors (#4367 ) * Fix: unpin flake8 and fix cs errors * Ok we still need to quote those	2020-05-14 13:14:26 -04:00
Julien Chaumond	c547f15a17	Use Filelock to ensure distributed barriers see context in https://github.com/huggingface/transformers/pull/4223	2020-05-14 11:58:32 -04:00