delete script

* Create README.md

Add newly trained `calbert-tiny-uncased` (complete rewrite with SentencePiece)

* Add Exbert link

* Apply suggestions from code review

Co-authored-by: Julien Chaumond <chaumond@gmail.com>

.github/ISSUE_TEMPLATE/question-help.md

@@ -1,6 +1,6 @@
 ---
 name: "❓ Questions & Help"
-about: Post your general questions on Stack Overflow tagged huggingface-transformers
+about: Post your general questions on the Hugging Face forum or Stack Overflow tagged huggingface-transformers
 title: ''
 labels: ''
 assignees: ''
@@ -11,19 +11,17 @@ assignees: ''
 
 <!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
      new models and benchmarks, and migration questions. For all other questions,
-     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
-     Tensorflow enthusiast can help you out. Make sure to tag your question with the
-     right deep learning framework as well as the huggingface-transformers tag: 
+     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
+     You can also try Stack Overflow (SO) where a whole community of PyTorch and
+     Tensorflow enthusiast can help you out. In this case, make sure to tag your
+     question with the right deep learning framework as well as the
+     huggingface-transformers tag: 
      https://stackoverflow.com/questions/tagged/huggingface-transformers 
-     
-     If your question wasn't answered after a period of time on Stack Overflow, you
-     can always open a question on GitHub. You should then link to the SO question 
-     that you posted.
      -->
 
 ## Details
 <!-- Description of your issue -->
 
-<!-- You should first ask your question on SO, and only if
+<!-- You should first ask your question on the forum or SO, and only if
      you didn't get an answer ask it here on GitHub. -->
-**A link to original question on Stack Overflow**:
+**A link to original question on Stack Overflow**:

docs/source/tokenizer_summary.rst

@@ -52,7 +52,7 @@ size of 267,735!
 
 A huge vocabulary size means a huge embedding matrix at the start of the model, which will cause memory problems.
 TransformerXL deals with it by using a special kind of embeddings called adaptive embeddings, but in general,
-transformers model rarely have a vocabulary size greater than 50,000, especially if they are trained on a single
+transformers models rarely have a vocabulary size greater than 50,000, especially if they are trained on a single
 language.
 
 So if tokenizing on words is unsatisfactory, we could go on the opposite direction and simply tokenize on characters.
@@ -69,7 +69,7 @@ decomposed as "annoying" and "ly". This is especially useful in agglutinative la
 form (almost) arbitrarily long complex words by stringing together some subwords.
 
 This allows the model to keep a reasonable vocabulary while still learning useful representations for common words or
-subwords. This also gives the ability to the model to process words it has never seen before, by decomposing them into
+subwords. This also enables the model to process words it has never seen before, by decomposing them into
 subwords it knows. For instance, the base :class:`~transformers.BertTokenizer` will tokenize "I have a new GPU!" like
 this:
 
@@ -110,7 +110,7 @@ splitting the training data into words, which can be a simple space tokenization
 (:doc:`GPT-2 <model_doc/gpt2>` and :doc:`Roberta <model_doc/roberta>` uses this for instance) or a rule-based tokenizer
 (:doc:`XLM <model_doc/xlm>` use Moses for most languages, as does :doc:`FlauBERT <model_doc/flaubert>`),
 
-:doc:`GPT <model_doc/gpt>` uses Spacy and ftfy) and, counts the frequency of each word in the training corpus.
+:doc:`GPT <model_doc/gpt>` uses Spacy and ftfy, and counts the frequency of each word in the training corpus.
 
 It then begins from the list of all characters, and will learn merge rules to form a new token from two symbols in the
 vocabulary until it has learned a vocabulary of the desired size (this is a hyperparameter to pick).
@@ -178,7 +178,7 @@ WordPiece is the subword tokenization algorithm used for :doc:`BERT <model_doc/b
 `this paper <https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf>`__. It relies
 on the same base as BPE, which is to initialize the vocabulary to every character present in the corpus and
 progressively learn a given number of merge rules, the difference is that it doesn't choose the pair that is the most
-frequent but the one that will maximize the likelihood on the corpus once merged. 
+frequent but the one that will maximize the likelihood on the corpus once merged.
 
 What does this mean? Well, in the previous example, it means we would only merge 'u' and 'g' if the probability of
 having 'ug' divided by the probability of having 'u' then 'g' is greater than for any other pair of symbols. It's
@@ -217,7 +217,7 @@ training corpus. You can then give a probability to each tokenization (which is
 tokens forming it) and pick the most likely one (or if you want to apply some data augmentation, you could sample one
 of the tokenization according to their probabilities).
 
-Those probabilities are what are used to define the loss that trains the tokenizer: if our corpus consists of the
+Those probabilities define the loss that trains the tokenizer: if our corpus consists of the
 words :math:`x_{1}, \dots, x_{N}` and if for the word :math:`x_{i}` we note :math:`S(x_{i})` the set of all possible
 tokenizations of :math:`x_{i}` (with the current vocabulary), then the loss is defined as
 
@@ -229,15 +229,15 @@ tokenizations of :math:`x_{i}` (with the current vocabulary), then the loss is d
 SentencePiece
 =============
 
-All the methods we have been looking at so far required some from of pretrokenization, which has a central problem: not
+All the methods we have been looking at so far required some form of pretokenization, which has a central problem: not
 all languages use spaces to separate words. This is a problem :doc:`XLM <model_doc/xlm>` solves by using specific
 pretokenizers for each of those languages (in this case, Chinese, Japanese and Thai). To solve this problem,
 SentencePiece (introduced in `this paper <https://arxiv.org/pdf/1808.06226.pdf>`__) treats the input as a raw stream,
 includes the space in the set of characters to use, then uses BPE or unigram to construct the appropriate vocabulary.
 
 That's why in the example we saw before using :class:`~transformers.XLNetTokenizer` (which uses SentencePiece), we had
-some '▁' characters, that represent spaces. Decoding a tokenized text is then super easy: we just have to concatenate
-all of them together and replace those '▁' by spaces.
+the '▁' character, that represents space. Decoding a tokenized text is then super easy: we just have to concatenate
+all of them together and replace '▁' with space.
 
 All transformers models in the library that use SentencePiece use it with unigram. Examples of models using it are
 :doc:`ALBERT <model_doc/albert>`, :doc:`XLNet <model_doc/xlnet>` or the :doc:`Marian framework <model_doc/marian>`.

model_cards/bashar-talafha/multi-dialect-bert-base-arabic/README.md

@@ -0,0 +1,63 @@
+# Multi-dialect-Arabic-BERT
+This is a repository of Multi-dialect Arabic BERT model.
+
+By [Mawdoo3-AI](https://ai.mawdoo3.com/). 
+
+<p align="center">
+    <br>
+    <img src="https://github.com/mawdoo3/Multi-dialect-Arabic-BERT/blob/master/multidialct_arabic_bert.png" alt="Background reference: http://www.qfi.org/wp-content/uploads/2018/02/Qfi_Infographic_Mother-Language_Final.pdf" width="500"/>
+    <br>
+<p>
+
+
+
+### About our Multi-dialect-Arabic-BERT model
+Instead of training the Multi-dialect Arabic BERT model from scratch, we initialized the weights of the model using [Arabic-BERT](https://github.com/alisafaya/Arabic-BERT) and trained it on 10M arabic tweets from the unlabled data of [The Nuanced Arabic Dialect Identification (NADI) shared task](https://sites.google.com/view/nadi-shared-task).
+
+### To cite this work
+We haven't published any paper on this work. Please cite this repository:
+```
+@inproceedings{talafha-etal-2020-nadi,
+    title ={{Multi-dialect Arabic BERT for Country-level Dialect Identification}},
+    author = {Talafha, Bashar, Ali, Mohammad, Za'ter, Muhy Eddin, Seelawi, Haitham, Tuffaha, Ibraheem, Samir, Mostafa, Farhan, Wael and Al-Natsheh, Hussein},
+    booktitle ={{Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP2020)}},
+    year = {2020},
+    address = {Barcelona, Spain}
+}
+```
+We will update the BibTeX once the paper published.
+
+### Usage
+The model weights can be loaded using `transformers` library by HuggingFace.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")
+model = AutoModel.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")
+```
+
+Example using `pipeline`:
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+    "fill-mask",
+    model="bashar-talafha/multi-dialect-bert-base-arabic ",
+    tokenizer="bashar-talafha/multi-dialect-bert-base-arabic "
+)
+
+fill_mask(" سافر الرحالة من مطار [MASK] ")
+```
+```
+[{'sequence': '[CLS] سافر الرحالة من مطار الكويت [SEP]', 'score': 0.08296813815832138, 'token': 3226},
+ {'sequence': '[CLS] سافر الرحالة من مطار دبي [SEP]', 'score': 0.05123933032155037, 'token': 4747},
+ {'sequence': '[CLS] سافر الرحالة من مطار مسقط [SEP]', 'score': 0.046838656067848206, 'token': 13205},
+ {'sequence': '[CLS] سافر الرحالة من مطار القاهرة [SEP]', 'score': 0.03234650194644928, 'token': 4003},
+ {'sequence': '[CLS] سافر الرحالة من مطار الرياض [SEP]', 'score': 0.02606341242790222, 'token': 2200}]
+```
+### Repository
+Please check the [original repository](https://github.com/mawdoo3/Multi-dialect-Arabic-BERT) for more information. 
+
+

model_cards/codegram/calbert-tiny-uncased/README.md

@@ -0,0 +1,91 @@
+---
+language: "ca"
+tags:
+  - lm-head
+  - masked-lm
+  - catalan
+  - exbert
+license: mit
+---
+
+# Calbert: a Catalan Language Model
+
+## Introduction
+
+CALBERT is an open-source language model for Catalan pretrained on the ALBERT architecture.
+
+It is now available on Hugging Face in its `tiny-uncased` version (the one you're looking at) and `base-uncased` as well, and was pretrained on the [OSCAR dataset](https://traces1.inria.fr/oscar/).
+
+For further information or requests, please go to the [GitHub repository](https://github.com/codegram/calbert)
+
+## Pre-trained models
+
+| Model                               | Arch.          | Training data          |
+| ----------------------------------- | -------------- | ---------------------- |
+| `codegram` / `calbert-tiny-uncased` | Tiny (uncased) | OSCAR (4.3 GB of text) |
+| `codegram` / `calbert-base-uncased` | Base (uncased) | OSCAR (4.3 GB of text) |
+
+## How to use Calbert with HuggingFace
+
+#### Load Calbert and its tokenizer:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("codegram/calbert-tiny-uncased")
+model = AutoModel.from_pretrained("codegram/calbert-tiny-uncased")
+
+model.eval() # disable dropout (or leave in train mode to finetune
+```
+
+#### Filling masks using pipeline
+
+```python
+from transformers import pipeline
+
+calbert_fill_mask  = pipeline("fill-mask", model="codegram/calbert-tiny-uncased", tokenizer="codegram/calbert-tiny-uncased")
+results = calbert_fill_mask("M'agrada [MASK] això")
+# results
+# [{'sequence': "[CLS] m'agrada molt aixo[SEP]", 'score': 0.4403671622276306, 'token': 61},
+#  {'sequence': "[CLS] m'agrada més aixo[SEP]", 'score': 0.050061386078596115, 'token': 43},
+#  {'sequence': "[CLS] m'agrada veure aixo[SEP]", 'score': 0.026286985725164413, 'token': 157},
+#  {'sequence': "[CLS] m'agrada bastant aixo[SEP]", 'score': 0.022483550012111664, 'token': 2143},
+#  {'sequence': "[CLS] m'agrada moltíssim aixo[SEP]", 'score': 0.014491282403469086, 'token': 4867}]
+
+```
+
+#### Extract contextual embedding features from Calbert output
+
+```python
+import torch
+# Tokenize in sub-words with SentencePiece
+tokenized_sentence = tokenizer.tokenize("M'és una mica igual")
+# ['▁m', "'", 'es', '▁una', '▁mica', '▁igual']
+
+# 1-hot encode and add special starting and end tokens
+encoded_sentence = tokenizer.encode(tokenized_sentence)
+# [2, 109, 7, 71, 36, 371, 1103, 3]
+# NB: Can be done in one step : tokenize.encode("M'és una mica igual")
+
+# Feed tokens to Calbert as a torch tensor (batch dim 1)
+encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
+embeddings, _ = model(encoded_sentence)
+embeddings.size()
+# torch.Size([1, 8, 312])
+embeddings.detach()
+# tensor([[[-0.2726, -0.9855,  0.9643,  ...,  0.3511,  0.3499, -0.1984],
+#         [-0.2824, -1.1693, -0.2365,  ..., -3.1866, -0.9386, -1.3718],
+#         [-2.3645, -2.2477, -1.6985,  ..., -1.4606, -2.7294,  0.2495],
+#         ...,
+#         [ 0.8800, -0.0244, -3.0446,  ...,  0.5148, -3.0903,  1.1879],
+#         [ 1.1300,  0.2425,  0.2162,  ..., -0.5722, -2.2004,  0.4045],
+#         [ 0.4549, -0.2378, -0.2290,  ..., -2.1247, -2.2769, -0.0820]]])
+```
+
+## Authors
+
+CALBERT was trained and evaluated by [Txus Bach](https://twitter.com/txustice), as part of [Codegram](https://www.codegram.com)'s applied research.
+
+<a href="https://huggingface.co/exbert/?model=codegram/calbert-tiny-uncased&modelKind=bidirectional&sentence=M%27agradaria%20força%20saber-ne%20més">
+	<img width="300px" src="https://hf-dinosaur.huggingface.co/exbert/button.png">
+</a>

model_cards/valhalla/t5-base-e2e-qg/README.md

@@ -0,0 +1,38 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "Python is a programming language. It is developed by Guido Van Rossum and released in 1991. </s>"
+license: "MIT"
+---
+
+## T5 for question-generation
+This is [t5-base](https://arxiv.org/abs/1910.10683) model trained for end-to-end question generation task. Simply input the text and the model will generate multile questions. 
+
+You can play with the model using the inference API, just put the text and see the results!
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+
+text = "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum \
+and first released in 1991, Python's design philosophy emphasizes code \
+readability with its notable use of significant whitespace."
+
+nlp = pipeline("e2e-qg", model="valhalla/t5-base-e2e-qg")
+nlp(text)
+=> [
+ 'Who created Python?',
+ 'When was Python first released?',
+ "What is Python's design philosophy?"
+]
+```

model_cards/valhalla/t5-base-qa-qg-hl/README.md

@@ -0,0 +1,50 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "generate question: <hl> 42 <hl> is the answer to life, the universe and everything. </s>"
+- text: "question: What is 42 context: 42 is the answer to life, the universe and everything. </s>"
+license: "MIT"
+---
+
+## T5 for multi-task QA and QG
+This is multi-task [t5-base](https://arxiv.org/abs/1910.10683) model trained for question answering and answer aware question generation tasks. 
+
+For question generation the answer spans are highlighted within the text with special highlight tokens (`<hl>`) and prefixed with 'generate question: '. For QA the input is processed like this `question: question_text context: context_text </s>` 
+
+You can play with the model using the inference API. Here's how you can use it
+
+For QG
+
+`generate question: <hl> 42 <hl> is the answer to life, the universe and everything. </s>`
+
+For QA
+
+`question: What is 42 context: 42 is the answer to life, the universe and everything. </s>`
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+nlp = pipeline("multitask-qa-qg", model="valhalla/t5-base-qa-qg-hl")
+
+# to generate questions simply pass the text
+nlp("42 is the answer to life, the universe and everything.")
+=> [{'answer': '42', 'question': 'What is the answer to life, the universe and everything?'}]
+
+# for qa pass a dict with "question" and "context"
+nlp({
+    "question": "What is 42 ?",
+    "context": "42 is the answer to life, the universe and everything."
+})
+=> 'the answer to life, the universe and everything'
+```

model_cards/valhalla/t5-base-qg-hl/README.md

@@ -0,0 +1,33 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "<hl> 42 <hl> is the answer to life, the universe and everything. </s>"
+- text: "Python is a programming language. It is developed by <hl> Guido Van Rossum <hl>. </s>"
+- text: "Although <hl> practicality <hl> beats purity </s>"
+license: "MIT"
+---
+
+## T5 for question-generation
+This is [t5-base](https://arxiv.org/abs/1910.10683) model trained for answer aware question generation task. The answer spans are highlighted within the text with special highlight tokens. 
+
+You can play with the model using the inference API, just highlight the answer spans with `<hl>` tokens and end the text with `</s>`. For example
+
+`<hl> 42 <hl> is the answer to life, the universe and everything. </s>`
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+nlp = pipeline("question-generation", model="valhalla/t5-base-qg-hl")
+nlp("42 is the answer to life, universe and everything.")
+=> [{'answer': '42', 'question': 'What is the answer to life, universe and everything?'}]
+```

model_cards/valhalla/t5-samll-qg-prepend/README.md

@@ -0,0 +1,36 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "answer: 42  context: 42 is the answer to life, the universe and everything. </s>"
+- text: "answer: Guido Van Rossum context: Python is a programming language. It is developed by Guido Van Rossum. </s>"
+- text: "answer: Explicit context: Explicit is better than implicit </s>"
+license: "MIT"
+---
+
+## T5 for question-generation
+This is [t5-small](https://arxiv.org/abs/1910.10683) model trained for answer aware question generation task. The answer text is prepended before the context text. 
+
+You can play with the model using the inference API, just get the input text in this format and see the results!
+`answer: answer_text context: context_text </s>`
+
+For example
+
+`answer: 42  context: 42 is the answer to life, the universe and everything. </s>`
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+nlp = pipeline("question-generation", qg_format="prepend")
+nlp("42 is the answer to life, universe and everything.")
+=> [{'answer': '42', 'question': 'What is the answer to life, universe and everything?'}]
+```

model_cards/valhalla/t5-small-e2e-qg/README.md

@@ -0,0 +1,38 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "Python is developed by Guido Van Rossum and released in 1991. </s>"
+license: "MIT"
+---
+
+## T5 for question-generation
+This is [t5-small](https://arxiv.org/abs/1910.10683) model trained for end-to-end question generation task. Simply input the text and the model will generate multile questions. 
+
+You can play with the model using the inference API, just put the text and see the results!
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+
+text = "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum \
+and first released in 1991, Python's design philosophy emphasizes code \
+readability with its notable use of significant whitespace."
+
+nlp = pipeline("e2e-qg")
+nlp(text)
+=> [
+ 'Who created Python?',
+ 'When was Python first released?',
+ "What is Python's design philosophy?"
+]
+```

model_cards/valhalla/t5-small-qa-qg-hl/README.md

@@ -0,0 +1,49 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "generate question: <hl> 42 <hl> is the answer to life, the universe and everything. </s>"
+- text: "question: What is 42 context: 42 is the answer to life, the universe and everything. </s>"
+license: "MIT"
+---
+
+## T5 for multi-task QA and QG
+This is multi-task [t5-small](https://arxiv.org/abs/1910.10683) model trained for question answering and answer aware question generation tasks. 
+
+For question generation the answer spans are highlighted within the text with special highlight tokens (`<hl>`) and prefixed with 'generate question: '. For QA the input is processed like this `question: question_text context: context_text </s>` 
+
+You can play with the model using the inference API. Here's how you can use it
+
+For QG
+
+`generate question: <hl> 42 <hl> is the answer to life, the universe and everything. </s>`
+
+For QA
+
+`question: What is 42 context: 42 is the answer to life, the universe and everything. </s>`
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+nlp = pipeline("multitask-qa-qg")
+
+# to generate questions simply pass the text
+nlp("42 is the answer to life, the universe and everything.")
+=> [{'answer': '42', 'question': 'What is the answer to life, the universe and everything?'}]
+
+# for qa pass a dict with "question" and "context"
+nlp({
+    "question": "What is 42 ?",
+    "context": "42 is the answer to life, the universe and everything."
+})
+=> 'the answer to life, the universe and everything'
+```

model_cards/valhalla/t5-small-qg-hl/README.md

@@ -0,0 +1,33 @@
+---
+datasets:
+- squad
+tags:
+- question-generation
+widget:
+- text: "<hl> 42 <hl> is the answer to life, the universe and everything. </s>"
+- text: "Python is a programming language. It is developed by <hl> Guido Van Rossum <hl>. </s>"
+- text: "Simple is better than <hl> complex <hl>. </s>"
+license: "MIT"
+---
+
+## T5 for question-generation
+This is [t5-small](https://arxiv.org/abs/1910.10683) model trained for answer aware question generation task. The answer spans are highlighted within the text with special highlight tokens. 
+
+You can play with the model using the inference API, just highlight the answer spans with `<hl>` tokens and end the text with `</s>`. For example
+
+`<hl> 42 <hl> is the answer to life, the universe and everything. </s>`
+
+For more deatils see [this](https://github.com/patil-suraj/question_generation) repo.
+
+### Model in action 🚀
+
+You'll need to clone the [repo](https://github.com/patil-suraj/question_generation).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb)
+
+```python3
+from pipelines import pipeline
+nlp = pipeline("question-generation")
+nlp("42 is the answer to life, universe and everything.")
+=> [{'answer': '42', 'question': 'What is the answer to life, universe and everything?'}]
+```

notebooks/05-benchmark.ipynb

@@ -6,7 +6,7 @@
       "name": "05-benchmark",
       "provenance": [],
       "collapsed_sections": [],
-      "authorship_tag": "ABX9TyNQ2BQG0erOGhTFF/2Mdn5a",
+      "authorship_tag": "ABX9TyOAUMA92fdE4FM6A349/FWI",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -272,7 +272,7 @@
         "colab_type": "text"
       },
       "source": [
-        "<a href=\"https://colab.research.google.com/github/huggingface/transformers/blob/add_benchmark_notebook/05_benchmark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+        "<a href=\"https://colab.research.google.com/github/huggingface/transformers/blob/update_notebook/notebooks/05_benchmark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
@@ -312,8 +312,8 @@
         ":-- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n",
         "**Speed - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ | ✔ |\n",
         "**Memory - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ | ✘ |\n",
-        "**Speed - Train** | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |\n",
-        "**Memory - Train** | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ |\n",
+        "**Speed - Train** | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✔ |\n",
+        "**Memory - Train** | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✘ |\n",
         "\n",
         "*   *eager execution* means that the function is run in the eager execution environment of TensorFlow 2, see [here](https://www.tensorflow.org/guide/eager).\n",
         "\n",
@@ -321,7 +321,7 @@
         "\n",
         "*   *FP16* stands for TensorFlow's mixed-precision package and is analogous to PyTorch's FP16 feature, see [here](https://www.tensorflow.org/guide/mixed_precision).\n",
         "\n",
-        "***Note***: In ~1,2 weeks it will also be possible to benchmark training in TensorFlow.\n",
+        "***Note***: Benchmark training in TensorFlow is not included in v3.0.2, but available in master.\n",
         "\n",
         "\n",
         "This notebook will show the user how to use `PyTorchBenchmark` and `TensorFlowBenchmark` for two different scenarios:\n",
@@ -407,7 +407,7 @@
         " print(\"GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB\".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))\n",
         "printm()"
       ],
-      "execution_count": 1,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -431,7 +431,7 @@
         "# If GPU RAM Util > 0% => crash notebook on purpose\n",
         "# !kill -9 -1"
       ],
-      "execution_count": 2,
+      "execution_count": null,
       "outputs": []
     },
     {
@@ -499,7 +499,7 @@
       "source": [
         "!python run_benchmark.py --help"
       ],
-      "execution_count": 4,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -615,7 +615,7 @@
         "# create plots folder in content\n",
         "!mkdir -p plots_pt"
       ],
-      "execution_count": 5,
+      "execution_count": null,
       "outputs": []
     },
     {
@@ -638,7 +638,7 @@
         "                                --inference_memory_csv_file plots_pt/required_memory.csv \\\n",
         "                                --env_info_csv_file plots_pt/env.csv >/dev/null 2>&1  # redirect all prints"
       ],
-      "execution_count": 6,
+      "execution_count": null,
       "outputs": []
     },
     {
@@ -666,7 +666,7 @@
         "df = pd.read_csv('plots_pt/required_memory.csv')\n",
         "df"
       ],
-      "execution_count": 7,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "execute_result",
@@ -901,7 +901,7 @@
         "df = pd.read_csv('plots_pt/env.csv')\n",
         "df"
       ],
-      "execution_count": 8,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "execute_result",
@@ -1086,7 +1086,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "22499f33-bafc-42b3-f1b7-fcb202df9cd2"
       },
@@ -1098,7 +1098,7 @@
         "from IPython.display import Image\n",
         "Image('plots_pt/required_memory_plot.png')"
       ],
-      "execution_count": 9,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1133,7 +1133,7 @@
         "In short, all memory that is allocated for a given *model identifier*, *batch size* and *sequence length* is measured in a separate process. This way it can be ensured that there is no previously unreleased memory falsely included in the measurement. One should also note that the measured memory even includes the memory allocated by the CUDA driver to load PyTorch and TensorFlow and is, therefore, higher than library-specific memory measurement function, *e.g.* this one for [PyTorch](https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated).\n",
         "\n",
         "Alright, let's analyze the results. It can be noted that the models `aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2` and `deepset/roberta-base-squad2` require significantly less memory than the other three models. Besides `mrm8488/longformer-base-4096-finetuned-squadv2` all models more or less follow the same memory consumption pattern with `aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2` seemingly being able to better scale to larger sequence lengths. \n",
-        "`mrm8488/longformer-base-4096-finetuned-squadv2` is a *Longformer* model, which makes use of *LocalAttention* (check this blog post to learn more about local attention) so that the model scales much better to longer input sequences.\n",
+        "`mrm8488/longformer-base-4096-finetuned-squadv2` is a *Longformer* model, which makes use of *LocalAttention* (check [this](https://huggingface.co/blog/reformer) blog post to learn more about local attention) so that the model scales much better to longer input sequences.\n",
         "\n",
         "For the sake of this notebook, we assume that the longest required input will be less than 512 tokens so that we settle on the models `aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2` and `deepset/roberta-base-squad2`. \n",
         "\n",
@@ -1161,7 +1161,7 @@
         "                                --batch_sizes 64 128 256 512\\\n",
         "                                --no_env_print"
       ],
-      "execution_count": 10,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1207,7 +1207,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "092c4dac-5002-4603-8eba-cd4bca727744"
       },
@@ -1223,7 +1223,7 @@
         "from IPython.display import Image\n",
         "Image('plots_pt/required_memory_plot_2.png')"
       ],
-      "execution_count": 11,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1283,7 +1283,7 @@
         "                                --batch_sizes 64 128 256 512 \\\n",
         "                                --no_env_print \\"
       ],
-      "execution_count": 12,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1351,7 +1351,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "3947ccf0-b91c-43bf-8569-d6afe0232185"
       },
@@ -1363,7 +1363,7 @@
         "from IPython.display import Image\n",
         "Image('plots_tf/required_memory_plot_2.png')"
       ],
-      "execution_count": 13,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1420,7 +1420,7 @@
         "                                --batch_sizes 256 \\\n",
         "                                --no_env_print \\"
       ],
-      "execution_count": 14,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1454,7 +1454,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "152f14c7-288a-4471-9cc0-5108cb24804c"
       },
@@ -1466,7 +1466,7 @@
         "from IPython.display import Image\n",
         "Image('plots_tf/time_plot_2.png')"
       ],
-      "execution_count": 15,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1523,7 +1523,7 @@
         "                                --no_env_print \\\n",
         "                                --use_xla"
       ],
-      "execution_count": 16,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1584,7 +1584,7 @@
         "# Imports\n",
         "from transformers import BartConfig, PyTorchBenchmark, PyTorchBenchmarkArguments"
       ],
-      "execution_count": 17,
+      "execution_count": null,
       "outputs": []
     },
     {
@@ -1622,7 +1622,7 @@
       "source": [
         "BartConfig.from_pretrained(\"facebook/bart-large-mnli\").to_diff_dict()"
       ],
-      "execution_count": 18,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "display_data",
@@ -1720,7 +1720,7 @@
         "config_10000_vocab = BartConfig.from_pretrained(\"facebook/bart-large-mnli\", vocab_size=10000)\n",
         "config_8_layers = BartConfig.from_pretrained(\"facebook/bart-large-mnli\", encoder_layers=8, decoder_layers=8)"
       ],
-      "execution_count": 19,
+      "execution_count": null,
       "outputs": []
     },
     {
@@ -1770,7 +1770,7 @@
         "# run benchmark\n",
         "result = benchmark.run()"
       ],
-      "execution_count": 20,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1829,7 +1829,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "5dbeb7f7-c996-4db2-a560-735354a5b76f"
       },
@@ -1841,7 +1841,7 @@
         "from IPython.display import Image\n",
         "Image('plots_pt/training_mem_fp16.png')"
       ],
-      "execution_count": 21,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1916,7 +1916,7 @@
         "# run benchmark\n",
         "result = benchmark.run()"
       ],
-      "execution_count": 22,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -1961,7 +1961,7 @@
         "colab_type": "code",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 514
+          "height": 534
         },
         "outputId": "8a4b4db7-abed-47c4-da61-c3b1ccae66f1"
       },
@@ -1973,7 +1973,7 @@
         "from IPython.display import Image\n",
         "Image('plots_pt/training_speed_fp16.png')"
       ],
-      "execution_count": 23,
+      "execution_count": null,
       "outputs": [
         {
           "output_type": "stream",
@@ -2017,7 +2017,7 @@
       "source": [
         "Alright, that's it! Now you should be able to benchmark your favorite models on your favorite configurations. \n",
         "\n",
-        "Transparency for the computational cost of a model is becoming more and more important. Feel free to share your results with the community on a shared spreadsheet or by tweeting us @huggingface 🤗."
+        "Feel free to share your results with the community [here](https://github.com/huggingface/transformers/blob/master/examples/benchmarking/README.md) or by tweeting us https://twitter.com/HuggingFace 🤗."
       ]
     }
   ]

src/transformers/benchmark/benchmark.py

@@ -157,7 +157,7 @@ class PyTorchBenchmark(Benchmark):
         else:
             train_model = model
 
-        model.eval()
+        model.train()
         model.to(self.args.device)
 
         # encoder-decoder has vocab size saved differently
@@ -175,12 +175,12 @@ class PyTorchBenchmark(Benchmark):
         def compute_loss_and_backprob_encoder():
             loss = train_model(input_ids, labels=input_ids)[0]
             loss.backward()
-            train_model.zero_grad()
+            return loss
 
         def compute_loss_and_backprob_encoder_decoder():
             loss = train_model(input_ids, decoder_input_ids=input_ids, labels=input_ids)[0]
             loss.backward()
-            train_model.zero_grad()
+            return loss
 
         _train = (
             compute_loss_and_backprob_encoder_decoder

src/transformers/benchmark/benchmark_tf.py

@@ -24,7 +24,13 @@ import timeit
 from functools import wraps
 from typing import Callable, Optional
 
-from transformers import TF_MODEL_MAPPING, PretrainedConfig, is_py3nvml_available, is_tf_available
+from transformers import (
+    TF_MODEL_MAPPING,
+    TF_MODEL_WITH_LM_HEAD_MAPPING,
+    PretrainedConfig,
+    is_py3nvml_available,
+    is_tf_available,
+)
 
 from .benchmark_utils import (
     Benchmark,
@@ -92,10 +98,11 @@ class TensorFlowBenchmark(Benchmark):
         _inference = self._prepare_inference_func(model_name, batch_size, sequence_length)
         return self._measure_speed(_inference)
 
-    def _train_speed(self, model_name, batch_size, sequence_length):
-        raise NotImplementedError(
-            "Training is currently not really implemented." "Wait for TFTrainer to support CLM and MLM."
-        )
+    def _train_speed(self, model_name: str, batch_size: int, sequence_length: int) -> float:
+        strategy = self.args.strategy
+        assert strategy is not None, "A device strategy has to be initialized before using TensorFlow."
+        _train = self._prepare_train_func(model_name, batch_size, sequence_length)
+        return self._measure_speed(_train)
 
     def _inference_memory(
         self, model_name: str, batch_size: int, sequence_length: int
@@ -108,10 +115,16 @@ class TensorFlowBenchmark(Benchmark):
         _inference = self._prepare_inference_func(model_name, batch_size, sequence_length)
         return self._measure_memory(_inference)
 
-    def _train_memory(self, model_name, batch_size, sequence_length):
-        raise NotImplementedError(
-            "Training is currently not really implemented. Wait for TFTrainer to support CLM and MLM."
-        )
+    def _train_memory(
+        self, model_name: str, batch_size: int, sequence_length: int
+    ) -> [Memory, Optional[MemorySummary]]:
+        if self.args.is_gpu:
+            tf.config.experimental.set_memory_growth(self.args.gpu_list[self.args.device_idx], True)
+        strategy = self.args.strategy
+        assert strategy is not None, "A device strategy has to be initialized before using TensorFlow."
+
+        _train = self._prepare_train_func(model_name, batch_size, sequence_length)
+        return self._measure_memory(_train)
 
     def _prepare_inference_func(self, model_name: str, batch_size: int, sequence_length: int) -> Callable[[], None]:
         config = self.config_dict[model_name]
@@ -149,6 +162,50 @@ class TensorFlowBenchmark(Benchmark):
 
         return _inference
 
+    def _prepare_train_func(self, model_name: str, batch_size: int, sequence_length: int) -> Callable[[], None]:
+        config = self.config_dict[model_name]
+
+        assert (
+            self.args.eager_mode is False
+        ), "Training cannot be done in eager mode. Please make sure that `args.eager_mode = False`."
+
+        if self.args.fp16:
+            raise NotImplementedError("Mixed precision is currently not supported.")
+
+        has_model_class_in_config = hasattr(config, "architecture") and len(config.architectures) > 1
+        if not self.args.only_pretrain_model and has_model_class_in_config:
+            try:
+                model_class = "TF" + config.architectures[0]  # prepend 'TF' for tensorflow model
+                transformers_module = __import__("transformers", fromlist=[model_class])
+                model_cls = getattr(transformers_module, model_class)
+                model = model_cls(config)
+            except ImportError:
+                raise ImportError(
+                    f"{model_class} does not exist. If you just want to test the pretrained model, you might want to set `--only_pretrain_model` or `args.only_pretrain_model=True`."
+                )
+        else:
+            model = TF_MODEL_WITH_LM_HEAD_MAPPING[config.__class__](config)
+
+        # encoder-decoder has vocab size saved differently
+        vocab_size = config.vocab_size if hasattr(config, "vocab_size") else config.encoder.vocab_size
+        input_ids = random_input_ids(batch_size, sequence_length, vocab_size)
+
+        @run_with_tf_optimizations(self.args.eager_mode, self.args.use_xla)
+        def encoder_decoder_train():
+            loss = model(input_ids, decoder_input_ids=input_ids, labels=input_ids, training=True)[0]
+            gradients = tf.gradients(loss, model.trainable_variables)
+            return gradients
+
+        @run_with_tf_optimizations(self.args.eager_mode, self.args.use_xla)
+        def encoder_train():
+            loss = model(input_ids, labels=input_ids, training=True)[0]
+            gradients = tf.gradients(loss, model.trainable_variables)
+            return gradients
+
+        _train = encoder_decoder_train if config.is_encoder_decoder else encoder_train
+
+        return _train
+
     def _measure_speed(self, func) -> float:
         with self.args.strategy.scope():
             try:

src/transformers/pipelines.py

@@ -1003,8 +1003,6 @@ class TokenClassificationPipeline(Pipeline):
             labels_idx = score.argmax(axis=-1)
 
             entities = []
-            entity_groups = []
-            entity_group_disagg = []
             # Filter to labels not in `self.ignore_labels`
             filtered_labels_idx = [
                 (idx, label_idx)
@@ -1020,37 +1018,13 @@ class TokenClassificationPipeline(Pipeline):
                     "entity": self.model.config.id2label[label_idx],
                     "index": idx,
                 }
-                last_idx, _ = filtered_labels_idx[-1]
-                if self.grouped_entities:
-                    if not entity_group_disagg:
-                        entity_group_disagg += [entity]
-                        if idx == last_idx:
-                            entity_groups += [self.group_entities(entity_group_disagg)]
-                        continue
-
-                    # If the current entity is similar and adjacent to the previous entity, append it to the disaggregated entity group
-                    if (
-                        entity["entity"] == entity_group_disagg[-1]["entity"]
-                        and entity["index"] == entity_group_disagg[-1]["index"] + 1
-                    ):
-                        entity_group_disagg += [entity]
-                        # Group the entities at the last entity
-                        if idx == last_idx:
-                            entity_groups += [self.group_entities(entity_group_disagg)]
-                    # If the current entity is different from the previous entity, aggregate the disaggregated entity group
-                    else:
-                        entity_groups += [self.group_entities(entity_group_disagg)]
-                        entity_group_disagg = [entity]
 
                 entities += [entity]
 
-            # Ensure if an entity is the latest one in the sequence it gets appended to the output
-            if len(entity_group_disagg) > 0:
-                entity_groups.append(self.group_entities(entity_group_disagg))
-
-            # Append
+            # Append grouped entities
             if self.grouped_entities:
-                answers += [entity_groups]
+                answers += [self.group_entities(entities)]
+            # Append ungrouped entities
             else:
                 answers += [entities]
 
@@ -1058,12 +1032,12 @@ class TokenClassificationPipeline(Pipeline):
             return answers[0]
         return answers
 
-    def group_entities(self, entities):
+    def group_sub_entities(self, entities: List[dict]) -> dict:
         """
-        Returns grouped entities
+        Returns grouped sub entities
         """
-        # Get the last entity in the entity group
-        entity = entities[-1]["entity"]
+        # Get the first entity in the entity group
+        entity = entities[0]["entity"]
         scores = np.mean([entity["score"] for entity in entities])
         tokens = [entity["word"] for entity in entities]
 
@@ -1074,6 +1048,45 @@ class TokenClassificationPipeline(Pipeline):
         }
         return entity_group
 
+    def group_entities(self, entities: List[dict]) -> List[dict]:
+        """
+        Returns grouped entities
+        """
+
+        entity_groups = []
+        entity_group_disagg = []
+
+        if entities:
+            last_idx = entities[-1]["index"]
+
+        for entity in entities:
+            is_last_idx = entity["index"] == last_idx
+            if not entity_group_disagg:
+                entity_group_disagg += [entity]
+                if is_last_idx:
+                    entity_groups += [self.group_sub_entities(entity_group_disagg)]
+                continue
+
+            # If the current entity is similar and adjacent to the previous entity, append it to the disaggregated entity group
+            # The split is meant to account for the "B" and "I" suffixes
+            if (
+                entity["entity"].split("-")[-1] == entity_group_disagg[-1]["entity"].split("-")[-1]
+                and entity["index"] == entity_group_disagg[-1]["index"] + 1
+            ):
+                entity_group_disagg += [entity]
+                # Group the entities at the last entity
+                if is_last_idx:
+                    entity_groups += [self.group_sub_entities(entity_group_disagg)]
+            # If the current entity is different from the previous entity, aggregate the disaggregated entity group
+            else:
+                entity_groups += [self.group_sub_entities(entity_group_disagg)]
+                entity_group_disagg = [entity]
+                # If it's the last entity, add it to the entity groups
+                if is_last_idx:
+                    entity_groups += [self.group_sub_entities(entity_group_disagg)]
+
+        return entity_groups
+
 
 NerPipeline = TokenClassificationPipeline
 

src/transformers/trainer.py

@@ -820,15 +820,29 @@ class Trainer:
                 inputs["mems"] = past
 
             with torch.no_grad():
-                outputs = model(**inputs)
-                if has_labels:
-                    step_eval_loss, logits = outputs[:2]
-                    eval_losses += [step_eval_loss.mean().item()]
-                else:
-                    logits = outputs[0]
-                if self.args.past_index >= 0:
-                    past = outputs[self.args.past_index if has_labels else self.args.past_index - 1]
+                if self.args.predict_from_generate:
+                    max_length = model.config.max_length
+                    logits_out = model.generate(inputs["input_ids"], attention_mask=inputs["attention_mask"])
+                    # in case the batch is shorter then max length, the output should be padded
+                    logits = model.config.eos_token_id * torch.ones(
+                        (logits_out.shape[0], max_length), dtype=logits_out.dtype, device=logits_out.device
+                    )
+                    logits[:, : logits_out.shape[-1]] = logits_out
 
+                    if has_labels:
+                        outputs = model(**inputs)
+                        step_eval_loss = outputs[0]
+                        eval_losses += [step_eval_loss.mean().item()]
+                else:
+                    outputs = model(**inputs)
+
+                    if has_labels:
+                        step_eval_loss, logits = outputs[:2]
+                        eval_losses += [step_eval_loss.mean().item()]
+                    else:
+                        logits = outputs[0]
+                    if self.args.past_index >= 0:
+                        past = outputs[self.args.past_index if has_labels else self.args.past_index - 1]
             if not prediction_loss_only:
                 if preds is None:
                     preds = logits.detach()

src/transformers/training_args.py

@@ -157,6 +157,12 @@ class TrainingArguments:
         default=1,
         metadata={"help": "Number of updates steps to accumulate before performing a backward/update pass."},
     )
+    predict_from_generate: bool = field(
+        default=False,
+        metadata={
+            "help": "Use generate function to predict logits. This is usually the case for summarization or translation."
+        },
+    )
 
     learning_rate: float = field(default=5e-5, metadata={"help": "The initial learning rate for Adam."})
     weight_decay: float = field(default=0.0, metadata={"help": "Weight decay if we apply some."})

tests/test_benchmark_tf.py

@@ -100,6 +100,37 @@ class TFBenchmarkTest(unittest.TestCase):
         self.check_results_dict_not_empty(results.time_inference_result)
         self.check_results_dict_not_empty(results.memory_inference_result)
 
+    def test_train_no_configs(self):
+        MODEL_ID = "sshleifer/tiny-gpt2"
+        benchmark_args = TensorFlowBenchmarkArguments(
+            models=[MODEL_ID],
+            training=True,
+            no_inference=True,
+            sequence_lengths=[8],
+            batch_sizes=[1],
+            no_multi_process=True,
+        )
+        benchmark = TensorFlowBenchmark(benchmark_args)
+        results = benchmark.run()
+        self.check_results_dict_not_empty(results.time_train_result)
+        self.check_results_dict_not_empty(results.memory_train_result)
+
+    def test_train_with_configs(self):
+        MODEL_ID = "sshleifer/tiny-gpt2"
+        config = AutoConfig.from_pretrained(MODEL_ID)
+        benchmark_args = TensorFlowBenchmarkArguments(
+            models=[MODEL_ID],
+            training=True,
+            no_inference=True,
+            sequence_lengths=[8],
+            batch_sizes=[1],
+            no_multi_process=True,
+        )
+        benchmark = TensorFlowBenchmark(benchmark_args, [config])
+        results = benchmark.run()
+        self.check_results_dict_not_empty(results.time_train_result)
+        self.check_results_dict_not_empty(results.memory_train_result)
+
     def test_inference_encoder_decoder_with_configs(self):
         MODEL_ID = "patrickvonplaten/t5-tiny-random"
         config = AutoConfig.from_pretrained(MODEL_ID)

tests/test_pipelines.py

@@ -38,6 +38,7 @@ expected_fill_mask_result = [
         {"sequence": "<s>The largest city in France is Lyon</s>", "score": 0.21112334728240967, "token": 12790},
     ],
 ]
+
 SUMMARIZATION_KWARGS = dict(num_beams=2, min_length=2, max_length=5)
 
 
@@ -156,34 +157,6 @@ class MonoColumnInputTestCase(unittest.TestCase):
 
         self.assertRaises(Exception, nlp, invalid_inputs)
 
-    @require_torch
-    def test_torch_ner(self):
-        mandatory_keys = {"entity", "word", "score"}
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name)
-            self._test_mono_column_pipeline(nlp, VALID_INPUTS, mandatory_keys)
-
-    @require_torch
-    def test_ner_grouped(self):
-        mandatory_keys = {"entity_group", "word", "score"}
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, grouped_entities=True)
-            self._test_mono_column_pipeline(nlp, VALID_INPUTS, mandatory_keys)
-
-    @require_tf
-    def test_tf_ner(self):
-        mandatory_keys = {"entity", "word", "score"}
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf")
-            self._test_mono_column_pipeline(nlp, VALID_INPUTS, mandatory_keys)
-
-    @require_tf
-    def test_tf_ner_grouped(self):
-        mandatory_keys = {"entity_group", "word", "score"}
-        for model_name in NER_FINETUNED_MODELS:
-            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf", grouped_entities=True)
-            self._test_mono_column_pipeline(nlp, VALID_INPUTS, mandatory_keys)
-
     @require_torch
     def test_torch_sentiment_analysis(self):
         mandatory_keys = {"label", "score"}
@@ -393,6 +366,100 @@ class QAPipelineTests(unittest.TestCase):
             self._test_qa_pipeline(nlp)
 
 
+class NerPipelineTests(unittest.TestCase):
+    def _test_ner_pipeline(
+        self, nlp: Pipeline, output_keys: Iterable[str],
+    ):
+
+        ungrouped_ner_inputs = [
+            [
+                {"entity": "B-PER", "index": 1, "score": 0.9994944930076599, "word": "Cons"},
+                {"entity": "B-PER", "index": 2, "score": 0.8025449514389038, "word": "##uelo"},
+                {"entity": "I-PER", "index": 3, "score": 0.9993102550506592, "word": "Ara"},
+                {"entity": "I-PER", "index": 4, "score": 0.9993743896484375, "word": "##új"},
+                {"entity": "I-PER", "index": 5, "score": 0.9992871880531311, "word": "##o"},
+                {"entity": "I-PER", "index": 6, "score": 0.9993029236793518, "word": "No"},
+                {"entity": "I-PER", "index": 7, "score": 0.9981776475906372, "word": "##guera"},
+                {"entity": "B-PER", "index": 15, "score": 0.9998136162757874, "word": "Andrés"},
+                {"entity": "I-PER", "index": 16, "score": 0.999740719795227, "word": "Pas"},
+                {"entity": "I-PER", "index": 17, "score": 0.9997414350509644, "word": "##tran"},
+                {"entity": "I-PER", "index": 18, "score": 0.9996136426925659, "word": "##a"},
+                {"entity": "B-ORG", "index": 28, "score": 0.9989739060401917, "word": "Far"},
+                {"entity": "I-ORG", "index": 29, "score": 0.7188422083854675, "word": "##c"},
+            ],
+            [
+                {"entity": "I-PER", "index": 1, "score": 0.9968166351318359, "word": "En"},
+                {"entity": "I-PER", "index": 2, "score": 0.9957635998725891, "word": "##zo"},
+                {"entity": "I-ORG", "index": 7, "score": 0.9986497163772583, "word": "UN"},
+            ],
+        ]
+        expected_grouped_ner_results = [
+            [
+                {"entity_group": "B-PER", "score": 0.9710702640669686, "word": "Consuelo Araújo Noguera"},
+                {"entity_group": "B-PER", "score": 0.9997273534536362, "word": "Andrés Pastrana"},
+                {"entity_group": "B-ORG", "score": 0.8589080572128296, "word": "Farc"},
+            ],
+            [
+                {"entity_group": "I-PER", "score": 0.9962901175022125, "word": "Enzo"},
+                {"entity_group": "I-ORG", "score": 0.9986497163772583, "word": "UN"},
+            ],
+        ]
+
+        self.assertIsNotNone(nlp)
+
+        mono_result = nlp(VALID_INPUTS[0])
+        self.assertIsInstance(mono_result, list)
+        self.assertIsInstance(mono_result[0], (dict, list))
+
+        if isinstance(mono_result[0], list):
+            mono_result = mono_result[0]
+
+        for key in output_keys:
+            self.assertIn(key, mono_result[0])
+
+        multi_result = [nlp(input) for input in VALID_INPUTS]
+        self.assertIsInstance(multi_result, list)
+        self.assertIsInstance(multi_result[0], (dict, list))
+
+        if isinstance(multi_result[0], list):
+            multi_result = multi_result[0]
+
+        for result in multi_result:
+            for key in output_keys:
+                self.assertIn(key, result)
+
+        for ungrouped_input, grouped_result in zip(ungrouped_ner_inputs, expected_grouped_ner_results):
+            self.assertEqual(nlp.group_entities(ungrouped_input), grouped_result)
+
+    @require_torch
+    def test_torch_ner(self):
+        mandatory_keys = {"entity", "word", "score"}
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name)
+            self._test_ner_pipeline(nlp, mandatory_keys)
+
+    @require_torch
+    def test_ner_grouped(self):
+        mandatory_keys = {"entity_group", "word", "score"}
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, grouped_entities=True)
+            self._test_ner_pipeline(nlp, mandatory_keys)
+
+    @require_tf
+    def test_tf_ner(self):
+        mandatory_keys = {"entity", "word", "score"}
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf")
+            self._test_ner_pipeline(nlp, mandatory_keys)
+
+    @require_tf
+    def test_tf_ner_grouped(self):
+        mandatory_keys = {"entity_group", "word", "score"}
+        for model_name in NER_FINETUNED_MODELS:
+            nlp = pipeline(task="ner", model=model_name, tokenizer=model_name, framework="tf", grouped_entities=True)
+            self._test_ner_pipeline(nlp, mandatory_keys)
+
+
 class PipelineCommonTests(unittest.TestCase):
 
     pipelines = SUPPORTED_TASKS.keys()