Add clear description of how to train T5

* force bleu

* fix wrong file name

* rename file

* different filenames for each example test

* test files should clean up after themselves

* test files should clean up after themselves

* do not force bleu

* correct typo

* fix isort

docs/source/model_doc/t5.rst

@@ -16,6 +16,37 @@ To facilitate future work on transfer learning for NLP, we release our dataset,
 
 The Authors' code can be found `here <https://github.com/google-research/text-to-text-transfer-transformer>`_ .
 
+Training
+~~~~~~~~~~~~~~~~~~~~
+T5 is an encoder-decoder model and casts all NLP problems as sequence to sequence tasks. It is trained using teacher forcing.
+This means that for training we always need an input sequence and a target sequence. 
+The input sequence is fed to the model using ``input_ids``. In teacher forcing style, the target sequence shifted to the right, *i.e.* prepended by the PAD "<pad>" token, is fed to the decoder and the target sequence appended by the EOS "</s>" token represents the ``lm_labels``.
+T5 can be trained / fine-tuned on two types of objectives:
+
+- Unsupervised denoising training
+  In this setup spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) 
+  and the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. 
+  *E.g.* the sentence "The cute dog walks in the park" and mask "cute dog" and "the" should be processed as follows: 
+
+::
+
+  input_ids = tokenizer.encode('The <extra_id_1> walks in <extra_id_2> park')
+  decoder_input_ids = tokenizer.encode('<pad> <extra_id_1> cute dog <extra_id_2> the <extra_id_3>')
+  lm_labels = tokenizer.encode('<extra_id_1> cute dog <extra_id_2> the <extra_id_3> </s>')
+  model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, lm_labels=lm_labels)
+
+- Supervised training
+  In this setup the input sequence and output sequence are standart sequence to sequence input output mapping.
+  In translation, *e.g.* the input sequence "The house is wonderful." and output sequence "Das Haus ist wunderbar." should 
+  be processed as follows:
+  
+::
+
+  input_ids = tokenizer.encode('The house is wonderful. </s>')
+  decoder_input_ids = tokenizer.encode('<pad> Das Haus ist wunderbar. ')
+  lm_labels = tokenizer.encode('Das Haus ist wunderbar. </s>')
+  model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, lm_labels=lm_labels)
+
 Tips
 ~~~~~~~~~~~~~~~~~~~~
 - T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised 
@@ -55,7 +86,7 @@ T5ForConditionalGeneration
     :members:
 
 
-TFBertModel
+TFT5Model
 ~~~~~~~~~~~~~~~~~~~~
 
 .. autoclass:: transformers.TFT5Model

docs/source/pretrained_models.rst

@@ -275,7 +275,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   |                                                            | | FlauBERT large architecture                                                                                                         |
 |                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 | Bart              | ``bart-large``                                             | | 12-layer, 1024-hidden, 16-heads, 406M parameters                                                                                    |
 |                   |                                                            | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/bart>`_)                                                       |
 |                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
@@ -285,6 +284,3 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``bart-large-cnn``                                         | | 12-layer, 1024-hidden, 16-heads, 406M parameters       (same as base)                                                               |
 |                   |                                                            | | bart-large base architecture finetuned on cnn summarization task                                                                    |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-
-
-.. <https://huggingface.co/transformers/examples.html>`__

examples/ner/utils_ner.py

@@ -112,12 +112,15 @@ def convert_examples_to_features(
         label_ids = []
         for word, label in zip(example.words, example.labels):
             word_tokens = tokenizer.tokenize(word)
-            tokens.extend(word_tokens)
-            # Use the real label id for the first token of the word, and padding ids for the remaining tokens
-            label_ids.extend([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))
+
+            # bert-base-multilingual-cased sometimes output "nothing ([]) when calling tokenize with just a space.
+            if len(word_tokens) > 0:
+                tokens.extend(word_tokens)
+                # Use the real label id for the first token of the word, and padding ids for the remaining tokens
+                label_ids.extend([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))
 
         # Account for [CLS] and [SEP] with "- 2" and with "- 3" for RoBERTa.
-        special_tokens_count = 3 if sep_token_extra else 2
+        special_tokens_count = tokenizer.num_added_tokens()
         if len(tokens) > max_seq_length - special_tokens_count:
             tokens = tokens[: (max_seq_length - special_tokens_count)]
             label_ids = label_ids[: (max_seq_length - special_tokens_count)]

examples/summarization/bart/test_bart_examples.py

@@ -1,4 +1,5 @@
 import logging
+import os
 import sys
 import tempfile
 import unittest
@@ -8,6 +9,8 @@ from unittest.mock import patch
 from .evaluate_cnn import _run_generate
 
 
+output_file_name = "output_bart_sum.txt"
+
 articles = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
 
 logging.basicConfig(level=logging.DEBUG)
@@ -19,10 +22,11 @@ class TestBartExamples(unittest.TestCase):
     def test_bart_cnn_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
+        tmp = Path(tempfile.gettempdir()) / "utest_generations_bart_sum.hypo"
         with tmp.open("w") as f:
             f.write("\n".join(articles))
-        testargs = ["evaluate_cnn.py", str(tmp), "output.txt"]
+        testargs = ["evaluate_cnn.py", str(tmp), output_file_name]
         with patch.object(sys, "argv", testargs):
             _run_generate()
-            self.assertTrue(Path("output.txt").exists())
+            self.assertTrue(Path(output_file_name).exists())
+            os.remove(Path(output_file_name))

examples/summarization/t5/README.md

@@ -1,4 +1,4 @@
-***This script evaluates the the multitask pre-trained checkpoint for ``t5-large`` (see paper [here](https://arxiv.org/pdf/1910.10683.pdf)) on the CNN/Daily Mail test dataset. Please note that the results in the paper were attained using a model fine-tuned on summarization, so that results will be worse here by approx. 0.5 ROUGE points***
+***This script evaluates the the multitask pre-trained checkpoint for ``t5-base`` (see paper [here](https://arxiv.org/pdf/1910.10683.pdf)) on the CNN/Daily Mail test dataset. Please note that the results in the paper were attained using a model fine-tuned on summarization, so that results will be worse here by approx. 0.5 ROUGE points***
 
 ### Get the CNN Data
 First, you need to download the CNN data. It's about ~400 MB and can be downloaded by 

examples/summarization/t5/evaluate_cnn.py

@@ -14,13 +14,13 @@ def chunks(lst, n):
         yield lst[i : i + n]
 
 
-def generate_summaries(lns, output_file_path, batch_size, device):
+def generate_summaries(lns, output_file_path, model_size, batch_size, device):
     output_file = Path(output_file_path).open("w")
 
-    model = T5ForConditionalGeneration.from_pretrained("t5-large")
+    model = T5ForConditionalGeneration.from_pretrained(model_size)
     model.to(device)
 
-    tokenizer = T5Tokenizer.from_pretrained("t5-large")
+    tokenizer = T5Tokenizer.from_pretrained(model_size)
 
     # update config with summarization specific params
     task_specific_params = model.config.task_specific_params
@@ -61,6 +61,12 @@ def calculate_rouge(output_lns, reference_lns, score_path):
 
 def run_generate():
     parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "model_size",
+        type=str,
+        help="T5 model size, either 't5-small', 't5-base' or 't5-large'. Defaults to base.",
+        default="t5-base",
+    )
     parser.add_argument(
         "input_path", type=str, help="like cnn_dm/test_articles_input.txt",
     )
@@ -83,7 +89,7 @@ def run_generate():
 
     source_lns = [x.rstrip() for x in open(args.input_path).readlines()]
 
-    generate_summaries(source_lns, args.output_path, args.batch_size, args.device)
+    generate_summaries(source_lns, args.output_path, args.model_size, args.batch_size, args.device)
 
     output_lns = [x.rstrip() for x in open(args.output_path).readlines()]
     reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()]

examples/summarization/t5/test_t5_examples.py

@@ -1,4 +1,5 @@
 import logging
+import os
 import sys
 import tempfile
 import unittest
@@ -8,6 +9,9 @@ from unittest.mock import patch
 from .evaluate_cnn import run_generate
 
 
+output_file_name = "output_t5_sum.txt"
+score_file_name = "score_t5_sum.txt"
+
 articles = ["New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
 
 logging.basicConfig(level=logging.DEBUG)
@@ -19,11 +23,13 @@ class TestT5Examples(unittest.TestCase):
     def test_t5_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
+        tmp = Path(tempfile.gettempdir()) / "utest_generations_t5_sum.hypo"
         with tmp.open("w") as f:
             f.write("\n".join(articles))
-        testargs = ["evaluate_cnn.py", str(tmp), "output.txt", str(tmp), "score.txt"]
+        testargs = ["evaluate_cnn.py", "t5-small", str(tmp), output_file_name, str(tmp), score_file_name]
         with patch.object(sys, "argv", testargs):
             run_generate()
-            self.assertTrue(Path("output.txt").exists())
-            self.assertTrue(Path("score.txt").exists())
+            self.assertTrue(Path(output_file_name).exists())
+            self.assertTrue(Path(score_file_name).exists())
+            os.remove(Path(output_file_name))
+            os.remove(Path(score_file_name))

examples/translation/t5/test_t5_examples.py

@@ -1,4 +1,5 @@
 import logging
+import os
 import sys
 import tempfile
 import unittest
@@ -8,7 +9,11 @@ from unittest.mock import patch
 from .evaluate_wmt import run_generate
 
 
-text = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
+text = ["When Liana Barrientos was 23 years old, she got married in Westchester County."]
+translation = ["Als Liana Barrientos 23 Jahre alt war, heiratete sie in Westchester County."]
+
+output_file_name = "output_t5_trans.txt"
+score_file_name = "score_t5_trans.txt"
 
 logging.basicConfig(level=logging.DEBUG)
 
@@ -19,10 +24,20 @@ class TestT5Examples(unittest.TestCase):
     def test_t5_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
-        with tmp.open("w") as f:
+
+        tmp_source = Path(tempfile.gettempdir()) / "utest_generations_t5_trans.hypo"
+        with tmp_source.open("w") as f:
             f.write("\n".join(text))
-        testargs = ["evaluate_cnn.py", str(tmp), "output.txt", str(tmp), "score.txt"]
+
+        tmp_target = Path(tempfile.gettempdir()) / "utest_generations_t5_trans.target"
+        with tmp_target.open("w") as f:
+            f.write("\n".join(translation))
+
+        testargs = ["evaluate_wmt.py", str(tmp_source), output_file_name, str(tmp_target), score_file_name]
+
         with patch.object(sys, "argv", testargs):
             run_generate()
-            self.assertTrue(Path("output.txt").exists())
+            self.assertTrue(Path(output_file_name).exists())
+            self.assertTrue(Path(score_file_name).exists())
+            os.remove(Path(output_file_name))
+            os.remove(Path(score_file_name))

model_cards/dbmdz/bert-base-german-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: german
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz German BERT models

model_cards/dbmdz/bert-base-german-europeana-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: german
+license: mit
 tags:
   - "historic german"
 ---

model_cards/dbmdz/bert-base-german-europeana-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: german
+license: mit
 tags:
   - "historic german"
 ---

model_cards/dbmdz/bert-base-german-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: german
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz German BERT models

model_cards/dbmdz/bert-base-italian-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: italian
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: italian
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-xxl-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: italian
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-xxl-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: italian
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-turkish-128k-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: turkish
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-128k-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: turkish
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: turkish
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-uncased/README.md

@@ -1,5 +1,6 @@
 ---
 language: turkish
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/distilbert-base-turkish-cased/README.md

@@ -1,5 +1,6 @@
 ---
 language: turkish
+license: mit
 ---
 
 # 🤗 + 📚 dbmdz Distilled Turkish BERT model

src/transformers/modeling_t5.py

@@ -724,7 +724,7 @@ T5_INPUTS_DOCSTRING = r"""
             `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
             Used in the cross-attention of the decoder.
         decoder_input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
-            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation
+            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation.
         decoder_attention_mask (:obj:`torch.BoolTensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
             Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
         inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
@@ -789,28 +789,30 @@ class T5Model(T5PreTrainedModel):
         head_mask=None,
     ):
         r"""
-        Return:
-            :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5`) and inputs:
-            last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
-                Sequence of hidden-states at the output of the last layer of the model.
-            hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-                Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
-                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+    Return:
+        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
+        last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+            Sequence of hidden-states at the output of the last layer of the model.
+        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-            attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
-                Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
-                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
-                heads.
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+            heads.
 
-        Examples::
+    Examples::
+
+            from transformers import T5Tokenizer, T5Model
 
             tokenizer = T5Tokenizer.from_pretrained('t5-small')
             model = T5Model.from_pretrained('t5-small')
-            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
-            outputs = model(input_ids=input_ids)
+            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
+            outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)
             last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
 
         """
@@ -884,34 +886,40 @@ class T5ForConditionalGeneration(T5PreTrainedModel):
     ):
         r"""
         lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`):
-            Labels for computing the sequence classification/regression loss.
-            Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
-            If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+                Labels for computing the sequence classification/regression loss.
+                Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
+                If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
 
-        Returns:
-            :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
-            loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
-                Classification loss (cross entropy)
-            prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
-                Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
-            hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-                Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
-                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+    Returns:
+        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
+        loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
+            Classification loss (cross entropy).
+        prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-            attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
-                Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
-                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.
 
-                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+    Examples::
 
-        Examples::
+        from transformers import T5Tokenizer, T5ForConditionalGeneration
 
-            tokenizer = T5Tokenizer.from_pretrained('t5-small')
-            model = T5ForConditionalGeneration.from_pretrained('t5-small')
-            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
-            outputs = model(input_ids=input_ids, lm_labels=input_ids)
-            loss, prediction_scores = outputs[:2]
+        tokenizer = T5Tokenizer.from_pretrained('t5-small')
+        model = T5ForConditionalGeneration.from_pretrained('t5-small')
+        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
+        outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, lm_labels=input_ids)
+        loss, prediction_scores = outputs[:2]
+
+        tokenizer = T5Tokenizer.from_pretrained('t5-small')
+        model = T5ForConditionalGeneration.from_pretrained('t5-small')
+        input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="pt")  # Batch size 1
+        outputs = model.generate(input_ids)
         """
 
         # Encode if needed (training, first prediction pass)

src/transformers/modeling_tf_t5.py

@@ -631,51 +631,50 @@ T5_START_DOCSTRING = r"""    The T5 model was proposed in
 
 T5_INPUTS_DOCSTRING = r"""
     Args:
-        decoder_input_ids (:obj: `list`,`tuple`, or `dict`):
-                decoder_input_ids are usually used as a `dict` (see T5 description above for more information) containing all the following:
-            decoder_input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
-                Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation
+        decoder_input_ids are usually used as a `dict` (see T5 description above for more information) containing all the following.
+        decoder_input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
+            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation.
 
-            input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`):
-                Indices of input sequence tokens in the vocabulary.
-                To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
+        input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary.
+            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
 
-                (a) For sequence pairs:
+            (a) For sequence pairs:
 
-                    ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
+                ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
 
-                (b) For single sequences:
+            (b) For single sequences:
 
-                    ``tokens:         [CLS] the dog is hairy . [SEP]``
+                ``tokens:         [CLS] the dog is hairy . [SEP]``
 
-                T5 is a model with relative position embeddings so you should be able to pad the inputs on
-                the right or the left.
+            T5 is a model with relative position embeddings so you should be able to pad the inputs on
+            the right or the left.
 
-                Indices can be obtained using :class:`transformers.T5Tokenizer`.
-                See :func:`transformers.PreTrainedTokenizer.encode` and
-                :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
-            attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
-                Mask to avoid performing attention on padding token indices.
-                Mask values selected in ``[0, 1]``:
-                ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
-            encoder_outputs (tuple(:obj:`tuple(tf.FloatTensor)`, `optional`, defaults to :obj:`None`):
-                Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`)
-                `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
-                Used in the cross-attention of the decoder.
-            decoder_attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
-                Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
-            inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
-                Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
-                This is useful if you want more control over how to convert `input_ids` indices into associated vectors
-                than the model's internal embedding lookup matrix.
-            decoder_inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
-                Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation.
-                This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors
-                than the model's internal embedding lookup matrix.
-            head_mask: (:obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
-                Mask to nullify selected heads of the self-attention modules.
-                Mask values selected in ``[0, 1]``:
-                ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
+            Indices can be obtained using :class:`transformers.T5Tokenizer`.
+            See :func:`transformers.PreTrainedTokenizer.encode` and
+            :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
+        attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
+            Mask to avoid performing attention on padding token indices.
+            Mask values selected in ``[0, 1]``:
+            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
+        encoder_outputs (tuple(:obj:`tuple(tf.FloatTensor)`, `optional`, defaults to :obj:`None`):
+            Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`)
+            `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
+            Used in the cross-attention of the decoder.
+        decoder_attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
+            Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
+        inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
+            Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
+            This is useful if you want more control over how to convert `input_ids` indices into associated vectors
+            than the model's internal embedding lookup matrix.
+        decoder_inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
+            Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation.
+            This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors
+            than the model's internal embedding lookup matrix.
+        head_mask: (:obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
+            Mask to nullify selected heads of the self-attention modules.
+            Mask values selected in ``[0, 1]``:
+            ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
 """
 
 
@@ -710,29 +709,31 @@ class TFT5Model(TFT5PreTrainedModel):
     @add_start_docstrings_to_callable(T5_INPUTS_DOCSTRING)
     def call(self, decoder_input_ids, **kwargs):
         r"""
-        Return:
-            :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
-            last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
-                Sequence of hidden-states at the output of the last layer of the model.
-            hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-                Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
-                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+    Return:
+        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
+        last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+            Sequence of hidden-states at the output of the last layer of the model.
+        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+            Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-            attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
-                Tuple of :obj:`tf.Tensor` (one for each layer) of shape
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`tf.Tensor` (one for each layer) of shape
                 :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
-                heads.
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+            heads.
 
-        Examples::
+    Examples::
 
-            tokenizer = T5Tokenizer.from_pretrained('t5-small')
-            model = TFT5Model.from_pretrained('t5-small')
-            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="tf")  # Batch size 1
-            outputs = model(input_ids)
-            last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
+        from transformers import T5Tokenizer, TFT5Model
+
+        tokenizer = T5Tokenizer.from_pretrained('t5-small')
+        model = TFT5Model.from_pretrained('t5-small')
+        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
+        outputs = model(input_ids, input_ids=input_ids)
+        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
 
         """
 
@@ -810,30 +811,38 @@ class TFT5ForConditionalGeneration(TFT5PreTrainedModel):
             Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
             If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
 
-        Returns:
-            :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
-            loss (:obj:`tf.Tensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
-                Classification loss (cross entropy)
-            prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
-                Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
-            hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-                Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
-                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+    Return:
+        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
+        loss (:obj:`tf.Tensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
+            Classification loss (cross entropy).
+        prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+            Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-            attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
-                Tuple of :obj:`tf.Tensor` (one for each layer) of shape
-                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`tf.Tensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.
 
-        Examples::
+    Examples::
+
+        from transformers import T5Tokenizer, TFT5ForConditionalGeneration
+
+        tokenizer = T5Tokenizer.from_pretrained('t5-small')
+        model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
+        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
+        outputs = model(input_ids, input_ids=input_ids, lm_labels=input_ids)
+        prediction_scores = outputs[:1]  # TODO: TFT5 still needs to implement
+
+        tokenizer = T5Tokenizer.from_pretrained('t5-small')
+        model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
+        input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="tf")  # Batch size 1
+        model.generate(input_ids)
 
-            tokenizer = T5Tokenizer.from_pretrained('t5-small')
-            model = T5ForConditionalGeneration.from_pretrained('t5-small')
-            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
-            outputs = model(input_ids, lm_labels=input_ids)
-            loss, prediction_scores = outputs[:2]
         """
 
         if isinstance(decoder_input_ids, dict):

src/transformers/modeling_tf_utils.py

@@ -231,7 +231,7 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin):
 
     def save_pretrained(self, save_directory):
         """ Save a model and its configuration file to a directory, so that it
-            can be re-loaded using the `:func:`~transformers.PreTrainedModel.from_pretrained`` class method.
+            can be re-loaded using the :func:`~transformers.PreTrainedModel.from_pretrained` class method.
         """
         assert os.path.isdir(
             save_directory
@@ -541,7 +541,7 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin):
                 ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
                 Defaults to `None`.
 
-            `What are attention masks? <../glossary.html#attention-mask>`__
+                `What are attention masks? <../glossary.html#attention-mask>`__
 
             decoder_start_token_id=None: (`optional`) int
                 If an encoder-decoder model starts decoding with a different token than BOS.

src/transformers/tokenization_t5.py

@@ -61,14 +61,14 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
 
 class T5Tokenizer(PreTrainedTokenizer):
     """
-        Constructs an XLNet tokenizer. Based on `SentencePiece <https://github.com/google/sentencepiece>`__
+        Constructs an XLNet tokenizer. Based on `SentencePiece <https://github.com/google/sentencepiece>`__ .
 
         This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users
         should refer to the superclass for more information regarding methods.
 
         Args:
             vocab_file (:obj:`string`):
-                `SentencePiece <https://github.com/google/sentencepiece>`__ file (generally has a .spm extension) that
+                `SentencePiece <https://github.com/google/sentencepiece>`__ file (generally has a `.spm` extension) that
                 contains the vocabulary necessary to instantiate a tokenizer.
             eos_token (:obj:`string`, `optional`, defaults to "</s>"):
                 The end of sequence token.
@@ -84,9 +84,8 @@ class T5Tokenizer(PreTrainedTokenizer):
                 The token used for padding, for example when batching sequences of different lengths.
             extra_ids (:obj:`List[str]`, `optional`, defaults to :obj:`100`):
                 Add a number of extra ids added to the end of the vocabulary for use as sentinels.
-                These tokens are accessible as `<extra_id_{%d}>` where `{%d}` is a number between 0 and extra_ids-1.
-                Extra tokens are indexed from the end of the vocabulary up to beginnning (<extra_id_0> is the last token in the vocabulary)
-                (like in T5 preprocessing
+                These tokens are accessible as "<extra_id_{%d}>" where "{%d}" is a number between 0 and extra_ids-1.
+                Extra tokens are indexed from the end of the vocabulary up to beginnning ("<extra_id_0>" is the last token in the vocabulary like in T5 preprocessing
                 see: https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)
             additional_special_tokens (:obj:`List[str]`, `optional`, defaults to :obj:`None`):
                 Additional special tokens used by the tokenizer.