make style

docs/source/model_doc/t5.rst

@@ -16,37 +16,6 @@ To facilitate future work on transfer learning for NLP, we release our dataset,
 
 The Authors' code can be found `here <https://github.com/google-research/text-to-text-transfer-transformer>`_ .
 
-Training
-~~~~~~~~~~~~~~~~~~~~
-T5 is an encoder-decoder model and casts all NLP problems as sequence to sequence tasks. It is trained using teacher forcing.
-This means that for training we always need an input sequence and a target sequence. 
-The input sequence is fed to the model using ``input_ids``. In teacher forcing style, the target sequence shifted to the right, *i.e.* prepended by the PAD "<pad>" token, is fed to the decoder and the target sequence appended by the EOS "</s>" token represents the ``lm_labels``.
-T5 can be trained / fine-tuned on two types of objectives:
-
-- Unsupervised denoising training
-  In this setup spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) 
-  and the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. 
-  *E.g.* the sentence "The cute dog walks in the park" and mask "cute dog" and "the" should be processed as follows: 
-
-::
-
-  input_ids = tokenizer.encode('The <extra_id_1> walks in <extra_id_2> park')
-  decoder_input_ids = tokenizer.encode('<pad> <extra_id_1> cute dog <extra_id_2> the <extra_id_3>')
-  lm_labels = tokenizer.encode('<extra_id_1> cute dog <extra_id_2> the <extra_id_3> </s>')
-  model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, lm_labels=lm_labels)
-
-- Supervised training
-  In this setup the input sequence and output sequence are standart sequence to sequence input output mapping.
-  In translation, *e.g.* the input sequence "The house is wonderful." and output sequence "Das Haus ist wunderbar." should 
-  be processed as follows:
-  
-::
-
-  input_ids = tokenizer.encode('The house is wonderful. </s>')
-  decoder_input_ids = tokenizer.encode('<pad> Das Haus ist wunderbar. ')
-  lm_labels = tokenizer.encode('Das Haus ist wunderbar. </s>')
-  model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, lm_labels=lm_labels)
-
 Tips
 ~~~~~~~~~~~~~~~~~~~~
 - T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised 
@@ -86,7 +55,7 @@ T5ForConditionalGeneration
     :members:
 
 
-TFT5Model
+TFBertModel
 ~~~~~~~~~~~~~~~~~~~~
 
 .. autoclass:: transformers.TFT5Model

docs/source/pretrained_models.rst

@@ -275,6 +275,7 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   |                                                            | | FlauBERT large architecture                                                                                                         |
 |                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
++-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 | Bart              | ``bart-large``                                             | | 12-layer, 1024-hidden, 16-heads, 406M parameters                                                                                    |
 |                   |                                                            | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/bart>`_)                                                       |
 |                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
@@ -284,3 +285,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``bart-large-cnn``                                         | | 12-layer, 1024-hidden, 16-heads, 406M parameters       (same as base)                                                               |
 |                   |                                                            | | bart-large base architecture finetuned on cnn summarization task                                                                    |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+
+
+.. <https://huggingface.co/transformers/examples.html>`__

examples/ner/utils_ner.py

@@ -112,15 +112,12 @@ def convert_examples_to_features(
         label_ids = []
         for word, label in zip(example.words, example.labels):
             word_tokens = tokenizer.tokenize(word)
-
-            # bert-base-multilingual-cased sometimes output "nothing ([]) when calling tokenize with just a space.
-            if len(word_tokens) > 0:
-                tokens.extend(word_tokens)
-                # Use the real label id for the first token of the word, and padding ids for the remaining tokens
-                label_ids.extend([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))
+            tokens.extend(word_tokens)
+            # Use the real label id for the first token of the word, and padding ids for the remaining tokens
+            label_ids.extend([label_map[label]] + [pad_token_label_id] * (len(word_tokens) - 1))
 
         # Account for [CLS] and [SEP] with "- 2" and with "- 3" for RoBERTa.
-        special_tokens_count = tokenizer.num_added_tokens()
+        special_tokens_count = 3 if sep_token_extra else 2
         if len(tokens) > max_seq_length - special_tokens_count:
             tokens = tokens[: (max_seq_length - special_tokens_count)]
             label_ids = label_ids[: (max_seq_length - special_tokens_count)]

examples/summarization/bart/test_bart_examples.py

@@ -1,5 +1,4 @@
 import logging
-import os
 import sys
 import tempfile
 import unittest
@@ -9,8 +8,6 @@ from unittest.mock import patch
 from .evaluate_cnn import _run_generate
 
 
-output_file_name = "output_bart_sum.txt"
-
 articles = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
 
 logging.basicConfig(level=logging.DEBUG)
@@ -22,11 +19,10 @@ class TestBartExamples(unittest.TestCase):
     def test_bart_cnn_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-        tmp = Path(tempfile.gettempdir()) / "utest_generations_bart_sum.hypo"
+        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
         with tmp.open("w") as f:
             f.write("\n".join(articles))
-        testargs = ["evaluate_cnn.py", str(tmp), output_file_name]
+        testargs = ["evaluate_cnn.py", str(tmp), "output.txt"]
         with patch.object(sys, "argv", testargs):
             _run_generate()
-            self.assertTrue(Path(output_file_name).exists())
-            os.remove(Path(output_file_name))
+            self.assertTrue(Path("output.txt").exists())

examples/summarization/t5/README.md

@@ -1,4 +1,4 @@
-***This script evaluates the the multitask pre-trained checkpoint for ``t5-base`` (see paper [here](https://arxiv.org/pdf/1910.10683.pdf)) on the CNN/Daily Mail test dataset. Please note that the results in the paper were attained using a model fine-tuned on summarization, so that results will be worse here by approx. 0.5 ROUGE points***
+***This script evaluates the the multitask pre-trained checkpoint for ``t5-large`` (see paper [here](https://arxiv.org/pdf/1910.10683.pdf)) on the CNN/Daily Mail test dataset. Please note that the results in the paper were attained using a model fine-tuned on summarization, so that results will be worse here by approx. 0.5 ROUGE points***
 
 ### Get the CNN Data
 First, you need to download the CNN data. It's about ~400 MB and can be downloaded by 

examples/summarization/t5/evaluate_cnn.py

@@ -14,13 +14,13 @@ def chunks(lst, n):
         yield lst[i : i + n]
 
 
-def generate_summaries(lns, output_file_path, model_size, batch_size, device):
+def generate_summaries(lns, output_file_path, batch_size, device):
     output_file = Path(output_file_path).open("w")
 
-    model = T5ForConditionalGeneration.from_pretrained(model_size)
+    model = T5ForConditionalGeneration.from_pretrained("t5-large")
     model.to(device)
 
-    tokenizer = T5Tokenizer.from_pretrained(model_size)
+    tokenizer = T5Tokenizer.from_pretrained("t5-large")
 
     # update config with summarization specific params
     task_specific_params = model.config.task_specific_params
@@ -61,12 +61,6 @@ def calculate_rouge(output_lns, reference_lns, score_path):
 
 def run_generate():
     parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "model_size",
-        type=str,
-        help="T5 model size, either 't5-small', 't5-base' or 't5-large'. Defaults to base.",
-        default="t5-base",
-    )
     parser.add_argument(
         "input_path", type=str, help="like cnn_dm/test_articles_input.txt",
     )
@@ -89,7 +83,7 @@ def run_generate():
 
     source_lns = [x.rstrip() for x in open(args.input_path).readlines()]
 
-    generate_summaries(source_lns, args.output_path, args.model_size, args.batch_size, args.device)
+    generate_summaries(source_lns, args.output_path, args.batch_size, args.device)
 
     output_lns = [x.rstrip() for x in open(args.output_path).readlines()]
     reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()]

examples/summarization/t5/test_t5_examples.py

@@ -1,5 +1,4 @@
 import logging
-import os
 import sys
 import tempfile
 import unittest
@@ -9,9 +8,6 @@ from unittest.mock import patch
 from .evaluate_cnn import run_generate
 
 
-output_file_name = "output_t5_sum.txt"
-score_file_name = "score_t5_sum.txt"
-
 articles = ["New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
 
 logging.basicConfig(level=logging.DEBUG)
@@ -23,13 +19,11 @@ class TestT5Examples(unittest.TestCase):
     def test_t5_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-        tmp = Path(tempfile.gettempdir()) / "utest_generations_t5_sum.hypo"
+        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
         with tmp.open("w") as f:
             f.write("\n".join(articles))
-        testargs = ["evaluate_cnn.py", "t5-small", str(tmp), output_file_name, str(tmp), score_file_name]
+        testargs = ["evaluate_cnn.py", str(tmp), "output.txt", str(tmp), "score.txt"]
         with patch.object(sys, "argv", testargs):
             run_generate()
-            self.assertTrue(Path(output_file_name).exists())
-            self.assertTrue(Path(score_file_name).exists())
-            os.remove(Path(output_file_name))
-            os.remove(Path(score_file_name))
+            self.assertTrue(Path("output.txt").exists())
+            self.assertTrue(Path("score.txt").exists())

examples/translation/t5/test_t5_examples.py

@@ -1,5 +1,4 @@
 import logging
-import os
 import sys
 import tempfile
 import unittest
@@ -9,11 +8,7 @@ from unittest.mock import patch
 from .evaluate_wmt import run_generate
 
 
-text = ["When Liana Barrientos was 23 years old, she got married in Westchester County."]
-translation = ["Als Liana Barrientos 23 Jahre alt war, heiratete sie in Westchester County."]
-
-output_file_name = "output_t5_trans.txt"
-score_file_name = "score_t5_trans.txt"
+text = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
 
 logging.basicConfig(level=logging.DEBUG)
 
@@ -24,20 +19,10 @@ class TestT5Examples(unittest.TestCase):
     def test_t5_cli(self):
         stream_handler = logging.StreamHandler(sys.stdout)
         logger.addHandler(stream_handler)
-
-        tmp_source = Path(tempfile.gettempdir()) / "utest_generations_t5_trans.hypo"
-        with tmp_source.open("w") as f:
+        tmp = Path(tempfile.gettempdir()) / "utest_generations.hypo"
+        with tmp.open("w") as f:
             f.write("\n".join(text))
-
-        tmp_target = Path(tempfile.gettempdir()) / "utest_generations_t5_trans.target"
-        with tmp_target.open("w") as f:
-            f.write("\n".join(translation))
-
-        testargs = ["evaluate_wmt.py", str(tmp_source), output_file_name, str(tmp_target), score_file_name]
-
+        testargs = ["evaluate_cnn.py", str(tmp), "output.txt", str(tmp), "score.txt"]
         with patch.object(sys, "argv", testargs):
             run_generate()
-            self.assertTrue(Path(output_file_name).exists())
-            self.assertTrue(Path(score_file_name).exists())
-            os.remove(Path(output_file_name))
-            os.remove(Path(score_file_name))
+            self.assertTrue(Path("output.txt").exists())

model_cards/dbmdz/bert-base-german-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: german
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz German BERT models

model_cards/dbmdz/bert-base-german-europeana-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: german
-license: mit
 tags:
   - "historic german"
 ---

model_cards/dbmdz/bert-base-german-europeana-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: german
-license: mit
 tags:
   - "historic german"
 ---

model_cards/dbmdz/bert-base-german-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: german
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz German BERT models

model_cards/dbmdz/bert-base-italian-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: italian
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: italian
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-xxl-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: italian
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-italian-xxl-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: italian
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz BERT models

model_cards/dbmdz/bert-base-turkish-128k-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: turkish
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-128k-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: turkish
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: turkish
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/bert-base-turkish-uncased/README.md

@@ -1,6 +1,5 @@
 ---
 language: turkish
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz Turkish BERT model

model_cards/dbmdz/distilbert-base-turkish-cased/README.md

@@ -1,6 +1,5 @@
 ---
 language: turkish
-license: mit
 ---
 
 # 🤗 + 📚 dbmdz Distilled Turkish BERT model

src/transformers/modeling_t5.py

@@ -724,7 +724,7 @@ T5_INPUTS_DOCSTRING = r"""
             `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
             Used in the cross-attention of the decoder.
         decoder_input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
-            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation.
+            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation
         decoder_attention_mask (:obj:`torch.BoolTensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
             Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
         inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
@@ -789,30 +789,28 @@ class T5Model(T5PreTrainedModel):
         head_mask=None,
     ):
         r"""
-    Return:
-        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
-        last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
-            Sequence of hidden-states at the output of the last layer of the model.
-        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
-            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+        Return:
+            :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5`) and inputs:
+            last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+                Sequence of hidden-states at the output of the last layer of the model.
+            hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+                Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
-            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
-            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+            attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+                Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
-            heads.
+                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+                heads.
 
-    Examples::
-
-            from transformers import T5Tokenizer, T5Model
+        Examples::
 
             tokenizer = T5Tokenizer.from_pretrained('t5-small')
             model = T5Model.from_pretrained('t5-small')
-            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
-            outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)
+            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
+            outputs = model(input_ids=input_ids)
             last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
 
         """
@@ -886,40 +884,34 @@ class T5ForConditionalGeneration(T5PreTrainedModel):
     ):
         r"""
         lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`):
-                Labels for computing the sequence classification/regression loss.
-                Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
-                If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+            Labels for computing the sequence classification/regression loss.
+            Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
+            If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
 
-    Returns:
-        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
-        loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
-            Classification loss (cross entropy).
-        prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
-            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
-        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
-            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+        Returns:
+            :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
+            loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
+                Classification loss (cross entropy)
+            prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+                Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+            hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+                Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
-            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
-            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
-            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.
+                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+            attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+                Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-    Examples::
+                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
 
-        from transformers import T5Tokenizer, T5ForConditionalGeneration
+        Examples::
 
-        tokenizer = T5Tokenizer.from_pretrained('t5-small')
-        model = T5ForConditionalGeneration.from_pretrained('t5-small')
-        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1
-        outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, lm_labels=input_ids)
-        loss, prediction_scores = outputs[:2]
-
-        tokenizer = T5Tokenizer.from_pretrained('t5-small')
-        model = T5ForConditionalGeneration.from_pretrained('t5-small')
-        input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="pt")  # Batch size 1
-        outputs = model.generate(input_ids)
+            tokenizer = T5Tokenizer.from_pretrained('t5-small')
+            model = T5ForConditionalGeneration.from_pretrained('t5-small')
+            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
+            outputs = model(input_ids=input_ids, lm_labels=input_ids)
+            loss, prediction_scores = outputs[:2]
         """
 
         # Encode if needed (training, first prediction pass)

src/transformers/modeling_tf_t5.py

@@ -631,50 +631,51 @@ T5_START_DOCSTRING = r"""    The T5 model was proposed in
 
 T5_INPUTS_DOCSTRING = r"""
     Args:
-        decoder_input_ids are usually used as a `dict` (see T5 description above for more information) containing all the following.
-        decoder_input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
-            Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation.
+        decoder_input_ids (:obj: `list`,`tuple`, or `dict`):
+                decoder_input_ids are usually used as a `dict` (see T5 description above for more information) containing all the following:
+            decoder_input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`, defaults to :obj:`None`):
+                Provide for sequence to sequence training. T5 uses the pad_token_id as the starting token for decoder_input_ids generation
 
-        input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`):
-            Indices of input sequence tokens in the vocabulary.
-            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
+            input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+                Indices of input sequence tokens in the vocabulary.
+                To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
 
-            (a) For sequence pairs:
+                (a) For sequence pairs:
 
-                ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
+                    ``tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]``
 
-            (b) For single sequences:
+                (b) For single sequences:
 
-                ``tokens:         [CLS] the dog is hairy . [SEP]``
+                    ``tokens:         [CLS] the dog is hairy . [SEP]``
 
-            T5 is a model with relative position embeddings so you should be able to pad the inputs on
-            the right or the left.
+                T5 is a model with relative position embeddings so you should be able to pad the inputs on
+                the right or the left.
 
-            Indices can be obtained using :class:`transformers.T5Tokenizer`.
-            See :func:`transformers.PreTrainedTokenizer.encode` and
-            :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
-        attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
-            Mask to avoid performing attention on padding token indices.
-            Mask values selected in ``[0, 1]``:
-            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
-        encoder_outputs (tuple(:obj:`tuple(tf.FloatTensor)`, `optional`, defaults to :obj:`None`):
-            Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`)
-            `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
-            Used in the cross-attention of the decoder.
-        decoder_attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
-            Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
-        inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
-            Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
-            This is useful if you want more control over how to convert `input_ids` indices into associated vectors
-            than the model's internal embedding lookup matrix.
-        decoder_inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
-            Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation.
-            This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors
-            than the model's internal embedding lookup matrix.
-        head_mask: (:obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
-            Mask to nullify selected heads of the self-attention modules.
-            Mask values selected in ``[0, 1]``:
-            ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
+                Indices can be obtained using :class:`transformers.T5Tokenizer`.
+                See :func:`transformers.PreTrainedTokenizer.encode` and
+                :func:`transformers.PreTrainedTokenizer.convert_tokens_to_ids` for details.
+            attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
+                Mask to avoid performing attention on padding token indices.
+                Mask values selected in ``[0, 1]``:
+                ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
+            encoder_outputs (tuple(:obj:`tuple(tf.FloatTensor)`, `optional`, defaults to :obj:`None`):
+                Tuple consists of (`last_hidden_state`, `optional`: `hidden_states`, `optional`: `attentions`)
+                `last_hidden_state` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`) is a sequence of hidden-states at the output of the last layer of the encoder.
+                Used in the cross-attention of the decoder.
+            decoder_attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, tgt_seq_len)`, `optional`, defaults to :obj:`None`):
+                Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also be used by default.
+            inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
+                Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
+                This is useful if you want more control over how to convert `input_ids` indices into associated vectors
+                than the model's internal embedding lookup matrix.
+            decoder_inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
+                Optionally, instead of passing :obj:`decoder_input_ids` you can choose to directly pass an embedded representation.
+                This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors
+                than the model's internal embedding lookup matrix.
+            head_mask: (:obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
+                Mask to nullify selected heads of the self-attention modules.
+                Mask values selected in ``[0, 1]``:
+                ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.
 """
 
 
@@ -709,31 +710,29 @@ class TFT5Model(TFT5PreTrainedModel):
     @add_start_docstrings_to_callable(T5_INPUTS_DOCSTRING)
     def call(self, decoder_input_ids, **kwargs):
         r"""
-    Return:
-        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
-        last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
-            Sequence of hidden-states at the output of the last layer of the model.
-        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-            Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
-            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+        Return:
+            :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
+            last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+                Sequence of hidden-states at the output of the last layer of the model.
+            hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+                Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
+                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
-            Tuple of :obj:`tf.Tensor` (one for each layer) of shape
+                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+            attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
+                Tuple of :obj:`tf.Tensor` (one for each layer) of shape
                 :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
-            heads.
+                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+                heads.
 
-    Examples::
+        Examples::
 
-        from transformers import T5Tokenizer, TFT5Model
-
-        tokenizer = T5Tokenizer.from_pretrained('t5-small')
-        model = TFT5Model.from_pretrained('t5-small')
-        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
-        outputs = model(input_ids, input_ids=input_ids)
-        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
+            tokenizer = T5Tokenizer.from_pretrained('t5-small')
+            model = TFT5Model.from_pretrained('t5-small')
+            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="tf")  # Batch size 1
+            outputs = model(input_ids)
+            last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
 
         """
 
@@ -811,38 +810,30 @@ class TFT5ForConditionalGeneration(TFT5PreTrainedModel):
             Indices should be in :obj:`[0, ..., config.vocab_size - 1]`.
             If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
 
-    Return:
-        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs.
-        loss (:obj:`tf.Tensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
-            Classification loss (cross entropy).
-        prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
-            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
-        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
-            Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
-            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+        Returns:
+            :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.T5Config`) and inputs:
+            loss (:obj:`tf.Tensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`lm_label` is provided):
+                Classification loss (cross entropy)
+            prediction_scores (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
+                Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+            hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+                Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
+                of shape :obj:`(batch_size, sequence_length, hidden_size)`.
 
-            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
-        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
-            Tuple of :obj:`tf.Tensor` (one for each layer) of shape
-            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+                Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+            attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``config.output_attentions=True``):
+                Tuple of :obj:`tf.Tensor` (one for each layer) of shape
+                :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
 
-            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention.
+                Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
 
-    Examples::
-
-        from transformers import T5Tokenizer, TFT5ForConditionalGeneration
-
-        tokenizer = T5Tokenizer.from_pretrained('t5-small')
-        model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
-        input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="tf")  # Batch size 1
-        outputs = model(input_ids, input_ids=input_ids, lm_labels=input_ids)
-        prediction_scores = outputs[:1]  # TODO: TFT5 still needs to implement
-
-        tokenizer = T5Tokenizer.from_pretrained('t5-small')
-        model = TFT5ForConditionalGeneration.from_pretrained('t5-small')
-        input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="tf")  # Batch size 1
-        model.generate(input_ids)
+        Examples::
 
+            tokenizer = T5Tokenizer.from_pretrained('t5-small')
+            model = T5ForConditionalGeneration.from_pretrained('t5-small')
+            input_ids = tokenizer.encode("Hello, my dog is cute", return_tensor="pt")  # Batch size 1
+            outputs = model(input_ids, lm_labels=input_ids)
+            loss, prediction_scores = outputs[:2]
         """
 
         if isinstance(decoder_input_ids, dict):

src/transformers/modeling_tf_utils.py

@@ -231,7 +231,7 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin):
 
     def save_pretrained(self, save_directory):
         """ Save a model and its configuration file to a directory, so that it
-            can be re-loaded using the :func:`~transformers.PreTrainedModel.from_pretrained` class method.
+            can be re-loaded using the `:func:`~transformers.PreTrainedModel.from_pretrained`` class method.
         """
         assert os.path.isdir(
             save_directory
@@ -541,7 +541,7 @@ class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin):
                 ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
                 Defaults to `None`.
 
-                `What are attention masks? <../glossary.html#attention-mask>`__
+            `What are attention masks? <../glossary.html#attention-mask>`__
 
             decoder_start_token_id=None: (`optional`) int
                 If an encoder-decoder model starts decoding with a different token than BOS.

src/transformers/tokenization_t5.py

@@ -61,14 +61,14 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
 
 class T5Tokenizer(PreTrainedTokenizer):
     """
-        Constructs an XLNet tokenizer. Based on `SentencePiece <https://github.com/google/sentencepiece>`__ .
+        Constructs an XLNet tokenizer. Based on `SentencePiece <https://github.com/google/sentencepiece>`__
 
         This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the methods. Users
         should refer to the superclass for more information regarding methods.
 
         Args:
             vocab_file (:obj:`string`):
-                `SentencePiece <https://github.com/google/sentencepiece>`__ file (generally has a `.spm` extension) that
+                `SentencePiece <https://github.com/google/sentencepiece>`__ file (generally has a .spm extension) that
                 contains the vocabulary necessary to instantiate a tokenizer.
             eos_token (:obj:`string`, `optional`, defaults to "</s>"):
                 The end of sequence token.
@@ -84,8 +84,9 @@ class T5Tokenizer(PreTrainedTokenizer):
                 The token used for padding, for example when batching sequences of different lengths.
             extra_ids (:obj:`List[str]`, `optional`, defaults to :obj:`100`):
                 Add a number of extra ids added to the end of the vocabulary for use as sentinels.
-                These tokens are accessible as "<extra_id_{%d}>" where "{%d}" is a number between 0 and extra_ids-1.
-                Extra tokens are indexed from the end of the vocabulary up to beginnning ("<extra_id_0>" is the last token in the vocabulary like in T5 preprocessing
+                These tokens are accessible as `<extra_id_{%d}>` where `{%d}` is a number between 0 and extra_ids-1.
+                Extra tokens are indexed from the end of the vocabulary up to beginnning (<extra_id_0> is the last token in the vocabulary)
+                (like in T5 preprocessing
                 see: https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)
             additional_special_tokens (:obj:`List[str]`, `optional`, defaults to :obj:`None`):
                 Additional special tokens used by the tokenizer.