using black -t py35

.circleci/config.yml

@@ -101,7 +101,7 @@ jobs:
             # we need a version of isort with https://github.com/timothycrosley/isort/pull/1000
             - run: sudo pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort
             - run: sudo pip install .[tf,torch,quality]
-            - run: black --check --line-length 119 --target-version py35 examples templates tests src utils
+            - run: black --check --line-length 119 examples templates tests src utils
             - run: isort --check-only --recursive examples templates tests src utils
             - run: flake8 examples templates tests src utils
     check_repository_consistency:

.github/ISSUE_TEMPLATE/bug-report.md

@@ -44,7 +44,7 @@ Steps to reproduce the behavior:
 * PyTorch version:
 * PyTorch Transformers version (or branch):
 * Using GPU ?
-* Distributed or parallel setup ?
+* Distributed of parallel setup ?
 * Any other relevant information:
 
 ## Additional context

.github/ISSUE_TEMPLATE/migration.md

@@ -34,7 +34,7 @@ Details of the issue:
 * PyTorch version:
 * PyTorch Transformers version (or branch):
 * Using GPU ?
-* Distributed or parallel setup ?
+* Distributed of parallel setup ?
 * Any other relevant information:
 
 ## Checklist

Makefile

@@ -3,14 +3,14 @@
 # Check that source code meets quality standards
 
 quality:
-	black --check --line-length 119 --target-version py35 examples templates tests src utils
+	black --check --line-length 119 examples templates tests src utils
 	isort --check-only --recursive examples templates tests src utils
 	flake8 examples templates tests src utils
 
 # Format source code automatically
 
 style:
-	black --line-length 119 --target-version py35 examples templates tests src utils
+	black --line-length 119 examples templates tests src utils
 	isort --recursive examples templates tests src utils
 
 # Run tests for the library

README.md

@@ -160,8 +160,7 @@ At some point in the future, you'll be able to seamlessly move from pre-training
 12. **[T5](https://github.com/google-research/text-to-text-transfer-transformer)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
 13. **[XLM-RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
 14. **[MMBT](https://github.com/facebookresearch/mmbt/)** (from Facebook), released together with the paper a [Supervised Multimodal Bitransformers for Classifying Images and Text](https://arxiv.org/pdf/1909.02950.pdf) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
-15. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
-16. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
+15. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
 
 These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).
 
@@ -500,7 +499,7 @@ model = AutoModel.from_pretrained("username/pretrained_model")
 
 Finally, list all your files on S3:
 ```shell
-transformers-cli s3 ls
+transformers-cli ls
 # List all your S3 objects.
 ```
 

docs/source/converting_tensorflow_models.rst

@@ -3,12 +3,6 @@ Converting Tensorflow Checkpoints
 
 A command-line interface is provided to convert original Bert/GPT/GPT-2/Transformer-XL/XLNet/XLM checkpoints in models than be loaded using the ``from_pretrained`` methods of the library.
 
-.. note::
-    Since 2.3.0 the conversion script is now part of the transformers CLI (**transformers-cli**)
-    available in any transformers >= 2.3.0 installation.
-
-    The documentation below reflects the **transformers-cli convert** command format.
-
 BERT
 ^^^^
 
@@ -26,10 +20,10 @@ Here is an example of the conversion process for a pre-trained ``BERT-Base Uncas
 
    export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12
 
-   transformers-cli convert --model_type bert \
-     --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
-     --config $BERT_BASE_DIR/bert_config.json \
-     --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin
+   transformers bert \
+     $BERT_BASE_DIR/bert_model.ckpt \
+     $BERT_BASE_DIR/bert_config.json \
+     $BERT_BASE_DIR/pytorch_model.bin
 
 You can download Google's pre-trained models for the conversion `here <https://github.com/google-research/bert#pre-trained-models>`__.
 
@@ -42,12 +36,10 @@ Here is an example of the conversion process for a pre-trained OpenAI GPT model,
 
    export OPENAI_GPT_CHECKPOINT_FOLDER_PATH=/path/to/openai/pretrained/numpy/weights
 
-   transformers-cli convert --model_type gpt \
-     --tf_checkpoint $OPENAI_GPT_CHECKPOINT_FOLDER_PATH \
-     --pytorch_dump_output $PYTORCH_DUMP_OUTPUT \
-     [--config OPENAI_GPT_CONFIG] \
-     [--finetuning_task_name OPENAI_GPT_FINETUNED_TASK] \
-
+   transformers gpt \
+     $OPENAI_GPT_CHECKPOINT_FOLDER_PATH \
+     $PYTORCH_DUMP_OUTPUT \
+     [OPENAI_GPT_CONFIG]
 
 OpenAI GPT-2
 ^^^^^^^^^^^^
@@ -58,11 +50,10 @@ Here is an example of the conversion process for a pre-trained OpenAI GPT-2 mode
 
    export OPENAI_GPT2_CHECKPOINT_PATH=/path/to/gpt2/pretrained/weights
 
-   transformers-cli convert --model_type gpt2 \
-     --tf_checkpoint $OPENAI_GPT2_CHECKPOINT_PATH \
-     --pytorch_dump_output $PYTORCH_DUMP_OUTPUT \
-     [--config OPENAI_GPT2_CONFIG] \
-     [--finetuning_task_name OPENAI_GPT2_FINETUNED_TASK]
+   transformers gpt2 \
+     $OPENAI_GPT2_CHECKPOINT_PATH \
+     $PYTORCH_DUMP_OUTPUT \
+     [OPENAI_GPT2_CONFIG]
 
 Transformer-XL
 ^^^^^^^^^^^^^^
@@ -73,28 +64,27 @@ Here is an example of the conversion process for a pre-trained Transformer-XL mo
 
    export TRANSFO_XL_CHECKPOINT_FOLDER_PATH=/path/to/transfo/xl/checkpoint
 
-   transformers-cli convert --model_type transfo_xl \
-     --tf_checkpoint $TRANSFO_XL_CHECKPOINT_FOLDER_PATH \
-     --pytorch_dump_output $PYTORCH_DUMP_OUTPUT \
-     [--config TRANSFO_XL_CONFIG] \
-     [--finetuning_task_name TRANSFO_XL_FINETUNED_TASK]
+   transformers transfo_xl \
+     $TRANSFO_XL_CHECKPOINT_FOLDER_PATH \
+     $PYTORCH_DUMP_OUTPUT \
+     [TRANSFO_XL_CONFIG]
 
 
 XLNet
 ^^^^^
 
-Here is an example of the conversion process for a pre-trained XLNet model:
+Here is an example of the conversion process for a pre-trained XLNet model, fine-tuned on STS-B using the TensorFlow script:
 
 .. code-block:: shell
 
    export TRANSFO_XL_CHECKPOINT_PATH=/path/to/xlnet/checkpoint
    export TRANSFO_XL_CONFIG_PATH=/path/to/xlnet/config
 
-   transformers-cli convert --model_type xlnet \
-     --tf_checkpoint $TRANSFO_XL_CHECKPOINT_PATH \
-     --config $TRANSFO_XL_CONFIG_PATH \
-     --pytorch_dump_output $PYTORCH_DUMP_OUTPUT \
-     [--finetuning_task_name XLNET_FINETUNED_TASK] \
+   transformers xlnet \
+     $TRANSFO_XL_CHECKPOINT_PATH \
+     $TRANSFO_XL_CONFIG_PATH \
+     $PYTORCH_DUMP_OUTPUT \
+     STS-B \
 
 
 XLM
@@ -106,8 +96,6 @@ Here is an example of the conversion process for a pre-trained XLM model:
 
    export XLM_CHECKPOINT_PATH=/path/to/xlm/checkpoint
 
-   transformers-cli convert --model_type xlm \
-     --tf_checkpoint $XLM_CHECKPOINT_PATH \
-     --pytorch_dump_output $PYTORCH_DUMP_OUTPUT
-    [--config XML_CONFIG] \
-    [--finetuning_task_name XML_FINETUNED_TASK]
+   transformers xlm \
+     $XLM_CHECKPOINT_PATH \
+     $PYTORCH_DUMP_OUTPUT \

docs/source/model_sharing.md

@@ -34,7 +34,7 @@ model = AutoModel.from_pretrained("username/pretrained_model")
 
 Finally, list all your files on S3:
 ```shell
-transformers-cli s3 ls
+transformers-cli ls
 # List all your S3 objects.
 ```
 

examples/contrib/run_camembert.py

@@ -30,7 +30,7 @@ def fill_mask(masked_input, model, tokenizer, topk=5):
             )
         else:
             topk_filled_outputs.append(
-                (masked_input.replace(masked_token, predicted_token), values[index].item(), predicted_token,)
+                (masked_input.replace(masked_token, predicted_token), values[index].item(), predicted_token)
             )
     return topk_filled_outputs
 

examples/contrib/run_openai_gpt.py

@@ -83,7 +83,7 @@ def pre_process_datasets(encoded_datasets, input_len, cap_length, start_token, d
         mc_token_ids = np.zeros((n_batch, 2), dtype=np.int64)
         lm_labels = np.full((n_batch, 2, input_len), fill_value=-100, dtype=np.int64)
         mc_labels = np.zeros((n_batch,), dtype=np.int64)
-        for i, (story, cont1, cont2, mc_label), in enumerate(dataset):
+        for i, (story, cont1, cont2, mc_label) in enumerate(dataset):
             with_cont1 = [start_token] + story[:cap_length] + [delimiter_token] + cont1[:cap_length] + [clf_token]
             with_cont2 = [start_token] + story[:cap_length] + [delimiter_token] + cont2[:cap_length] + [clf_token]
             input_ids[i, 0, : len(with_cont1)] = with_cont1

examples/contrib/run_swag.py

@@ -51,9 +51,7 @@ logger = logging.getLogger(__name__)
 
 ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in [BertConfig]), ())
 
-MODEL_CLASSES = {
-    "bert": (BertConfig, BertForMultipleChoice, BertTokenizer),
-}
+MODEL_CLASSES = {"bert": (BertConfig, BertForMultipleChoice, BertTokenizer)}
 
 
 class SwagExample(object):
@@ -63,12 +61,7 @@ class SwagExample(object):
         self.swag_id = swag_id
         self.context_sentence = context_sentence
         self.start_ending = start_ending
-        self.endings = [
-            ending_0,
-            ending_1,
-            ending_2,
-            ending_3,
-        ]
+        self.endings = [ending_0, ending_1, ending_2, ending_3]
         self.label = label
 
     def __str__(self):

examples/distillation/run_squad_w_distillation.py

@@ -13,20 +13,20 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" This is the exact same script as `examples/run_squad.py` (as of 2020, January 8th) with an additional and optional step of distillation."""
+""" This is the exact same script as `examples/run_squad.py` (as of 2019, October 4th) with an additional and optional step of distillation."""
+
 
 import argparse
 import glob
 import logging
 import os
 import random
-import timeit
 
 import numpy as np
 import torch
 import torch.nn as nn
 import torch.nn.functional as F
-from torch.utils.data import DataLoader, RandomSampler, SequentialSampler
+from torch.utils.data import DataLoader, RandomSampler, SequentialSampler, TensorDataset
 from torch.utils.data.distributed import DistributedSampler
 from tqdm import tqdm, trange
 
@@ -46,14 +46,22 @@ from transformers import (
     XLNetForQuestionAnswering,
     XLNetTokenizer,
     get_linear_schedule_with_warmup,
-    squad_convert_examples_to_features,
 )
-from transformers.data.metrics.squad_metrics import (
-    compute_predictions_log_probs,
-    compute_predictions_logits,
-    squad_evaluate,
+
+from ..utils_squad import (
+    RawResult,
+    RawResultExtended,
+    convert_examples_to_features,
+    read_squad_examples,
+    write_predictions,
+    write_predictions_extended,
 )
-from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor
+
+# The follwing import is the official SQuAD evaluation script (2.0).
+# You can remove it from the dependencies if you are using this script outside of the library
+# We've added it here for automated tests (see examples/test_examples.py file)
+from ..utils_squad_evaluate import EVAL_OPTS
+from ..utils_squad_evaluate import main as evaluate_on_squad
 
 
 try:
@@ -116,21 +124,11 @@ def train(args, train_dataset, model, tokenizer, teacher=None):
     scheduler = get_linear_schedule_with_warmup(
         optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
     )
-
-    # Check if saved optimizer or scheduler states exist
-    if os.path.isfile(os.path.join(args.model_name_or_path, "optimizer.pt")) and os.path.isfile(
-        os.path.join(args.model_name_or_path, "scheduler.pt")
-    ):
-        # Load in optimizer and scheduler states
-        optimizer.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "optimizer.pt")))
-        scheduler.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "scheduler.pt")))
-
     if args.fp16:
         try:
             from apex import amp
         except ImportError:
             raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
-
         model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)
 
     # multi-gpu training (should be after apex fp16 initialization)
@@ -157,47 +155,18 @@ def train(args, train_dataset, model, tokenizer, teacher=None):
     logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
     logger.info("  Total optimization steps = %d", t_total)
 
-    global_step = 1
-    epochs_trained = 0
-    steps_trained_in_current_epoch = 0
-    # Check if continuing training from a checkpoint
-    if os.path.exists(args.model_name_or_path):
-        try:
-            # set global_step to gobal_step of last saved checkpoint from model path
-            checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0]
-            global_step = int(checkpoint_suffix)
-            epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
-            steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)
-
-            logger.info("  Continuing training from checkpoint, will skip to saved global_step")
-            logger.info("  Continuing training from epoch %d", epochs_trained)
-            logger.info("  Continuing training from global step %d", global_step)
-            logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
-        except ValueError:
-            logger.info("  Starting fine-tuning.")
-
+    global_step = 0
     tr_loss, logging_loss = 0.0, 0.0
     model.zero_grad()
-    train_iterator = trange(
-        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]
-    )
-    # Added here for reproductibility
-    set_seed(args)
-
+    train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0])
+    set_seed(args)  # Added here for reproductibility
     for _ in train_iterator:
         epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
         for step, batch in enumerate(epoch_iterator):
-
-            # Skip past any already trained steps if resuming training
-            if steps_trained_in_current_epoch > 0:
-                steps_trained_in_current_epoch -= 1
-                continue
-
             model.train()
             if teacher is not None:
                 teacher.eval()
             batch = tuple(t.to(args.device) for t in batch)
-
             inputs = {
                 "input_ids": batch[0],
                 "attention_mask": batch[1],
@@ -208,8 +177,6 @@ def train(args, train_dataset, model, tokenizer, teacher=None):
                 inputs["token_type_ids"] = None if args.model_type == "xlm" else batch[2]
             if args.model_type in ["xlnet", "xlm"]:
                 inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
-                if args.version_2_with_negative:
-                    inputs.update({"is_impossible": batch[7]})
             outputs = model(**inputs)
             loss, start_logits_stu, end_logits_stu = outputs
 
@@ -247,25 +214,23 @@ def train(args, train_dataset, model, tokenizer, teacher=None):
             if args.fp16:
                 with amp.scale_loss(loss, optimizer) as scaled_loss:
                     scaled_loss.backward()
+                torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
             else:
                 loss.backward()
+                torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
 
             tr_loss += loss.item()
             if (step + 1) % args.gradient_accumulation_steps == 0:
-                if args.fp16:
-                    torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
-                else:
-                    torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
-
                 optimizer.step()
                 scheduler.step()  # Update learning rate schedule
                 model.zero_grad()
                 global_step += 1
 
-                # Log metrics
                 if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
-                    # Only evaluate when single GPU otherwise metrics may not average well
-                    if args.local_rank == -1 and args.evaluate_during_training:
+                    # Log metrics
+                    if (
+                        args.local_rank == -1 and args.evaluate_during_training
+                    ):  # Only evaluate when single GPU otherwise metrics may not average well
                         results = evaluate(args, model, tokenizer)
                         for key, value in results.items():
                             tb_writer.add_scalar("eval_{}".format(key), value, global_step)
@@ -282,15 +247,9 @@ def train(args, train_dataset, model, tokenizer, teacher=None):
                         model.module if hasattr(model, "module") else model
                     )  # Take care of distributed/parallel training
                     model_to_save.save_pretrained(output_dir)
-                    tokenizer.save_pretrained(output_dir)
-
                     torch.save(args, os.path.join(output_dir, "training_args.bin"))
                     logger.info("Saving model checkpoint to %s", output_dir)
 
-                    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
-                    torch.save(scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
-                    logger.info("Saving optimizer and scheduler states to %s", output_dir)
-
             if args.max_steps > 0 and global_step > args.max_steps:
                 epoch_iterator.close()
                 break
@@ -311,27 +270,18 @@ def evaluate(args, model, tokenizer, prefix=""):
         os.makedirs(args.output_dir)
 
     args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
-
     # Note that DistributedSampler samples randomly
-    eval_sampler = SequentialSampler(dataset)
+    eval_sampler = SequentialSampler(dataset) if args.local_rank == -1 else DistributedSampler(dataset)
     eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)
 
-    # multi-gpu evaluate
-    if args.n_gpu > 1 and not isinstance(model, torch.nn.DataParallel):
-        model = torch.nn.DataParallel(model)
-
     # Eval!
     logger.info("***** Running evaluation {} *****".format(prefix))
     logger.info("  Num examples = %d", len(dataset))
     logger.info("  Batch size = %d", args.eval_batch_size)
-
     all_results = []
-    start_time = timeit.default_timer()
-
     for batch in tqdm(eval_dataloader, desc="Evaluating"):
         model.eval()
         batch = tuple(t.to(args.device) for t in batch)
-
         with torch.no_grad():
             inputs = {"input_ids": batch[0], "attention_mask": batch[1]}
             if args.model_type != "distilbert":
@@ -339,46 +289,30 @@ def evaluate(args, model, tokenizer, prefix=""):
             example_indices = batch[3]
             if args.model_type in ["xlnet", "xlm"]:
                 inputs.update({"cls_index": batch[4], "p_mask": batch[5]})
-
             outputs = model(**inputs)
 
         for i, example_index in enumerate(example_indices):
             eval_feature = features[example_index.item()]
             unique_id = int(eval_feature.unique_id)
-
-            output = [to_list(output[i]) for output in outputs]
-
-            # Some models (XLNet, XLM) use 5 arguments for their predictions, while the other "simpler"
-            # models only use two.
-            if len(output) >= 5:
-                start_logits = output[0]
-                start_top_index = output[1]
-                end_logits = output[2]
-                end_top_index = output[3]
-                cls_logits = output[4]
-
-                result = SquadResult(
-                    unique_id,
-                    start_logits,
-                    end_logits,
-                    start_top_index=start_top_index,
-                    end_top_index=end_top_index,
-                    cls_logits=cls_logits,
+            if args.model_type in ["xlnet", "xlm"]:
+                # XLNet uses a more complex post-processing procedure
+                result = RawResultExtended(
+                    unique_id=unique_id,
+                    start_top_log_probs=to_list(outputs[0][i]),
+                    start_top_index=to_list(outputs[1][i]),
+                    end_top_log_probs=to_list(outputs[2][i]),
+                    end_top_index=to_list(outputs[3][i]),
+                    cls_logits=to_list(outputs[4][i]),
                 )
-
             else:
-                start_logits, end_logits = output
-                result = SquadResult(unique_id, start_logits, end_logits)
-
+                result = RawResult(
+                    unique_id=unique_id, start_logits=to_list(outputs[0][i]), end_logits=to_list(outputs[1][i])
+                )
             all_results.append(result)
 
-    evalTime = timeit.default_timer() - start_time
-    logger.info("  Evaluation done in total %f secs (%f sec per example)", evalTime, evalTime / len(dataset))
-
     # Compute predictions
     output_prediction_file = os.path.join(args.output_dir, "predictions_{}.json".format(prefix))
     output_nbest_file = os.path.join(args.output_dir, "nbest_predictions_{}.json".format(prefix))
-
     if args.version_2_with_negative:
         output_null_log_odds_file = os.path.join(args.output_dir, "null_odds_{}.json".format(prefix))
     else:
@@ -386,7 +320,7 @@ def evaluate(args, model, tokenizer, prefix=""):
 
     if args.model_type in ["xlnet", "xlm"]:
         # XLNet uses a more complex post-processing procedure
-        predictions = compute_predictions_log_probs(
+        write_predictions_extended(
             examples,
             features,
             all_results,
@@ -395,6 +329,7 @@ def evaluate(args, model, tokenizer, prefix=""):
             output_prediction_file,
             output_nbest_file,
             output_null_log_odds_file,
+            args.predict_file,
             model.config.start_n_top,
             model.config.end_n_top,
             args.version_2_with_negative,
@@ -402,7 +337,7 @@ def evaluate(args, model, tokenizer, prefix=""):
             args.verbose_logging,
         )
     else:
-        predictions = compute_predictions_logits(
+        write_predictions(
             examples,
             features,
             all_results,
@@ -415,70 +350,76 @@ def evaluate(args, model, tokenizer, prefix=""):
             args.verbose_logging,
             args.version_2_with_negative,
             args.null_score_diff_threshold,
-            tokenizer,
         )
 
-    # Compute the F1 and exact scores.
-    results = squad_evaluate(examples, predictions)
+    # Evaluate with the official SQuAD script
+    evaluate_options = EVAL_OPTS(
+        data_file=args.predict_file, pred_file=output_prediction_file, na_prob_file=output_null_log_odds_file
+    )
+    results = evaluate_on_squad(evaluate_options)
     return results
 
 
 def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
     if args.local_rank not in [-1, 0] and not evaluate:
-        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
-        torch.distributed.barrier()
+        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache
 
     # Load data features from cache or dataset file
     input_file = args.predict_file if evaluate else args.train_file
     cached_features_file = os.path.join(
         os.path.dirname(input_file),
-        "cached_distillation_{}_{}_{}".format(
+        "cached_{}_{}_{}".format(
             "dev" if evaluate else "train",
             list(filter(None, args.model_name_or_path.split("/"))).pop(),
             str(args.max_seq_length),
         ),
     )
-    if os.path.exists(cached_features_file) and not args.overwrite_cache:
+    if os.path.exists(cached_features_file) and not args.overwrite_cache and not output_examples:
         logger.info("Loading features from cached file %s", cached_features_file)
-        features_and_dataset = torch.load(cached_features_file)
-
-        try:
-            features, dataset, examples = (
-                features_and_dataset["features"],
-                features_and_dataset["dataset"],
-                features_and_dataset["examples"],
-            )
-        except KeyError:
-            raise DeprecationWarning(
-                "You seem to be loading features from an older version of this script please delete the "
-                "file %s in order for it to be created again" % cached_features_file
-            )
+        features = torch.load(cached_features_file)
     else:
         logger.info("Creating features from dataset file at %s", input_file)
-        processor = SquadV2Processor() if args.version_2_with_negative else SquadV1Processor()
-        if evaluate:
-            examples = processor.get_dev_examples(args.data_dir, filename=args.predict_file)
-        else:
-            examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
-
-        features, dataset = squad_convert_examples_to_features(
+        examples = read_squad_examples(
+            input_file=input_file, is_training=not evaluate, version_2_with_negative=args.version_2_with_negative
+        )
+        features = convert_examples_to_features(
             examples=examples,
             tokenizer=tokenizer,
             max_seq_length=args.max_seq_length,
             doc_stride=args.doc_stride,
             max_query_length=args.max_query_length,
             is_training=not evaluate,
-            return_dataset="pt",
-            threads=args.threads,
         )
-
         if args.local_rank in [-1, 0]:
             logger.info("Saving features into cached file %s", cached_features_file)
-            torch.save({"features": features, "dataset": dataset, "examples": examples}, cached_features_file)
+            torch.save(features, cached_features_file)
 
     if args.local_rank == 0 and not evaluate:
-        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
-        torch.distributed.barrier()
+        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache
+
+    # Convert to Tensors and build dataset
+    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
+    all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)
+    all_segment_ids = torch.tensor([f.segment_ids for f in features], dtype=torch.long)
+    all_cls_index = torch.tensor([f.cls_index for f in features], dtype=torch.long)
+    all_p_mask = torch.tensor([f.p_mask for f in features], dtype=torch.float)
+    if evaluate:
+        all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)
+        dataset = TensorDataset(
+            all_input_ids, all_input_mask, all_segment_ids, all_example_index, all_cls_index, all_p_mask
+        )
+    else:
+        all_start_positions = torch.tensor([f.start_position for f in features], dtype=torch.long)
+        all_end_positions = torch.tensor([f.end_position for f in features], dtype=torch.long)
+        dataset = TensorDataset(
+            all_input_ids,
+            all_input_mask,
+            all_segment_ids,
+            all_start_positions,
+            all_end_positions,
+            all_cls_index,
+            all_p_mask,
+        )
 
     if output_examples:
         return dataset, examples, features
@@ -489,6 +430,16 @@ def main():
     parser = argparse.ArgumentParser()
 
     # Required parameters
+    parser.add_argument(
+        "--train_file", default=None, type=str, required=True, help="SQuAD json for training. E.g., train-v1.1.json"
+    )
+    parser.add_argument(
+        "--predict_file",
+        default=None,
+        type=str,
+        required=True,
+        help="SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json",
+    )
     parser.add_argument(
         "--model_type",
         default=None,
@@ -535,27 +486,6 @@ def main():
     )
 
     # Other parameters
-    parser.add_argument(
-        "--data_dir",
-        default=None,
-        type=str,
-        help="The input data dir. Should contain the .json files for the task."
-        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
-    )
-    parser.add_argument(
-        "--train_file",
-        default=None,
-        type=str,
-        help="The input training file. If a data dir is specified, will look for the file there"
-        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
-    )
-    parser.add_argument(
-        "--predict_file",
-        default=None,
-        type=str,
-        help="The input evaluation file. If a data dir is specified, will look for the file there"
-        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
-    )
     parser.add_argument(
         "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
     )
@@ -624,7 +554,7 @@ def main():
         default=1,
         help="Number of updates steps to accumulate before performing a backward/update pass.",
     )
-    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
+    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight deay if we apply some.")
     parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
     parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
     parser.add_argument(
@@ -688,8 +618,6 @@ def main():
     )
     parser.add_argument("--server_ip", type=str, default="", help="Can be used for distant debugging.")
     parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")
-
-    parser.add_argument("--threads", type=int, default=1, help="multiple threads for converting example to features")
     args = parser.parse_args()
 
     if (
@@ -744,8 +672,7 @@ def main():
 
     # Load pretrained model and tokenizer
     if args.local_rank not in [-1, 0]:
-        # Make sure only the first process in distributed training will download model & vocab
-        torch.distributed.barrier()
+        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
 
     args.model_type = args.model_type.lower()
     config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
@@ -782,24 +709,12 @@ def main():
         teacher = None
 
     if args.local_rank == 0:
-        # Make sure only the first process in distributed training will download model & vocab
-        torch.distributed.barrier()
+        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
 
     model.to(args.device)
 
     logger.info("Training/evaluation parameters %s", args)
 
-    # Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum if args.fp16 is set.
-    # Otherwise it'll default to "promote" mode, and we'll get fp32 operations. Note that running `--fp16_opt_level="O2"` will
-    # remove the need for this code, but it is still valid.
-    if args.fp16:
-        try:
-            import apex
-
-            apex.amp.register_half_function(torch, "einsum")
-        except ImportError:
-            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
-
     # Training
     if args.do_train:
         train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
@@ -825,15 +740,15 @@ def main():
         torch.save(args, os.path.join(args.output_dir, "training_args.bin"))
 
         # Load a trained model and vocabulary that you have fine-tuned
-        model = model_class.from_pretrained(args.output_dir)
-        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+        model = model_class.from_pretrained(args.output_dir, cache_dir=args.cache_dir if args.cache_dir else None)
+        tokenizer = tokenizer_class.from_pretrained(
+            args.output_dir, do_lower_case=args.do_lower_case, cache_dir=args.cache_dir if args.cache_dir else None
+        )
         model.to(args.device)
 
     # Evaluation - we can ask to evaluate all the checkpoints (sub-directories) in a directory
     results = {}
     if args.do_eval and args.local_rank in [-1, 0]:
-        if args.do_train:
-            logger.info("Loading checkpoints saved during training for evaluation")
         checkpoints = [args.output_dir]
         if args.eval_all_checkpoints:
             checkpoints = list(
@@ -846,7 +761,7 @@ def main():
         for checkpoint in checkpoints:
             # Reload the model
             global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
-            model = model_class.from_pretrained(checkpoint)
+            model = model_class.from_pretrained(checkpoint, cache_dir=args.cache_dir if args.cache_dir else None)
             model.to(args.device)
 
             # Evaluate

examples/distillation/utils.py

@@ -117,9 +117,7 @@ def init_gpu_params(params):
     # initialize multi-GPU
     if params.multi_gpu:
         logger.info("Initializing PyTorch distributed")
-        torch.distributed.init_process_group(
-            init_method="env://", backend="nccl",
-        )
+        torch.distributed.init_process_group(init_method="env://", backend="nccl")
 
 
 def set_seed(args):

examples/mm-imdb/utils_mmimdb.py

@@ -138,6 +138,6 @@ def get_image_transforms():
             transforms.Resize(256),
             transforms.CenterCrop(224),
             transforms.ToTensor(),
-            transforms.Normalize(mean=[0.46777044, 0.44531429, 0.40661017], std=[0.12221994, 0.12145835, 0.14380469],),
+            transforms.Normalize(mean=[0.46777044, 0.44531429, 0.40661017], std=[0.12221994, 0.12145835, 0.14380469]),
         ]
     )

examples/pplm/run_pplm.py

@@ -718,7 +718,7 @@ if __name__ == "__main__":
     parser.add_argument("--cond_text", type=str, default="The lake", help="Prefix texts to condition on")
     parser.add_argument("--uncond", action="store_true", help="Generate from end-of-text as prefix")
     parser.add_argument(
-        "--num_samples", type=int, default=1, help="Number of samples to generate from the modified latents",
+        "--num_samples", type=int, default=1, help="Number of samples to generate from the modified latents"
     )
     parser.add_argument(
         "--bag_of_words",
@@ -741,9 +741,7 @@ if __name__ == "__main__":
     parser.add_argument(
         "--discrim_meta", type=str, default=None, help="Meta information for the generic discriminator"
     )
-    parser.add_argument(
-        "--class_label", type=int, default=-1, help="Class label used for the discriminator",
-    )
+    parser.add_argument("--class_label", type=int, default=-1, help="Class label used for the discriminator")
     parser.add_argument("--length", type=int, default=100)
     parser.add_argument("--stepsize", type=float, default=0.02)
     parser.add_argument("--temperature", type=float, default=1.0)
@@ -757,9 +755,7 @@ if __name__ == "__main__":
         default=0,
         help="Length of past which is being optimized; " "0 corresponds to infinite window length",
     )
-    parser.add_argument(
-        "--horizon_length", type=int, default=1, help="Length of future to optimize over",
-    )
+    parser.add_argument("--horizon_length", type=int, default=1, help="Length of future to optimize over")
     parser.add_argument("--decay", action="store_true", help="whether to decay or not")
     parser.add_argument("--gamma", type=float, default=1.5)
     parser.add_argument("--gm_scale", type=float, default=0.9)

examples/pplm/run_pplm_discrim_train.py

@@ -242,7 +242,7 @@ def train_discriminator(
 
         text = torchtext_data.Field()
         label = torchtext_data.Field(sequential=False)
-        train_data, val_data, test_data = datasets.SST.splits(text, label, fine_grained=True, train_subtrees=True,)
+        train_data, val_data, test_data = datasets.SST.splits(text, label, fine_grained=True, train_subtrees=True)
 
         x = []
         y = []

examples/run_generation.py

@@ -41,7 +41,7 @@ from transformers import (
 
 
 logging.basicConfig(
-    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s", datefmt="%m/%d/%Y %H:%M:%S", level=logging.INFO,
+    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s", datefmt="%m/%d/%Y %H:%M:%S", level=logging.INFO
 )
 logger = logging.getLogger(__name__)
 
@@ -212,7 +212,6 @@ def main():
         prepare_input = PREPROCESSING_FUNCTIONS.get(args.model_type)
         prompt_text = prepare_input(args, model, tokenizer, prompt_text)
     encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
-    encoded_prompt = encoded_prompt.to(args.device)
 
     output_sequences = model.generate(
         input_ids=encoded_prompt,

examples/run_glue.py

@@ -13,7 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa, Albert, XLM-RoBERTa)."""
+""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
 
 
 import argparse
@@ -72,15 +72,7 @@ logger = logging.getLogger(__name__)
 ALL_MODELS = sum(
     (
         tuple(conf.pretrained_config_archive_map.keys())
-        for conf in (
-            BertConfig,
-            XLNetConfig,
-            XLMConfig,
-            RobertaConfig,
-            DistilBertConfig,
-            AlbertConfig,
-            XLMRobertaConfig,
-        )
+        for conf in (BertConfig, XLNetConfig, XLMConfig, RobertaConfig, DistilBertConfig)
     ),
     (),
 )
@@ -156,7 +148,7 @@ def train(args, train_dataset, model, tokenizer):
     # Distributed training (should be after apex fp16 initialization)
     if args.local_rank != -1:
         model = torch.nn.parallel.DistributedDataParallel(
-            model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True,
+            model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True
         )
 
     # Train!
@@ -191,7 +183,7 @@ def train(args, train_dataset, model, tokenizer):
     tr_loss, logging_loss = 0.0, 0.0
     model.zero_grad()
     train_iterator = trange(
-        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0],
+        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]
     )
     set_seed(args)  # Added here for reproductibility
     for _ in train_iterator:
@@ -208,8 +200,8 @@ def train(args, train_dataset, model, tokenizer):
             inputs = {"input_ids": batch[0], "attention_mask": batch[1], "labels": batch[3]}
             if args.model_type != "distilbert":
                 inputs["token_type_ids"] = (
-                    batch[2] if args.model_type in ["bert", "xlnet", "albert"] else None
-                )  # XLM, DistilBERT, RoBERTa, and XLM-RoBERTa don't use segment_ids
+                    batch[2] if args.model_type in ["bert", "xlnet"] else None
+                )  # XLM, DistilBERT and RoBERTa don't use segment_ids
             outputs = model(**inputs)
             loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
 
@@ -324,8 +316,8 @@ def evaluate(args, model, tokenizer, prefix=""):
                 inputs = {"input_ids": batch[0], "attention_mask": batch[1], "labels": batch[3]}
                 if args.model_type != "distilbert":
                     inputs["token_type_ids"] = (
-                        batch[2] if args.model_type in ["bert", "xlnet", "albert"] else None
-                    )  # XLM, DistilBERT, RoBERTa, and XLM-RoBERTa don't use segment_ids
+                        batch[2] if args.model_type in ["bert", "xlnet"] else None
+                    )  # XLM, DistilBERT and RoBERTa don't use segment_ids
                 outputs = model(**inputs)
                 tmp_eval_loss, logits = outputs[:2]
 
@@ -456,7 +448,7 @@ def main():
 
     # Other parameters
     parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name",
+        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
     )
     parser.add_argument(
         "--tokenizer_name",
@@ -480,17 +472,15 @@ def main():
     parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
     parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
     parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step.",
+        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
     )
     parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model.",
+        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
     )
 
+    parser.add_argument("--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
     parser.add_argument(
-        "--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.",
-    )
-    parser.add_argument(
-        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation.",
+        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation."
     )
     parser.add_argument(
         "--gradient_accumulation_steps",
@@ -503,7 +493,7 @@ def main():
     parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
     parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
     parser.add_argument(
-        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform.",
+        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform."
     )
     parser.add_argument(
         "--max_steps",
@@ -522,10 +512,10 @@ def main():
     )
     parser.add_argument("--no_cuda", action="store_true", help="Avoid using CUDA when available")
     parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory",
+        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
     )
     parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets",
+        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
     )
     parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
 

examples/run_lm_finetuning.py

@@ -28,7 +28,6 @@ import pickle
 import random
 import re
 import shutil
-from typing import Tuple
 
 import numpy as np
 import torch
@@ -54,7 +53,6 @@ from transformers import (
     OpenAIGPTConfig,
     OpenAIGPTLMHeadModel,
     OpenAIGPTTokenizer,
-    PreTrainedTokenizer,
     RobertaConfig,
     RobertaForMaskedLM,
     RobertaTokenizer,
@@ -166,7 +164,7 @@ def _rotate_checkpoints(args, checkpoint_prefix, use_mtime=False):
         shutil.rmtree(checkpoint)
 
 
-def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> Tuple[torch.Tensor, torch.Tensor]:
+def mask_tokens(inputs, tokenizer, args):
     """ Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. """
     labels = inputs.clone()
     # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
@@ -264,19 +262,15 @@ def train(args, train_dataset, model, tokenizer):
     steps_trained_in_current_epoch = 0
     # Check if continuing training from a checkpoint
     if os.path.exists(args.model_name_or_path):
-        try:
-            # set global_step to gobal_step of last saved checkpoint from model path
-            checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0]
-            global_step = int(checkpoint_suffix)
-            epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
-            steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)
+        # set global_step to gobal_step of last saved checkpoint from model path
+        global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
+        epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
+        steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)
 
-            logger.info("  Continuing training from checkpoint, will skip to saved global_step")
-            logger.info("  Continuing training from epoch %d", epochs_trained)
-            logger.info("  Continuing training from global step %d", global_step)
-            logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
-        except ValueError:
-            logger.info("  Starting fine-tuning.")
+        logger.info("  Continuing training from checkpoint, will skip to saved global_step")
+        logger.info("  Continuing training from epoch %d", epochs_trained)
+        logger.info("  Continuing training from global step %d", global_step)
+        logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
 
     tr_loss, logging_loss = 0.0, 0.0
 
@@ -478,7 +472,7 @@ def main():
         "--cache_dir",
         default="",
         type=str,
-        help="Optional directory to store the pre-trained models downloaded from s3 (instead of the default one)",
+        help="Optional directory to store the pre-trained models downloaded from s3 (instread of the default one)",
     )
     parser.add_argument(
         "--block_size",

examples/run_squad.py

@@ -170,19 +170,15 @@ def train(args, train_dataset, model, tokenizer):
     steps_trained_in_current_epoch = 0
     # Check if continuing training from a checkpoint
     if os.path.exists(args.model_name_or_path):
-        try:
-            # set global_step to gobal_step of last saved checkpoint from model path
-            checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0]
-            global_step = int(checkpoint_suffix)
-            epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
-            steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)
+        # set global_step to gobal_step of last saved checkpoint from model path
+        global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
+        epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
+        steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)
 
-            logger.info("  Continuing training from checkpoint, will skip to saved global_step")
-            logger.info("  Continuing training from epoch %d", epochs_trained)
-            logger.info("  Continuing training from global step %d", global_step)
-            logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
-        except ValueError:
-            logger.info("  Starting fine-tuning.")
+        logger.info("  Continuing training from checkpoint, will skip to saved global_step")
+        logger.info("  Continuing training from epoch %d", epochs_trained)
+        logger.info("  Continuing training from global step %d", global_step)
+        logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
 
     tr_loss, logging_loss = 0.0, 0.0
     model.zero_grad()
@@ -207,14 +203,11 @@ def train(args, train_dataset, model, tokenizer):
             inputs = {
                 "input_ids": batch[0],
                 "attention_mask": batch[1],
-                "token_type_ids": batch[2],
+                "token_type_ids": None if args.model_type in ["xlm", "roberta", "distilbert"] else batch[2],
                 "start_positions": batch[3],
                 "end_positions": batch[4],
             }
 
-            if args.model_type in ["xlm", "roberta", "distilbert"]:
-                del inputs["token_type_ids"]
-
             if args.model_type in ["xlnet", "xlm"]:
                 inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
                 if args.version_2_with_negative:
@@ -319,12 +312,8 @@ def evaluate(args, model, tokenizer, prefix=""):
             inputs = {
                 "input_ids": batch[0],
                 "attention_mask": batch[1],
-                "token_type_ids": batch[2],
+                "token_type_ids": None if args.model_type in ["xlm", "roberta", "distilbert"] else batch[2],
             }
-
-            if args.model_type in ["xlm", "roberta", "distilbert"]:
-                del inputs["token_type_ids"]
-
             example_indices = batch[3]
 
             # XLNet and XLM use more arguments for their predictions
@@ -434,14 +423,10 @@ def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=Fal
     )
 
     # Init features and dataset from cache if it exists
-    if os.path.exists(cached_features_file) and not args.overwrite_cache:
+    if os.path.exists(cached_features_file) and not args.overwrite_cache and not output_examples:
         logger.info("Loading features from cached file %s", cached_features_file)
         features_and_dataset = torch.load(cached_features_file)
-        features, dataset, examples = (
-            features_and_dataset["features"],
-            features_and_dataset["dataset"],
-            features_and_dataset["examples"],
-        )
+        features, dataset = features_and_dataset["features"], features_and_dataset["dataset"]
     else:
         logger.info("Creating features from dataset file at %s", input_dir)
 
@@ -476,7 +461,7 @@ def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=Fal
 
         if args.local_rank in [-1, 0]:
             logger.info("Saving features into cached file %s", cached_features_file)
-            torch.save({"features": features, "dataset": dataset, "examples": examples}, cached_features_file)
+            torch.save({"features": features, "dataset": dataset}, cached_features_file)
 
     if args.local_rank == 0 and not evaluate:
         # Make sure only the first process in distributed training process the dataset, and the others will use the cache
@@ -787,7 +772,7 @@ def main():
         torch.save(args, os.path.join(args.output_dir, "training_args.bin"))
 
         # Load a trained model and vocabulary that you have fine-tuned
-        model = model_class.from_pretrained(args.output_dir)  # , force_download=True)
+        model = model_class.from_pretrained(args.output_dir, force_download=True)
         tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
         model.to(args.device)
 
@@ -812,7 +797,7 @@ def main():
         for checkpoint in checkpoints:
             # Reload the model
             global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
-            model = model_class.from_pretrained(checkpoint)  # , force_download=True)
+            model = model_class.from_pretrained(checkpoint, force_download=True)
             model.to(args.device)
 
             # Evaluate

examples/run_tf_ner.py

@@ -9,6 +9,7 @@ import re
 import numpy as np
 import tensorflow as tf
 from absl import app, flags, logging
+from fastprogress import master_bar, progress_bar
 from seqeval import metrics
 
 from transformers import (
@@ -28,12 +29,6 @@ from transformers import (
 from utils_ner import convert_examples_to_features, get_labels, read_examples_from_file
 
 
-try:
-    from fastprogress import master_bar, progress_bar
-except ImportError:
-    from fastprogress.fastprogress import master_bar, progress_bar
-
-
 ALL_MODELS = sum(
     (tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, RobertaConfig, DistilBertConfig)), ()
 )

examples/summarization/configuration_bertabs.py

@@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
 
 
 BERTABS_FINETUNED_CONFIG_MAP = {
-    "bertabs-finetuned-cnndm": "https://s3.amazonaws.com/models.huggingface.co/bert/remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization-config.json",
+    "bertabs-finetuned-cnndm": "https://s3.amazonaws.com/models.huggingface.co/bert/remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization-config.json"
 }
 
 

examples/summarization/convert_bertabs_original_pytorch_checkpoint.py

@@ -164,13 +164,11 @@ def convert_bertabs_checkpoints(path_to_checkpoints, dump_path):
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument(
-        "--bertabs_checkpoint_path", default=None, type=str, required=True, help="Path the official PyTorch dump.",
+        "--bertabs_checkpoint_path", default=None, type=str, required=True, help="Path the official PyTorch dump."
     )
     parser.add_argument(
-        "--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model.",
+        "--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
     )
     args = parser.parse_args()
 
-    convert_bertabs_checkpoints(
-        args.bertabs_checkpoint_path, args.pytorch_dump_folder_path,
-    )
+    convert_bertabs_checkpoints(args.bertabs_checkpoint_path, args.pytorch_dump_folder_path)

examples/summarization/modeling_bertabs.py

@@ -34,7 +34,7 @@ from transformers import BertConfig, BertModel, PreTrainedModel
 MAX_SIZE = 5000
 
 BERTABS_FINETUNED_MODEL_MAP = {
-    "bertabs-finetuned-cnndm": "https://s3.amazonaws.com/models.huggingface.co/bert/remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization-pytorch_model.bin",
+    "bertabs-finetuned-cnndm": "https://s3.amazonaws.com/models.huggingface.co/bert/remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization-pytorch_model.bin"
 }
 
 
@@ -106,10 +106,10 @@ class BertAbs(BertAbsPreTrainedModel):
                 p.data.zero_()
 
     def forward(
-        self, encoder_input_ids, decoder_input_ids, token_type_ids, encoder_attention_mask, decoder_attention_mask,
+        self, encoder_input_ids, decoder_input_ids, token_type_ids, encoder_attention_mask, decoder_attention_mask
     ):
         encoder_output = self.bert(
-            input_ids=encoder_input_ids, token_type_ids=token_type_ids, attention_mask=encoder_attention_mask,
+            input_ids=encoder_input_ids, token_type_ids=token_type_ids, attention_mask=encoder_attention_mask
         )
         encoder_hidden_states = encoder_output[0]
         dec_state = self.decoder.init_decoder_state(encoder_input_ids, encoder_hidden_states)
@@ -308,7 +308,7 @@ class TransformerDecoderLayer(nn.Module):
         self.register_buffer("mask", mask)
 
     def forward(
-        self, inputs, memory_bank, src_pad_mask, tgt_pad_mask, previous_input=None, layer_cache=None, step=None,
+        self, inputs, memory_bank, src_pad_mask, tgt_pad_mask, previous_input=None, layer_cache=None, step=None
     ):
         """
         Args:
@@ -332,13 +332,13 @@ class TransformerDecoderLayer(nn.Module):
             all_input = torch.cat((previous_input, input_norm), dim=1)
             dec_mask = None
 
-        query = self.self_attn(all_input, all_input, input_norm, mask=dec_mask, layer_cache=layer_cache, type="self",)
+        query = self.self_attn(all_input, all_input, input_norm, mask=dec_mask, layer_cache=layer_cache, type="self")
 
         query = self.drop(query) + inputs
 
         query_norm = self.layer_norm_2(query)
         mid = self.context_attn(
-            memory_bank, memory_bank, query_norm, mask=src_pad_mask, layer_cache=layer_cache, type="context",
+            memory_bank, memory_bank, query_norm, mask=src_pad_mask, layer_cache=layer_cache, type="context"
         )
         output = self.feed_forward(self.drop(mid) + query)
 
@@ -422,9 +422,7 @@ class MultiHeadedAttention(nn.Module):
         if self.use_final_linear:
             self.final_linear = nn.Linear(model_dim, model_dim)
 
-    def forward(
-        self, key, value, query, mask=None, layer_cache=None, type=None, predefined_graph_1=None,
-    ):
+    def forward(self, key, value, query, mask=None, layer_cache=None, type=None, predefined_graph_1=None):
         """
         Compute the context vector and the attention vectors.
 
@@ -458,11 +456,7 @@ class MultiHeadedAttention(nn.Module):
         # 1) Project key, value, and query.
         if layer_cache is not None:
             if type == "self":
-                query, key, value = (
-                    self.linear_query(query),
-                    self.linear_keys(query),
-                    self.linear_values(query),
-                )
+                query, key, value = (self.linear_query(query), self.linear_keys(query), self.linear_values(query))
 
                 key = shape(key)
                 value = shape(value)
@@ -483,10 +477,7 @@ class MultiHeadedAttention(nn.Module):
                         key = shape(key)
                         value = shape(value)
                     else:
-                        key, value = (
-                            layer_cache["memory_keys"],
-                            layer_cache["memory_values"],
-                        )
+                        key, value = (layer_cache["memory_keys"], layer_cache["memory_values"])
                     layer_cache["memory_keys"] = key
                     layer_cache["memory_values"] = value
                 else:
@@ -999,12 +990,8 @@ class BertSumOptimizer(object):
         self.warmup_steps = warmup_steps
 
         self.optimizers = {
-            "encoder": torch.optim.Adam(
-                model.encoder.parameters(), lr=lr["encoder"], betas=(beta_1, beta_2), eps=eps,
-            ),
-            "decoder": torch.optim.Adam(
-                model.decoder.parameters(), lr=lr["decoder"], betas=(beta_1, beta_2), eps=eps,
-            ),
+            "encoder": torch.optim.Adam(model.encoder.parameters(), lr=lr["encoder"], betas=(beta_1, beta_2), eps=eps),
+            "decoder": torch.optim.Adam(model.decoder.parameters(), lr=lr["decoder"], betas=(beta_1, beta_2), eps=eps),
         }
 
         self._step = 0

examples/summarization/run_summarization.py

@@ -188,7 +188,7 @@ def build_data_iterator(args, tokenizer):
     def collate_fn(data):
         return collate(data, tokenizer, block_size=512, device=args.device)
 
-    iterator = DataLoader(dataset, sampler=sampler, batch_size=args.batch_size, collate_fn=collate_fn,)
+    iterator = DataLoader(dataset, sampler=sampler, batch_size=args.batch_size, collate_fn=collate_fn)
 
     return iterator
 
@@ -265,24 +265,14 @@ def main():
         help="Compute the ROUGE metrics during evaluation. Only available for the CNN/DailyMail dataset.",
     )
     # EVALUATION options
-    parser.add_argument(
-        "--no_cuda", default=False, type=bool, help="Whether to force the execution on CPU.",
-    )
-    parser.add_argument(
-        "--batch_size", default=4, type=int, help="Batch size per GPU/CPU for training.",
-    )
+    parser.add_argument("--no_cuda", default=False, type=bool, help="Whether to force the execution on CPU.")
+    parser.add_argument("--batch_size", default=4, type=int, help="Batch size per GPU/CPU for training.")
     # BEAM SEARCH arguments
+    parser.add_argument("--min_length", default=50, type=int, help="Minimum number of tokens for the summaries.")
+    parser.add_argument("--max_length", default=200, type=int, help="Maixmum number of tokens for the summaries.")
+    parser.add_argument("--beam_size", default=5, type=int, help="The number of beams to start with for each example.")
     parser.add_argument(
-        "--min_length", default=50, type=int, help="Minimum number of tokens for the summaries.",
-    )
-    parser.add_argument(
-        "--max_length", default=200, type=int, help="Maixmum number of tokens for the summaries.",
-    )
-    parser.add_argument(
-        "--beam_size", default=5, type=int, help="The number of beams to start with for each example.",
-    )
-    parser.add_argument(
-        "--alpha", default=0.95, type=float, help="The value of alpha for the length penalty in the beam search.",
+        "--alpha", default=0.95, type=float, help="The value of alpha for the length penalty in the beam search."
     )
     parser.add_argument(
         "--block_trigram",

examples/utils_multiple_choice.py

@@ -320,7 +320,7 @@ def convert_examples_to_features(
             else:
                 text_b = example.question + " " + ending
 
-            inputs = tokenizer.encode_plus(text_a, text_b, add_special_tokens=True, max_length=max_length,)
+            inputs = tokenizer.encode_plus(text_a, text_b, add_special_tokens=True, max_length=max_length)
             if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                 logger.info(
                     "Attention! you are cropping tokens (swag task is ok). "
@@ -362,7 +362,7 @@ def convert_examples_to_features(
                 logger.info("token_type_ids: {}".format(" ".join(map(str, token_type_ids))))
                 logger.info("label: {}".format(label))
 
-        features.append(InputFeatures(example_id=example.example_id, choices_features=choices_features, label=label,))
+        features.append(InputFeatures(example_id=example.example_id, choices_features=choices_features, label=label))
 
     return features
 

setup.py

@@ -62,6 +62,7 @@ extras["mecab"] = ["mecab-python3"]
 extras["sklearn"] = ["scikit-learn"]
 extras["tf"] = ["tensorflow"]
 extras["torch"] = ["torch"]
+extras["fast"] = ["tokenizers == 0.0.10"]
 
 extras["serving"] = ["pydantic", "uvicorn", "fastapi"]
 extras["all"] = extras["serving"] + ["tensorflow", "torch"]
@@ -86,7 +87,6 @@ setup(
     packages=find_packages("src"),
     install_requires=[
         "numpy",
-        "tokenizers == 0.0.11",
         # accessing files from S3 directly
         "boto3",
         # filesystem locks e.g. to prevent parallel downloads

src/transformers/__init__.py

@@ -70,6 +70,7 @@ from .file_utils import (
     add_end_docstrings,
     add_start_docstrings,
     cached_path,
+    is_fast_tokenizers_available,
     is_tf_available,
     is_torch_available,
 )

src/transformers/__main__.py

@@ -0,0 +1,42 @@
+# coding: utf8
+
+
+def main():
+    import sys
+
+    if len(sys.argv) < 2 or sys.argv[1] not in ["convert", "train", "predict", "serve"]:
+        print(
+            "First argument to `transformers` command line interface should be one of: \n"
+            ">> convert serve train predict"
+        )
+    if sys.argv[1] == "convert":
+        from transformers.commands import convert
+
+        convert(sys.argv)
+    elif sys.argv[1] == "train":
+        from transformers.commands import train
+
+        train(sys.argv)
+    elif sys.argv[1] == "serve":
+        pass
+        # from argparse import ArgumentParser
+        # from transformers.commands.serving import ServeCommand
+        # parser = ArgumentParser('Transformers CLI tool', usage='transformers serve <command> [<args>]')
+        # commands_parser = parser.add_subparsers(help='transformers-cli command helpers')
+
+        # # Register commands
+        # ServeCommand.register_subcommand(commands_parser)
+
+        # # Let's go
+        # args = parser.parse_args()
+
+        # if not hasattr(args, 'func'):
+        #     parser.print_help()
+        #     exit(1)
+        # # Run
+        # service = args.func(args)
+        # service.run()
+
+
+if __name__ == "__main__":
+    main()

src/transformers/commands/convert.py

@@ -25,7 +25,7 @@ class ConvertCommand(BaseTransformersCLICommand):
         train_parser = parser.add_parser(
             "convert",
             help="CLI tool to run convert model from original "
-            "author checkpoints to Transformers PyTorch checkpoints.",
+            "author checkpoints to Transformesr PyTorch checkpoints.",
         )
         train_parser.add_argument("--model_type", type=str, required=True, help="Model's type.")
         train_parser.add_argument(

src/transformers/commands/user.py

@@ -9,26 +9,17 @@ from transformers.commands import BaseTransformersCLICommand
 from transformers.hf_api import HfApi, HfFolder
 
 
-UPLOAD_MAX_FILES = 15
-
-
 class UserCommands(BaseTransformersCLICommand):
     @staticmethod
     def register_subcommand(parser: ArgumentParser):
-        login_parser = parser.add_parser("login", help="Log in using the same credentials as on huggingface.co")
+        login_parser = parser.add_parser("login")
         login_parser.set_defaults(func=lambda args: LoginCommand(args))
-        whoami_parser = parser.add_parser("whoami", help="Find out which huggingface.co account you are logged in as.")
+        whoami_parser = parser.add_parser("whoami")
         whoami_parser.set_defaults(func=lambda args: WhoamiCommand(args))
-        logout_parser = parser.add_parser("logout", help="Log out")
+        logout_parser = parser.add_parser("logout")
         logout_parser.set_defaults(func=lambda args: LogoutCommand(args))
-        # s3
-        s3_parser = parser.add_parser("s3", help="{ls, rm} Commands to interact with the files you upload on S3.")
-        s3_subparsers = s3_parser.add_subparsers(help="s3 related commands")
-        ls_parser = s3_subparsers.add_parser("ls")
-        ls_parser.set_defaults(func=lambda args: ListObjsCommand(args))
-        rm_parser = s3_subparsers.add_parser("rm")
-        rm_parser.add_argument("filename", type=str, help="individual object filename to delete from S3.")
-        rm_parser.set_defaults(func=lambda args: DeleteObjCommand(args))
+        list_parser = parser.add_parser("ls")
+        list_parser.set_defaults(func=lambda args: ListObjsCommand(args))
         # upload
         upload_parser = parser.add_parser("upload")
         upload_parser.add_argument("path", type=str, help="Local path of the folder or individual file to upload.")
@@ -140,27 +131,13 @@ class ListObjsCommand(BaseUserCommand):
         print(self.tabulate(rows, headers=["Filename", "LastModified", "ETag", "Size"]))
 
 
-class DeleteObjCommand(BaseUserCommand):
-    def run(self):
-        token = HfFolder.get_token()
-        if token is None:
-            print("Not logged in")
-            exit(1)
-        try:
-            self._api.delete_obj(token, filename=self.args.filename)
-        except HTTPError as e:
-            print(e)
-            exit(1)
-        print("Done")
-
-
 class UploadCommand(BaseUserCommand):
     def walk_dir(self, rel_path):
         """
         Recursively list all files in a folder.
         """
         entries: List[os.DirEntry] = list(os.scandir(rel_path))
-        files = [(os.path.join(os.getcwd(), f.path), f.path) for f in entries if f.is_file()]  # (filepath, filename)
+        files = [(os.path.join(os.getcwd(), f.path), f.path) for f in entries if f.is_file()]  # filepath  # filename
         for f in entries:
             if f.is_dir():
                 files += self.walk_dir(f.path)
@@ -183,14 +160,6 @@ class UploadCommand(BaseUserCommand):
         else:
             raise ValueError("Not a valid file or directory: {}".format(local_path))
 
-        if len(files) > UPLOAD_MAX_FILES:
-            print(
-                "About to upload {} files to S3. This is probably wrong. Please filter files before uploading.".format(
-                    ANSI.bold(len(files))
-                )
-            )
-            exit(1)
-
         for filepath, filename in files:
             print("About to upload file {} to S3 under filename {}".format(ANSI.bold(filepath), ANSI.bold(filename)))
 

src/transformers/configuration_camembert.py

@@ -24,7 +24,7 @@ from .configuration_roberta import RobertaConfig
 logger = logging.getLogger(__name__)
 
 CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
-    "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-config.json",
+    "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-config.json"
 }
 
 

src/transformers/configuration_transfo_xl.py

@@ -24,7 +24,7 @@ from .configuration_utils import PretrainedConfig
 logger = logging.getLogger(__name__)
 
 TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP = {
-    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-config.json",
+    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-config.json"
 }
 
 

src/transformers/data/metrics/__init__.py

@@ -35,20 +35,12 @@ if _has_sklearn:
     def acc_and_f1(preds, labels):
         acc = simple_accuracy(preds, labels)
         f1 = f1_score(y_true=labels, y_pred=preds)
-        return {
-            "acc": acc,
-            "f1": f1,
-            "acc_and_f1": (acc + f1) / 2,
-        }
+        return {"acc": acc, "f1": f1, "acc_and_f1": (acc + f1) / 2}
 
     def pearson_and_spearman(preds, labels):
         pearson_corr = pearsonr(preds, labels)[0]
         spearman_corr = spearmanr(preds, labels)[0]
-        return {
-            "pearson": pearson_corr,
-            "spearmanr": spearman_corr,
-            "corr": (pearson_corr + spearman_corr) / 2,
-        }
+        return {"pearson": pearson_corr, "spearmanr": spearman_corr, "corr": (pearson_corr + spearman_corr) / 2}
 
     def glue_compute_metrics(task_name, preds, labels):
         assert len(preds) == len(labels)

src/transformers/data/processors/glue.py

@@ -81,12 +81,12 @@ def glue_convert_examples_to_features(
     features = []
     for (ex_index, example) in enumerate(examples):
         if ex_index % 10000 == 0:
-            logger.info("Writing example %d/%d" % (ex_index, len(examples)))
+            logger.info("Writing example %d" % (ex_index))
         if is_tf_dataset:
             example = processor.get_example_from_tensor_dict(example)
             example = processor.tfds_map(example)
 
-        inputs = tokenizer.encode_plus(example.text_a, example.text_b, add_special_tokens=True, max_length=max_length,)
+        inputs = tokenizer.encode_plus(example.text_a, example.text_b, add_special_tokens=True, max_length=max_length)
         input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
 
         # The mask has 1 for real tokens and 0 for padding tokens. Only real

src/transformers/data/processors/utils.py

@@ -93,33 +93,6 @@ class InputFeatures(object):
 class DataProcessor(object):
     """Base class for data converters for sequence classification data sets."""
 
-    def get_example_from_tensor_dict(self, tensor_dict):
-        """Gets an example from a dict with tensorflow tensors
-        Args:
-            tensor_dict: Keys and values should match the corresponding Glue
-                tensorflow_dataset examples.
-        """
-        raise NotImplementedError()
-
-    def get_train_examples(self, data_dir):
-        """Gets a collection of `InputExample`s for the train set."""
-        raise NotImplementedError()
-
-    def get_dev_examples(self, data_dir):
-        """Gets a collection of `InputExample`s for the dev set."""
-        raise NotImplementedError()
-
-    def get_labels(self):
-        """Gets the list of labels for this data set."""
-        raise NotImplementedError()
-
-    def tfds_map(self, example):
-        """Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are.
-        This method converts examples to the correct format."""
-        if len(self.get_labels()) > 1:
-            example.label = self.get_labels()[int(example.label)]
-        return example
-
     @classmethod
     def _read_tsv(cls, input_file, quotechar=None):
         """Reads a tab separated value file."""
@@ -271,7 +244,7 @@ class SingleSentenceClassificationProcessor(DataProcessor):
                 logger.info("Tokenizing example %d", ex_index)
 
             input_ids = tokenizer.encode(
-                example.text_a, add_special_tokens=True, max_length=min(max_length, tokenizer.max_len),
+                example.text_a, add_special_tokens=True, max_length=min(max_length, tokenizer.max_len)
             )
             all_input_ids.append(input_ids)
 
@@ -280,7 +253,7 @@ class SingleSentenceClassificationProcessor(DataProcessor):
         features = []
         for (ex_index, (input_ids, example)) in enumerate(zip(all_input_ids, self.examples)):
             if ex_index % 10000 == 0:
-                logger.info("Writing example %d/%d" % (ex_index, len(self.examples)))
+                logger.info("Writing example %d", ex_index)
             # The mask has 1 for real tokens and 0 for padding tokens. Only real
             # tokens are attended to.
             attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)

src/transformers/data/processors/xnli.py

@@ -72,14 +72,8 @@ class XnliProcessor(DataProcessor):
         return ["contradiction", "entailment", "neutral"]
 
 
-xnli_processors = {
-    "xnli": XnliProcessor,
-}
+xnli_processors = {"xnli": XnliProcessor}
 
-xnli_output_modes = {
-    "xnli": "classification",
-}
+xnli_output_modes = {"xnli": "classification"}
 
-xnli_tasks_num_labels = {
-    "xnli": 3,
-}
+xnli_tasks_num_labels = {"xnli": 3}

src/transformers/file_utils.py

@@ -55,6 +55,20 @@ try:
 except (ImportError, AssertionError):
     _tf_available = False  # pylint: disable=invalid-name
 
+try:
+    os.environ.setdefault("USE_FAST_TOKENIZERS", "YES")
+    if os.environ["USE_FAST_TOKENIZERS"].upper() in ("1", "ON", "YES"):
+        import tokenizers
+
+        _fast_tokenizers_available = True  # pylint: disable=invalid-name
+        logger.info("Fast Tokenizers version {} available.".format(tokenizers.__version__))
+    else:
+        logger.info("USE_FAST_TOKENIZERS override through env variable, disabling fast Tokenizers")
+        _fast_tokenizers_available = False
+except ImportError:
+    _fast_tokenizers_available = False  # pylint: disable=invalid-name
+
+
 try:
     from torch.hub import _get_torch_home
 
@@ -97,7 +111,12 @@ def is_torch_available():
     return _torch_available
 
 
+def is_fast_tokenizers_available():
+    return _fast_tokenizers_available
+
+
 def is_tf_available():
+
     return _tf_available
 
 
@@ -273,10 +292,6 @@ def s3_get(url, temp_file, proxies=None):
 
 def http_get(url, temp_file, proxies=None, resume_size=0, user_agent=None):
     ua = "transformers/{}; python/{}".format(__version__, sys.version.split()[0])
-    if is_torch_available():
-        ua += "; torch/{}".format(torch.__version__)
-    if is_tf_available():
-        ua += "; tensorflow/{}".format(tf.__version__)
     if isinstance(user_agent, dict):
         ua += "; " + "; ".join("{}/{}".format(k, v) for k, v in user_agent.items())
     elif isinstance(user_agent, str):

src/transformers/hf_api.py

@@ -79,7 +79,7 @@ class HfApi:
         r = requests.post(path, headers={"authorization": "Bearer {}".format(token)})
         r.raise_for_status()
 
-    def presign(self, token: str, filename: str) -> PresignedUrl:
+    def presign(self, token: str, filename) -> PresignedUrl:
         """
         Call HF API to get a presigned url to upload `filename` to S3.
         """
@@ -89,7 +89,7 @@ class HfApi:
         d = r.json()
         return PresignedUrl(**d)
 
-    def presign_and_upload(self, token: str, filename: str, filepath: str) -> str:
+    def presign_and_upload(self, token: str, filename, filepath) -> str:
         """
         Get a presigned url, then upload file to S3.
 
@@ -111,7 +111,7 @@ class HfApi:
             pf.close()
         return urls.access
 
-    def list_objs(self, token: str) -> List[S3Obj]:
+    def list_objs(self, token) -> List[S3Obj]:
         """
         Call HF API to list all stored files for user.
         """
@@ -121,14 +121,6 @@ class HfApi:
         d = r.json()
         return [S3Obj(**x) for x in d]
 
-    def delete_obj(self, token: str, filename: str):
-        """
-        Call HF API to delete a file stored by user
-        """
-        path = "{}/api/deleteObj".format(self.endpoint)
-        r = requests.delete(path, headers={"authorization": "Bearer {}".format(token)}, json={"filename": filename})
-        r.raise_for_status()
-
 
 class TqdmProgressFileReader:
     """

src/transformers/modeling_albert.py

@@ -355,7 +355,7 @@ class AlbertTransformer(nn.Module):
 
 class AlbertPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = AlbertConfig
@@ -597,7 +597,7 @@ class AlbertForMaskedLM(AlbertPreTrainedModel):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
 
@@ -837,7 +837,7 @@ class AlbertForQuestionAnswering(AlbertPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/modeling_bert.py

@@ -521,7 +521,7 @@ class BertPreTrainingHeads(nn.Module):
 
 class BertPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = BertConfig
@@ -810,7 +810,7 @@ class BertModel(BertPreTrainedModel):
         sequence_output = encoder_outputs[0]
         pooled_output = self.pooler(sequence_output)
 
-        outputs = (sequence_output, pooled_output,) + encoder_outputs[
+        outputs = (sequence_output, pooled_output) + encoder_outputs[
             1:
         ]  # add hidden_states and attentions if they are here
         return outputs  # sequence_output, pooled_output, (hidden_states), (attentions)
@@ -826,7 +826,7 @@ class BertForPreTraining(BertPreTrainedModel):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
         **next_sentence_label**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size,)``:
@@ -895,7 +895,7 @@ class BertForPreTraining(BertPreTrainedModel):
         sequence_output, pooled_output = outputs[:2]
         prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
 
-        outputs = (prediction_scores, seq_relationship_score,) + outputs[
+        outputs = (prediction_scores, seq_relationship_score) + outputs[
             2:
         ]  # add hidden states and attention if they are here
 
@@ -916,12 +916,12 @@ class BertForMaskedLM(BertPreTrainedModel):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
         **lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the left-to-right language modeling loss (next word prediction).
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
 
@@ -1392,7 +1392,8 @@ class BertForQuestionAnswering(BertPreTrainedModel):
         tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
         model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
         question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
-        input_ids = tokenizer.encode(question, text)
+        input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
+        input_ids = tokenizer.encode(input_text)
         token_type_ids = [0 if i <= input_ids.index(102) else 1 for i in range(len(input_ids))]
         start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
         all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
@@ -1439,7 +1440,7 @@ class BertForQuestionAnswering(BertPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/modeling_camembert.py

@@ -32,7 +32,7 @@ from .modeling_roberta import (
 logger = logging.getLogger(__name__)
 
 CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
-    "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-pytorch_model.bin",
+    "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-pytorch_model.bin"
 }
 
 
@@ -167,7 +167,7 @@ class CamembertForMaskedLM(RobertaForMaskedLM):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
 

src/transformers/modeling_ctrl.py

@@ -163,7 +163,7 @@ class EncoderLayer(torch.nn.Module):
 
 class CTRLPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = CTRLConfig
@@ -444,7 +444,7 @@ class CTRLLMHeadModel(CTRLPreTrainedModel):
         **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for language modeling.
             Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids``
-            Indices are selected in ``[-100, 0, ..., config.vocab_size]``
+            Indices are selected in ``[-1, 0, ..., config.vocab_size]``
             All labels set to ``-100`` are ignored (masked), the loss is only
             computed for labels in ``[0, ..., config.vocab_size]``
 

src/transformers/modeling_distilbert.py

@@ -496,7 +496,7 @@ class DistilBertForMaskedLM(DistilBertPreTrainedModel):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
 
@@ -707,7 +707,7 @@ class DistilBertForQuestionAnswering(DistilBertPreTrainedModel):
         start_logits = start_logits.squeeze(-1)  # (bs, max_query_len)
         end_logits = end_logits.squeeze(-1)  # (bs, max_query_len)
 
-        outputs = (start_logits, end_logits,) + distilbert_output[1:]
+        outputs = (start_logits, end_logits) + distilbert_output[1:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/modeling_gpt2.py

@@ -240,7 +240,7 @@ class Block(nn.Module):
 
 class GPT2PreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = GPT2Config
@@ -513,7 +513,7 @@ class GPT2LMHeadModel(GPT2PreTrainedModel):
         **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for language modeling.
             Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids``
-            Indices are selected in ``[-100, 0, ..., config.vocab_size]``
+            Indices are selected in ``[-1, 0, ..., config.vocab_size]``
             All labels set to ``-100`` are ignored (masked), the loss is only
             computed for labels in ``[0, ..., config.vocab_size]``
 

src/transformers/modeling_mmbt.py

@@ -310,7 +310,7 @@ class MMBTModel(nn.Module):
         sequence_output = encoder_outputs[0]
         pooled_output = self.transformer.pooler(sequence_output)
 
-        outputs = (sequence_output, pooled_output,) + encoder_outputs[
+        outputs = (sequence_output, pooled_output) + encoder_outputs[
             1:
         ]  # add hidden_states and attentions if they are here
         return outputs  # sequence_output, pooled_output, (hidden_states), (attentions)

src/transformers/modeling_openai.py

@@ -257,7 +257,7 @@ class Block(nn.Module):
 
 class OpenAIGPTPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = OpenAIGPTConfig
@@ -490,7 +490,7 @@ class OpenAIGPTLMHeadModel(OpenAIGPTPreTrainedModel):
         **labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for language modeling.
             Note that the labels **are shifted** inside the model, i.e. you can set ``labels = input_ids``
-            Indices are selected in ``[-100, 0, ..., config.vocab_size]``
+            Indices are selected in ``[-1, 0, ..., config.vocab_size]``
             All labels set to ``-100`` are ignored (masked), the loss is only
             computed for labels in ``[0, ..., config.vocab_size]``
 
@@ -578,7 +578,7 @@ class OpenAIGPTDoubleHeadsModel(OpenAIGPTPreTrainedModel):
         **lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for language modeling.
             Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids``
-            Indices are selected in ``[-100, 0, ..., config.vocab_size]``
+            Indices are selected in ``[-1, 0, ..., config.vocab_size]``
             All labels set to ``-100`` are ignored (masked), the loss is only
             computed for labels in ``[0, ..., config.vocab_size]``
         **mc_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size)``:

src/transformers/modeling_roberta.py

@@ -223,7 +223,7 @@ class RobertaForMaskedLM(BertPreTrainedModel):
     r"""
         **masked_lm_labels**: (`optional`) ``torch.LongTensor`` of shape ``(batch_size, sequence_length)``:
             Labels for computing the masked language modeling loss.
-            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
+            Indices should be in ``[-1, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
             Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
             in ``[0, ..., config.vocab_size]``
 
@@ -715,7 +715,7 @@ class RobertaForQuestionAnswering(BertPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/modeling_t5.py

@@ -446,7 +446,7 @@ class T5Block(nn.Module):
 
 class T5PreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = T5Config
@@ -905,7 +905,7 @@ class T5WithLMHeadModel(T5PreTrainedModel):
         if lm_labels is not None:
             shift_logits = lm_logits[..., :-1, :].contiguous()
             shift_labels = lm_labels[..., 1:].contiguous()
-            loss_fct = CrossEntropyLoss(ignore_index=-100)
+            loss_fct = CrossEntropyLoss(ignore_index=-1)
             loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
             decoder_outputs = (
                 loss,

src/transformers/modeling_tf_albert.py

@@ -435,7 +435,7 @@ class TFAlbertTransformer(tf.keras.layers.Layer):
 
 class TFAlbertPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = AlbertConfig
@@ -698,7 +698,7 @@ class TFAlbertModel(TFAlbertPreTrainedModel):
         pooled_output = self.pooler(sequence_output[:, 0])
 
         # add hidden_states and attentions if they are here
-        outputs = (sequence_output, pooled_output,) + encoder_outputs[1:]
+        outputs = (sequence_output, pooled_output) + encoder_outputs[1:]
         # sequence_output, pooled_output, (hidden_states), (attentions)
         return outputs
 

src/transformers/modeling_tf_bert.py

@@ -568,7 +568,7 @@ class TFBertMainLayer(tf.keras.layers.Layer):
         sequence_output = encoder_outputs[0]
         pooled_output = self.pooler(sequence_output)
 
-        outputs = (sequence_output, pooled_output,) + encoder_outputs[
+        outputs = (sequence_output, pooled_output) + encoder_outputs[
             1:
         ]  # add hidden_states and attentions if they are here
         return outputs  # sequence_output, pooled_output, (hidden_states), (attentions)
@@ -576,7 +576,7 @@ class TFBertMainLayer(tf.keras.layers.Layer):
 
 class TFBertPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = BertConfig
@@ -766,7 +766,7 @@ class TFBertForPreTraining(TFBertPreTrainedModel):
         prediction_scores = self.mlm(sequence_output, training=kwargs.get("training", False))
         seq_relationship_score = self.nsp(pooled_output)
 
-        outputs = (prediction_scores, seq_relationship_score,) + outputs[
+        outputs = (prediction_scores, seq_relationship_score) + outputs[
             2:
         ]  # add hidden states and attention if they are here
 
@@ -1139,6 +1139,6 @@ class TFBertForQuestionAnswering(TFBertPreTrainedModel):
         start_logits = tf.squeeze(start_logits, axis=-1)
         end_logits = tf.squeeze(end_logits, axis=-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
 
         return outputs  # start_logits, end_logits, (hidden_states), (attentions)

src/transformers/modeling_tf_ctrl.py

@@ -344,7 +344,7 @@ class TFCTRLMainLayer(tf.keras.layers.Layer):
 
 class TFCTRLPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = CTRLConfig

src/transformers/modeling_tf_distilbert.py

@@ -822,5 +822,5 @@ class TFDistilBertForQuestionAnswering(TFDistilBertPreTrainedModel):
         start_logits = tf.squeeze(start_logits, axis=-1)
         end_logits = tf.squeeze(end_logits, axis=-1)
 
-        outputs = (start_logits, end_logits,) + distilbert_output[1:]
+        outputs = (start_logits, end_logits) + distilbert_output[1:]
         return outputs  # start_logits, end_logits, (hidden_states), (attentions)

src/transformers/modeling_tf_gpt2.py

@@ -360,7 +360,7 @@ class TFGPT2MainLayer(tf.keras.layers.Layer):
 
 class TFGPT2PreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = GPT2Config

src/transformers/modeling_tf_openai.py

@@ -346,7 +346,7 @@ class TFOpenAIGPTMainLayer(tf.keras.layers.Layer):
 
 class TFOpenAIGPTPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = OpenAIGPTConfig

src/transformers/modeling_tf_roberta.py

@@ -98,7 +98,7 @@ class TFRobertaMainLayer(TFBertMainLayer):
 
 class TFRobertaPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = RobertaConfig

src/transformers/modeling_tf_t5.py

@@ -514,7 +514,7 @@ class TFT5MainLayer(tf.keras.layers.Layer):
 ####################################################
 class TFT5PreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = T5Config

src/transformers/modeling_tf_transfo_xl.py

@@ -30,7 +30,7 @@ from .modeling_tf_utils import TFPreTrainedModel, get_initializer, shape_list
 logger = logging.getLogger(__name__)
 
 TF_TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP = {
-    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-tf_model.h5",
+    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-tf_model.h5"
 }
 
 
@@ -622,7 +622,7 @@ class TFTransfoXLMainLayer(tf.keras.layers.Layer):
 
 class TFTransfoXLPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = TransfoXLConfig

src/transformers/modeling_tf_transfo_xl_utilities.py

@@ -64,7 +64,7 @@ class TFAdaptiveSoftmaxMask(tf.keras.layers.Layer):
                 else:
                     self.out_projs.append(None)
                 weight = self.add_weight(
-                    shape=(self.vocab_size, self.d_embed,),
+                    shape=(self.vocab_size, self.d_embed),
                     initializer="zeros",
                     trainable=True,
                     name="out_layers_._{}_._weight".format(i),
@@ -86,7 +86,7 @@ class TFAdaptiveSoftmaxMask(tf.keras.layers.Layer):
                 )
                 self.out_projs.append(weight)
                 weight = self.add_weight(
-                    shape=(r_idx - l_idx, d_emb_i,),
+                    shape=(r_idx - l_idx, d_emb_i),
                     initializer="zeros",
                     trainable=True,
                     name="out_layers_._{}_._weight".format(i),

src/transformers/modeling_tf_xlm.py

@@ -465,7 +465,7 @@ class TFXLMMainLayer(tf.keras.layers.Layer):
 
 class TFXLMPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XLMConfig
@@ -800,7 +800,7 @@ class TFXLMForQuestionAnsweringSimple(TFXLMPreTrainedModel):
         start_logits = tf.squeeze(start_logits, axis=-1)
         end_logits = tf.squeeze(end_logits, axis=-1)
 
-        outputs = (start_logits, end_logits,) + transformer_outputs[
+        outputs = (start_logits, end_logits) + transformer_outputs[
             1:
         ]  # Keep mems, hidden states, attentions if there are in it
 

src/transformers/modeling_tf_xlnet.py

@@ -686,7 +686,7 @@ class TFXLNetMainLayer(tf.keras.layers.Layer):
 
 class TFXLNetPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XLNetConfig
@@ -1074,7 +1074,7 @@ class TFXLNetForQuestionAnsweringSimple(TFXLNetPreTrainedModel):
         start_logits = tf.squeeze(start_logits, axis=-1)
         end_logits = tf.squeeze(end_logits, axis=-1)
 
-        outputs = (start_logits, end_logits,) + transformer_outputs[
+        outputs = (start_logits, end_logits) + transformer_outputs[
             1:
         ]  # Keep mems, hidden states, attentions if there are in it
 

src/transformers/modeling_transfo_xl.py

@@ -34,7 +34,7 @@ from .modeling_utils import PreTrainedModel
 logger = logging.getLogger(__name__)
 
 TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP = {
-    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-pytorch_model.bin",
+    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-pytorch_model.bin"
 }
 
 
@@ -449,7 +449,7 @@ class AdaptiveEmbedding(nn.Module):
 
 class TransfoXLPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = TransfoXLConfig

src/transformers/modeling_xlm.py

@@ -213,7 +213,7 @@ class TransformerFFN(nn.Module):
 
 class XLMPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XLMConfig
@@ -896,10 +896,7 @@ class XLMForQuestionAnsweringSimple(XLMPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (
-            start_logits,
-            end_logits,
-        )
+        outputs = (start_logits, end_logits)
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/modeling_xlnet.py

@@ -468,7 +468,7 @@ class XLNetLayer(nn.Module):
 
 class XLNetPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XLNetConfig
@@ -514,7 +514,7 @@ XLNET_START_DOCSTRING = r"""    The XLNet model was proposed in
 
     The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
 
-    Due to the difficulty of training a fully auto-regressive model over various factorization order,
+    Do to the difficulty of training a fully auto-regressive model over various factorization order,
     XLNet is pretrained using only a sub-set of the output tokens as target which are selected
     with the `target_mapping` input.
 
@@ -1505,7 +1505,7 @@ class XLNetForQuestionAnsweringSimple(XLNetPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

src/transformers/pipelines.py

@@ -335,13 +335,13 @@ class Pipeline(_ScikitCompat):
         self.tokenizer = tokenizer
         self.modelcard = modelcard
         self.framework = framework
-        self.device = device if framework == "tf" else torch.device("cpu" if device < 0 else "cuda:{}".format(device))
+        self.device = device
         self.binary_output = binary_output
         self._args_parser = args_parser or DefaultArgumentHandler()
 
         # Special handling
-        if self.framework == "pt" and self.device.type == "cuda":
-            self.model = self.model.to(self.device)
+        if self.device >= 0 and self.framework == "pt":
+            self.model = self.model.to("cuda:{}".format(self.device))
 
     def save_pretrained(self, save_directory):
         """
@@ -385,19 +385,11 @@ class Pipeline(_ScikitCompat):
             with tf.device("/CPU:0" if self.device == -1 else "/device:GPU:{}".format(self.device)):
                 yield
         else:
-            if self.device.type == "cuda":
+            if self.device >= 0:
                 torch.cuda.set_device(self.device)
 
             yield
 
-    def ensure_tensor_on_device(self, **inputs):
-        """
-        Ensure PyTorch tensors are on the specified device.
-        :param inputs:
-        :return:
-        """
-        return {name: tensor.to(self.device) for name, tensor in inputs.items()}
-
     def inputs_for_model(self, features: Union[dict, List[dict]]) -> Dict:
         """
         Generates the input dictionary with model-specific parameters.
@@ -423,13 +415,16 @@ class Pipeline(_ScikitCompat):
     def __call__(self, *texts, **kwargs):
         # Parse arguments
         inputs = self._args_parser(*texts, **kwargs)
-        inputs = self.tokenizer.batch_encode_plus(
-            inputs, add_special_tokens=True, return_tensors=self.framework, max_length=self.tokenizer.max_len
-        )
 
-        # Filter out features not available on specific models
-        inputs = self.inputs_for_model(inputs)
-        return self._forward(inputs)
+        # Encode for forward
+        with self.device_placement():
+            inputs = self.tokenizer.batch_encode_plus(
+                inputs, add_special_tokens=True, return_tensors=self.framework, max_length=self.tokenizer.max_len
+            )
+
+            # Filter out features not available on specific models
+            inputs = self.inputs_for_model(inputs)
+            return self._forward(inputs)
 
     def _forward(self, inputs):
         """
@@ -439,15 +434,12 @@ class Pipeline(_ScikitCompat):
         Returns:
             Numpy array
         """
-        # Encode for forward
-        with self.device_placement():
-            if self.framework == "tf":
-                # TODO trace model
-                predictions = self.model(inputs, training=False)[0]
-            else:
-                with torch.no_grad():
-                    inputs = self.ensure_tensor_on_device(**inputs)
-                    predictions = self.model(**inputs)[0].cpu()
+        if self.framework == "tf":
+            # TODO trace model
+            predictions = self.model(inputs, training=False)[0]
+        else:
+            with torch.no_grad():
+                predictions = self.model(**inputs)[0].cpu()
 
         return predictions.numpy()
 
@@ -542,7 +534,6 @@ class NerPipeline(Pipeline):
                     input_ids = tokens["input_ids"].numpy()[0]
                 else:
                     with torch.no_grad():
-                        tokens = self.ensure_tensor_on_device(**tokens)
                         entities = self.model(**tokens)[0][0].cpu().numpy()
                         input_ids = tokens["input_ids"].cpu().numpy()[0]
 
@@ -719,7 +710,7 @@ class QuestionAnsweringPipeline(Pipeline):
             else:
                 with torch.no_grad():
                     # Retrieve the score for the context tokens only (removing question tokens)
-                    fw_args = {k: torch.tensor(v, device=self.device) for (k, v) in fw_args.items()}
+                    fw_args = {k: torch.tensor(v) for (k, v) in fw_args.items()}
                     start, end = self.model(**fw_args)
                     start, end = start.cpu().numpy(), end.cpu().numpy()
 

src/transformers/tokenization_bert.py

@@ -20,11 +20,15 @@ import logging
 import os
 import unicodedata
 
-import tokenizers as tk
-
+from .file_utils import is_fast_tokenizers_available
 from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
 
 
+if is_fast_tokenizers_available():
+    import tokenizers as tk
+else:
+    tk = None
+
 logger = logging.getLogger(__name__)
 
 VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"}
@@ -563,7 +567,15 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
             **kwargs,
         )
 
-        self._tokenizer = tk.Tokenizer(tk.models.WordPiece.from_files(vocab_file, unk_token=unk_token))
+        if is_fast_tokenizers_available():
+            self._tokenizer = tk.Tokenizer(tk.models.WordPiece.from_files(vocab_file, unk_token=unk_token))
+        else:
+            logger.error(
+                "Using fast Tokenizers requires the `tokenizers` library. "
+                "Please install it with `pip install tokenizers`."
+            )
+            raise ImportError()
+
         self._update_special_tokens()
         self._tokenizer.with_pre_tokenizer(
             tk.pre_tokenizers.BertPreTokenizer.new(

src/transformers/tokenization_camembert.py

@@ -32,13 +32,11 @@ VOCAB_FILES_NAMES = {"vocab_file": "sentencepiece.bpe.model"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {
-        "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-sentencepiece.bpe.model",
+        "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-sentencepiece.bpe.model"
     }
 }
 
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "camembert-base": None,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"camembert-base": None}
 
 
 class CamembertTokenizer(PreTrainedTokenizer):

src/transformers/tokenization_ctrl.py

@@ -26,19 +26,14 @@ from .tokenization_utils import PreTrainedTokenizer
 
 logger = logging.getLogger(__name__)
 
-VOCAB_FILES_NAMES = {
-    "vocab_file": "vocab.json",
-    "merges_file": "merges.txt",
-}
+VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json"},
     "merges_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt"},
 }
 
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "ctrl": 256,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"ctrl": 256}
 
 CONTROL_CODES = {
     "Pregnancy": 168629,

src/transformers/tokenization_distilbert.py

@@ -41,14 +41,6 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
 }
 
 
-PRETRAINED_INIT_CONFIGURATION = {
-    "distilbert-base-uncased": {"do_lower_case": True},
-    "distilbert-base-uncased-distilled-squad": {"do_lower_case": True},
-    "distilbert-base-german-cased": {"do_lower_case": False},
-    "distilbert-base-multilingual-cased": {"do_lower_case": False},
-}
-
-
 class DistilBertTokenizer(BertTokenizer):
     r"""
     Constructs a DistilBertTokenizer.
@@ -67,4 +59,3 @@ class DistilBertTokenizer(BertTokenizer):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
-    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION

src/transformers/tokenization_gpt2.py

@@ -21,17 +21,19 @@ import os
 from functools import lru_cache
 
 import regex as re
-import tokenizers as tk
 
+from .file_utils import is_fast_tokenizers_available
 from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
 
 
+if is_fast_tokenizers_available():
+    import tokenizers as tk
+else:
+    tk = None
+
 logger = logging.getLogger(__name__)
 
-VOCAB_FILES_NAMES = {
-    "vocab_file": "vocab.json",
-    "merges_file": "merges.txt",
-}
+VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {
@@ -272,7 +274,15 @@ class GPT2TokenizerFast(PreTrainedTokenizerFast):
             bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs
         )
 
-        self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
+        if is_fast_tokenizers_available():
+            self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
+        else:
+            logger.error(
+                "Using fast Tokenizers requires the `tokenizers` library. "
+                "Please install it with `pip install tokenizers`."
+            )
+            raise ImportError()
+
         self._update_special_tokens()
         self._tokenizer.with_pre_tokenizer(tk.pre_tokenizers.ByteLevel.new(add_prefix_space=add_prefix_space))
         self._tokenizer.with_decoder(tk.decoders.ByteLevel.new())

src/transformers/tokenization_openai.py

@@ -26,19 +26,14 @@ from .tokenization_utils import PreTrainedTokenizer
 
 logger = logging.getLogger(__name__)
 
-VOCAB_FILES_NAMES = {
-    "vocab_file": "vocab.json",
-    "merges_file": "merges.txt",
-}
+VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {"openai-gpt": "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-vocab.json"},
     "merges_file": {"openai-gpt": "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-merges.txt"},
 }
 
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "openai-gpt": 512,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"openai-gpt": 512}
 
 
 def get_pairs(word):

src/transformers/tokenization_roberta.py

@@ -22,10 +22,7 @@ from .tokenization_gpt2 import GPT2Tokenizer
 
 logger = logging.getLogger(__name__)
 
-VOCAB_FILES_NAMES = {
-    "vocab_file": "vocab.json",
-    "merges_file": "merges.txt",
-}
+VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {

src/transformers/tokenization_transfo_xl.py

@@ -40,16 +40,14 @@ VOCAB_FILES_NAMES = {"pretrained_vocab_file": "vocab.bin", "vocab_file": "vocab.
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "pretrained_vocab_file": {
-        "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-vocab.bin",
+        "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-vocab.bin"
     }
 }
 
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "transfo-xl-wt103": None,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"transfo-xl-wt103": None}
 
 PRETRAINED_CORPUS_ARCHIVE_MAP = {
-    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-corpus.bin",
+    "transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-corpus.bin"
 }
 CORPUS_NAME = "corpus.bin"
 

src/transformers/tokenization_utils.py

@@ -39,7 +39,7 @@ TOKENIZER_CONFIG_FILE = "tokenizer_config.json"
 
 class PreTrainedTokenizer(object):
     """ Base class for all tokenizers.
-    Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.
+    Handle all the shared methods for tokenization and special tokens as well as methods dowloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.
 
     This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).
 
@@ -460,7 +460,7 @@ class PreTrainedTokenizer(object):
         try:
             tokenizer = cls(*init_inputs, **init_kwargs)
         except OSError:
-            raise OSError(
+            OSError(
                 "Unable to load vocabulary from file. "
                 "Please check that the provided vocabulary is accessible and not corrupted."
             )
@@ -1495,9 +1495,7 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
         return_overflowing_tokens=False,
         return_special_tokens_mask=False,
     ):
-        encoding_dict = {
-            "input_ids": encoding.ids,
-        }
+        encoding_dict = {"input_ids": encoding.ids}
         if return_token_type_ids:
             encoding_dict["token_type_ids"] = encoding.type_ids
         if return_attention_mask:
@@ -1511,16 +1509,14 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
         # Prepare inputs as tensors if asked
         if return_tensors == "tf" and is_tf_available():
             encoding_dict["input_ids"] = tf.constant([encoding_dict["input_ids"]])
-            if "token_type_ids" in encoding_dict:
-                encoding_dict["token_type_ids"] = tf.constant([encoding_dict["token_type_ids"]])
+            encoding_dict["token_type_ids"] = tf.constant([encoding_dict["token_type_ids"]])
 
             if "attention_mask" in encoding_dict:
                 encoding_dict["attention_mask"] = tf.constant([encoding_dict["attention_mask"]])
 
         elif return_tensors == "pt" and is_torch_available():
             encoding_dict["input_ids"] = torch.tensor([encoding_dict["input_ids"]])
-            if "token_type_ids" in encoding_dict:
-                encoding_dict["token_type_ids"] = torch.tensor([encoding_dict["token_type_ids"]])
+            encoding_dict["token_type_ids"] = torch.tensor([encoding_dict["token_type_ids"]])
 
             if "attention_mask" in encoding_dict:
                 encoding_dict["attention_mask"] = torch.tensor([encoding_dict["attention_mask"]])

src/transformers/tokenization_xlm.py

@@ -29,10 +29,7 @@ from .tokenization_utils import PreTrainedTokenizer
 
 logger = logging.getLogger(__name__)
 
-VOCAB_FILES_NAMES = {
-    "vocab_file": "vocab.json",
-    "merges_file": "merges.txt",
-}
+VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt"}
 
 PRETRAINED_VOCAB_FILES_MAP = {
     "vocab_file": {
@@ -474,7 +471,7 @@ def replace_unicode_punct(text):
     text = text.replace("！", "!")
     text = text.replace("（", "(")
     text = text.replace("；", ";")
-    text = text.replace("１", "1")
+    text = text.replace("１", '"')
     text = text.replace("」", '"')
     text = text.replace("「", '"')
     text = text.replace("０", "0")
@@ -845,7 +842,7 @@ class XLMTokenizer(PreTrainedTokenizer):
                     "You should not supply a second sequence if the provided sequence of "
                     "ids is already formated with special tokens for the model."
                 )
-            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0,))
+            return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))
 
         if token_ids_1 is not None:
             return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]

src/transformers/tokenization_xlnet.py

@@ -34,10 +34,7 @@ PRETRAINED_VOCAB_FILES_MAP = {
     }
 }
 
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "xlnet-base-cased": None,
-    "xlnet-large-cased": None,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"xlnet-base-cased": None, "xlnet-large-cased": None}
 
 SPIECE_UNDERLINE = "▁"
 

templates/adding_a_new_model/README.md

@@ -56,7 +56,7 @@ You can then finish the addition step by adding imports for your classes in the
 - [ ] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
 - [ ] add your tokenizer in `tokenization_auto.py`
 - [ ] add your models and tokenizer to `pipeline.py`
-- [ ] add a link to your conversion script in the main conversion utility (in `commands/convert.py`)
+- [ ] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
 - [ ] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
 - [ ] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
 - [ ] upload the pretrained weigths, configurations and vocabulary files.

templates/adding_a_new_model/modeling_tf_xxx.py

@@ -176,7 +176,7 @@ class TFXxxMainLayer(tf.keras.layers.Layer):
 ####################################################
 class TFXxxPreTrainedModel(TFPreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XxxConfig
@@ -527,6 +527,6 @@ class TFXxxForQuestionAnswering(TFXxxPreTrainedModel):
         start_logits = tf.squeeze(start_logits, axis=-1)
         end_logits = tf.squeeze(end_logits, axis=-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
 
         return outputs  # start_logits, end_logits, (hidden_states), (attentions)

templates/adding_a_new_model/modeling_xxx.py

@@ -173,7 +173,7 @@ XxxPooler = nn.Module
 
 class XxxPreTrainedModel(PreTrainedModel):
     """ An abstract class to handle weights initialization and
-        a simple interface for downloading and loading pretrained models.
+        a simple interface for dowloading and loading pretrained models.
     """
 
     config_class = XxxConfig
@@ -728,7 +728,7 @@ class XxxForQuestionAnswering(XxxPreTrainedModel):
         start_logits = start_logits.squeeze(-1)
         end_logits = end_logits.squeeze(-1)
 
-        outputs = (start_logits, end_logits,) + outputs[2:]
+        outputs = (start_logits, end_logits) + outputs[2:]
         if start_positions is not None and end_positions is not None:
             # If we are on multi-GPU, split add a dimension
             if len(start_positions.size()) > 1:

templates/adding_a_new_model/tests/test_modeling_tf_xxx.py

@@ -144,10 +144,7 @@ class TFXxxModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-                "pooled_output": pooled_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy(), "pooled_output": pooled_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -159,9 +156,7 @@ class TFXxxModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFXxxForMaskedLM(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (prediction_scores,) = model(inputs)
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -173,9 +168,7 @@ class TFXxxModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFXxxForSequenceClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.num_labels])
 
         def create_and_check_xxx_for_token_classification(
@@ -185,9 +178,7 @@ class TFXxxModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFXxxForTokenClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(
                 list(result["logits"].shape), [self.batch_size, self.seq_length, self.num_labels]
             )
@@ -198,10 +189,7 @@ class TFXxxModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFXxxForQuestionAnswering(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             start_logits, end_logits = model(inputs)
-            result = {
-                "start_logits": start_logits.numpy(),
-                "end_logits": end_logits.numpy(),
-            }
+            result = {"start_logits": start_logits.numpy(), "end_logits": end_logits.numpy()}
             self.parent.assertListEqual(list(result["start_logits"].shape), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].shape), [self.batch_size, self.seq_length])
 

templates/adding_a_new_model/tests/test_modeling_xxx.py

@@ -141,10 +141,7 @@ class XxxModelTest(ModelTesterMixin, unittest.TestCase):
             sequence_output, pooled_output = model(input_ids, token_type_ids=token_type_ids)
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "pooled_output": pooled_output,
-            }
+            result = {"sequence_output": sequence_output, "pooled_output": pooled_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -159,10 +156,7 @@ class XxxModelTest(ModelTesterMixin, unittest.TestCase):
             loss, prediction_scores = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, masked_lm_labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -181,11 +175,7 @@ class XxxModelTest(ModelTesterMixin, unittest.TestCase):
                 start_positions=sequence_labels,
                 end_positions=sequence_labels,
             )
-            result = {
-                "loss": loss,
-                "start_logits": start_logits,
-                "end_logits": end_logits,
-            }
+            result = {"loss": loss, "start_logits": start_logits, "end_logits": end_logits}
             self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
             self.check_loss_output(result)
@@ -200,10 +190,7 @@ class XxxModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=sequence_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_labels])
             self.check_loss_output(result)
 
@@ -217,10 +204,7 @@ class XxxModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(
                 list(result["logits"].size()), [self.batch_size, self.seq_length, self.num_labels]
             )

templates/adding_a_new_model/tokenization_xxx.py

@@ -48,10 +48,7 @@ PRETRAINED_VOCAB_FILES_MAP = {
 ####################################################
 # Mapping from model shortcut names to max length of inputs
 ####################################################
-PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
-    "xxx-base-uncased": 512,
-    "xxx-large-uncased": 512,
-}
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"xxx-base-uncased": 512, "xxx-large-uncased": 512}
 
 ####################################################
 # Mapping from model shortcut names to a dictionary of additional

tests/test_hf_api.py

@@ -60,11 +60,6 @@ class HfApiEndpointsTest(HfApiCommonTest):
         """
         cls._token = cls._api.login(username=USER, password=PASS)
 
-    @classmethod
-    def tearDownClass(cls):
-        for FILE_KEY, FILE_PATH in FILES:
-            cls._api.delete_obj(token=cls._token, filename=FILE_KEY)
-
     def test_whoami(self):
         user = self._api.whoami(token=self._token)
         self.assertEqual(user, USER)

tests/test_modeling_albert.py

@@ -141,10 +141,7 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
             sequence_output, pooled_output = model(input_ids, token_type_ids=token_type_ids)
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "pooled_output": pooled_output,
-            }
+            result = {"sequence_output": sequence_output, "pooled_output": pooled_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -159,10 +156,7 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, prediction_scores = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, masked_lm_labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -181,11 +175,7 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
                 start_positions=sequence_labels,
                 end_positions=sequence_labels,
             )
-            result = {
-                "loss": loss,
-                "start_logits": start_logits,
-                "end_logits": end_logits,
-            }
+            result = {"loss": loss, "start_logits": start_logits, "end_logits": end_logits}
             self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
             self.check_loss_output(result)
@@ -200,10 +190,7 @@ class AlbertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=sequence_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_labels])
             self.check_loss_output(result)
 

tests/test_modeling_bert.py

@@ -180,10 +180,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
             sequence_output, pooled_output = model(input_ids, token_type_ids=token_type_ids)
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "pooled_output": pooled_output,
-            }
+            result = {"sequence_output": sequence_output, "pooled_output": pooled_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -219,10 +216,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
             )
             sequence_output, pooled_output = model(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "pooled_output": pooled_output,
-            }
+            result = {"sequence_output": sequence_output, "pooled_output": pooled_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -237,10 +231,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, prediction_scores = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, masked_lm_labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -276,10 +267,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
                 masked_lm_labels=token_labels,
                 encoder_hidden_states=encoder_hidden_states,
             )
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -297,10 +285,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
                 token_type_ids=token_type_ids,
                 next_sentence_label=sequence_labels,
             )
-            result = {
-                "loss": loss,
-                "seq_relationship_score": seq_relationship_score,
-            }
+            result = {"loss": loss, "seq_relationship_score": seq_relationship_score}
             self.parent.assertListEqual(list(result["seq_relationship_score"].size()), [self.batch_size, 2])
             self.check_loss_output(result)
 
@@ -341,11 +326,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
                 start_positions=sequence_labels,
                 end_positions=sequence_labels,
             )
-            result = {
-                "loss": loss,
-                "start_logits": start_logits,
-                "end_logits": end_logits,
-            }
+            result = {"loss": loss, "start_logits": start_logits, "end_logits": end_logits}
             self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
             self.check_loss_output(result)
@@ -360,10 +341,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=sequence_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_labels])
             self.check_loss_output(result)
 
@@ -377,10 +355,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(
                 list(result["logits"].size()), [self.batch_size, self.seq_length, self.num_labels]
             )
@@ -402,10 +377,7 @@ class BertModelTest(ModelTesterMixin, unittest.TestCase):
                 token_type_ids=multiple_choice_token_type_ids,
                 labels=choice_labels,
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_choices])
             self.check_loss_output(result)
 

tests/test_modeling_ctrl.py

@@ -150,10 +150,7 @@ class CTRLModelTest(ModelTesterMixin, unittest.TestCase):
             model(input_ids, token_type_ids=token_type_ids)
             sequence_output, presents = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "presents": presents,
-            }
+            result = {"sequence_output": sequence_output, "presents": presents}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )

tests/test_modeling_distilbert.py

@@ -138,9 +138,7 @@ class DistilBertModelTest(ModelTesterMixin, unittest.TestCase):
             (sequence_output,) = model(input_ids, input_mask)
             (sequence_output,) = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-            }
+            result = {"sequence_output": sequence_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -152,10 +150,7 @@ class DistilBertModelTest(ModelTesterMixin, unittest.TestCase):
             model.to(torch_device)
             model.eval()
             loss, prediction_scores = model(input_ids, attention_mask=input_mask, masked_lm_labels=token_labels)
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -170,11 +165,7 @@ class DistilBertModelTest(ModelTesterMixin, unittest.TestCase):
             loss, start_logits, end_logits = model(
                 input_ids, attention_mask=input_mask, start_positions=sequence_labels, end_positions=sequence_labels
             )
-            result = {
-                "loss": loss,
-                "start_logits": start_logits,
-                "end_logits": end_logits,
-            }
+            result = {"loss": loss, "start_logits": start_logits, "end_logits": end_logits}
             self.parent.assertListEqual(list(result["start_logits"].size()), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].size()), [self.batch_size, self.seq_length])
             self.check_loss_output(result)
@@ -187,10 +178,7 @@ class DistilBertModelTest(ModelTesterMixin, unittest.TestCase):
             model.to(torch_device)
             model.eval()
             loss, logits = model(input_ids, attention_mask=input_mask, labels=sequence_labels)
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(list(result["logits"].size()), [self.batch_size, self.num_labels])
             self.check_loss_output(result)
 
@@ -203,10 +191,7 @@ class DistilBertModelTest(ModelTesterMixin, unittest.TestCase):
             model.eval()
 
             loss, logits = model(input_ids, attention_mask=input_mask, labels=token_labels)
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(
                 list(result["logits"].size()), [self.batch_size, self.seq_length, self.num_labels]
             )

tests/test_modeling_gpt2.py

@@ -153,10 +153,7 @@ class GPT2ModelTest(ModelTesterMixin, unittest.TestCase):
             model(input_ids, token_type_ids=token_type_ids)
             sequence_output, presents = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "presents": presents,
-            }
+            result = {"sequence_output": sequence_output, "presents": presents}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )

tests/test_modeling_roberta.py

@@ -138,10 +138,7 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
             sequence_output, pooled_output = model(input_ids, token_type_ids=token_type_ids)
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output,
-                "pooled_output": pooled_output,
-            }
+            result = {"sequence_output": sequence_output, "pooled_output": pooled_output}
             self.parent.assertListEqual(
                 list(result["sequence_output"].size()), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -156,10 +153,7 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
             loss, prediction_scores = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, masked_lm_labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -175,10 +169,7 @@ class RobertaModelTest(ModelTesterMixin, unittest.TestCase):
             loss, logits = model(
                 input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, labels=token_labels
             )
-            result = {
-                "loss": loss,
-                "logits": logits,
-            }
+            result = {"loss": loss, "logits": logits}
             self.parent.assertListEqual(
                 list(result["logits"].size()), [self.batch_size, self.seq_length, self.num_labels]
             )

tests/test_modeling_t5.py

@@ -136,10 +136,7 @@ class T5ModelTest(ModelTesterMixin, unittest.TestCase):
                 encoder_input_ids=encoder_input_ids, decoder_input_ids=decoder_input_ids
             )
 
-            result = {
-                "encoder_output": encoder_output,
-                "decoder_output": decoder_output,
-            }
+            result = {"encoder_output": encoder_output, "decoder_output": decoder_output}
             self.parent.assertListEqual(
                 list(result["encoder_output"].size()), [self.batch_size, self.encoder_seq_length, self.hidden_size]
             )
@@ -165,10 +162,7 @@ class T5ModelTest(ModelTesterMixin, unittest.TestCase):
                 decoder_lm_labels=decoder_lm_labels,
             )
             loss, prediction_scores = outputs[0], outputs[1]
-            result = {
-                "loss": loss,
-                "prediction_scores": prediction_scores,
-            }
+            result = {"loss": loss, "prediction_scores": prediction_scores}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].size()), [self.batch_size, self.decoder_seq_length, self.vocab_size]
             )

tests/test_modeling_tf_albert.py

@@ -141,10 +141,7 @@ class TFAlbertModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-                "pooled_output": pooled_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy(), "pooled_output": pooled_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -156,9 +153,7 @@ class TFAlbertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFAlbertForMaskedLM(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (prediction_scores,) = model(inputs)
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -170,9 +165,7 @@ class TFAlbertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFAlbertForSequenceClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.num_labels])
 
         def prepare_config_and_inputs_for_common(self):

tests/test_modeling_tf_bert.py

@@ -150,10 +150,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output, pooled_output = model(input_ids)
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-                "pooled_output": pooled_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy(), "pooled_output": pooled_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -165,9 +162,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFBertForMaskedLM(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (prediction_scores,) = model(inputs)
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -178,9 +173,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFBertForNextSentencePrediction(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (seq_relationship_score,) = model(inputs)
-            result = {
-                "seq_relationship_score": seq_relationship_score.numpy(),
-            }
+            result = {"seq_relationship_score": seq_relationship_score.numpy()}
             self.parent.assertListEqual(list(result["seq_relationship_score"].shape), [self.batch_size, 2])
 
         def create_and_check_bert_for_pretraining(
@@ -205,9 +198,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFBertForSequenceClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.num_labels])
 
         def create_and_check_bert_for_multiple_choice(
@@ -224,9 +215,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
                 "token_type_ids": multiple_choice_token_type_ids,
             }
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.num_choices])
 
         def create_and_check_bert_for_token_classification(
@@ -236,9 +225,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFBertForTokenClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(
                 list(result["logits"].shape), [self.batch_size, self.seq_length, self.num_labels]
             )
@@ -249,10 +236,7 @@ class TFBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFBertForQuestionAnswering(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             start_logits, end_logits = model(inputs)
-            result = {
-                "start_logits": start_logits.numpy(),
-                "end_logits": end_logits.numpy(),
-            }
+            result = {"start_logits": start_logits.numpy(), "end_logits": end_logits.numpy()}
             self.parent.assertListEqual(list(result["start_logits"].shape), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].shape), [self.batch_size, self.seq_length])
 

tests/test_modeling_tf_common.py

@@ -115,13 +115,6 @@ class TFModelTesterMixin:
             tf_hidden_states[np.isnan(tf_hidden_states)] = 0
             pt_hidden_states[np.isnan(pt_hidden_states)] = 0
             max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
-            # Debug info (remove when fixed)
-            if max_diff >= 2e-2:
-                print("===")
-                print(model_class)
-                print(config)
-                print(inputs_dict)
-                print(pt_inputs_dict)
             self.assertLessEqual(max_diff, 2e-2)
 
             # Check we can load pt model in tf and vice-versa with checkpoint => model functions

tests/test_modeling_tf_ctrl.py

@@ -145,9 +145,7 @@ class TFCTRLModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output = model(input_ids)[0]
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -156,9 +154,7 @@ class TFCTRLModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFCTRLLMHeadModel(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             prediction_scores = model(inputs)[0]
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )

tests/test_modeling_tf_distilbert.py

@@ -142,9 +142,7 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
 
             (sequence_output,) = model(inputs)
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -155,9 +153,7 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFDistilBertForMaskedLM(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask}
             (prediction_scores,) = model(inputs)
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -168,10 +164,7 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFDistilBertForQuestionAnswering(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask}
             start_logits, end_logits = model(inputs)
-            result = {
-                "start_logits": start_logits.numpy(),
-                "end_logits": end_logits.numpy(),
-            }
+            result = {"start_logits": start_logits.numpy(), "end_logits": end_logits.numpy()}
             self.parent.assertListEqual(list(result["start_logits"].shape), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].shape), [self.batch_size, self.seq_length])
 
@@ -182,9 +175,7 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFDistilBertForSequenceClassification(config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.num_labels])
 
         def prepare_config_and_inputs_for_common(self):

tests/test_modeling_tf_gpt2.py

@@ -152,9 +152,7 @@ class TFGPT2ModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output = model(input_ids)[0]
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -163,9 +161,7 @@ class TFGPT2ModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFGPT2LMHeadModel(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             prediction_scores = model(inputs)[0]
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )

tests/test_modeling_tf_openai_gpt.py

@@ -153,9 +153,7 @@ class TFOpenAIGPTModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output = model(input_ids)[0]
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -164,9 +162,7 @@ class TFOpenAIGPTModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFOpenAIGPTLMHeadModel(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             prediction_scores = model(inputs)[0]
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )

tests/test_modeling_tf_roberta.py

@@ -138,9 +138,7 @@ class TFRobertaModelTest(TFModelTesterMixin, unittest.TestCase):
 
             sequence_output = model(input_ids)[0]
 
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -150,9 +148,7 @@ class TFRobertaModelTest(TFModelTesterMixin, unittest.TestCase):
         ):
             model = TFRobertaForMaskedLM(config=config)
             prediction_scores = model([input_ids, input_mask, token_type_ids])[0]
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )
@@ -164,9 +160,7 @@ class TFRobertaModelTest(TFModelTesterMixin, unittest.TestCase):
             model = TFRobertaForTokenClassification(config=config)
             inputs = {"input_ids": input_ids, "attention_mask": input_mask, "token_type_ids": token_type_ids}
             (logits,) = model(inputs)
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
             self.parent.assertListEqual(
                 list(result["logits"].shape), [self.batch_size, self.seq_length, self.num_labels]
             )

tests/test_modeling_tf_t5.py

@@ -109,10 +109,7 @@ class TFT5ModelTest(TFModelTesterMixin, unittest.TestCase):
                 input_ids, decoder_attention_mask=input_mask, encoder_input_ids=input_ids
             )
 
-            result = {
-                "encoder_output": encoder_output.numpy(),
-                "decoder_output": decoder_output.numpy(),
-            }
+            result = {"encoder_output": encoder_output.numpy(), "decoder_output": decoder_output.numpy()}
             self.parent.assertListEqual(
                 list(result["encoder_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -128,9 +125,7 @@ class TFT5ModelTest(TFModelTesterMixin, unittest.TestCase):
                 "decoder_attention_mask": input_mask,
             }
             prediction_scores, decoder_output = model(inputs)
-            result = {
-                "prediction_scores": prediction_scores.numpy(),
-            }
+            result = {"prediction_scores": prediction_scores.numpy()}
             self.parent.assertListEqual(
                 list(result["prediction_scores"].shape), [self.batch_size, self.seq_length, self.vocab_size]
             )

tests/test_modeling_tf_xlm.py

@@ -176,9 +176,7 @@ class TFXLMModelTest(TFModelTesterMixin, unittest.TestCase):
             inputs = [input_ids, input_mask]
             outputs = model(inputs)
             sequence_output = outputs[0]
-            result = {
-                "sequence_output": sequence_output.numpy(),
-            }
+            result = {"sequence_output": sequence_output.numpy()}
             self.parent.assertListEqual(
                 list(result["sequence_output"].shape), [self.batch_size, self.seq_length, self.hidden_size]
             )
@@ -201,9 +199,7 @@ class TFXLMModelTest(TFModelTesterMixin, unittest.TestCase):
 
             logits = outputs[0]
 
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
 
             self.parent.assertListEqual(
                 list(result["logits"].shape), [self.batch_size, self.seq_length, self.vocab_size]
@@ -226,10 +222,7 @@ class TFXLMModelTest(TFModelTesterMixin, unittest.TestCase):
 
             start_logits, end_logits = model(inputs)
 
-            result = {
-                "start_logits": start_logits.numpy(),
-                "end_logits": end_logits.numpy(),
-            }
+            result = {"start_logits": start_logits.numpy(), "end_logits": end_logits.numpy()}
 
             self.parent.assertListEqual(list(result["start_logits"].shape), [self.batch_size, self.seq_length])
             self.parent.assertListEqual(list(result["end_logits"].shape), [self.batch_size, self.seq_length])
@@ -251,9 +244,7 @@ class TFXLMModelTest(TFModelTesterMixin, unittest.TestCase):
 
             (logits,) = model(inputs)
 
-            result = {
-                "logits": logits.numpy(),
-            }
+            result = {"logits": logits.numpy()}
 
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.type_sequence_label_size])
 

tests/test_modeling_tf_xlnet.py

@@ -183,10 +183,7 @@ class TFXLNetModelTest(TFModelTesterMixin, unittest.TestCase):
 
             outputs, mems_1 = model(inputs)
 
-            result = {
-                "mems_1": [mem.numpy() for mem in mems_1],
-                "outputs": outputs.numpy(),
-            }
+            result = {"mems_1": [mem.numpy() for mem in mems_1], "outputs": outputs.numpy()}
 
             config.mem_len = 0
             model = TFXLNetModel(config)
@@ -302,10 +299,7 @@ class TFXLNetModelTest(TFModelTesterMixin, unittest.TestCase):
 
             logits, mems_1 = model(input_ids_1)
 
-            result = {
-                "mems_1": [mem.numpy() for mem in mems_1],
-                "logits": logits.numpy(),
-            }
+            result = {"mems_1": [mem.numpy() for mem in mems_1], "logits": logits.numpy()}
 
             self.parent.assertListEqual(list(result["logits"].shape), [self.batch_size, self.type_sequence_label_size])
             self.parent.assertListEqual(
@@ -335,10 +329,7 @@ class TFXLNetModelTest(TFModelTesterMixin, unittest.TestCase):
                 # 'token_type_ids': token_type_ids
             }
             logits, mems_1 = model(inputs)
-            result = {
-                "mems_1": [mem.numpy() for mem in mems_1],
-                "logits": logits.numpy(),
-            }
+            result = {"mems_1": [mem.numpy() for mem in mems_1], "logits": logits.numpy()}
             self.parent.assertListEqual(
                 list(result["logits"].shape), [self.batch_size, self.seq_length, config.num_labels]
             )