make style and quality

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

.github/ISSUE_TEMPLATE/---new-benchmark.md

@@ -1,17 +1,17 @@
 ---
-name: "\U0001F5A5 New benchmark"
-about: Benchmark a part of this library and share your results
+name: "\U0001F5A5 New Benchmark"
+about: You benchmark a part of this library and would like to share your results
 title: "[Benchmark]"
 labels: ''
 assignees: ''
 
 ---
 
-# 🖥 Benchmarking `transformers`
+# Benchmarking Transformers
 
 ## Benchmark
 
-Which part of `transformers` did you benchmark?
+Which part of Transformers did you benchmark?
 
 ## Set-up
 

.github/ISSUE_TEMPLATE/--new-model-addition.md

@@ -1,5 +1,5 @@
 ---
-name: "\U0001F31F New model addition"
+name: "\U0001F31FNew model addition"
 about: Submit a proposal/request to implement a new Transformer-based model
 title: ''
 labels: ''
@@ -7,14 +7,18 @@ assignees: ''
 
 ---
 
-# 🌟 New model addition
+# 🌟New model addition
 
 ## Model description
 
 <!-- Important information -->
 
-## Open source status
+## Open Source status
 
 * [ ] the model implementation is available: (give details)
 * [ ] the model weights are available: (give details)
-* [ ] who are the authors: (mention them, if possible by @gh-username)
+* [ ] who are the authors: (mention them)
+
+## Additional context
+
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,29 +1,29 @@
 ---
 name: "\U0001F41B Bug Report"
-about: Submit a bug report to help us improve transformers
+about: Submit a bug report to help us improve PyTorch Transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-# 🐛 Bug
+## 🐛 Bug
 
-## Information
+<!-- Important information -->
 
-Model I am using (Bert, XLNet ...):
+Model I am using (Bert, XLNet....):
 
-Language I am using the model on (English, Chinese ...):
+Language I am using the model on (English, Chinese....):
 
-The problem arises when using:
-* [ ] the official example scripts: (give details below)
-* [ ] my own modified scripts: (give details below)
+The problem arise when using:
+* [ ] the official example scripts: (give details)
+* [ ] my own modified scripts: (give details)
 
 The tasks I am working on is:
 * [ ] an official GLUE/SQUaD task: (give the name)
-* [ ] my own task or dataset: (give details below)
+* [ ] my own task or dataset: (give details)
 
-## To reproduce
+## To Reproduce
 
 Steps to reproduce the behavior:
 
@@ -31,20 +31,22 @@ Steps to reproduce the behavior:
 2.
 3.
 
-<!-- If you have code snippets, error messages, stack traces please provide them here as well.
-     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
-     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
+<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
 
 ## Expected behavior
 
-<!-- A clear and concise description of what you would expect to happen. -->
+<!-- A clear and concise description of what you expected to happen. -->
 
 ## Environment
 
 * OS:
 * Python version:
 * PyTorch version:
-* `transformers` version (or branch):
+* PyTorch Transformers version (or branch):
 * Using GPU ?
 * Distributed or parallel setup ?
 * Any other relevant information:
+
+## Additional context
+
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/feature-request.md

@@ -1,25 +1,20 @@
 ---
-name: "\U0001F680 Feature request"
-about: Submit a proposal/request for a new transformers feature
+name: "\U0001F680 Feature Request"
+about: Submit a proposal/request for a new PyTorch Transformers feature
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-# 🚀 Feature request
+## 🚀 Feature
 
-<!-- A clear and concise description of the feature proposal.
-     Please provide a link to the paper and code in case they exist. -->
+<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
 
 ## Motivation
 
-<!-- Please outline the motivation for the proposal. Is your feature request
-     related to a problem? e.g., I'm always frustrated when [...]. If this is related
-     to another GitHub issue, please link here too. -->
+<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->
 
-## Your contribution
+## Additional context
 
-<!-- Is there any way that you could help, e.g. by submitting a PR?
-     Make sure to read the CONTRIBUTING.MD readme:
-     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
+<!-- Add any other context or screenshots about the feature request here. -->

.github/ISSUE_TEMPLATE/migration.md

@@ -1,52 +1,47 @@
 ---
-name: "\U0001F4DA Migration from pytorch-pretrained-bert or pytorch-transformers"
-about: Report a problem when migrating from pytorch-pretrained-bert or pytorch-transformers to transformers
+name: "\U0001F4DA Migration from PyTorch-pretrained-Bert"
+about: Report a problem when migrating from PyTorch-pretrained-Bert to Transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-# 📚 Migration
-
-## Information
+## 📚 Migration
 
 <!-- Important information -->
 
-Model I am using (Bert, XLNet ...):
+Model I am using (Bert, XLNet....):
 
-Language I am using the model on (English, Chinese ...):
+Language I am using the model on (English, Chinese....):
 
-The problem arises when using:
-* [ ] the official example scripts: (give details below)
-* [ ] my own modified scripts: (give details below)
+The problem arise when using:
+* [ ] the official example scripts: (give details)
+* [ ] my own modified scripts: (give details)
 
 The tasks I am working on is:
 * [ ] an official GLUE/SQUaD task: (give the name)
-* [ ] my own task or dataset: (give details below)
+* [ ] my own task or dataset: (give details)
 
-## Details
+Details of the issue:
 
-<!-- A clear and concise description of the migration issue.
-    If you have code snippets, please provide it here as well.
-    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
-    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
-    -->
+<!-- A clear and concise description of the migration issue. If you have code snippets, please provide it here as well. -->
 
 ## Environment
 
 * OS:
 * Python version:
 * PyTorch version:
-* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
-* `transformers` version (or branch):
-* Using GPU?
-* Distributed or parallel setup?
+* PyTorch Transformers version (or branch):
+* Using GPU ?
+* Distributed or parallel setup ?
 * Any other relevant information:
 
 ## Checklist
 
 - [ ] I have read the migration guide in the readme.
- ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
-  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
 - [ ] I checked if a related official extension example runs on my machine.
+
+## Additional context
+
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/question-help.md

@@ -1,29 +1,12 @@
 ---
-name: "❓ Questions & Help"
-about: Post your general questions on Stack Overflow tagged huggingface-transformers
+name: "❓Questions & Help"
+about: Start a general discussion related to PyTorch Transformers
 title: ''
 labels: ''
 assignees: ''
 
 ---
 
-# ❓ Questions & Help
+## ❓ Questions & Help
 
-<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
-     new models and benchmarks, and migration questions. For all other questions,
-     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
-     Tensorflow enthusiast can help you out. Make sure to tag your question with the
-     right deep learning framework as well as the huggingface-transformers tag: 
-     https://stackoverflow.com/questions/tagged/huggingface-transformers 
-     
-     If your question wasn't answered after a period of time on Stack Overflow, you
-     can always open a question on GitHub. You should then link to the SO question 
-     that you posted.
-     -->
-
-## Details
-<!-- Description of your issue -->
-
-<!-- You should first ask your question on SO, and only if
-     you didn't get an answer ask it here on GitHub. -->
-**A link to original question on Stack Overflow**: 
+<!-- A clear and concise description of the question. -->

docs/source/model_doc/auto.rst

@@ -29,13 +29,6 @@ Instantiating one of ``AutoModel``, ``AutoConfig`` and ``AutoTokenizer`` will di
     :members:
 
 
-``AutoModelForPreTraining``
-~~~~~~~~~~~~~~~~~~~~~
-
-.. autoclass:: transformers.AutoModelForPreTraining
-    :members:
-
-
 ``AutoModelWithLMHead``
 ~~~~~~~~~~~~~~~~~~~~~
 

docs/source/pretrained_models.rst

@@ -88,10 +88,6 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``bert-base-finnish-uncased-v1``                           | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
 |                   |                                                            | | Trained on uncased Finnish text.                                                                                                    |
 |                   |                                                            | (see `details on turkunlp.org <http://turkunlp.org/FinBERT/>`__).                                                                     |
-|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
-|                   | ``bert-base-dutch-cased``                                  | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
-|                   |                                                            | | Trained on cased Dutch text.                                                                                                        |
-|                   |                                                            | (see `details on wietsedv repository <https://github.com/wietsedv/bertje/>`__).                                                       |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 | GPT               | ``openai-gpt``                                             | | 12-layer, 768-hidden, 12-heads, 110M parameters.                                                                                    |
 |                   |                                                            | | OpenAI GPT English model                                                                                                            |

examples/run_glue.py

@@ -513,8 +513,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_lm_finetuning.py

@@ -202,9 +202,8 @@ def mask_tokens(inputs: torch.Tensor, tokenizer: PreTrainedTokenizer, args) -> T
         tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
     ]
     probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
-    if tokenizer._pad_token is not None:
-        padding_mask = labels.eq(tokenizer.pad_token_id)
-        probability_matrix.masked_fill_(padding_mask, value=0.0)
+    padding_mask = labels.eq(tokenizer.pad_token_id)
+    probability_matrix.masked_fill_(padding_mask, value=0.0)
     masked_indices = torch.bernoulli(probability_matrix).bool()
     labels[~masked_indices] = -100  # We only compute loss on masked tokens
 
@@ -229,8 +228,6 @@ def train(args, train_dataset, model: PreTrainedModel, tokenizer: PreTrainedToke
     args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
 
     def collate(examples: List[torch.Tensor]):
-        if tokenizer._pad_token is None:
-            return pad_sequence(examples, batch_first=True)
         return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
 
     train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
@@ -424,8 +421,6 @@ def evaluate(args, model: PreTrainedModel, tokenizer: PreTrainedTokenizer, prefi
     # Note that DistributedSampler samples randomly
 
     def collate(examples: List[torch.Tensor]):
-        if tokenizer._pad_token is None:
-            return pad_sequence(examples, batch_first=True)
         return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
 
     eval_sampler = SequentialSampler(eval_dataset)
@@ -574,8 +569,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--save_total_limit",
         type=int,

examples/run_multiple_choice.py

@@ -478,8 +478,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_ner.py

@@ -485,8 +485,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_squad.py

@@ -636,8 +636,8 @@ def main():
         "A number of warnings are expected for a normal SQuAD evaluation.",
     )
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

examples/run_tf_glue.py

@@ -1,22 +1,3 @@
-# coding=utf-8
-# Copyright 2018 The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
-
-
-import argparse
-import logging
 import os
 
 import tensorflow as tf
@@ -24,309 +5,104 @@ import tensorflow_datasets
 
 from transformers import (
     BertConfig,
+    BertForSequenceClassification,
     BertTokenizer,
-    DistilBertConfig,
-    DistilBertTokenizer,
-    RobertaConfig,
-    RobertaTokenizer,
     TFBertForSequenceClassification,
-    TFDistilBertForSequenceClassification,
-    TFRobertaForSequenceClassification,
-    TFXLMForSequenceClassification,
-    TFXLNetForSequenceClassification,
-    XLMConfig,
-    XLMTokenizer,
-    XLNetConfig,
-    XLNetTokenizer,
-)
-from transformers import glue_convert_examples_to_features as convert_examples_to_features
-from transformers import glue_output_modes as output_modes
-from transformers import glue_processors as processors
-
-
-logger = logging.getLogger(__name__)
-
-ALL_MODELS = sum(
-    (
-        tuple(conf.pretrained_config_archive_map.keys())
-        for conf in (BertConfig, XLNetConfig, XLMConfig, RobertaConfig, DistilBertConfig)
-    ),
-    (),
+    glue_convert_examples_to_features,
+    glue_processors,
 )
 
-MODEL_CLASSES = {
-    "bert": (BertConfig, TFBertForSequenceClassification, BertTokenizer),
-    "xlnet": (XLNetConfig, TFXLNetForSequenceClassification, XLNetTokenizer),
-    "xlm": (XLMConfig, TFXLMForSequenceClassification, XLMTokenizer),
-    "roberta": (RobertaConfig, TFRobertaForSequenceClassification, RobertaTokenizer),
-    "distilbert": (DistilBertConfig, TFDistilBertForSequenceClassification, DistilBertTokenizer),
-}
+
+# script parameters
+BATCH_SIZE = 32
+EVAL_BATCH_SIZE = BATCH_SIZE * 2
+USE_XLA = False
+USE_AMP = False
+EPOCHS = 3
+
+TASK = "mrpc"
+
+if TASK == "sst-2":
+    TFDS_TASK = "sst2"
+elif TASK == "sts-b":
+    TFDS_TASK = "stsb"
+else:
+    TFDS_TASK = TASK
+
+num_labels = len(glue_processors[TASK]().get_labels())
+print(num_labels)
+
+tf.config.optimizer.set_jit(USE_XLA)
+tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})
+
+# Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
+config = BertConfig.from_pretrained("bert-base-cased", num_labels=num_labels)
+tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
+model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
+
+# Load dataset via TensorFlow Datasets
+data, info = tensorflow_datasets.load(f"glue/{TFDS_TASK}", with_info=True)
+train_examples = info.splits["train"].num_examples
+
+# MNLI expects either validation_matched or validation_mismatched
+valid_examples = info.splits["validation"].num_examples
+
+# Prepare dataset for GLUE as a tf.data.Dataset instance
+train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, 128, TASK)
+
+# MNLI expects either validation_matched or validation_mismatched
+valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, 128, TASK)
+train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
+valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)
+
+# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
+opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
+if USE_AMP:
+    # loss scaling is currently required when using mixed precision
+    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
 
 
-def load_and_cache_examples(args, data, task, tokenizer, split):
-    if task == "mnli" and split == "validation":
-        split = "validation_matched"
+if num_labels == 1:
+    loss = tf.keras.losses.MeanSquaredError()
+else:
+    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
 
-    features_output_dir = os.path.join(args.output_dir, "features")
-    cached_features_file = os.path.join(
-        features_output_dir,
-        "cached_{}_{}_{}_{}.tfrecord".format(
-            split, list(filter(None, args.model_name_or_path.split("/"))).pop(), str(args.max_seq_length), str(task)
-        ),
-    )
+metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
+model.compile(optimizer=opt, loss=loss, metrics=[metric])
 
-    if not os.path.exists(cached_features_file) or args.overwrite_cache:
-        logger.info("Converting examples to features")
-        dataset = convert_examples_to_features(data[split], tokenizer, args.max_seq_length, task)
+# Train and evaluate using tf.keras.Model.fit()
+train_steps = train_examples // BATCH_SIZE
+valid_steps = valid_examples // EVAL_BATCH_SIZE
 
-        if not os.path.exists(features_output_dir):
-            os.makedirs(features_output_dir)
+history = model.fit(
+    train_dataset,
+    epochs=EPOCHS,
+    steps_per_epoch=train_steps,
+    validation_data=valid_dataset,
+    validation_steps=valid_steps,
+)
 
-        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
-            for feature in dataset:
-                example, label = feature
-                feature_key_value_pair = {
-                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
-                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
-                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
-                    "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
-                }
-                features = tf.train.Features(feature=feature_key_value_pair)
-                example = tf.train.Example(features=features)
+# Save TF2 model
+os.makedirs("./save/", exist_ok=True)
+model.save_pretrained("./save/")
 
-                tfwriter.write(example.SerializeToString())
+if TASK == "mrpc":
+    # Load the TensorFlow model in PyTorch for inspection
+    # This is to demo the interoperability between the two frameworks, you don't have to
+    # do this in real life (you can run the inference on the TF model).
+    pytorch_model = BertForSequenceClassification.from_pretrained("./save/", from_tf=True)
 
-        logger.info("Features saved to cache")
+    # Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
+    sentence_0 = "This research was consistent with his findings."
+    sentence_1 = "His findings were compatible with this research."
+    sentence_2 = "His findings were not compatible with this research."
+    inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors="pt")
+    inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors="pt")
 
-    features = {
-        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "label": tf.io.FixedLenFeature([], tf.int64),
-    }
+    del inputs_1["special_tokens_mask"]
+    del inputs_2["special_tokens_mask"]
 
-    def select_data_from_record(record):
-        record = tf.io.parse_single_example(record, features)
-        x = {
-            "input_ids": record["input_ids"],
-            "attention_mask": record["attention_mask"],
-            "token_type_ids": record["token_type_ids"],
-        }
-        y = record["label"]
-        return (x, y)
-
-    dataset = tf.data.TFRecordDataset(cached_features_file)
-    dataset = dataset.map(select_data_from_record)
-
-    logger.info("Created dataset %s from TFRecord" % split)
-    return dataset
-
-
-def main():
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument(
-        "--model_type",
-        default=None,
-        type=str,
-        required=True,
-        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--model_name_or_path",
-        default=None,
-        type=str,
-        required=True,
-        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
-    )
-    parser.add_argument(
-        "--task_name",
-        default=None,
-        type=str,
-        required=True,
-        help="The name of the task to train selected in the list: " + ", ".join(processors.keys()),
-    )
-    parser.add_argument(
-        "--output_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The output directory where the model predictions and checkpoints will be written.",
-    )
-    parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
-    )
-
-    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
-    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
-    parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
-    )
-
-    parser.add_argument("--train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
-    parser.add_argument(
-        "--valid_batch_size", default=8, type=int, help="Batch size per GPU/CPU for validation during training."
-    )
-    parser.add_argument(
-        "--test_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation after training."
-    )
-    parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
-    )
-
-    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
-    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
-    parser.add_argument(
-        "--max_seq_length",
-        default=128,
-        type=int,
-        help="The maximum total input sequence length after tokenization. Sequences longer "
-        "than this will be truncated, sequences shorter will be padded.",
-    )
-    parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
-    )
-
-    parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
-    )
-    parser.add_argument(
-        "--tokenizer_name",
-        default="",
-        type=str,
-        help="Pretrained tokenizer name or path if not the same as model_name",
-    )
-    parser.add_argument(
-        "--cache_dir",
-        default="",
-        type=str,
-        help="Where do you want to store the pre-trained models downloaded from s3",
-    )
-    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
-    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
-    parser.add_argument(
-        "--force_download",
-        action="store_true",
-        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
-    )
-    args = parser.parse_args()
-
-    # Setup logging
-    logging.basicConfig(
-        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
-        datefmt="%m/%d/%Y %H:%M:%S",
-        level=logging.INFO,
-    )
-
-    if os.path.exists(args.output_dir) and args.do_train:
-        if not args.overwrite_output_dir and bool(
-            [file for file in os.listdir(args.output_dir) if "features" not in file]
-        ):
-            raise ValueError(
-                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
-                    args.output_dir
-                )
-            )
-
-    args.model_type = args.model_type.lower()
-    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
-
-    TASK = args.task_name.lower()
-
-    if TASK not in processors:
-        raise ValueError("Task not found: %s" % (TASK))
-
-    if TASK == "sst-2":
-        TFDS_TASK = "sst2"
-    elif TASK == "sts-b":
-        TFDS_TASK = "stsb"
-    else:
-        TFDS_TASK = TASK
-
-    num_labels = len(processors[TASK]().get_labels())
-    print(num_labels)
-
-    tf.config.optimizer.set_jit(args.xla)
-    tf.config.optimizer.set_experimental_options({"auto_mixed_precision": args.amp})
-
-    # Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
-    config = config_class.from_pretrained(
-        args.config_name if args.config_name else args.model_name_or_path,
-        num_labels=num_labels,
-        finetuning_task=args.task_name,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    tokenizer = tokenizer_class.from_pretrained(
-        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
-        do_lower_case=args.do_lower_case,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    model = model_class.from_pretrained(
-        args.model_name_or_path,
-        config=config,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    # Load dataset via TensorFlow Datasets
-    data, info = tensorflow_datasets.load("glue/%s" % TFDS_TASK, with_info=True)
-
-    # Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
-    opt = tf.keras.optimizers.Adam(learning_rate=args.learning_rate, epsilon=args.adam_epsilon)
-
-    if args.amp:
-        # loss scaling is currently required when using mixed precision
-        opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
-
-    if num_labels == 1:
-        loss = tf.keras.losses.MeanSquaredError()
-    else:
-        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
-
-    metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
-    model.compile(optimizer=opt, loss=loss, metrics=[metric])
-
-    class save_model(tf.keras.callbacks.Callback):
-        def on_epoch_end(self, epoch, logs=None):
-            print("Saving model at epoch {}".format(epoch))
-            output_dir = os.path.join(args.output_dir, "checkpoint-epoch-{}".format(epoch))
-            if not os.path.exists(output_dir):
-                os.makedirs(output_dir)
-            self.model.save_pretrained(output_dir)
-
-    if args.do_train:
-        train_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="train")
-        train_dataset = train_dataset.batch(args.train_batch_size).repeat(args.num_train_epochs)
-        train_examples = info.splits["train"].num_examples / args.train_batch_size
-
-        validation_identifier = "validation_mismatched" if TASK == "mnli" else "validation"
-        valid_dataset = load_and_cache_examples(
-            args, data=data, task=TASK, tokenizer=tokenizer, split=validation_identifier
-        )
-        valid_dataset = valid_dataset.batch(args.valid_batch_size)
-        valid_examples = info.splits[validation_identifier].num_examples / args.valid_batch_size
-
-        history = model.fit(
-            train_dataset,
-            steps_per_epoch=train_examples,
-            epochs=args.num_train_epochs,
-            validation_data=valid_dataset if args.evaluate_during_training else None,
-            validation_steps=valid_examples if args.evaluate_during_training else None,
-            callbacks=[save_model()],
-        )
-
-    if args.do_eval:
-        test_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="test")
-        test_dataset = test_dataset.batch(args.test_batch_size)
-        test_examples = info.splits["test"].num_examples / args.test_batch_size
-
-        results = model.evaluate(test_dataset, steps=test_examples)
-
-
-if __name__ == "__main__":
-    main()
+    pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
+    pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
+    print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
+    print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")

examples/run_tf_squad.py

@@ -1,699 +0,0 @@
-# Copyright 2018 The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
-from __future__ import absolute_import, division, print_function
-
-import argparse
-import datetime
-import glob
-import logging
-import math
-import os
-import pickle
-
-import regex as re
-import tensorflow as tf
-from fastprogress import master_bar, progress_bar
-from seqeval import metrics
-from tqdm import tqdm, trange
-
-from transformers import (
-    TF2_WEIGHTS_NAME,
-    BertConfig,
-    BertTokenizer,
-    DistilBertConfig,
-    DistilBertTokenizer,
-    GradientAccumulator,
-    SquadV1Processor,
-    SquadV2Processor,
-    TFBertForQuestionAnswering,
-    TFDistilBertForQuestionAnswering,
-    TFXLMForQuestionAnsweringSimple,
-    TFXLNetForQuestionAnsweringSimple,
-    XLMConfig,
-    XLMTokenizer,
-    XLNetConfig,
-    XLNetTokenizer,
-    create_optimizer,
-    squad_convert_examples_to_features,
-)
-from transformers.data.metrics.squad_metrics import compute_predictions_logits, squad_evaluate
-from transformers.data.processors.squad import SquadResult
-
-
-logger = logging.getLogger(__name__)
-
-ALL_MODELS = sum(
-    (
-        tuple(conf.pretrained_config_archive_map.keys())
-        for conf in (BertConfig, XLNetConfig, XLMConfig, DistilBertConfig)
-    ),
-    (),
-)
-
-MODEL_CLASSES = {
-    "bert": (BertConfig, TFBertForQuestionAnswering, BertTokenizer),
-    "xlnet": (XLNetConfig, TFXLNetForQuestionAnsweringSimple, XLNetTokenizer),
-    "xlm": (XLMConfig, TFXLMForQuestionAnsweringSimple, XLMTokenizer),
-    "distilbert": (DistilBertConfig, TFDistilBertForQuestionAnswering, DistilBertTokenizer),
-}
-
-
-def train(args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size):
-    if args.max_steps > 0:
-        num_train_steps = args.max_steps * args.gradient_accumulation_steps
-        args.num_train_epochs = 1
-    else:
-        num_train_steps = (
-            math.ceil(num_train_examples / train_batch_size)
-            // args.gradient_accumulation_steps
-            * args.num_train_epochs
-        )
-
-    writer = tf.summary.create_file_writer("/tmp/mylogs")
-
-    with strategy.scope():
-        loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(
-            reduction=tf.keras.losses.Reduction.NONE, from_logits=True
-        )
-        optimizer = create_optimizer(args.learning_rate, num_train_steps, args.warmup_steps)
-
-        if args.xla:
-            tf.config.optimizer.set_jit(True)
-
-        if args.amp:
-            optimizer = tf.keras.mixed_precision.experimental.LossScaleOptimizer(optimizer, "dynamic")
-
-        loss_metric = tf.keras.metrics.Mean(name="loss", dtype=tf.float32)
-        gradient_accumulator = GradientAccumulator()
-
-    logging.info("***** Running training *****")
-    logging.info("  Num examples = %d", num_train_examples)
-    logging.info("  Num Epochs = %d", args.num_train_epochs)
-    logging.info("  Instantaneous batch size per device = %d", args.per_device_train_batch_size)
-    logging.info(
-        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
-        train_batch_size * args.gradient_accumulation_steps,
-    )
-    logging.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
-    logging.info("  Total training steps = %d", num_train_steps)
-
-    model.summary()
-
-    @tf.function
-    def apply_gradients():
-        grads_and_vars = []
-
-        for gradient, variable in zip(gradient_accumulator.gradients, model.trainable_variables):
-            if gradient is not None:
-                scaled_gradient = gradient / (args.n_device * args.gradient_accumulation_steps)
-                grads_and_vars.append((scaled_gradient, variable))
-            else:
-                grads_and_vars.append((gradient, variable))
-
-        optimizer.apply_gradients(grads_and_vars, args.max_grad_norm)
-        gradient_accumulator.reset()
-
-    @tf.function
-    def train_step(train_features, train_labels):
-        def step_fn(train_features, train_labels):
-            with tf.GradientTape() as tape:
-                start_logits, end_logits = model(train_features)
-                start_logits = tf.multiply(
-                    start_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32)
-                )
-                end_logits = tf.multiply(end_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32))
-                start_loss = loss_fct(train_labels["start_position"], start_logits)
-                end_loss = loss_fct(train_labels["end_position"], end_logits)
-                total_loss = (start_loss + end_loss) / 2
-
-                loss = tf.reduce_sum(total_loss) * (1.0 / train_batch_size)
-                grads = tape.gradient(loss, model.trainable_variables)
-
-                gradient_accumulator(grads)
-
-            return total_loss
-
-        per_example_losses = strategy.experimental_run_v2(step_fn, args=(train_features, train_labels))
-        mean_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_example_losses, axis=0)
-
-        return mean_loss
-
-    current_time = datetime.datetime.now()
-    train_iterator = master_bar(range(args.num_train_epochs))
-    global_step = 0
-    logging_loss = 0.0
-
-    for epoch in train_iterator:
-        epoch_iterator = progress_bar(
-            train_dataset,
-            total=num_train_steps / args.num_train_epochs,
-            parent=train_iterator,
-            display=args.n_device > 1,
-        )
-        step = 1
-
-        with strategy.scope():
-            for train_features, train_labels in tqdm(
-                epoch_iterator,
-                desc="Training, epoch {}".format(epoch),
-                total=int(num_train_steps / args.num_train_epochs),
-            ):
-                loss = train_step(train_features, train_labels)
-
-                if step % args.gradient_accumulation_steps == 0:
-                    strategy.experimental_run_v2(apply_gradients)
-
-                    loss_metric(loss)
-
-                    global_step += 1
-
-                    if args.logging_steps > 0 and global_step % args.logging_steps == 0:
-                        # Log metrics
-                        if args.n_device == 1 and args.evaluate_during_training:
-                            # Only evaluate when single GPU otherwise metrics may not average well
-                            results = evaluate(args, strategy, model, tokenizer, global_step)
-
-                            with writer.as_default():
-                                tf.summary.scalar("exact", results[0], global_step)
-                                tf.summary.scalar("f1", results[1], global_step)
-
-                        lr = optimizer.learning_rate
-                        learning_rate = lr(step)
-
-                        with writer.as_default():
-                            tf.summary.scalar("lr", learning_rate, global_step)
-                            tf.summary.scalar(
-                                "loss", (loss_metric.result() - logging_loss) / args.logging_steps, global_step
-                            )
-
-                        logging_loss = loss_metric.result()
-
-                    with writer.as_default():
-                        tf.summary.scalar("loss", loss_metric.result(), step=step)
-
-                    if args.save_steps > 0 and global_step % args.save_steps == 0:
-                        # Save model checkpoint
-                        output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
-
-                        if not os.path.exists(output_dir):
-                            os.makedirs(output_dir)
-
-                        model.save_pretrained(output_dir)
-                        logging.info("Saving model checkpoint to %s", output_dir)
-
-                train_iterator.child.comment = f"loss : {loss_metric.result()}"
-                step += 1
-
-        train_iterator.write(f"loss epoch {epoch + 1}: {loss_metric.result()}")
-
-        loss_metric.reset_states()
-
-    logging.info("  Training took time = {}".format(datetime.datetime.now() - current_time))
-
-
-def evaluate(args, strategy, model, tokenizer, prefix):
-    eval_batch_size = args.per_device_eval_batch_size * args.n_device
-    eval_dataset, size, examples, features = load_and_cache_examples(
-        args, tokenizer, evaluate=True, output_examples=True
-    )
-    eval_dataset = eval_dataset.batch(args.per_device_eval_batch_size)
-    eval_dataset = strategy.experimental_distribute_dataset(eval_dataset)
-    num_eval_steps = math.ceil(size / eval_batch_size)
-    master = master_bar(range(1))
-    eval_iterator = progress_bar(eval_dataset, total=num_eval_steps, parent=master, display=args.n_device > 1)
-
-    logging.info("***** Running evaluation *****")
-    logging.info("  Num examples = %d", size)
-    logging.info("  Batch size = %d", eval_batch_size)
-
-    all_results = []
-    for index, (eval_features, eval_labels) in tqdm(
-        enumerate(eval_iterator), total=size / args.per_device_eval_batch_size
-    ):
-        with strategy.scope():
-            start_logits, end_logits = model(eval_features)
-
-            for sample_index in range(args.per_device_eval_batch_size):
-                if index * args.per_device_eval_batch_size + sample_index < len(features):
-                    result = SquadResult(
-                        features[index * args.per_device_eval_batch_size + sample_index].unique_id,
-                        start_logits[sample_index],
-                        end_logits[sample_index],
-                    )
-                    all_results.append(result)
-
-    output_prediction_file = os.path.join(args.output_dir, "predictions_{}.json".format(prefix))
-    output_nbest_file = os.path.join(args.output_dir, "nbest_predictions_{}.json".format(prefix))
-
-    if args.version_2_with_negative:
-        output_null_log_odds_file = os.path.join(args.output_dir, "null_odds_{}.json".format(prefix))
-    else:
-        output_null_log_odds_file = None
-
-    predictions = compute_predictions_logits(
-        examples,
-        features,
-        all_results,
-        args.n_best_size,
-        args.max_answer_length,
-        args.do_lower_case,
-        output_prediction_file,
-        output_nbest_file,
-        output_null_log_odds_file,
-        args.verbose_logging,
-        args.version_2_with_negative,
-        args.null_score_diff_threshold,
-        tokenizer,
-    )
-
-    # Compute the F1 and exact scores.
-    results = squad_evaluate(examples, predictions)
-    return results
-
-
-def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
-    features_output_dir = os.path.join(args.output_dir, "features")
-    cached_features_file = os.path.join(
-        features_output_dir,
-        "cached_{}_{}_{}.tfrecord".format(
-            "dev" if evaluate else "train",
-            list(filter(None, args.model_name_or_path.split("/"))).pop(),
-            str(args.max_seq_length),
-        ),
-    )
-
-    if not os.path.exists(cached_features_file) or args.overwrite_cache:
-        if args.version_2_with_negative:
-            processor = SquadV2Processor()
-        else:
-            processor = SquadV1Processor()
-
-        if args.data_dir:
-            directory = args.directory
-            examples = processor.get_dev_examples(directory) if evaluate else processor.get_train_examples(directory)
-        else:
-            try:
-                import tensorflow_datasets as tfds
-            except ImportError:
-                raise ImportError("If not data_dir is specified, tensorflow_datasets needs to be installed.")
-
-            if args.version_2_with_negative:
-                logger.warning("tensorflow_datasets does not handle version 2 of SQuAD.")
-
-            examples = processor.get_examples_from_dataset(tfds.load("squad"), evaluate=evaluate)
-            examples = examples
-
-        logger.info("Converting examples to features")
-        features, dataset = squad_convert_examples_to_features(
-            examples,
-            tokenizer,
-            args.max_seq_length,
-            args.doc_stride,
-            args.max_query_length,
-            is_training=not evaluate,
-            return_dataset="tf",
-        )
-
-        if not os.path.exists(features_output_dir):
-            os.makedirs(features_output_dir)
-
-        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
-            for feature in tqdm(dataset, desc="Building tfrecord dataset", total=len(features)):
-                example, result = feature
-                feature_key_value_pair = {
-                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
-                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
-                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
-                    "start_position": tf.train.Feature(
-                        int64_list=tf.train.Int64List(value=[result["start_position"]])
-                    ),
-                    "end_position": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["end_position"]])),
-                    "cls_index": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["cls_index"]])),
-                    "p_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=result["p_mask"])),
-                }
-                feature_skeleton = tf.train.Features(feature=feature_key_value_pair)
-                example = tf.train.Example(features=feature_skeleton)
-
-                tfwriter.write(example.SerializeToString())
-
-            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "wb") as handle:
-                pickle.dump(examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
-
-            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "wb") as handle:
-                pickle.dump(features, handle, protocol=pickle.HIGHEST_PROTOCOL)
-
-        logger.info("Features saved to cache")
-
-    feature_skeleton = {
-        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "start_position": tf.io.FixedLenFeature([], tf.int64),
-        "end_position": tf.io.FixedLenFeature([], tf.int64),
-        "cls_index": tf.io.FixedLenFeature([], tf.int64),
-        "p_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-    }
-
-    def select_data_from_record(record):
-        record = tf.io.parse_single_example(record, feature_skeleton)
-        x = {
-            "input_ids": record["input_ids"],
-            "attention_mask": record["attention_mask"],
-            "token_type_ids": record["token_type_ids"],
-        }
-        y = {
-            "start_position": record["start_position"],
-            "end_position": record["end_position"],
-            "cls_index": record["cls_index"],
-            "p_mask": record["p_mask"],
-        }
-        return x, y
-
-    dataset = tf.data.TFRecordDataset(cached_features_file)
-    dataset = dataset.map(select_data_from_record)
-
-    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "rb") as handle:
-        examples = pickle.load(handle)
-
-    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "rb") as handle:
-        features = pickle.load(handle)
-
-    logger.info("Created dataset %s from TFRecord" % "dev" if evaluate else "train")
-
-    if output_examples:
-        return dataset, len(list(dataset.__iter__())), examples, features
-    return dataset, len(list(dataset.__iter__()))
-
-
-def main():
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument(
-        "--model_type",
-        default=None,
-        type=str,
-        required=True,
-        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--model_name_or_path",
-        default=None,
-        type=str,
-        required=True,
-        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
-    )
-    parser.add_argument(
-        "--output_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The output directory where the model predictions and checkpoints will be written.",
-    )
-    parser.add_argument(
-        "--data_dir",
-        default=None,
-        type=str,
-        required=False,
-        help="The input data directory containing the .json files. If no data dir is specified, uses tensorflow_datasets to load the data."
-        + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
-    )
-
-    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
-    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
-    parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
-    )
-
-    parser.add_argument(
-        "--per_device_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training."
-    )
-    parser.add_argument(
-        "--per_device_eval_batch_size",
-        default=8,
-        type=int,
-        help="Batch size per GPU/CPU for validation during training.",
-    )
-    parser.add_argument(
-        "--per_device_test_batch_size",
-        default=8,
-        type=int,
-        help="Batch size per GPU/CPU for evaluation after training.",
-    )
-    parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
-    )
-
-    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
-    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
-    parser.add_argument(
-        "--max_seq_length",
-        default=128,
-        type=int,
-        help="The maximum total input sequence length after tokenization. Sequences longer "
-        "than this will be truncated, sequences shorter will be padded.",
-    )
-    parser.add_argument(
-        "--doc_stride",
-        default=128,
-        type=int,
-        help="When splitting up a long document into chunks, how much stride to take between chunks.",
-    )
-    parser.add_argument(
-        "--max_query_length",
-        default=64,
-        type=int,
-        help="The maximum number of tokens for the question. Questions longer than this will "
-        "be truncated to this length.",
-    )
-    parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
-    )
-
-    parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
-    )
-    parser.add_argument(
-        "--tokenizer_name",
-        default="",
-        type=str,
-        help="Pretrained tokenizer name or path if not the same as model_name",
-    )
-    parser.add_argument(
-        "--cache_dir",
-        default="",
-        type=str,
-        help="Where do you want to store the pre-trained models downloaded from s3",
-    )
-    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
-    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
-    parser.add_argument(
-        "--force_download",
-        action="store_true",
-        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
-    )
-
-    parser.add_argument(
-        "--tpu",
-        default=None,
-        help="The Cloud TPU to use for training. This should be either the name "
-        "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
-        "url.",
-    )
-    parser.add_argument("--num_tpu_cores", default="8", help="Total number of TPU cores to use.")
-    parser.add_argument(
-        "--gpus",
-        default="0",
-        help="Comma separated list of gpus devices. If only one, switch to single gpu strategy, if None takes all the gpus available.",
-    )
-    parser.add_argument("--no_cuda", action="store_true", help="Whether not to use CUDA when available")
-    parser.add_argument(
-        "--version_2_with_negative",
-        action="store_true",
-        help="If true, the SQuAD examples contain some that do not have an answer.",
-    )
-    parser.add_argument(
-        "--max_steps",
-        default=-1,
-        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
-    )
-    parser.add_argument(
-        "--gradient_accumulation_steps",
-        default=1,
-        help="Number of updates steps to accumulate before performing a backward/update pass.",
-    )
-    parser.add_argument(
-        "--warmup_steps", default=0, help="Linear warmup over warmup_steps.",
-    )
-    parser.add_argument("--max_grad_norm", default=1.0, help="Max gradient norm.")
-    parser.add_argument("--logging_steps", default=500, type=int, help="Log every X updates.")
-    parser.add_argument("--save_steps", default=500, type=int, help="Save checkpoint every X updates.")
-    parser.add_argument(
-        "--eval_all_checkpoints",
-        action="store_true",
-        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
-    )
-    parser.add_argument(
-        "--n_best_size",
-        default=20,
-        type=int,
-        help="The total number of n-best predictions to generate in the nbest_predictions.json output file.",
-    )
-    parser.add_argument(
-        "--max_answer_length",
-        default=30,
-        type=int,
-        help="The maximum length of an answer that can be generated. This is needed because the start "
-        "and end predictions are not conditioned on one another.",
-    )
-    parser.add_argument(
-        "--verbose_logging",
-        action="store_true",
-        help="If true, all of the warnings related to data processing will be printed. "
-        "A number of warnings are expected for a normal SQuAD evaluation.",
-    )
-    parser.add_argument(
-        "--null_score_diff_threshold",
-        type=float,
-        default=0.0,
-        help="If null_score - best_non_null is greater than the threshold predict null.",
-    )
-    args = parser.parse_args()
-
-    # Setup logging
-    logging.basicConfig(
-        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
-        datefmt="%m/%d/%Y %H:%M:%S",
-        level=logging.INFO,
-    )
-
-    if os.path.exists(args.output_dir) and args.do_train:
-        if not args.overwrite_output_dir and bool(
-            [file for file in os.listdir(args.output_dir) if "features" not in file]
-        ):
-            raise ValueError(
-                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
-                    args.output_dir
-                )
-            )
-
-        if args.amp:
-            tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})
-
-    if args.tpu:
-        resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=args.tpu)
-        tf.config.experimental_connect_to_cluster(resolver)
-        tf.tpu.experimental.initialize_tpu_system(resolver)
-        strategy = tf.distribute.experimental.TPUStrategy(resolver)
-        args.n_device = args.num_tpu_cores
-    elif len(args.gpus.split(",")) > 1:
-        args.n_device = len([f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
-        strategy = tf.distribute.MirroredStrategy(devices=[f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
-    elif args.no_cuda:
-        args.n_device = 1
-        strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
-    else:
-        args.n_device = len(args.gpus.split(","))
-        strategy = tf.distribute.OneDeviceStrategy(device="/gpu:" + args.gpus.split(",")[0])
-
-    logging.warning(
-        "n_device: %s, distributed training: %s, 16-bits training: %s",
-        args.n_device,
-        bool(args.n_device > 1),
-        args.amp,
-    )
-
-    args.model_type = args.model_type.lower()
-    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
-
-    config = config_class.from_pretrained(
-        args.config_name if args.config_name else args.model_name_or_path,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-    )
-
-    logging.info("Training/evaluation parameters %s", args)
-
-    if args.do_train:
-        tokenizer = tokenizer_class.from_pretrained(
-            args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
-            do_lower_case=args.do_lower_case,
-            cache_dir=args.cache_dir if args.cache_dir else None,
-        )
-
-        with strategy.scope():
-            model = model_class.from_pretrained(
-                args.model_name_or_path,
-                from_pt=bool(".bin" in args.model_name_or_path),
-                config=config,
-                cache_dir=args.cache_dir if args.cache_dir else None,
-            )
-            model.layers[-1].activation = tf.keras.activations.softmax
-
-        train_batch_size = args.per_device_train_batch_size * args.n_device
-        train_dataset, num_train_examples = load_and_cache_examples(args, tokenizer, evaluate=False)
-
-        train_dataset = train_dataset.batch(train_batch_size)
-        train_dataset = train_dataset.prefetch(buffer_size=train_batch_size)
-        train_dataset = strategy.experimental_distribute_dataset(train_dataset)
-
-        train(
-            args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size,
-        )
-
-        if not os.path.exists(args.output_dir):
-            os.makedirs(args.output_dir)
-
-        logging.info("Saving model to %s", args.output_dir)
-
-        model.save_pretrained(args.output_dir)
-        tokenizer.save_pretrained(args.output_dir)
-
-    if args.do_eval:
-        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
-        checkpoints = []
-        results = []
-
-        if args.eval_all_checkpoints:
-            checkpoints = list(
-                os.path.dirname(c)
-                for c in sorted(
-                    glob.glob(args.output_dir + "/**/" + TF2_WEIGHTS_NAME, recursive=True),
-                    key=lambda f: int("".join(filter(str.isdigit, f)) or -1),
-                )
-            )
-
-        logging.info("Evaluate the following checkpoints: %s", checkpoints)
-
-        if len(checkpoints) == 0:
-            checkpoints.append(args.output_dir)
-
-        for checkpoint in checkpoints:
-            global_step = checkpoint.split("-")[-1] if re.match(".*checkpoint-[0-9]", checkpoint) else "final"
-
-            with strategy.scope():
-                model = model_class.from_pretrained(checkpoint)
-                results = evaluate(args, strategy, model, tokenizer, prefix=global_step)
-
-            result = dict((k + ("_{}".format(global_step) if global_step else ""), v) for k, v in results.items())
-            results.update(result)
-
-        logger.info("Results: {}".format(results))
-
-
-if __name__ == "__main__":
-    main()

examples/run_xnli.py

@@ -473,8 +473,8 @@ def main():
     )
     parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
 
-    parser.add_argument("--logging_steps", type=int, default=500, help="Log every X updates steps.")
-    parser.add_argument("--save_steps", type=int, default=500, help="Save checkpoint every X updates steps.")
+    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
+    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
     parser.add_argument(
         "--eval_all_checkpoints",
         action="store_true",

setup.py

@@ -63,7 +63,7 @@ extras["sklearn"] = ["scikit-learn"]
 extras["tf"] = ["tensorflow"]
 extras["torch"] = ["torch"]
 
-extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
+extras["serving"] = ["pydantic", "uvicorn", "fastapi"]
 extras["all"] = extras["serving"] + ["tensorflow", "torch"]
 
 extras["testing"] = ["pytest", "pytest-xdist"]
@@ -86,7 +86,7 @@ setup(
     packages=find_packages("src"),
     install_requires=[
         "numpy",
-        "tokenizers == 0.0.11",
+        "tokenizers == 0.2.1",
         # accessing files from S3 directly
         "boto3",
         # filesystem locks e.g. to prevent parallel downloads

src/transformers/__init__.py

@@ -133,7 +133,6 @@ if is_torch_available():
     from .modeling_utils import PreTrainedModel, prune_layer, Conv1D
     from .modeling_auto import (
         AutoModel,
-        AutoModelForPreTraining,
         AutoModelForSequenceClassification,
         AutoModelForQuestionAnswering,
         AutoModelWithLMHead,
@@ -268,7 +267,6 @@ if is_tf_available():
     from .modeling_tf_utils import TFPreTrainedModel, TFSharedEmbeddings, TFSequenceSummary, shape_list
     from .modeling_tf_auto import (
         TFAutoModel,
-        TFAutoModelForPreTraining,
         TFAutoModelForSequenceClassification,
         TFAutoModelForQuestionAnswering,
         TFAutoModelWithLMHead,

src/transformers/commands/serving.py

@@ -2,6 +2,7 @@ import logging
 from argparse import ArgumentParser, Namespace
 from typing import Any, List, Optional
 
+from starlette.responses import JSONResponse
 from transformers import Pipeline
 from transformers.commands import BaseTransformersCLICommand
 from transformers.pipelines import SUPPORTED_TASKS, pipeline
@@ -12,16 +13,15 @@ try:
     from fastapi import FastAPI, HTTPException, Body
     from fastapi.routing import APIRoute
     from pydantic import BaseModel
-    from starlette.responses import JSONResponse
 
-    _serve_dependencies_installed = True
+    _serve_dependancies_installed = True
 except (ImportError, AttributeError):
     BaseModel = object
 
     def Body(*x, **y):
         pass
 
-    _serve_dependencies_installed = False
+    _serve_dependancies_installed = False
 
 
 logger = logging.getLogger("transformers-cli/serving")
@@ -111,7 +111,7 @@ class ServeCommand(BaseTransformersCLICommand):
         self.port = port
         self.workers = workers
 
-        if not _serve_dependencies_installed:
+        if not _serve_dependancies_installed:
             raise RuntimeError(
                 "Using serve command requires FastAPI and unicorn. "
                 'Please install transformers with [serving]: pip install "transformers[serving]".'

src/transformers/configuration_bert.py

@@ -45,7 +45,6 @@ BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-config.json",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/config.json",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/config.json",
-    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/config.json",
 }
 
 

src/transformers/data/metrics/squad_metrics.py

@@ -555,10 +555,10 @@ def compute_predictions_logits(
         all_nbest_json[example.qas_id] = nbest_json
 
     with open(output_prediction_file, "w") as writer:
-        writer.write(json.dumps(str(all_predictions), indent=4) + "\n")
+        writer.write(json.dumps(all_predictions, indent=4) + "\n")
 
     with open(output_nbest_file, "w") as writer:
-        writer.write(json.dumps(str(all_nbest_json), indent=4) + "\n")
+        writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
 
     if version_2_with_negative:
         with open(output_null_log_odds_file, "w") as writer:

src/transformers/data/processors/squad.py

@@ -306,15 +306,13 @@ def squad_convert_examples_to_features(
             tqdm(
                 p.imap(annotate_, examples, chunksize=32),
                 total=len(examples),
-                desc="Converting squad examples to features",
+                desc="convert squad examples to features",
             )
         )
-
-    print("Converted {} examples into {} features".format(len(examples), len(features)))
     new_features = []
     unique_id = 1000000000
     example_index = 0
-    for example_features in features:
+    for example_features in tqdm(features, total=len(features), desc="add example index and unique id"):
         if not example_features:
             continue
         for example_feature in example_features:
@@ -378,34 +376,31 @@ def squad_convert_examples_to_features(
                     },
                 )
 
-        return (
-            features,
-            tf.data.Dataset.from_generator(
-                gen,
-                (
-                    {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
-                    {
-                        "start_position": tf.int64,
-                        "end_position": tf.int64,
-                        "cls_index": tf.int64,
-                        "p_mask": tf.int32,
-                        "is_impossible": tf.int32,
-                    },
-                ),
-                (
-                    {
-                        "input_ids": tf.TensorShape([None]),
-                        "attention_mask": tf.TensorShape([None]),
-                        "token_type_ids": tf.TensorShape([None]),
-                    },
-                    {
-                        "start_position": tf.TensorShape([]),
-                        "end_position": tf.TensorShape([]),
-                        "cls_index": tf.TensorShape([]),
-                        "p_mask": tf.TensorShape([None]),
-                        "is_impossible": tf.TensorShape([]),
-                    },
-                ),
+        return tf.data.Dataset.from_generator(
+            gen,
+            (
+                {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
+                {
+                    "start_position": tf.int64,
+                    "end_position": tf.int64,
+                    "cls_index": tf.int64,
+                    "p_mask": tf.int32,
+                    "is_impossible": tf.int32,
+                },
+            ),
+            (
+                {
+                    "input_ids": tf.TensorShape([None]),
+                    "attention_mask": tf.TensorShape([None]),
+                    "token_type_ids": tf.TensorShape([None]),
+                },
+                {
+                    "start_position": tf.TensorShape([]),
+                    "end_position": tf.TensorShape([]),
+                    "cls_index": tf.TensorShape([]),
+                    "p_mask": tf.TensorShape([None]),
+                    "is_impossible": tf.TensorShape([]),
+                },
             ),
         )
 

src/transformers/modeling_auto.py

@@ -45,7 +45,6 @@ from .modeling_albert import (
 from .modeling_bert import (
     BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     BertForMaskedLM,
-    BertForPreTraining,
     BertForQuestionAnswering,
     BertForSequenceClassification,
     BertForTokenClassification,
@@ -72,7 +71,6 @@ from .modeling_openai import OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP, OpenAIGPTL
 from .modeling_roberta import (
     ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
     RobertaForMaskedLM,
-    RobertaForQuestionAnswering,
     RobertaForSequenceClassification,
     RobertaForTokenClassification,
     RobertaModel,
@@ -144,24 +142,6 @@ MODEL_MAPPING = OrderedDict(
     ]
 )
 
-MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
-    [
-        (T5Config, T5WithLMHeadModel),
-        (DistilBertConfig, DistilBertForMaskedLM),
-        (AlbertConfig, AlbertForMaskedLM),
-        (CamembertConfig, CamembertForMaskedLM),
-        (XLMRobertaConfig, XLMRobertaForMaskedLM),
-        (RobertaConfig, RobertaForMaskedLM),
-        (BertConfig, BertForPreTraining),
-        (OpenAIGPTConfig, OpenAIGPTLMHeadModel),
-        (GPT2Config, GPT2LMHeadModel),
-        (TransfoXLConfig, TransfoXLLMHeadModel),
-        (XLNetConfig, XLNetLMHeadModel),
-        (XLMConfig, XLMWithLMHeadModel),
-        (CTRLConfig, CTRLLMHeadModel),
-    ]
-)
-
 MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
     [
         (T5Config, T5WithLMHeadModel),
@@ -197,7 +177,6 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
     [
         (DistilBertConfig, DistilBertForQuestionAnswering),
         (AlbertConfig, AlbertForQuestionAnswering),
-        (RobertaConfig, RobertaForQuestionAnswering),
         (BertConfig, BertForQuestionAnswering),
         (XLNetConfig, XLNetForQuestionAnswering),
         (XLMConfig, XLMForQuestionAnswering),
@@ -367,156 +346,6 @@ class AutoModel(object):
         )
 
 
-class AutoModelForPreTraining(object):
-    r"""
-        :class:`~transformers.AutoModelForPreTraining` is a generic model class
-        that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
-        class method.
-
-        This class cannot be instantiated using `__init__()` (throws an error).
-    """
-
-    def __init__(self):
-        raise EnvironmentError(
-            "AutoModelForPreTraining is designed to be instantiated "
-            "using the `AutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
-            "`AutoModelForPreTraining.from_config(config)` methods."
-        )
-
-    @classmethod
-    def from_config(cls, config):
-        r""" Instantiates one of the base model classes of the library
-        from a configuration.
-
-        Args:
-            config (:class:`~transformers.PretrainedConfig`):
-                The model class to instantiate is selected based on the configuration class:
-
-                - isInstance of `distilbert` configuration class: :class:`~transformers.DistilBertModelForMaskedLM` (DistilBERT model)
-                - isInstance of `roberta` configuration class: :class:`~transformers.RobertaModelForMaskedLM` (RoBERTa model)
-                - isInstance of `bert` configuration class: :class:`~transformers.BertForPreTraining` (Bert model)
-                - isInstance of `openai-gpt` configuration class: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
-                - isInstance of `gpt2` configuration class: :class:`~transformers.GPT2ModelLMHeadModel` (OpenAI GPT-2 model)
-                - isInstance of `ctrl` configuration class: :class:`~transformers.CTRLModelLMHeadModel` (Salesforce CTRL  model)
-                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
-                - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
-                - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
-
-        Examples::
-
-            config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
-            model = AutoModelForPreTraining.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
-        """
-        for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
-            if isinstance(config, config_class):
-                return model_class(config)
-        raise ValueError(
-            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
-            "Model type should be one of {}.".format(
-                config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
-            )
-        )
-
-    @classmethod
-    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
-        r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
-
-        The `from_pretrained()` method takes care of returning the correct model class instance
-        based on the `model_type` property of the config object, or when it's missing,
-        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
-
-        The model class to instantiate is selected as the first pattern matching
-        in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: :class:`~transformers.T5ModelWithLMHead` (T5 model)
-            - contains `distilbert`: :class:`~transformers.DistilBertForMaskedLM` (DistilBERT model)
-            - contains `albert`: :class:`~transformers.AlbertForMaskedLM` (ALBERT model)
-            - contains `camembert`: :class:`~transformers.CamembertForMaskedLM` (CamemBERT model)
-            - contains `xlm-roberta`: :class:`~transformers.XLMRobertaForMaskedLM` (XLM-RoBERTa model)
-            - contains `roberta`: :class:`~transformers.RobertaForMaskedLM` (RoBERTa model)
-            - contains `bert`: :class:`~transformers.BertForPreTraining` (Bert model)
-            - contains `openai-gpt`: :class:`~transformers.OpenAIGPTLMHeadModel` (OpenAI GPT model)
-            - contains `gpt2`: :class:`~transformers.GPT2LMHeadModel` (OpenAI GPT-2 model)
-            - contains `transfo-xl`: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
-            - contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
-            - contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
-            - contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
-
-        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
-        To train the model, you should first set it back in training mode with `model.train()`
-
-        Args:
-            pretrained_model_name_or_path:
-                Either:
-
-                - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
-                - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
-                - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
-                - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
-            model_args: (`optional`) Sequence of positional arguments:
-                All remaning positional arguments will be passed to the underlying model's ``__init__`` method
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
-                Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
-
-                - the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
-                - the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
-                - the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
-
-            state_dict: (`optional`) dict:
-                an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
-                This option can be used if you want to create a model from a pretrained configuration but load your own weights.
-                In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
-            cache_dir: (`optional`) string:
-                Path to a directory in which a downloaded pre-trained model
-                configuration should be cached if the standard cache should not be used.
-            force_download: (`optional`) boolean, default False:
-                Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
-            resume_download: (`optional`) boolean, default False:
-                Do not delete incompletely received file. Attempt to resume the download if such a file exists.
-            proxies: (`optional`) dict, default None:
-                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
-                The proxies are used on each request.
-            output_loading_info: (`optional`) boolean:
-                Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
-            kwargs: (`optional`) Remaining dictionary of keyword arguments:
-                Can be used to update the configuration object (after it being loaded) and initiate the model.
-                (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
-                automatically loaded:
-
-                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
-                  underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
-                  already been done)
-                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
-                  initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
-                  ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
-                  with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
-                  attribute will be passed to the underlying model's ``__init__`` function.
-
-        Examples::
-
-            model = AutoModelForPreTraining.from_pretrained('bert-base-uncased')    # Download model and configuration from S3 and cache.
-            model = AutoModelForPreTraining.from_pretrained('./test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
-            model = AutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True)  # Update configuration during loading
-            assert model.config.output_attention == True
-            # Loading from a TF checkpoint file instead of a PyTorch model (slower)
-            config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
-            model = AutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
-
-        """
-        config = kwargs.pop("config", None)
-        if not isinstance(config, PretrainedConfig):
-            config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
-
-        for config_class, model_class in MODEL_FOR_PRETRAINING_MAPPING.items():
-            if isinstance(config, config_class):
-                return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
-        raise ValueError(
-            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
-            "Model type should be one of {}.".format(
-                config.__class__, cls.__name__, ", ".join(c.__name__ for c in MODEL_FOR_PRETRAINING_MAPPING.keys())
-            )
-        )
-
-
 class AutoModelWithLMHead(object):
     r"""
         :class:`~transformers.AutoModelWithLMHead` is a generic model class

src/transformers/modeling_bert.py

@@ -53,7 +53,6 @@ BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-pytorch_model.bin",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/pytorch_model.bin",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/pytorch_model.bin",
-    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/pytorch_model.bin",
 }
 
 

src/transformers/modeling_ctrl.py

@@ -30,7 +30,9 @@ from .modeling_utils import Conv1D, PreTrainedModel
 
 logger = logging.getLogger(__name__)
 
-CTRL_PRETRAINED_MODEL_ARCHIVE_MAP = {"ctrl": "https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin"}
+CTRL_PRETRAINED_MODEL_ARCHIVE_MAP = {
+    "ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-pytorch_model.bin"
+}
 
 
 def angle_defn(pos, i, d_model_size):

src/transformers/modeling_tf_auto.py

@@ -42,7 +42,6 @@ from .modeling_tf_albert import (
 from .modeling_tf_bert import (
     TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     TFBertForMaskedLM,
-    TFBertForPreTraining,
     TFBertForQuestionAnswering,
     TFBertForSequenceClassification,
     TFBertForTokenClassification,
@@ -126,22 +125,6 @@ TF_MODEL_MAPPING = OrderedDict(
     ]
 )
 
-TF_MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
-    [
-        (T5Config, TFT5WithLMHeadModel),
-        (DistilBertConfig, TFDistilBertForMaskedLM),
-        (AlbertConfig, TFAlbertForMaskedLM),
-        (RobertaConfig, TFRobertaForMaskedLM),
-        (BertConfig, TFBertForPreTraining),
-        (OpenAIGPTConfig, TFOpenAIGPTLMHeadModel),
-        (GPT2Config, TFGPT2LMHeadModel),
-        (TransfoXLConfig, TFTransfoXLLMHeadModel),
-        (XLNetConfig, TFXLNetLMHeadModel),
-        (XLMConfig, TFXLMWithLMHeadModel),
-        (CTRLConfig, TFCTRLLMHeadModel),
-    ]
-)
-
 TF_MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
     [
         (T5Config, TFT5WithLMHeadModel),
@@ -346,154 +329,6 @@ class TFAutoModel(object):
         )
 
 
-class TFAutoModelForPreTraining(object):
-    r"""
-        :class:`~transformers.TFAutoModelForPreTraining` is a generic model class
-        that will be instantiated as one of the model classes of the library -with the architecture used for pretraining this model– when created with the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)`
-        class method.
-
-        This class cannot be instantiated using `__init__()` (throws an error).
-    """
-
-    def __init__(self):
-        raise EnvironmentError(
-            "TFAutoModelForPreTraining is designed to be instantiated "
-            "using the `TFAutoModelForPreTraining.from_pretrained(pretrained_model_name_or_path)` or "
-            "`TFAutoModelForPreTraining.from_config(config)` methods."
-        )
-
-    @classmethod
-    def from_config(cls, config):
-        r""" Instantiates one of the base model classes of the library
-        from a configuration.
-
-        Args:
-            config (:class:`~transformers.PretrainedConfig`):
-                The model class to instantiate is selected based on the configuration class:
-
-                - isInstance of `distilbert` configuration class: :class:`~transformers.TFDistilBertModelForMaskedLM` (DistilBERT model)
-                - isInstance of `roberta` configuration class: :class:`~transformers.TFRobertaModelForMaskedLM` (RoBERTa model)
-                - isInstance of `bert` configuration class: :class:`~transformers.TFBertForPreTraining` (Bert model)
-                - isInstance of `openai-gpt` configuration class: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
-                - isInstance of `gpt2` configuration class: :class:`~transformers.TFGPT2ModelLMHeadModel` (OpenAI GPT-2 model)
-                - isInstance of `ctrl` configuration class: :class:`~transformers.TFCTRLModelLMHeadModel` (Salesforce CTRL  model)
-                - isInstance of `transfo-xl` configuration class: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
-                - isInstance of `xlnet` configuration class: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
-                - isInstance of `xlm` configuration class: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
-
-        Examples::
-
-            config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
-            model = TFAutoModelForPreTraining.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
-        """
-        for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
-            if isinstance(config, config_class):
-                return model_class(config)
-        raise ValueError(
-            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
-            "Model type should be one of {}.".format(
-                config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
-            )
-        )
-
-    @classmethod
-    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
-        r""" Instantiates one of the model classes of the library -with the architecture used for pretraining this model– from a pre-trained model configuration.
-
-        The `from_pretrained()` method takes care of returning the correct model class instance
-        based on the `model_type` property of the config object, or when it's missing,
-        falling back to using pattern matching on the `pretrained_model_name_or_path` string.
-
-        The model class to instantiate is selected as the first pattern matching
-        in the `pretrained_model_name_or_path` string (in the following order):
-            - contains `t5`: :class:`~transformers.TFT5ModelWithLMHead` (T5 model)
-            - contains `distilbert`: :class:`~transformers.TFDistilBertForMaskedLM` (DistilBERT model)
-            - contains `albert`: :class:`~transformers.TFAlbertForMaskedLM` (ALBERT model)
-            - contains `roberta`: :class:`~transformers.TFRobertaForMaskedLM` (RoBERTa model)
-            - contains `bert`: :class:`~transformers.TFBertForPreTraining` (Bert model)
-            - contains `openai-gpt`: :class:`~transformers.TFOpenAIGPTLMHeadModel` (OpenAI GPT model)
-            - contains `gpt2`: :class:`~transformers.TFGPT2LMHeadModel` (OpenAI GPT-2 model)
-            - contains `transfo-xl`: :class:`~transformers.TFTransfoXLLMHeadModel` (Transformer-XL model)
-            - contains `xlnet`: :class:`~transformers.TFXLNetLMHeadModel` (XLNet model)
-            - contains `xlm`: :class:`~transformers.TFXLMWithLMHeadModel` (XLM model)
-            - contains `ctrl`: :class:`~transformers.TFCTRLLMHeadModel` (Salesforce CTRL model)
-
-        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
-        To train the model, you should first set it back in training mode with `model.train()`
-
-        Args:
-            pretrained_model_name_or_path:
-                Either:
-
-                - a string with the `shortcut name` of a pre-trained model to load from cache or download, e.g.: ``bert-base-uncased``.
-                - a string with the `identifier name` of a pre-trained model that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
-                - a path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g.: ``./my_model_directory/``.
-                - a path or url to a `tensorflow index checkpoint file` (e.g. `./tf_model/model.ckpt.index`). In this case, ``from_tf`` should be set to True and a configuration object should be provided as ``config`` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
-            model_args: (`optional`) Sequence of positional arguments:
-                All remaning positional arguments will be passed to the underlying model's ``__init__`` method
-            config: (`optional`) instance of a class derived from :class:`~transformers.PretrainedConfig`:
-                Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
-
-                - the model is a model provided by the library (loaded with the ``shortcut-name`` string of a pretrained model), or
-                - the model was saved using :func:`~transformers.PreTrainedModel.save_pretrained` and is reloaded by suppling the save directory.
-                - the model is loaded by suppling a local directory as ``pretrained_model_name_or_path`` and a configuration JSON file named `config.json` is found in the directory.
-
-            state_dict: (`optional`) dict:
-                an optional state dictionnary for the model to use instead of a state dictionary loaded from saved weights file.
-                This option can be used if you want to create a model from a pretrained configuration but load your own weights.
-                In this case though, you should check if using :func:`~transformers.PreTrainedModel.save_pretrained` and :func:`~transformers.PreTrainedModel.from_pretrained` is not a simpler option.
-            cache_dir: (`optional`) string:
-                Path to a directory in which a downloaded pre-trained model
-                configuration should be cached if the standard cache should not be used.
-            force_download: (`optional`) boolean, default False:
-                Force to (re-)download the model weights and configuration files and override the cached versions if they exists.
-            resume_download: (`optional`) boolean, default False:
-                Do not delete incompletely received file. Attempt to resume the download if such a file exists.
-            proxies: (`optional`) dict, default None:
-                A dictionary of proxy servers to use by protocol or endpoint, e.g.: {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
-                The proxies are used on each request.
-            output_loading_info: (`optional`) boolean:
-                Set to ``True`` to also return a dictionnary containing missing keys, unexpected keys and error messages.
-            kwargs: (`optional`) Remaining dictionary of keyword arguments:
-                Can be used to update the configuration object (after it being loaded) and initiate the model.
-                (e.g. ``output_attention=True``). Behave differently depending on whether a `config` is provided or
-                automatically loaded:
-
-                - If a configuration is provided with ``config``, ``**kwargs`` will be directly passed to the
-                  underlying model's ``__init__`` method (we assume all relevant updates to the configuration have
-                  already been done)
-                - If a configuration is not provided, ``kwargs`` will be first passed to the configuration class
-                  initialization function (:func:`~transformers.PretrainedConfig.from_pretrained`). Each key of
-                  ``kwargs`` that corresponds to a configuration attribute will be used to override said attribute
-                  with the supplied ``kwargs`` value. Remaining keys that do not correspond to any configuration
-                  attribute will be passed to the underlying model's ``__init__`` function.
-
-        Examples::
-
-            model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased')    # Download model and configuration from S3 and cache.
-            model = TFAutoModelForPreTraining.from_pretrained('./test/bert_model/')  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
-            model = TFAutoModelForPreTraining.from_pretrained('bert-base-uncased', output_attention=True)  # Update configuration during loading
-            assert model.config.output_attention == True
-            # Loading from a TF checkpoint file instead of a PyTorch model (slower)
-            config = AutoConfig.from_json_file('./tf_model/bert_tf_model_config.json')
-            model = TFAutoModelForPreTraining.from_pretrained('./tf_model/bert_tf_checkpoint.ckpt.index', from_tf=True, config=config)
-
-        """
-        config = kwargs.pop("config", None)
-        if not isinstance(config, PretrainedConfig):
-            config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
-
-        for config_class, model_class in TF_MODEL_FOR_PRETRAINING_MAPPING.items():
-            if isinstance(config, config_class):
-                return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
-        raise ValueError(
-            "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"
-            "Model type should be one of {}.".format(
-                config.__class__, cls.__name__, ", ".join(c.__name__ for c in TF_MODEL_FOR_PRETRAINING_MAPPING.keys())
-            )
-        )
-
-
 class TFAutoModelWithLMHead(object):
     r"""
         :class:`~transformers.TFAutoModelWithLMHead` is a generic model class
@@ -548,7 +383,7 @@ class TFAutoModelWithLMHead(object):
         Examples::
 
             config = BertConfig.from_pretrained('bert-base-uncased')    # Download configuration from S3 and cache.
-            model = TFAutoModelWithLMHead.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
+            model = AutoModelWithLMHead.from_config(config)  # E.g. model was saved using `save_pretrained('./test/saved_model/')`
         """
         for config_class, model_class in TF_MODEL_WITH_LM_HEAD_MAPPING.items():
             if isinstance(config, config_class):

src/transformers/modeling_tf_bert.py

@@ -49,7 +49,6 @@ TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
     "bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-tf_model.h5",
     "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/tf_model.h5",
     "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/tf_model.h5",
-    "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/tf_model.h5",
 }
 
 

src/transformers/modeling_xlnet.py

@@ -756,7 +756,7 @@ class XLNetModel(XLNetPreTrainedModel):
             input_ids = input_ids.transpose(0, 1).contiguous()
             qlen, bsz = input_ids.shape[0], input_ids.shape[1]
         elif inputs_embeds is not None:
-            inputs_embeds = inputs_embeds.transpose(0, 1).contiguous()
+            inputs_embeds.transpose(0, 1).contiguous()
             qlen, bsz = inputs_embeds.shape[0], inputs_embeds.shape[1]
         else:
             raise ValueError("You have to specify either input_ids or inputs_embeds")

src/transformers/optimization_tf.py

@@ -56,16 +56,7 @@ class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
         }
 
 
-def create_optimizer(
-    init_lr,
-    num_train_steps,
-    num_warmup_steps,
-    weight_decay=0.0,
-    adam_epsilon=1e-6,
-    beta_1=0.9,
-    beta_2=0.999,
-    exclude_from_weight_decay=("layer_norm", "bias"),
-):
+def create_optimizer(init_lr, num_train_steps, num_warmup_steps):
     """Creates an optimizer with learning rate schedule."""
     # Implements linear decay of the learning rate.
     learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(
@@ -77,11 +68,11 @@ def create_optimizer(
         )
     optimizer = AdamWeightDecay(
         learning_rate=learning_rate_fn,
-        weight_decay_rate=weight_decay,
-        beta_1=beta_1,
-        beta_2=beta_2,
-        epsilon=adam_epsilon,
-        exclude_from_weight_decay=exclude_from_weight_decay,
+        weight_decay_rate=0.01,
+        beta_1=0.9,
+        beta_2=0.999,
+        epsilon=1e-6,
+        exclude_from_weight_decay=["layer_norm", "bias"],
     )
     return optimizer
 

src/transformers/tokenization_auto.py

@@ -36,14 +36,14 @@ from .configuration_auto import (
 )
 from .configuration_utils import PretrainedConfig
 from .tokenization_albert import AlbertTokenizer
-from .tokenization_bert import BertTokenizer
+from .tokenization_bert import BertTokenizer, BertTokenizerFast
 from .tokenization_bert_japanese import BertJapaneseTokenizer
 from .tokenization_camembert import CamembertTokenizer
-from .tokenization_ctrl import CTRLTokenizer
-from .tokenization_distilbert import DistilBertTokenizer
-from .tokenization_gpt2 import GPT2Tokenizer
-from .tokenization_openai import OpenAIGPTTokenizer
-from .tokenization_roberta import RobertaTokenizer
+from .tokenization_ctrl import CTRLTokenizer, CTRLTokenizerFast
+from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast
+from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
+from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
+from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
 from .tokenization_t5 import T5Tokenizer
 from .tokenization_transfo_xl import TransfoXLTokenizer
 from .tokenization_xlm import XLMTokenizer
@@ -56,19 +56,19 @@ logger = logging.getLogger(__name__)
 
 TOKENIZER_MAPPING = OrderedDict(
     [
-        (T5Config, T5Tokenizer),
-        (DistilBertConfig, DistilBertTokenizer),
-        (AlbertConfig, AlbertTokenizer),
-        (CamembertConfig, CamembertTokenizer),
-        (XLMRobertaConfig, XLMRobertaTokenizer),
-        (RobertaConfig, RobertaTokenizer),
-        (BertConfig, BertTokenizer),
-        (OpenAIGPTConfig, OpenAIGPTTokenizer),
-        (GPT2Config, GPT2Tokenizer),
-        (TransfoXLConfig, TransfoXLTokenizer),
-        (XLNetConfig, XLNetTokenizer),
-        (XLMConfig, XLMTokenizer),
-        (CTRLConfig, CTRLTokenizer),
+        (T5Config, (T5Tokenizer, None)),
+        (DistilBertConfig, (DistilBertTokenizer, DistilBertTokenizerFast)),
+        (AlbertConfig, (AlbertTokenizer, None)),
+        (CamembertConfig, (CamembertTokenizer, None)),
+        (XLMRobertaConfig, (XLMRobertaTokenizer, None)),
+        (RobertaConfig, (RobertaTokenizer, RobertaTokenizerFast)),
+        (BertConfig, (BertTokenizer, BertTokenizerFast)),
+        (OpenAIGPTConfig, (OpenAIGPTTokenizer, OpenAIGPTTokenizerFast)),
+        (GPT2Config, (GPT2Tokenizer, GPT2TokenizerFast)),
+        (TransfoXLConfig, (TransfoXLTokenizer, None)),
+        (XLNetConfig, (XLNetTokenizer, None)),
+        (XLMConfig, (XLMTokenizer, None)),
+        (CTRLConfig, (CTRLTokenizer, CTRLTokenizerFast)),
     ]
 )
 
@@ -174,9 +174,12 @@ class AutoTokenizer(object):
         if "bert-base-japanese" in pretrained_model_name_or_path:
             return BertJapaneseTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
 
-        for config_class, tokenizer_class in TOKENIZER_MAPPING.items():
+        for config_class, (tokenizer_class_py, tokenizer_class_ru) in TOKENIZER_MAPPING.items():
             if isinstance(config, config_class):
-                return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
+                if tokenizer_class_ru:
+                    return tokenizer_class_ru.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
+                else:
+                    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
 
         raise ValueError(
             "Unrecognized configuration class {} to build an AutoTokenizer.\n"

src/transformers/tokenization_bert.py

@@ -48,7 +48,6 @@ PRETRAINED_VOCAB_FILES_MAP = {
         "bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-vocab.txt",
         "bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/vocab.txt",
         "bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/vocab.txt",
-        "bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/vocab.txt",
     }
 }
 
@@ -70,7 +69,6 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
     "bert-base-german-dbmdz-uncased": 512,
     "bert-base-finnish-cased-v1": 512,
     "bert-base-finnish-uncased-v1": 512,
-    "bert-base-dutch-cased": 512,
 }
 
 PRETRAINED_INIT_CONFIGURATION = {
@@ -91,7 +89,6 @@ PRETRAINED_INIT_CONFIGURATION = {
     "bert-base-german-dbmdz-uncased": {"do_lower_case": True},
     "bert-base-finnish-cased-v1": {"do_lower_case": False},
     "bert-base-finnish-uncased-v1": {"do_lower_case": True},
-    "bert-base-dutch-cased": {"do_lower_case": False},
 }
 
 
@@ -558,6 +555,15 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
         **kwargs
     ):
         super().__init__(
+            tk.implementations.BertWordPieceTokenizer(
+                vocab_file,
+                add_special_tokens,
+                unk_token,
+                sep_token,
+                cls_token,
+                handle_chinese_chars=tokenize_chinese_chars,
+                lowercase=do_lower_case,
+            ),
             unk_token=unk_token,
             sep_token=sep_token,
             pad_token=pad_token,
@@ -565,33 +571,3 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
             mask_token=mask_token,
             **kwargs,
         )
-
-        self._tokenizer = tk.Tokenizer(tk.models.WordPiece.from_files(vocab_file, unk_token=unk_token))
-        self._update_special_tokens()
-        self._tokenizer.with_pre_tokenizer(
-            tk.pre_tokenizers.BertPreTokenizer.new(
-                do_basic_tokenize=do_basic_tokenize,
-                do_lower_case=do_lower_case,
-                tokenize_chinese_chars=tokenize_chinese_chars,
-                never_split=never_split if never_split is not None else [],
-            )
-        )
-        self._tokenizer.with_decoder(tk.decoders.WordPiece.new())
-
-        if add_special_tokens:
-            self._tokenizer.with_post_processor(
-                tk.processors.BertProcessing.new(
-                    (sep_token, self._tokenizer.token_to_id(sep_token)),
-                    (cls_token, self._tokenizer.token_to_id(cls_token)),
-                )
-            )
-        if max_length is not None:
-            self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
-        self._tokenizer.with_padding(
-            max_length=max_length if pad_to_max_length else None,
-            direction=self.padding_side,
-            pad_id=self.pad_token_id,
-            pad_type_id=self.pad_token_type_id,
-            pad_token=self.pad_token,
-        )
-        self._decoder = tk.decoders.WordPiece.new()

src/transformers/tokenization_ctrl.py

@@ -20,8 +20,9 @@ import logging
 import os
 
 import regex as re
+from tokenizers import BPETokenizer
 
-from .tokenization_utils import PreTrainedTokenizer
+from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
 
 
 logger = logging.getLogger(__name__)
@@ -32,8 +33,8 @@ VOCAB_FILES_NAMES = {
 }
 
 PRETRAINED_VOCAB_FILES_MAP = {
-    "vocab_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json"},
-    "merges_file": {"ctrl": "https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt"},
+    "vocab_file": {"ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-vocab.json"},
+    "merges_file": {"ctrl": "https://s3.amazonaws.com/models.huggingface.co/bert/ctrl-merges.txt"},
 }
 
 PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
@@ -148,14 +149,14 @@ class CTRLTokenizer(PreTrainedTokenizer):
         return len(self.encoder)
 
     def bpe(self, token):
-        if token in self.cache:
-            return self.cache[token]
         word = tuple(token)
         word = tuple(list(word[:-1]) + [word[-1] + "</w>"])
+        if token in self.cache:
+            return self.cache[token]
         pairs = get_pairs(word)
 
         if not pairs:
-            return token
+            return token + "</w>"
 
         while True:
             bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
@@ -186,8 +187,9 @@ class CTRLTokenizer(PreTrainedTokenizer):
                 break
             else:
                 pairs = get_pairs(word)
-        word = "@@ ".join(word)
-        word = word[:-4]
+        word = " ".join(word)
+        if word == "\n  </w>":
+            word = "\n</w>"
         self.cache[token] = word
         return word
 
@@ -212,7 +214,7 @@ class CTRLTokenizer(PreTrainedTokenizer):
 
     def convert_tokens_to_string(self, tokens):
         """ Converts a sequence of tokens (string) in a single string. """
-        out_string = " ".join(tokens).replace("@@ ", "").strip()
+        out_string = "".join(tokens).replace("</w>", " ").strip()
         return out_string
 
     def save_vocabulary(self, save_directory):
@@ -246,3 +248,13 @@ class CTRLTokenizer(PreTrainedTokenizer):
     #     tokens_generated_so_far = re.sub('(@@ )', '', string=filtered_tokens)
     #     tokens_generated_so_far = re.sub('(@@ ?$)', '', string=tokens_generated_so_far)
     #     return ''.join(tokens_generated_so_far)
+
+
+class CTRLTokenizerFast(PreTrainedTokenizerFast):
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    control_codes = CONTROL_CODES
+
+    def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
+        super().__init__(BPETokenizer(vocab_file, merges_file, unk_token), **kwargs)

src/transformers/tokenization_distilbert.py

@@ -17,7 +17,7 @@
 
 import logging
 
-from .tokenization_bert import BertTokenizer
+from .tokenization_bert import BertTokenizer, BertTokenizerFast
 
 
 logger = logging.getLogger(__name__)
@@ -68,3 +68,10 @@ class DistilBertTokenizer(BertTokenizer):
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
     pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
+
+
+class DistilBertTokenizerFast(BertTokenizerFast):
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION

src/transformers/tokenization_gpt2.py

@@ -22,6 +22,7 @@ from functools import lru_cache
 
 import regex as re
 import tokenizers as tk
+from tokenizers import ByteLevelBPETokenizer
 
 from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
 
@@ -268,19 +269,25 @@ class GPT2TokenizerFast(PreTrainedTokenizerFast):
         truncation_strategy="longest_first",
         **kwargs
     ):
-        super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
-
-        self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
-        self._update_special_tokens()
-        self._tokenizer.with_pre_tokenizer(tk.pre_tokenizers.ByteLevel.new(add_prefix_space=add_prefix_space))
-        self._tokenizer.with_decoder(tk.decoders.ByteLevel.new())
-        if max_length:
-            self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
-        self._tokenizer.with_padding(
-            max_length=max_length if pad_to_max_length else None,
-            direction=self.padding_side,
-            pad_id=self.pad_token_id if self.pad_token_id is not None else 0,
-            pad_type_id=self.pad_token_type_id,
-            pad_token=self.pad_token if self.pad_token is not None else "",
+        super().__init__(
+            ByteLevelBPETokenizer(vocab_file, merges_file, add_prefix_space),
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            **kwargs,
         )
-        self._decoder = tk.decoders.ByteLevel.new()
+
+        # self._tokenizer = tk.Tokenizer(tk.models.BPE.from_files(vocab_file, merges_file))
+        # self._update_special_tokens()
+        # self._tokenizer.with_pre_tokenizer(tk.pre_tokenizers.ByteLevel.new(add_prefix_space=add_prefix_space))
+        # self._tokenizer.with_decoder(tk.decoders.ByteLevel.new())
+        # if max_length:
+        #     self._tokenizer.with_truncation(max_length, stride=stride, strategy=truncation_strategy)
+        # self._tokenizer.with_padding(
+        #     max_length=max_length if pad_to_max_length else None,
+        #     direction=self.padding_side,
+        #     pad_id=self.pad_token_id if self.pad_token_id is not None else 0,
+        #     pad_type_id=self.pad_token_type_id,
+        #     pad_token=self.pad_token if self.pad_token is not None else "",
+        # )
+        # self._decoder = tk.decoders.ByteLevel.new()

src/transformers/tokenization_openai.py

@@ -20,8 +20,10 @@ import logging
 import os
 import re
 
+from tokenizers import BPETokenizer
+
 from .tokenization_bert import BasicTokenizer
-from .tokenization_utils import PreTrainedTokenizer
+from .tokenization_utils import PreTrainedTokenizer, PreTrainedTokenizerFast
 
 
 logger = logging.getLogger(__name__)
@@ -213,3 +215,12 @@ class OpenAIGPTTokenizer(PreTrainedTokenizer):
                 index += 1
 
         return vocab_file, merge_file
+
+
+class OpenAIGPTTokenizerFast(PreTrainedTokenizerFast):
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
+        super().__init__(BPETokenizer(vocab_file, merges_file, unk_token), **kwargs)

src/transformers/tokenization_roberta.py

@@ -17,7 +17,7 @@
 
 import logging
 
-from .tokenization_gpt2 import GPT2Tokenizer
+from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
 
 
 logger = logging.getLogger(__name__)
@@ -154,3 +154,30 @@ class RobertaTokenizer(GPT2Tokenizer):
         if token_ids_1 is None:
             return len(cls + token_ids_0 + sep) * [0]
         return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
+
+
+class RobertaTokenizerFast(GPT2TokenizerFast):
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(
+        self,
+        vocab_file,
+        merges_file,
+        errors="replace",
+        bos_token="<s>",
+        eos_token="</s>",
+        sep_token="</s>",
+        cls_token="<s>",
+        unk_token="<unk>",
+        pad_token="<pad>",
+        mask_token="<mask>",
+        **kwargs
+    ):
+        kwargs["pad_token"] = pad_token
+        kwargs["sep_token"] = sep_token
+        kwargs["cls_token"] = cls_token
+        kwargs["mask_token"] = mask_token
+
+        super().__init__(vocab_file, merges_file, unk_token, bos_token, eos_token, add_prefix_space=True)

src/transformers/tokenization_utils.py

@@ -21,6 +21,9 @@ import json
 import logging
 import os
 import re
+from contextlib import contextmanager
+
+from tokenizers.implementations import BaseTokenizer
 
 from .file_utils import cached_path, hf_bucket_url, is_remote_url, is_tf_available, is_torch_available
 
@@ -37,6 +40,56 @@ ADDED_TOKENS_FILE = "added_tokens.json"
 TOKENIZER_CONFIG_FILE = "tokenizer_config.json"
 
 
+@contextmanager
+def truncate_and_pad(
+    tokenizer: BaseTokenizer,
+    max_length: int,
+    stride: int,
+    strategy: str,
+    pad_to_max_length: bool,
+    padding_side: str,
+    pad_token_id: int,
+    pad_token_type_id: int,
+    pad_token: str,
+):
+    """
+    This contextmanager is in charge of defining the truncation and the padding strategies and then
+    restore the tokenizer settings afterwards.
+
+    :param tokenizer:
+    :param max_length:
+    :param stride:
+    :param strategy:
+    :param pad_to_max_length:
+    :param padding_side:
+    :param pad_token_id:
+    :param pad_token_type_id:
+    :param pad_token:
+    :return:
+    """
+
+    # Handle all the truncation and padding stuff
+    if max_length is not None:
+        tokenizer.enable_truncation(max_length, stride=stride, strategy=strategy)
+
+        if pad_to_max_length:
+            tokenizer.enable_padding(
+                max_length=max_length,
+                direction=padding_side,
+                pad_id=pad_token_id,
+                pad_type_id=pad_token_type_id,
+                pad_token=pad_token,
+            )
+
+    yield
+
+    if max_length is not None:
+        tokenizer.no_truncation()
+
+        if pad_to_max_length:
+            tokenizer.no_padding()
+
+
 class PreTrainedTokenizer(object):
     """ Base class for all tokenizers.
     Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.
@@ -832,6 +885,7 @@ class PreTrainedTokenizer(object):
         return_attention_mask=True,
         return_overflowing_tokens=False,
         return_special_tokens_mask=False,
+        return_offsets_mapping=False,
         **kwargs
     ):
         """
@@ -905,6 +959,9 @@ class PreTrainedTokenizer(object):
                     "Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers."
                 )
 
+        if return_offsets_mapping:
+            logger.warning("offset mapping is not available on Python tokenizers.")
+
         first_ids = get_input_ids(text)
         second_ids = get_input_ids(text_pair) if text_pair is not None else None
 
@@ -998,7 +1055,7 @@ class PreTrainedTokenizer(object):
             for key, value in batch_outputs.items():
 
                 padded_value = value
-                if key != "input_len" and self._pad_token is not None:
+                if key != "input_len":
                     # Padding handle
                     padded_value = [
                         v + [self.pad_token_id if key == "input_ids" else 1] * (max_seq_len - len(v))
@@ -1417,30 +1474,27 @@ class PreTrainedTokenizer(object):
 
 
 class PreTrainedTokenizerFast(PreTrainedTokenizer):
-    _tokenizer = None
-    _decoder = None
+    def __init__(self, tokenizer: BaseTokenizer, **kwargs):
+        if tokenizer is None:
+            raise ValueError("Provided tokenizer cannot be None")
+        self._tokenizer = tokenizer
 
-    def __init__(self, **kwargs):
         super().__init__(**kwargs)
 
     @property
     def tokenizer(self):
-        if self._tokenizer is None:
-            raise NotImplementedError
         return self._tokenizer
 
     @property
     def decoder(self):
-        if self._decoder is None:
-            raise NotImplementedError
-        return self._decoder
+        return self._tokenizer._tokenizer.decoder
 
     @property
     def vocab_size(self):
-        return self.tokenizer.get_vocab_size(with_added_tokens=False)
+        return self._tokenizer.get_vocab_size(with_added_tokens=False)
 
     def __len__(self):
-        return self.tokenizer.get_vocab_size(with_added_tokens=True)
+        return self._tokenizer.get_vocab_size(with_added_tokens=True)
 
     @PreTrainedTokenizer.bos_token.setter
     def bos_token(self, value):
@@ -1494,36 +1548,59 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
         return_attention_mask=True,
         return_overflowing_tokens=False,
         return_special_tokens_mask=False,
+        return_offsets_mapping=False,
+        pad_token_id: int = 0,
+        pad_to_length: int = -1,
     ):
+        if return_overflowing_tokens and encoding.overflowing is not None:
+            encodings = [encoding] + encoding.overflowing
+        else:
+            encodings = [encoding]
+
         encoding_dict = {
-            "input_ids": encoding.ids,
+            "input_ids": [e.ids for e in encodings],
         }
+
         if return_token_type_ids:
-            encoding_dict["token_type_ids"] = encoding.type_ids
+            encoding_dict["token_type_ids"] = [e.type_ids for e in encodings]
         if return_attention_mask:
-            encoding_dict["attention_mask"] = encoding.attention_mask
-        if return_overflowing_tokens:
-            overflowing = encoding.overflowing
-            encoding_dict["overflowing_tokens"] = overflowing.ids if overflowing is not None else []
+            encoding_dict["attention_mask"] = [e.attention_mask for e in encodings]
         if return_special_tokens_mask:
-            encoding_dict["special_tokens_mask"] = encoding.special_tokens_mask
+            encoding_dict["special_tokens_mask"] = [e.special_tokens_mask for e in encodings]
+        if return_offsets_mapping:
+            encoding_dict["offset_mapping"] = [e.offsets for e in encodings]
+
+        if pad_to_length > 0:
+            for i in range(len(encoding_dict["input_ids"])):
+                if len(encoding_dict["input_ids"][i]) < pad_to_length:
+                    padding = pad_to_length - len(encoding_dict["input_ids"][i])
+                    encoding_dict["input_ids"][i] += [pad_token_id] * padding
+
+                    if return_attention_mask:
+                        encoding_dict["attention_mask"][i] += [0] * padding
+
+                    if return_special_tokens_mask:
+                        encoding_dict["special_tokens_mask"][i] += [1] * padding
+
+                    if return_token_type_ids:
+                        encoding_dict["token_type_ids"][i] += [1] * padding
 
         # Prepare inputs as tensors if asked
         if return_tensors == "tf" and is_tf_available():
-            encoding_dict["input_ids"] = tf.constant([encoding_dict["input_ids"]])
+            encoding_dict["input_ids"] = tf.constant(encoding_dict["input_ids"])
             if "token_type_ids" in encoding_dict:
-                encoding_dict["token_type_ids"] = tf.constant([encoding_dict["token_type_ids"]])
+                encoding_dict["token_type_ids"] = tf.constant(encoding_dict["token_type_ids"])
 
             if "attention_mask" in encoding_dict:
-                encoding_dict["attention_mask"] = tf.constant([encoding_dict["attention_mask"]])
+                encoding_dict["attention_mask"] = tf.constant(encoding_dict["attention_mask"])
 
         elif return_tensors == "pt" and is_torch_available():
-            encoding_dict["input_ids"] = torch.tensor([encoding_dict["input_ids"]])
+            encoding_dict["input_ids"] = torch.tensor(encoding_dict["input_ids"])
             if "token_type_ids" in encoding_dict:
-                encoding_dict["token_type_ids"] = torch.tensor([encoding_dict["token_type_ids"]])
+                encoding_dict["token_type_ids"] = torch.tensor(encoding_dict["token_type_ids"])
 
             if "attention_mask" in encoding_dict:
-                encoding_dict["attention_mask"] = torch.tensor([encoding_dict["attention_mask"]])
+                encoding_dict["attention_mask"] = torch.tensor(encoding_dict["attention_mask"])
         elif return_tensors is not None:
             logger.warning(
                 "Unable to convert output to tensors format {}, PyTorch or TensorFlow is not available.".format(
@@ -1531,73 +1608,110 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
                 )
             )
 
-        return encoding_dict
-
-    def encode_plus(
-        self,
-        text,
-        text_pair=None,
-        return_tensors=None,
-        return_token_type_ids=True,
-        return_attention_mask=True,
-        return_overflowing_tokens=False,
-        return_special_tokens_mask=False,
-        **kwargs
-    ):
-        encoding = self.tokenizer.encode(text, text_pair)
-        return self._convert_encoding(
-            encoding,
-            return_tensors=return_tensors,
-            return_token_type_ids=return_token_type_ids,
-            return_attention_mask=return_attention_mask,
-            return_overflowing_tokens=return_overflowing_tokens,
-            return_special_tokens_mask=return_special_tokens_mask,
-        )
-
-    def tokenize(self, text):
-        return self.tokenizer.encode(text).tokens
+        return {k: v if len(v) > 1 else v[0] for k, v in encoding_dict.items()}
 
     def _convert_token_to_id_with_added_voc(self, token):
-        id = self.tokenizer.token_to_id(token)
+        id = self._tokenizer.token_to_id(token)
         if id is None:
             return self.unk_token_id
         return id
 
     def _convert_id_to_token(self, index):
-        return self.tokenizer.id_to_token(int(index))
+        return self._tokenizer.id_to_token(int(index))
 
     def convert_tokens_to_string(self, tokens):
-        return self.decoder.decode(tokens)
+        return self._tokenizer.decode(tokens)
 
     def add_tokens(self, new_tokens):
-        self.tokenizer.add_tokens(new_tokens)
+        self._tokenizer.add_tokens(new_tokens)
 
     def add_special_tokens(self, special_tokens_dict):
         added = super().add_special_tokens(special_tokens_dict)
         self._update_special_tokens()
         return added
 
-    def encode_batch(
+    def encode_plus(
         self,
-        texts,
+        text,
+        text_pair=None,
+        add_special_tokens=True,
+        max_length=None,
+        stride=0,
+        truncation_strategy="longest_first",
+        pad_to_max_length=False,
         return_tensors=None,
         return_token_type_ids=True,
         return_attention_mask=True,
         return_overflowing_tokens=False,
         return_special_tokens_mask=False,
+        return_offsets_mapping=False,
+        **kwargs
     ):
-        return [
+        # Ensure we have text defined as [str]
+        if text is not None and not isinstance(text, list):
+            text = [text]
+
+        if text_pair is not None and not isinstance(text_pair, list):
+            text_pair = [text_pair]
+
+            # Ensure we have all the pairs
+            if len(text_pair) != len(text):
+                raise ValueError(
+                    "Number of text_pair ({}) doesn't match number of text ({})".format(len(text_pair), len(text))
+                )
+
+        # Set the truncation and padding strategy and restore the initial configuration
+        with truncate_and_pad(
+            self._tokenizer,
+            max_length,
+            stride,
+            truncation_strategy,
+            pad_to_max_length,
+            self.padding_side,
+            self.pad_token_id,
+            self.pad_token_type_id,
+            self._pad_token,
+        ):
+
+            if text_pair is None:
+                tokens = self._tokenizer.encode_batch(text)
+            else:
+                tokens = self._tokenizer.encode_batch(list(zip(text, text_pair)))
+
+        # Convert encoding to dict
+        max_length = max(map(lambda e: len(e.ids), tokens))
+        tokens = [
             self._convert_encoding(
                 encoding,
-                return_tensors=return_tensors,
-                return_token_type_ids=return_token_type_ids,
-                return_attention_mask=return_attention_mask,
-                return_overflowing_tokens=return_overflowing_tokens,
-                return_special_tokens_mask=return_special_tokens_mask,
+                return_tensors,
+                return_token_type_ids,
+                return_attention_mask,
+                return_overflowing_tokens,
+                return_special_tokens_mask,
+                return_offsets_mapping,
+                self.pad_token_id,
+                max_length,
             )
-            for encoding in self.tokenizer.encode_batch(texts)
+            for encoding in tokens
         ]
 
+        # Unwrap from the list if only on sample
+        if len(tokens) == 1:
+            return tokens[0]
+
+        # Sanitize the output to have dict[list] from list[dict]
+        sanitized = {}
+        for key in tokens[0].keys():
+            stack = [item[key] for item in tokens]
+
+            if return_tensors == "tf":
+                stack = tf.concat(stack, axis=0)
+            elif return_tensors == "pt":
+                stack = torch.cat(stack, dim=0)
+
+            sanitized[key] = stack
+        return sanitized
+
     def decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=True):
         text = self.tokenizer.decode(token_ids, skip_special_tokens)
 
@@ -1607,8 +1721,5 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
         else:
             return text
 
-    def decode_batch(self, ids_batch, skip_special_tokens=False, clear_up_tokenization_spaces=True):
-        return [
-            self.clean_up_tokenization(text) if clear_up_tokenization_spaces else text
-            for text in self.tokenizer.decode_batch(ids_batch, skip_special_tokens)
-        ]
+    def save_vocabulary(self, save_directory):
+        self._tokenizer.save(save_directory)

src/transformers/tokenization_xlm_roberta.py

@@ -176,7 +176,10 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
     def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):
         """
         Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
-        RoBERTa does not make use of token type ids, therefore a list of zeros is returned.
+        A RoBERTa sequence pair mask has the following format:
+        0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
+        | first sequence    | second sequence
+
         if token_ids_1 is None, only returns the first portion of the mask (0's).
         """
         sep = [self.sep_token_id]
@@ -184,7 +187,7 @@ class XLMRobertaTokenizer(PreTrainedTokenizer):
 
         if token_ids_1 is None:
             return len(cls + token_ids_0 + sep) * [0]
-        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
+        return len(cls + token_ids_0 + sep + sep) * [0] + len(token_ids_1 + sep) * [1]
 
     @property
     def vocab_size(self):

tests/test_modeling_auto.py

@@ -28,8 +28,6 @@ if is_torch_available():
         BertConfig,
         AutoModel,
         BertModel,
-        AutoModelForPreTraining,
-        BertForPreTraining,
         AutoModelWithLMHead,
         BertForMaskedLM,
         RobertaForMaskedLM,
@@ -58,21 +56,6 @@ class AutoModelTest(unittest.TestCase):
             for value in loading_info.values():
                 self.assertEqual(len(value), 0)
 
-    @slow
-    def test_model_for_pretraining_from_pretrained(self):
-        logging.basicConfig(level=logging.INFO)
-        for model_name in list(BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
-            config = AutoConfig.from_pretrained(model_name)
-            self.assertIsNotNone(config)
-            self.assertIsInstance(config, BertConfig)
-
-            model = AutoModelForPreTraining.from_pretrained(model_name)
-            model, loading_info = AutoModelForPreTraining.from_pretrained(model_name, output_loading_info=True)
-            self.assertIsNotNone(model)
-            self.assertIsInstance(model, BertForPreTraining)
-            for value in loading_info.values():
-                self.assertEqual(len(value), 0)
-
     @slow
     def test_lmhead_model_from_pretrained(self):
         logging.basicConfig(level=logging.INFO)

tests/test_modeling_tf_auto.py

@@ -28,8 +28,6 @@ if is_tf_available():
         BertConfig,
         TFAutoModel,
         TFBertModel,
-        TFAutoModelForPreTraining,
-        TFBertForPreTraining,
         TFAutoModelWithLMHead,
         TFBertForMaskedLM,
         TFRobertaForMaskedLM,
@@ -59,23 +57,6 @@ class TFAutoModelTest(unittest.TestCase):
             self.assertIsNotNone(model)
             self.assertIsInstance(model, TFBertModel)
 
-    @slow
-    def test_model_for_pretraining_from_pretrained(self):
-        import h5py
-
-        self.assertTrue(h5py.version.hdf5_version.startswith("1.10"))
-
-        logging.basicConfig(level=logging.INFO)
-        # for model_name in list(TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
-        for model_name in ["bert-base-uncased"]:
-            config = AutoConfig.from_pretrained(model_name)
-            self.assertIsNotNone(config)
-            self.assertIsInstance(config, BertConfig)
-
-            model = TFAutoModelForPreTraining.from_pretrained(model_name)
-            self.assertIsNotNone(model)
-            self.assertIsInstance(model, TFBertForPreTraining)
-
     @slow
     def test_lmhead_model_from_pretrained(self):
         logging.basicConfig(level=logging.INFO)

tests/test_modeling_tf_common.py

@@ -112,15 +112,8 @@ class TFModelTesterMixin:
             tfo = tf_model(inputs_dict, training=False)
             tf_hidden_states = tfo[0].numpy()
             pt_hidden_states = pto[0].numpy()
-
-            tf_nans = np.copy(np.isnan(tf_hidden_states))
-            pt_nans = np.copy(np.isnan(pt_hidden_states))
-
-            pt_hidden_states[tf_nans] = 0
-            tf_hidden_states[tf_nans] = 0
-            pt_hidden_states[pt_nans] = 0
-            tf_hidden_states[pt_nans] = 0
-
+            tf_hidden_states[np.isnan(tf_hidden_states)] = 0
+            pt_hidden_states[np.isnan(pt_hidden_states)] = 0
             max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
             # Debug info (remove when fixed)
             if max_diff >= 2e-2:
@@ -151,14 +144,8 @@ class TFModelTesterMixin:
             tfo = tf_model(inputs_dict)
             tfo = tfo[0].numpy()
             pto = pto[0].numpy()
-            tf_nans = np.copy(np.isnan(tfo))
-            pt_nans = np.copy(np.isnan(pto))
-
-            pto[tf_nans] = 0
-            tfo[tf_nans] = 0
-            pto[pt_nans] = 0
-            tfo[pt_nans] = 0
-
+            tfo[np.isnan(tfo)] = 0
+            pto[np.isnan(pto)] = 0
             max_diff = np.amax(np.abs(tfo - pto))
             self.assertLessEqual(max_diff, 2e-2)
 

tests/test_modeling_tf_distilbert.py

@@ -219,5 +219,5 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
     # @slow
     # def test_model_from_pretrained(self):
     #     for model_name in list(DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
-    #         model = DistilBertModesss.from_pretrained(model_name, cache_dir=CACHE_DIR)
+    #         model = DistilBertModel.from_pretrained(model_name, cache_dir=CACHE_DIR)
     #         self.assertIsNotNone(model)

tests/test_tokenization_fast.py

@@ -0,0 +1,100 @@
+import unittest
+
+from transformers import (
+    BertTokenizer,
+    BertTokenizerFast,
+    CTRLTokenizer,
+    DistilBertTokenizer,
+    GPT2Tokenizer,
+    GPT2TokenizerFast,
+    OpenAIGPTTokenizer,
+    RobertaTokenizer,
+)
+from transformers.tokenization_ctrl import CTRLTokenizerFast
+from transformers.tokenization_distilbert import DistilBertTokenizerFast
+from transformers.tokenization_openai import OpenAIGPTTokenizerFast
+from transformers.tokenization_roberta import RobertaTokenizerFast
+
+
+class FastTokenizerMatchingTest(unittest.TestCase):
+    def setUp(self) -> None:
+        with open("fixtures/sample_text.txt") as f_data:
+            self._data = f_data.read()
+
+    def _tokenize_inputs_and_check_matching(self, tokenizer_p, tokenizer_r):
+        # Ensure basic input match
+        input_p = tokenizer_p.encode_plus(self._data)
+        input_r = tokenizer_r.encode_plus(self._data)
+
+        self.assertSequenceEqual(input_p["input_ids"], input_r["input_ids"])
+        self.assertSequenceEqual(input_p["token_type_ids"], input_r["token_type_ids"])
+        self.assertSequenceEqual(input_p["attention_mask"], input_r["attention_mask"])
+
+        input_pairs_p = tokenizer_p.encode_plus(self._data, self._data)
+        input_pairs_r = tokenizer_r.encode_plus(self._data, self._data)
+
+        self.assertSequenceEqual(input_pairs_p["input_ids"], input_pairs_r["input_ids"])
+        self.assertSequenceEqual(input_pairs_p["token_type_ids"], input_pairs_r["token_type_ids"])
+        self.assertSequenceEqual(input_pairs_p["attention_mask"], input_pairs_r["attention_mask"])
+
+        # Ensure truncation match
+        input_p = tokenizer_p.encode_plus(self._data, max_length=512, pad_to_max_length=True)
+        input_r = tokenizer_r.encode_plus(self._data, max_length=512, pad_to_max_length=True)
+
+        self.assertSequenceEqual(input_p["input_ids"], input_r["input_ids"])
+        self.assertSequenceEqual(input_p["token_type_ids"], input_r["token_type_ids"])
+        self.assertSequenceEqual(input_p["attention_mask"], input_r["attention_mask"])
+
+        # Ensure truncation with stride match
+        # input_p = tokenizer_p.encode_plus(self._data, max_length=512, stride=3, return_overflowing_tokens=True)
+        # input_r = tokenizer_r.encode_plus(self._data, max_length=512, stride=3, return_overflowing_tokens=True)
+        #
+        # self.assertSequenceEqual(input_p['input_ids'], input_r['input_ids'])
+        # self.assertSequenceEqual(input_p['token_type_ids'], input_r['token_type_ids'])
+        # self.assertSequenceEqual(input_p['attention_mask'], input_r['attention_mask'])
+
+    def test_bert(self):
+        for tokenizer_name in BertTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = BertTokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = BertTokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+    def test_ctrl(self):
+        for tokenizer_name in CTRLTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = CTRLTokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = CTRLTokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+    def test_distilbert(self):
+        for tokenizer_name in DistilBertTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = DistilBertTokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = DistilBertTokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+    def test_gpt2(self):
+        for tokenizer_name in GPT2Tokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = GPT2Tokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = GPT2TokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+    def test_roberta(self):
+        for tokenizer_name in RobertaTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = RobertaTokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = RobertaTokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+    def test_openai(self):
+        for tokenizer_name in OpenAIGPTTokenizer.pretrained_vocab_files_map["vocab_file"].keys():
+            tokenizer_p = OpenAIGPTTokenizer.from_pretrained(tokenizer_name)
+            tokenizer_r = OpenAIGPTTokenizerFast.from_pretrained(tokenizer_name)
+
+            self._tokenize_inputs_and_check_matching(tokenizer_p, tokenizer_r)
+
+
+if __name__ == "__main__":
+    unittest.main()