Release: v2.4.0

* [Umberto] model shortcuts

cc @loretoparisi @simonefrancia

see #2485

* Ensure that tokenizers will be correctly configured

README.md

@@ -160,8 +160,9 @@ At some point in the future, you'll be able to seamlessly move from pre-training
 12. **[T5](https://github.com/google-research/text-to-text-transfer-transformer)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
 13. **[XLM-RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
 14. **[MMBT](https://github.com/facebookresearch/mmbt/)** (from Facebook), released together with the paper a [Supervised Multimodal Bitransformers for Classifying Images and Text](https://arxiv.org/pdf/1909.02950.pdf) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
-15. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
-16. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
+15. **[FlauBERT](https://github.com/getalp/Flaubert)** (from CNRS) released with the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
+16. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
+17. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
 
 These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).
 
@@ -520,8 +521,9 @@ You can create `Pipeline` objects for the following down-stream tasks:
  - `feature-extraction`: Generates a tensor representation for the input sequence
  - `ner`: Generates named entity mapping for each word in the input sequence.
  - `sentiment-analysis`: Gives the polarity (positive / negative) of the whole input sequence.
- - `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question
- in the context.
+ - `text-classification`: Initialize a `TextClassificationPipeline` directly, or see `sentiment-analysis` for an example.
+ - `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question in the context.
+ - `fill-mask`: Takes an input sequence containing a masked token (e.g. `<mask>`) and return list of most probable filled sequences, with their probabilities.
 
 ```python
 from transformers import pipeline

docs/source/conf.py

@@ -26,7 +26,7 @@ author = u'huggingface'
 # The short X.Y version
 version = u''
 # The full version, including alpha/beta/rc tags
-release = u'2.3.0'
+release = u'2.4.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/source/index.rst

@@ -51,6 +51,7 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
 10. `CamemBERT <https://huggingface.co/transformers/model_doc/camembert.html>`_ (from FAIR, Inria, Sorbonne Université) released together with the paper `CamemBERT: a Tasty French Language Model <https://arxiv.org/abs/1911.03894>`_ by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suarez, Yoann Dupont, Laurent Romary, Eric Villemonte de la Clergerie, Djame Seddah, and Benoît Sagot.
 11. `ALBERT <https://github.com/google-research/ALBERT>`_ (from Google Research), released together with the paper a `ALBERT: A Lite BERT for Self-supervised Learning of Language Representations <https://arxiv.org/abs/1909.11942>`_ by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
 12. `XLM-RoBERTa <https://github.com/pytorch/fairseq/tree/master/examples/xlmr>`_ (from Facebook AI), released together with the paper `Unsupervised Cross-lingual Representation Learning at Scale <https://arxiv.org/abs/1911.02116>`_ by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
+13. `FlauBERT <https://github.com/getalp/Flaubert>`_ (from CNRS) released with the paper `FlauBERT: Unsupervised Language Model Pre-training for French <https://arxiv.org/abs/1912.05372>`_ by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
 
 .. toctree::
     :maxdepth: 2
@@ -97,4 +98,5 @@ The library currently contains PyTorch and Tensorflow implementations, pre-train
     model_doc/ctrl
     model_doc/camembert
     model_doc/albert
-    model_doc/xlmroberta
+    model_doc/xlmroberta
+    model_doc/flaubert

docs/source/model_doc/camembert.rst

@@ -69,3 +69,31 @@ CamembertForTokenClassification
 
 .. autoclass:: transformers.CamembertForTokenClassification
     :members:
+
+
+TFCamembertModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFCamembertModel
+    :members:
+
+
+TFCamembertForMaskedLM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFCamembertForMaskedLM
+    :members:
+
+
+TFCamembertForSequenceClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFCamembertForSequenceClassification
+    :members:
+
+
+TFCamembertForTokenClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFCamembertForTokenClassification
+    :members:

docs/source/model_doc/flaubert.rst

@@ -0,0 +1,72 @@
+FlauBERT
+----------------------------------------------------
+
+The FlauBERT model was proposed in the paper
+`FlauBERT: Unsupervised Language Model Pre-training for French <https://arxiv.org/abs/1912.05372>`__ by Hang Le et al.
+It's a transformer pre-trained using a masked language modeling (MLM) objective (BERT-like).
+
+The abstract from the paper is the following:
+
+*Language models have become a key step to achieve state-of-the art results in many different Natural Language
+Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient
+way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
+contextualization at the sentence level. This has been widely demonstrated for English using contextualized
+representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et
+al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large
+and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre
+for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
+classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most
+of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified
+evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared
+to the research community for further reproducible experiments in French NLP.*
+
+
+FlaubertConfig
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertConfig
+    :members:
+
+
+FlaubertTokenizer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertTokenizer
+    :members:
+
+
+FlaubertModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertModel
+    :members:
+
+
+FlaubertWithLMHeadModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertWithLMHeadModel
+    :members:
+
+
+FlaubertForSequenceClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertForSequenceClassification
+    :members:
+
+
+FlaubertForQuestionAnsweringSimple
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertForQuestionAnsweringSimple
+    :members:
+
+
+FlaubertForQuestionAnswering
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.FlaubertForQuestionAnswering
+    :members:
+
+

docs/source/model_doc/xlmroberta.rst

@@ -73,3 +73,30 @@ XLMRobertaForTokenClassification
 .. autoclass:: transformers.XLMRobertaForTokenClassification
     :members:
 
+
+TFXLMRobertaModel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFXLMRobertaModel
+    :members:
+
+
+TFXLMRobertaForMaskedLM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFXLMRobertaForMaskedLM
+    :members:
+
+
+TFXLMRobertaForSequenceClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFXLMRobertaForSequenceClassification
+    :members:
+
+
+TFXLMRobertaForTokenClassification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: transformers.TFXLMRobertaForTokenClassification
+    :members:

docs/source/pretrained_models.rst

@@ -251,6 +251,22 @@ For a list that includes community-uploaded models, refer to `https://huggingfac
 |                   | ``xlm-roberta-large``                                      | | ~355M parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads,                                       |
 |                   |                                                            | | Trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages                                                          |
 +-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+| FlauBERT          | ``flaubert-small-cased``                                   | | 6-layer, 512-hidden, 8-heads, 54M parameters                                                                                        |
+|                   |                                                            | | FlauBERT small architecture                                                                                                         |
+|                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``flaubert-base-uncased``                                  | | 12-layer, 768-hidden, 12-heads, 137M parameters                                                                                     |
+|                   |                                                            | | FlauBERT base architecture with uncased vocabulary                                                                                  |
+|                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``flaubert-base-cased``                                    | | 12-layer, 768-hidden, 12-heads, 138M parameters                                                                                     |
+|                   |                                                            | | FlauBERT base architecture with cased vocabulary                                                                                    |
+|                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
+|                   +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
+|                   | ``flaubert-large-cased``                                   | | 24-layer, 1024-hidden, 16-heads, 373M parameters                                                                                    |
+|                   |                                                            | | FlauBERT large architecture                                                                                                         |
+|                   |                                                            | (see `details <https://github.com/getalp/Flaubert>`__)                                                                                |
++-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
 
 
 .. <https://huggingface.co/transformers/examples.html>`__

examples/README.md

@@ -404,12 +404,12 @@ exact_match = 81.22
 #### Distributed training
 
 
-Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.0:
+Here is an example using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD1.1:
 
 ```bash
-python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
+python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
     --model_type bert \
-    --model_name_or_path bert-base-cased \
+    --model_name_or_path bert-large-uncased-whole-word-masking \
     --do_train \
     --do_eval \
     --do_lower_case \
@@ -419,9 +419,9 @@ python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
     --num_train_epochs 2 \
     --max_seq_length 384 \
     --doc_stride 128 \
-    --output_dir ../models/wwm_uncased_finetuned_squad/ \
-    --per_gpu_train_batch_size 24 \
-    --gradient_accumulation_steps 12
+    --output_dir ./examples/models/wwm_uncased_finetuned_squad/ \
+    --per_gpu_eval_batch_size=3   \
+    --per_gpu_train_batch_size=3   \
 ```
 
 Training with the previously defined hyper-parameters yields the following results:

examples/run_glue.py

@@ -41,6 +41,9 @@ from transformers import (
     DistilBertConfig,
     DistilBertForSequenceClassification,
     DistilBertTokenizer,
+    FlaubertConfig,
+    FlaubertForSequenceClassification,
+    FlaubertTokenizer,
     RobertaConfig,
     RobertaForSequenceClassification,
     RobertaTokenizer,
@@ -80,6 +83,7 @@ ALL_MODELS = sum(
             DistilBertConfig,
             AlbertConfig,
             XLMRobertaConfig,
+            FlaubertConfig,
         )
     ),
     (),
@@ -93,6 +97,7 @@ MODEL_CLASSES = {
     "distilbert": (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
     "albert": (AlbertConfig, AlbertForSequenceClassification, AlbertTokenizer),
     "xlmroberta": (XLMRobertaConfig, XLMRobertaForSequenceClassification, XLMRobertaTokenizer),
+    "flaubert": (FlaubertConfig, FlaubertForSequenceClassification, FlaubertTokenizer),
 }
 
 
@@ -480,7 +485,7 @@ def main():
     parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
     parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
     parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step.",
+        "--evaluate_during_training", action="store_true", help="Run evaluation during training at each logging step.",
     )
     parser.add_argument(
         "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model.",

examples/run_tf_glue.py

@@ -1,22 +1,3 @@
-# coding=utf-8
-# Copyright 2018 The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
-
-
-import argparse
-import logging
 import os
 
 import tensorflow as tf
@@ -24,309 +5,101 @@ import tensorflow_datasets
 
 from transformers import (
     BertConfig,
+    BertForSequenceClassification,
     BertTokenizer,
-    DistilBertConfig,
-    DistilBertTokenizer,
-    RobertaConfig,
-    RobertaTokenizer,
     TFBertForSequenceClassification,
-    TFDistilBertForSequenceClassification,
-    TFRobertaForSequenceClassification,
-    TFXLMForSequenceClassification,
-    TFXLNetForSequenceClassification,
-    XLMConfig,
-    XLMTokenizer,
-    XLNetConfig,
-    XLNetTokenizer,
-)
-from transformers import glue_convert_examples_to_features as convert_examples_to_features
-from transformers import glue_output_modes as output_modes
-from transformers import glue_processors as processors
-
-
-logger = logging.getLogger(__name__)
-
-ALL_MODELS = sum(
-    (
-        tuple(conf.pretrained_config_archive_map.keys())
-        for conf in (BertConfig, XLNetConfig, XLMConfig, RobertaConfig, DistilBertConfig)
-    ),
-    (),
+    glue_convert_examples_to_features,
+    glue_processors,
 )
 
-MODEL_CLASSES = {
-    "bert": (BertConfig, TFBertForSequenceClassification, BertTokenizer),
-    "xlnet": (XLNetConfig, TFXLNetForSequenceClassification, XLNetTokenizer),
-    "xlm": (XLMConfig, TFXLMForSequenceClassification, XLMTokenizer),
-    "roberta": (RobertaConfig, TFRobertaForSequenceClassification, RobertaTokenizer),
-    "distilbert": (DistilBertConfig, TFDistilBertForSequenceClassification, DistilBertTokenizer),
-}
+
+# script parameters
+BATCH_SIZE = 32
+EVAL_BATCH_SIZE = BATCH_SIZE * 2
+USE_XLA = False
+USE_AMP = False
+EPOCHS = 3
+
+TASK = "mrpc"
+
+if TASK == "sst-2":
+    TFDS_TASK = "sst2"
+elif TASK == "sts-b":
+    TFDS_TASK = "stsb"
+else:
+    TFDS_TASK = TASK
+
+num_labels = len(glue_processors[TASK]().get_labels())
+print(num_labels)
+
+tf.config.optimizer.set_jit(USE_XLA)
+tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})
+
+# Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
+config = BertConfig.from_pretrained("bert-base-cased", num_labels=num_labels)
+tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
+model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
+
+# Load dataset via TensorFlow Datasets
+data, info = tensorflow_datasets.load(f"glue/{TFDS_TASK}", with_info=True)
+train_examples = info.splits["train"].num_examples
+
+# MNLI expects either validation_matched or validation_mismatched
+valid_examples = info.splits["validation"].num_examples
+
+# Prepare dataset for GLUE as a tf.data.Dataset instance
+train_dataset = glue_convert_examples_to_features(data["train"], tokenizer, 128, TASK)
+
+# MNLI expects either validation_matched or validation_mismatched
+valid_dataset = glue_convert_examples_to_features(data["validation"], tokenizer, 128, TASK)
+train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
+valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)
+
+# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
+opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
+if USE_AMP:
+    # loss scaling is currently required when using mixed precision
+    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
 
 
-def load_and_cache_examples(args, data, task, tokenizer, split):
-    if task == "mnli" and split == "validation":
-        split = "validation_matched"
+if num_labels == 1:
+    loss = tf.keras.losses.MeanSquaredError()
+else:
+    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
 
-    features_output_dir = os.path.join(args.output_dir, "features")
-    cached_features_file = os.path.join(
-        features_output_dir,
-        "cached_{}_{}_{}_{}.tfrecord".format(
-            split, list(filter(None, args.model_name_or_path.split("/"))).pop(), str(args.max_seq_length), str(task)
-        ),
-    )
+metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
+model.compile(optimizer=opt, loss=loss, metrics=[metric])
 
-    if not os.path.exists(cached_features_file) or args.overwrite_cache:
-        logger.info("Converting examples to features")
-        dataset = convert_examples_to_features(data[split], tokenizer, args.max_seq_length, task)
+# Train and evaluate using tf.keras.Model.fit()
+train_steps = train_examples // BATCH_SIZE
+valid_steps = valid_examples // EVAL_BATCH_SIZE
 
-        if not os.path.exists(features_output_dir):
-            os.makedirs(features_output_dir)
+history = model.fit(
+    train_dataset,
+    epochs=EPOCHS,
+    steps_per_epoch=train_steps,
+    validation_data=valid_dataset,
+    validation_steps=valid_steps,
+)
 
-        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
-            for feature in dataset:
-                example, label = feature
-                feature_key_value_pair = {
-                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
-                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
-                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
-                    "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
-                }
-                features = tf.train.Features(feature=feature_key_value_pair)
-                example = tf.train.Example(features=features)
+# Save TF2 model
+os.makedirs("./save/", exist_ok=True)
+model.save_pretrained("./save/")
 
-                tfwriter.write(example.SerializeToString())
+if TASK == "mrpc":
+    # Load the TensorFlow model in PyTorch for inspection
+    # This is to demo the interoperability between the two frameworks, you don't have to
+    # do this in real life (you can run the inference on the TF model).
+    pytorch_model = BertForSequenceClassification.from_pretrained("./save/", from_tf=True)
 
-        logger.info("Features saved to cache")
+    # Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
+    sentence_0 = "This research was consistent with his findings."
+    sentence_1 = "His findings were compatible with this research."
+    sentence_2 = "His findings were not compatible with this research."
+    inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors="pt")
+    inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors="pt")
 
-    features = {
-        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "label": tf.io.FixedLenFeature([], tf.int64),
-    }
-
-    def select_data_from_record(record):
-        record = tf.io.parse_single_example(record, features)
-        x = {
-            "input_ids": record["input_ids"],
-            "attention_mask": record["attention_mask"],
-            "token_type_ids": record["token_type_ids"],
-        }
-        y = record["label"]
-        return (x, y)
-
-    dataset = tf.data.TFRecordDataset(cached_features_file)
-    dataset = dataset.map(select_data_from_record)
-
-    logger.info("Created dataset %s from TFRecord" % split)
-    return dataset
-
-
-def main():
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument(
-        "--model_type",
-        default=None,
-        type=str,
-        required=True,
-        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--model_name_or_path",
-        default=None,
-        type=str,
-        required=True,
-        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
-    )
-    parser.add_argument(
-        "--task_name",
-        default=None,
-        type=str,
-        required=True,
-        help="The name of the task to train selected in the list: " + ", ".join(processors.keys()),
-    )
-    parser.add_argument(
-        "--output_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The output directory where the model predictions and checkpoints will be written.",
-    )
-    parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
-    )
-
-    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
-    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
-    parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
-    )
-
-    parser.add_argument("--train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
-    parser.add_argument(
-        "--valid_batch_size", default=8, type=int, help="Batch size per GPU/CPU for validation during training."
-    )
-    parser.add_argument(
-        "--test_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation after training."
-    )
-    parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
-    )
-
-    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
-    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
-    parser.add_argument(
-        "--max_seq_length",
-        default=128,
-        type=int,
-        help="The maximum total input sequence length after tokenization. Sequences longer "
-        "than this will be truncated, sequences shorter will be padded.",
-    )
-    parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
-    )
-
-    parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
-    )
-    parser.add_argument(
-        "--tokenizer_name",
-        default="",
-        type=str,
-        help="Pretrained tokenizer name or path if not the same as model_name",
-    )
-    parser.add_argument(
-        "--cache_dir",
-        default="",
-        type=str,
-        help="Where do you want to store the pre-trained models downloaded from s3",
-    )
-    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
-    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
-    parser.add_argument(
-        "--force_download",
-        action="store_true",
-        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
-    )
-    args = parser.parse_args()
-
-    # Setup logging
-    logging.basicConfig(
-        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
-        datefmt="%m/%d/%Y %H:%M:%S",
-        level=logging.INFO,
-    )
-
-    if os.path.exists(args.output_dir) and args.do_train:
-        if not args.overwrite_output_dir and bool(
-            [file for file in os.listdir(args.output_dir) if "features" not in file]
-        ):
-            raise ValueError(
-                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
-                    args.output_dir
-                )
-            )
-
-    args.model_type = args.model_type.lower()
-    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
-
-    TASK = args.task_name.lower()
-
-    if TASK not in processors:
-        raise ValueError("Task not found: %s" % (TASK))
-
-    if TASK == "sst-2":
-        TFDS_TASK = "sst2"
-    elif TASK == "sts-b":
-        TFDS_TASK = "stsb"
-    else:
-        TFDS_TASK = TASK
-
-    num_labels = len(processors[TASK]().get_labels())
-    print(num_labels)
-
-    tf.config.optimizer.set_jit(args.xla)
-    tf.config.optimizer.set_experimental_options({"auto_mixed_precision": args.amp})
-
-    # Load tokenizer and model from pretrained model/vocabulary. Specify the number of labels to classify (2+: classification, 1: regression)
-    config = config_class.from_pretrained(
-        args.config_name if args.config_name else args.model_name_or_path,
-        num_labels=num_labels,
-        finetuning_task=args.task_name,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    tokenizer = tokenizer_class.from_pretrained(
-        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
-        do_lower_case=args.do_lower_case,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    model = model_class.from_pretrained(
-        args.model_name_or_path,
-        config=config,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-        force_download=args.force_download,
-    )
-
-    # Load dataset via TensorFlow Datasets
-    data, info = tensorflow_datasets.load("glue/%s" % TFDS_TASK, with_info=True)
-
-    # Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
-    opt = tf.keras.optimizers.Adam(learning_rate=args.learning_rate, epsilon=args.adam_epsilon)
-
-    if args.amp:
-        # loss scaling is currently required when using mixed precision
-        opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
-
-    if num_labels == 1:
-        loss = tf.keras.losses.MeanSquaredError()
-    else:
-        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
-
-    metric = tf.keras.metrics.SparseCategoricalAccuracy("accuracy")
-    model.compile(optimizer=opt, loss=loss, metrics=[metric])
-
-    class save_model(tf.keras.callbacks.Callback):
-        def on_epoch_end(self, epoch, logs=None):
-            print("Saving model at epoch {}".format(epoch))
-            output_dir = os.path.join(args.output_dir, "checkpoint-epoch-{}".format(epoch))
-            if not os.path.exists(output_dir):
-                os.makedirs(output_dir)
-            self.model.save_pretrained(output_dir)
-
-    if args.do_train:
-        train_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="train")
-        train_dataset = train_dataset.batch(args.train_batch_size).repeat(args.num_train_epochs)
-        train_examples = info.splits["train"].num_examples / args.train_batch_size
-
-        validation_identifier = "validation_mismatched" if TASK == "mnli" else "validation"
-        valid_dataset = load_and_cache_examples(
-            args, data=data, task=TASK, tokenizer=tokenizer, split=validation_identifier
-        )
-        valid_dataset = valid_dataset.batch(args.valid_batch_size)
-        valid_examples = info.splits[validation_identifier].num_examples / args.valid_batch_size
-
-        history = model.fit(
-            train_dataset,
-            steps_per_epoch=train_examples,
-            epochs=args.num_train_epochs,
-            validation_data=valid_dataset if args.evaluate_during_training else None,
-            validation_steps=valid_examples if args.evaluate_during_training else None,
-            callbacks=[save_model()],
-        )
-
-    if args.do_eval:
-        test_dataset = load_and_cache_examples(args, data=data, task=TASK, tokenizer=tokenizer, split="test")
-        test_dataset = test_dataset.batch(args.test_batch_size)
-        test_examples = info.splits["test"].num_examples / args.test_batch_size
-
-        results = model.evaluate(test_dataset, steps=test_examples)
-
-
-if __name__ == "__main__":
-    main()
+    pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
+    pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
+    print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
+    print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")

examples/run_tf_squad.py

@@ -1,699 +0,0 @@
-# Copyright 2018 The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-""" Finetuning the library models for sequence classification on GLUE (Bert, XLM, XLNet, RoBERTa)."""
-from __future__ import absolute_import, division, print_function
-
-import argparse
-import datetime
-import glob
-import logging
-import math
-import os
-import pickle
-
-import regex as re
-import tensorflow as tf
-from fastprogress import master_bar, progress_bar
-from seqeval import metrics
-from tqdm import tqdm, trange
-
-from transformers import (
-    TF2_WEIGHTS_NAME,
-    BertConfig,
-    BertTokenizer,
-    DistilBertConfig,
-    DistilBertTokenizer,
-    GradientAccumulator,
-    SquadV1Processor,
-    SquadV2Processor,
-    TFBertForQuestionAnswering,
-    TFDistilBertForQuestionAnswering,
-    TFXLMForQuestionAnsweringSimple,
-    TFXLNetForQuestionAnsweringSimple,
-    XLMConfig,
-    XLMTokenizer,
-    XLNetConfig,
-    XLNetTokenizer,
-    create_optimizer,
-    squad_convert_examples_to_features,
-)
-from transformers.data.metrics.squad_metrics import compute_predictions_logits, squad_evaluate
-from transformers.data.processors.squad import SquadResult
-
-
-logger = logging.getLogger(__name__)
-
-ALL_MODELS = sum(
-    (
-        tuple(conf.pretrained_config_archive_map.keys())
-        for conf in (BertConfig, XLNetConfig, XLMConfig, DistilBertConfig)
-    ),
-    (),
-)
-
-MODEL_CLASSES = {
-    "bert": (BertConfig, TFBertForQuestionAnswering, BertTokenizer),
-    "xlnet": (XLNetConfig, TFXLNetForQuestionAnsweringSimple, XLNetTokenizer),
-    "xlm": (XLMConfig, TFXLMForQuestionAnsweringSimple, XLMTokenizer),
-    "distilbert": (DistilBertConfig, TFDistilBertForQuestionAnswering, DistilBertTokenizer),
-}
-
-
-def train(args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size):
-    if args.max_steps > 0:
-        num_train_steps = args.max_steps * args.gradient_accumulation_steps
-        args.num_train_epochs = 1
-    else:
-        num_train_steps = (
-            math.ceil(num_train_examples / train_batch_size)
-            // args.gradient_accumulation_steps
-            * args.num_train_epochs
-        )
-
-    writer = tf.summary.create_file_writer("/tmp/mylogs")
-
-    with strategy.scope():
-        loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(
-            reduction=tf.keras.losses.Reduction.NONE, from_logits=True
-        )
-        optimizer = create_optimizer(args.learning_rate, num_train_steps, args.warmup_steps)
-
-        if args.xla:
-            tf.config.optimizer.set_jit(True)
-
-        if args.amp:
-            optimizer = tf.keras.mixed_precision.experimental.LossScaleOptimizer(optimizer, "dynamic")
-
-        loss_metric = tf.keras.metrics.Mean(name="loss", dtype=tf.float32)
-        gradient_accumulator = GradientAccumulator()
-
-    logging.info("***** Running training *****")
-    logging.info("  Num examples = %d", num_train_examples)
-    logging.info("  Num Epochs = %d", args.num_train_epochs)
-    logging.info("  Instantaneous batch size per device = %d", args.per_device_train_batch_size)
-    logging.info(
-        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
-        train_batch_size * args.gradient_accumulation_steps,
-    )
-    logging.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
-    logging.info("  Total training steps = %d", num_train_steps)
-
-    model.summary()
-
-    @tf.function
-    def apply_gradients():
-        grads_and_vars = []
-
-        for gradient, variable in zip(gradient_accumulator.gradients, model.trainable_variables):
-            if gradient is not None:
-                scaled_gradient = gradient / (args.n_device * args.gradient_accumulation_steps)
-                grads_and_vars.append((scaled_gradient, variable))
-            else:
-                grads_and_vars.append((gradient, variable))
-
-        optimizer.apply_gradients(grads_and_vars, args.max_grad_norm)
-        gradient_accumulator.reset()
-
-    @tf.function
-    def train_step(train_features, train_labels):
-        def step_fn(train_features, train_labels):
-            with tf.GradientTape() as tape:
-                start_logits, end_logits = model(train_features)
-                start_logits = tf.multiply(
-                    start_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32)
-                )
-                end_logits = tf.multiply(end_logits, tf.dtypes.cast((train_features["attention_mask"]), tf.float32))
-                start_loss = loss_fct(train_labels["start_position"], start_logits)
-                end_loss = loss_fct(train_labels["end_position"], end_logits)
-                total_loss = (start_loss + end_loss) / 2
-
-                loss = tf.reduce_sum(total_loss) * (1.0 / train_batch_size)
-                grads = tape.gradient(loss, model.trainable_variables)
-
-                gradient_accumulator(grads)
-
-            return total_loss
-
-        per_example_losses = strategy.experimental_run_v2(step_fn, args=(train_features, train_labels))
-        mean_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_example_losses, axis=0)
-
-        return mean_loss
-
-    current_time = datetime.datetime.now()
-    train_iterator = master_bar(range(args.num_train_epochs))
-    global_step = 0
-    logging_loss = 0.0
-
-    for epoch in train_iterator:
-        epoch_iterator = progress_bar(
-            train_dataset,
-            total=num_train_steps / args.num_train_epochs,
-            parent=train_iterator,
-            display=args.n_device > 1,
-        )
-        step = 1
-
-        with strategy.scope():
-            for train_features, train_labels in tqdm(
-                epoch_iterator,
-                desc="Training, epoch {}".format(epoch),
-                total=int(num_train_steps / args.num_train_epochs),
-            ):
-                loss = train_step(train_features, train_labels)
-
-                if step % args.gradient_accumulation_steps == 0:
-                    strategy.experimental_run_v2(apply_gradients)
-
-                    loss_metric(loss)
-
-                    global_step += 1
-
-                    if args.logging_steps > 0 and global_step % args.logging_steps == 0:
-                        # Log metrics
-                        if args.n_device == 1 and args.evaluate_during_training:
-                            # Only evaluate when single GPU otherwise metrics may not average well
-                            results = evaluate(args, strategy, model, tokenizer, global_step)
-
-                            with writer.as_default():
-                                tf.summary.scalar("exact", results[0], global_step)
-                                tf.summary.scalar("f1", results[1], global_step)
-
-                        lr = optimizer.learning_rate
-                        learning_rate = lr(step)
-
-                        with writer.as_default():
-                            tf.summary.scalar("lr", learning_rate, global_step)
-                            tf.summary.scalar(
-                                "loss", (loss_metric.result() - logging_loss) / args.logging_steps, global_step
-                            )
-
-                        logging_loss = loss_metric.result()
-
-                    with writer.as_default():
-                        tf.summary.scalar("loss", loss_metric.result(), step=step)
-
-                    if args.save_steps > 0 and global_step % args.save_steps == 0:
-                        # Save model checkpoint
-                        output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
-
-                        if not os.path.exists(output_dir):
-                            os.makedirs(output_dir)
-
-                        model.save_pretrained(output_dir)
-                        logging.info("Saving model checkpoint to %s", output_dir)
-
-                train_iterator.child.comment = f"loss : {loss_metric.result()}"
-                step += 1
-
-        train_iterator.write(f"loss epoch {epoch + 1}: {loss_metric.result()}")
-
-        loss_metric.reset_states()
-
-    logging.info("  Training took time = {}".format(datetime.datetime.now() - current_time))
-
-
-def evaluate(args, strategy, model, tokenizer, prefix):
-    eval_batch_size = args.per_device_eval_batch_size * args.n_device
-    eval_dataset, size, examples, features = load_and_cache_examples(
-        args, tokenizer, evaluate=True, output_examples=True
-    )
-    eval_dataset = eval_dataset.batch(args.per_device_eval_batch_size)
-    eval_dataset = strategy.experimental_distribute_dataset(eval_dataset)
-    num_eval_steps = math.ceil(size / eval_batch_size)
-    master = master_bar(range(1))
-    eval_iterator = progress_bar(eval_dataset, total=num_eval_steps, parent=master, display=args.n_device > 1)
-
-    logging.info("***** Running evaluation *****")
-    logging.info("  Num examples = %d", size)
-    logging.info("  Batch size = %d", eval_batch_size)
-
-    all_results = []
-    for index, (eval_features, eval_labels) in tqdm(
-        enumerate(eval_iterator), total=size / args.per_device_eval_batch_size
-    ):
-        with strategy.scope():
-            start_logits, end_logits = model(eval_features)
-
-            for sample_index in range(args.per_device_eval_batch_size):
-                if index * args.per_device_eval_batch_size + sample_index < len(features):
-                    result = SquadResult(
-                        features[index * args.per_device_eval_batch_size + sample_index].unique_id,
-                        start_logits[sample_index],
-                        end_logits[sample_index],
-                    )
-                    all_results.append(result)
-
-    output_prediction_file = os.path.join(args.output_dir, "predictions_{}.json".format(prefix))
-    output_nbest_file = os.path.join(args.output_dir, "nbest_predictions_{}.json".format(prefix))
-
-    if args.version_2_with_negative:
-        output_null_log_odds_file = os.path.join(args.output_dir, "null_odds_{}.json".format(prefix))
-    else:
-        output_null_log_odds_file = None
-
-    predictions = compute_predictions_logits(
-        examples,
-        features,
-        all_results,
-        args.n_best_size,
-        args.max_answer_length,
-        args.do_lower_case,
-        output_prediction_file,
-        output_nbest_file,
-        output_null_log_odds_file,
-        args.verbose_logging,
-        args.version_2_with_negative,
-        args.null_score_diff_threshold,
-        tokenizer,
-    )
-
-    # Compute the F1 and exact scores.
-    results = squad_evaluate(examples, predictions)
-    return results
-
-
-def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
-    features_output_dir = os.path.join(args.output_dir, "features")
-    cached_features_file = os.path.join(
-        features_output_dir,
-        "cached_{}_{}_{}.tfrecord".format(
-            "dev" if evaluate else "train",
-            list(filter(None, args.model_name_or_path.split("/"))).pop(),
-            str(args.max_seq_length),
-        ),
-    )
-
-    if not os.path.exists(cached_features_file) or args.overwrite_cache:
-        if args.version_2_with_negative:
-            processor = SquadV2Processor()
-        else:
-            processor = SquadV1Processor()
-
-        if args.data_dir:
-            directory = args.directory
-            examples = processor.get_dev_examples(directory) if evaluate else processor.get_train_examples(directory)
-        else:
-            try:
-                import tensorflow_datasets as tfds
-            except ImportError:
-                raise ImportError("If not data_dir is specified, tensorflow_datasets needs to be installed.")
-
-            if args.version_2_with_negative:
-                logger.warning("tensorflow_datasets does not handle version 2 of SQuAD.")
-
-            examples = processor.get_examples_from_dataset(tfds.load("squad"), evaluate=evaluate)
-            examples = examples
-
-        logger.info("Converting examples to features")
-        features, dataset = squad_convert_examples_to_features(
-            examples,
-            tokenizer,
-            args.max_seq_length,
-            args.doc_stride,
-            args.max_query_length,
-            is_training=not evaluate,
-            return_dataset="tf",
-        )
-
-        if not os.path.exists(features_output_dir):
-            os.makedirs(features_output_dir)
-
-        with tf.compat.v1.python_io.TFRecordWriter(cached_features_file) as tfwriter:
-            for feature in tqdm(dataset, desc="Building tfrecord dataset", total=len(features)):
-                example, result = feature
-                feature_key_value_pair = {
-                    "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["input_ids"])),
-                    "attention_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=example["attention_mask"])),
-                    "token_type_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=example["token_type_ids"])),
-                    "start_position": tf.train.Feature(
-                        int64_list=tf.train.Int64List(value=[result["start_position"]])
-                    ),
-                    "end_position": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["end_position"]])),
-                    "cls_index": tf.train.Feature(int64_list=tf.train.Int64List(value=[result["cls_index"]])),
-                    "p_mask": tf.train.Feature(int64_list=tf.train.Int64List(value=result["p_mask"])),
-                }
-                feature_skeleton = tf.train.Features(feature=feature_key_value_pair)
-                example = tf.train.Example(features=feature_skeleton)
-
-                tfwriter.write(example.SerializeToString())
-
-            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "wb") as handle:
-                pickle.dump(examples, handle, protocol=pickle.HIGHEST_PROTOCOL)
-
-            with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "wb") as handle:
-                pickle.dump(features, handle, protocol=pickle.HIGHEST_PROTOCOL)
-
-        logger.info("Features saved to cache")
-
-    feature_skeleton = {
-        "input_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "attention_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "token_type_ids": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-        "start_position": tf.io.FixedLenFeature([], tf.int64),
-        "end_position": tf.io.FixedLenFeature([], tf.int64),
-        "cls_index": tf.io.FixedLenFeature([], tf.int64),
-        "p_mask": tf.io.FixedLenFeature([args.max_seq_length], tf.int64),
-    }
-
-    def select_data_from_record(record):
-        record = tf.io.parse_single_example(record, feature_skeleton)
-        x = {
-            "input_ids": record["input_ids"],
-            "attention_mask": record["attention_mask"],
-            "token_type_ids": record["token_type_ids"],
-        }
-        y = {
-            "start_position": record["start_position"],
-            "end_position": record["end_position"],
-            "cls_index": record["cls_index"],
-            "p_mask": record["p_mask"],
-        }
-        return x, y
-
-    dataset = tf.data.TFRecordDataset(cached_features_file)
-    dataset = dataset.map(select_data_from_record)
-
-    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_examples")), "rb") as handle:
-        examples = pickle.load(handle)
-
-    with open("{}.pickle".format(cached_features_file.replace(".tfrecord", "_features")), "rb") as handle:
-        features = pickle.load(handle)
-
-    logger.info("Created dataset %s from TFRecord" % "dev" if evaluate else "train")
-
-    if output_examples:
-        return dataset, len(list(dataset.__iter__())), examples, features
-    return dataset, len(list(dataset.__iter__()))
-
-
-def main():
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument(
-        "--model_type",
-        default=None,
-        type=str,
-        required=True,
-        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--model_name_or_path",
-        default=None,
-        type=str,
-        required=True,
-        help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(ALL_MODELS),
-    )
-    parser.add_argument(
-        "--output_dir",
-        default=None,
-        type=str,
-        required=True,
-        help="The output directory where the model predictions and checkpoints will be written.",
-    )
-    parser.add_argument(
-        "--data_dir",
-        default=None,
-        type=str,
-        required=False,
-        help="The input data directory containing the .json files. If no data dir is specified, uses tensorflow_datasets to load the data."
-        + ", ".join(MODEL_CLASSES.keys()),
-    )
-    parser.add_argument(
-        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
-    )
-
-    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
-    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
-    parser.add_argument(
-        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
-    )
-
-    parser.add_argument(
-        "--per_device_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training."
-    )
-    parser.add_argument(
-        "--per_device_eval_batch_size",
-        default=8,
-        type=int,
-        help="Batch size per GPU/CPU for validation during training.",
-    )
-    parser.add_argument(
-        "--per_device_test_batch_size",
-        default=8,
-        type=int,
-        help="Batch size per GPU/CPU for evaluation after training.",
-    )
-    parser.add_argument(
-        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
-    )
-
-    parser.add_argument("--num_train_epochs", default=3, type=int, help="Total number of training epochs to perform.")
-    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
-    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
-    parser.add_argument(
-        "--max_seq_length",
-        default=128,
-        type=int,
-        help="The maximum total input sequence length after tokenization. Sequences longer "
-        "than this will be truncated, sequences shorter will be padded.",
-    )
-    parser.add_argument(
-        "--doc_stride",
-        default=128,
-        type=int,
-        help="When splitting up a long document into chunks, how much stride to take between chunks.",
-    )
-    parser.add_argument(
-        "--max_query_length",
-        default=64,
-        type=int,
-        help="The maximum number of tokens for the question. Questions longer than this will "
-        "be truncated to this length.",
-    )
-    parser.add_argument(
-        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
-    )
-
-    parser.add_argument(
-        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
-    )
-    parser.add_argument(
-        "--tokenizer_name",
-        default="",
-        type=str,
-        help="Pretrained tokenizer name or path if not the same as model_name",
-    )
-    parser.add_argument(
-        "--cache_dir",
-        default="",
-        type=str,
-        help="Where do you want to store the pre-trained models downloaded from s3",
-    )
-    parser.add_argument("--xla", action="store_true", help="Whether to use XLA (Accelerated Linear Algebra).")
-    parser.add_argument("--amp", action="store_true", help="Whether to use AMP (Automatic Mixed Precision).")
-    parser.add_argument(
-        "--force_download",
-        action="store_true",
-        help="Whether to force download the weights from S3 (useful if the file is corrupted).",
-    )
-
-    parser.add_argument(
-        "--tpu",
-        default=None,
-        help="The Cloud TPU to use for training. This should be either the name "
-        "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
-        "url.",
-    )
-    parser.add_argument("--num_tpu_cores", default="8", help="Total number of TPU cores to use.")
-    parser.add_argument(
-        "--gpus",
-        default="0",
-        help="Comma separated list of gpus devices. If only one, switch to single gpu strategy, if None takes all the gpus available.",
-    )
-    parser.add_argument("--no_cuda", action="store_true", help="Whether not to use CUDA when available")
-    parser.add_argument(
-        "--version_2_with_negative",
-        action="store_true",
-        help="If true, the SQuAD examples contain some that do not have an answer.",
-    )
-    parser.add_argument(
-        "--max_steps",
-        default=-1,
-        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
-    )
-    parser.add_argument(
-        "--gradient_accumulation_steps",
-        default=1,
-        help="Number of updates steps to accumulate before performing a backward/update pass.",
-    )
-    parser.add_argument(
-        "--warmup_steps", default=0, help="Linear warmup over warmup_steps.",
-    )
-    parser.add_argument("--max_grad_norm", default=1.0, help="Max gradient norm.")
-    parser.add_argument("--logging_steps", default=500, type=int, help="Log every X updates.")
-    parser.add_argument("--save_steps", default=500, type=int, help="Save checkpoint every X updates.")
-    parser.add_argument(
-        "--eval_all_checkpoints",
-        action="store_true",
-        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
-    )
-    parser.add_argument(
-        "--n_best_size",
-        default=20,
-        type=int,
-        help="The total number of n-best predictions to generate in the nbest_predictions.json output file.",
-    )
-    parser.add_argument(
-        "--max_answer_length",
-        default=30,
-        type=int,
-        help="The maximum length of an answer that can be generated. This is needed because the start "
-        "and end predictions are not conditioned on one another.",
-    )
-    parser.add_argument(
-        "--verbose_logging",
-        action="store_true",
-        help="If true, all of the warnings related to data processing will be printed. "
-        "A number of warnings are expected for a normal SQuAD evaluation.",
-    )
-    parser.add_argument(
-        "--null_score_diff_threshold",
-        type=float,
-        default=0.0,
-        help="If null_score - best_non_null is greater than the threshold predict null.",
-    )
-    args = parser.parse_args()
-
-    # Setup logging
-    logging.basicConfig(
-        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
-        datefmt="%m/%d/%Y %H:%M:%S",
-        level=logging.INFO,
-    )
-
-    if os.path.exists(args.output_dir) and args.do_train:
-        if not args.overwrite_output_dir and bool(
-            [file for file in os.listdir(args.output_dir) if "features" not in file]
-        ):
-            raise ValueError(
-                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
-                    args.output_dir
-                )
-            )
-
-        if args.amp:
-            tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})
-
-    if args.tpu:
-        resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=args.tpu)
-        tf.config.experimental_connect_to_cluster(resolver)
-        tf.tpu.experimental.initialize_tpu_system(resolver)
-        strategy = tf.distribute.experimental.TPUStrategy(resolver)
-        args.n_device = args.num_tpu_cores
-    elif len(args.gpus.split(",")) > 1:
-        args.n_device = len([f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
-        strategy = tf.distribute.MirroredStrategy(devices=[f"/gpu:{gpu}" for gpu in args.gpus.split(",")])
-    elif args.no_cuda:
-        args.n_device = 1
-        strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
-    else:
-        args.n_device = len(args.gpus.split(","))
-        strategy = tf.distribute.OneDeviceStrategy(device="/gpu:" + args.gpus.split(",")[0])
-
-    logging.warning(
-        "n_device: %s, distributed training: %s, 16-bits training: %s",
-        args.n_device,
-        bool(args.n_device > 1),
-        args.amp,
-    )
-
-    args.model_type = args.model_type.lower()
-    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
-
-    config = config_class.from_pretrained(
-        args.config_name if args.config_name else args.model_name_or_path,
-        cache_dir=args.cache_dir if args.cache_dir else None,
-    )
-
-    logging.info("Training/evaluation parameters %s", args)
-
-    if args.do_train:
-        tokenizer = tokenizer_class.from_pretrained(
-            args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
-            do_lower_case=args.do_lower_case,
-            cache_dir=args.cache_dir if args.cache_dir else None,
-        )
-
-        with strategy.scope():
-            model = model_class.from_pretrained(
-                args.model_name_or_path,
-                from_pt=bool(".bin" in args.model_name_or_path),
-                config=config,
-                cache_dir=args.cache_dir if args.cache_dir else None,
-            )
-            model.layers[-1].activation = tf.keras.activations.softmax
-
-        train_batch_size = args.per_device_train_batch_size * args.n_device
-        train_dataset, num_train_examples = load_and_cache_examples(args, tokenizer, evaluate=False)
-
-        train_dataset = train_dataset.batch(train_batch_size)
-        train_dataset = train_dataset.prefetch(buffer_size=train_batch_size)
-        train_dataset = strategy.experimental_distribute_dataset(train_dataset)
-
-        train(
-            args, strategy, train_dataset, tokenizer, model, num_train_examples, train_batch_size,
-        )
-
-        if not os.path.exists(args.output_dir):
-            os.makedirs(args.output_dir)
-
-        logging.info("Saving model to %s", args.output_dir)
-
-        model.save_pretrained(args.output_dir)
-        tokenizer.save_pretrained(args.output_dir)
-
-    if args.do_eval:
-        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
-        checkpoints = []
-        results = []
-
-        if args.eval_all_checkpoints:
-            checkpoints = list(
-                os.path.dirname(c)
-                for c in sorted(
-                    glob.glob(args.output_dir + "/**/" + TF2_WEIGHTS_NAME, recursive=True),
-                    key=lambda f: int("".join(filter(str.isdigit, f)) or -1),
-                )
-            )
-
-        logging.info("Evaluate the following checkpoints: %s", checkpoints)
-
-        if len(checkpoints) == 0:
-            checkpoints.append(args.output_dir)
-
-        for checkpoint in checkpoints:
-            global_step = checkpoint.split("-")[-1] if re.match(".*checkpoint-[0-9]", checkpoint) else "final"
-
-            with strategy.scope():
-                model = model_class.from_pretrained(checkpoint)
-                results = evaluate(args, strategy, model, tokenizer, prefix=global_step)
-
-            result = dict((k + ("_{}".format(global_step) if global_step else ""), v) for k, v in results.items())
-            results.update(result)
-
-        logger.info("Results: {}".format(results))
-
-
-if __name__ == "__main__":
-    main()

setup.py

@@ -73,7 +73,7 @@ extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "sciki
 
 setup(
     name="transformers",
-    version="2.3.0",
+    version="2.4.0",
     author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",
     author_email="thomas@huggingface.co",
     description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",

src/transformers/__init__.py

@@ -2,7 +2,7 @@
 # There's no way to ignore "F401 '...' imported but unused" warnings in this
 # module, but to preserve other warnings. So, don't check this module at all.
 
-__version__ = "2.3.0"
+__version__ = "2.4.0"
 
 # Work around to update TensorFlow's absl.logging threshold which alters the
 # default Python logging output behavior when present.
@@ -25,6 +25,7 @@ from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
 from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
 from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
 from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
+from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
 from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
 from .configuration_mmbt import MMBTConfig
 from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
@@ -92,6 +93,7 @@ from .modeling_tf_pytorch_utils import (
 from .pipelines import (
     CsvPipelineDataFormat,
     FeatureExtractionPipeline,
+    FillMaskPipeline,
     JsonPipelineDataFormat,
     NerPipeline,
     PipedPipelineDataFormat,
@@ -108,6 +110,7 @@ from .tokenization_bert_japanese import BertJapaneseTokenizer, CharacterTokenize
 from .tokenization_camembert import CamembertTokenizer
 from .tokenization_ctrl import CTRLTokenizer
 from .tokenization_distilbert import DistilBertTokenizer
+from .tokenization_flaubert import FlaubertTokenizer
 from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
 from .tokenization_openai import OpenAIGPTTokenizer
 from .tokenization_roberta import RobertaTokenizer
@@ -209,6 +212,13 @@ if is_torch_available():
         RobertaForQuestionAnswering,
         ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
     )
+    from .modeling_camembert import (
+        CamembertForMaskedLM,
+        CamembertModel,
+        CamembertForSequenceClassification,
+        CamembertForTokenClassification,
+        CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+    )
     from .modeling_distilbert import (
         DistilBertPreTrainedModel,
         DistilBertForMaskedLM,
@@ -249,9 +259,19 @@ if is_torch_available():
         XLMRobertaForMultipleChoice,
         XLMRobertaForSequenceClassification,
         XLMRobertaForTokenClassification,
+        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
     )
     from .modeling_mmbt import ModalEmbeddings, MMBTModel, MMBTForClassification
 
+    from .modeling_flaubert import (
+        FlaubertModel,
+        FlaubertWithLMHeadModel,
+        FlaubertForSequenceClassification,
+        FlaubertForQuestionAnswering,
+        FlaubertForQuestionAnsweringSimple,
+        FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+    )
+
     # Optimization
     from .optimization import (
         AdamW,
@@ -338,6 +358,14 @@ if is_tf_available():
         TF_XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
     )
 
+    from .modeling_tf_xlm_roberta import (
+        TFXLMRobertaForMaskedLM,
+        TFXLMRobertaModel,
+        TFXLMRobertaForSequenceClassification,
+        TFXLMRobertaForTokenClassification,
+        TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+    )
+
     from .modeling_tf_roberta import (
         TFRobertaPreTrainedModel,
         TFRobertaMainLayer,
@@ -348,6 +376,14 @@ if is_tf_available():
         TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
     )
 
+    from .modeling_tf_camembert import (
+        TFCamembertModel,
+        TFCamembertForMaskedLM,
+        TFCamembertForSequenceClassification,
+        TFCamembertForTokenClassification,
+        TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+    )
+
     from .modeling_tf_distilbert import (
         TFDistilBertPreTrainedModel,
         TFDistilBertMainLayer,
@@ -374,7 +410,12 @@ if is_tf_available():
         TF_ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     )
 
-    from .modeling_tf_t5 import TFT5PreTrainedModel, TFT5Model, TFT5WithLMHeadModel, TF_T5_PRETRAINED_MODEL_ARCHIVE_MAP
+    from .modeling_tf_t5 import (
+        TFT5PreTrainedModel,
+        TFT5Model,
+        TFT5WithLMHeadModel,
+        TF_T5_PRETRAINED_MODEL_ARCHIVE_MAP,
+    )
 
     # Optimization
     from .optimization_tf import WarmUp, create_optimizer, AdamWeightDecay, GradientAccumulator

src/transformers/configuration_albert.py

@@ -76,6 +76,8 @@ class AlbertConfig(PretrainedConfig):
                 The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
             layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
                 The epsilon used by the layer normalization layers.
+            classifier_dropout_prob (:obj:`float`, optional, defaults to 0.1):
+                The dropout ratio for attached classifiers.
 
         Example::
 
@@ -121,6 +123,7 @@ class AlbertConfig(PretrainedConfig):
         type_vocab_size=2,
         initializer_range=0.02,
         layer_norm_eps=1e-12,
+        classifier_dropout_prob=0.1,
         **kwargs
     ):
         super().__init__(**kwargs)
@@ -140,3 +143,4 @@ class AlbertConfig(PretrainedConfig):
         self.type_vocab_size = type_vocab_size
         self.initializer_range = initializer_range
         self.layer_norm_eps = layer_norm_eps
+        self.classifier_dropout_prob = classifier_dropout_prob

src/transformers/configuration_auto.py

@@ -23,6 +23,7 @@ from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
 from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
 from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
 from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
+from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
 from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
 from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
 from .configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
@@ -53,6 +54,7 @@ ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict(
         CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
         T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
         XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
+        FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     ]
     for key, value, in pretrained_map.items()
 )
@@ -66,6 +68,7 @@ CONFIG_MAPPING = OrderedDict(
         ("camembert", CamembertConfig,),
         ("xlm-roberta", XLMRobertaConfig,),
         ("roberta", RobertaConfig,),
+        ("flaubert", FlaubertConfig,),
         ("bert", BertConfig,),
         ("openai-gpt", OpenAIGPTConfig,),
         ("gpt2", GPT2Config,),
@@ -77,7 +80,7 @@ CONFIG_MAPPING = OrderedDict(
 )
 
 
-class AutoConfig(object):
+class AutoConfig:
     r"""
         :class:`~transformers.AutoConfig` is a generic configuration class
         that will be instantiated as one of the configuration classes of the library
@@ -126,6 +129,7 @@ class AutoConfig(object):
             - contains `xlnet`: :class:`~transformers.XLNetConfig` (XLNet model)
             - contains `xlm`: :class:`~transformers.XLMConfig` (XLM model)
             - contains `ctrl` : :class:`~transformers.CTRLConfig` (CTRL model)
+            - contains `flaubert` : :class:`~transformers.FlaubertConfig` (Flaubert model)
 
 
         Args:

src/transformers/configuration_camembert.py

@@ -25,6 +25,8 @@ logger = logging.getLogger(__name__)
 
 CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
     "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-config.json",
+    "umberto-commoncrawl-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-commoncrawl-cased-v1/config.json",
+    "umberto-wikipedia-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-wikipedia-uncased-v1/config.json",
 }
 
 

src/transformers/configuration_flaubert.py

@@ -0,0 +1,152 @@
+# coding=utf-8
+# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Flaubert configuration, based on XLM. """
+
+
+import logging
+
+from .configuration_xlm import XLMConfig
+
+
+logger = logging.getLogger(__name__)
+
+FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
+    "flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/config.json",
+    "flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/config.json",
+    "flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/config.json",
+    "flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/config.json",
+}
+
+
+class FlaubertConfig(XLMConfig):
+    """
+        Configuration class to store the configuration of a `FlaubertModel`.
+        This is the configuration class to store the configuration of a :class:`~transformers.XLMModel`.
+        It is used to instantiate an XLM model according to the specified arguments, defining the model
+        architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
+        the `xlm-mlm-en-2048 <https://huggingface.co/xlm-mlm-en-2048>`__ architecture.
+
+        Configuration objects inherit from  :class:`~transformers.PretrainedConfig` and can be used
+        to control the model outputs. Read the documentation from  :class:`~transformers.PretrainedConfig`
+        for more information.
+
+        Args:
+            pre_norm (:obj:`bool`, `optional`, defaults to :obj:`False`):
+                Whether to apply the layer normalization before or after the feed forward layer following the
+                attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)
+            layerdrop (:obj:`float`, `optional`, defaults to 0.0):
+                Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand
+                with Structured Dropout. ICLR 2020)
+            vocab_size (:obj:`int`, optional, defaults to 30145):
+                Vocabulary size of the XLM model. Defines the different tokens that
+                can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLMModel`.
+            emb_dim (:obj:`int`, optional, defaults to 2048):
+                Dimensionality of the encoder layers and the pooler layer.
+            n_layer (:obj:`int`, optional, defaults to 12):
+                Number of hidden layers in the Transformer encoder.
+            n_head (:obj:`int`, optional, defaults to 16):
+                Number of attention heads for each attention layer in the Transformer encoder.
+            dropout (:obj:`float`, optional, defaults to 0.1):
+                The dropout probability for all fully connected
+                layers in the embeddings, encoder, and pooler.
+            attention_dropout (:obj:`float`, optional, defaults to 0.1):
+                The dropout probability for the attention mechanism
+            gelu_activation (:obj:`boolean`, optional, defaults to :obj:`True`):
+                The non-linear activation function (function or string) in the
+                encoder and pooler. If set to `True`, "gelu" will be used instead of "relu".
+            sinusoidal_embeddings (:obj:`boolean`, optional, defaults to :obj:`False`):
+                Whether to use sinusoidal positional embeddings instead of absolute positional embeddings.
+            causal (:obj:`boolean`, optional, defaults to :obj:`False`):
+                Set this to `True` for the model to behave in a causal manner.
+                Causal models use a triangular attention mask in order to only attend to the left-side context instead
+                if a bidirectional context.
+            asm (:obj:`boolean`, optional, defaults to :obj:`False`):
+                Whether to use an adaptive log softmax projection layer instead of a linear layer for the prediction
+                layer.
+            n_langs (:obj:`int`, optional, defaults to 1):
+                The number of languages the model handles. Set to 1 for monolingual models.
+            use_lang_emb (:obj:`boolean`, optional, defaults to :obj:`True`)
+                Whether to use language embeddings. Some models use additional language embeddings, see
+                `the multilingual models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__
+                for information on how to use them.
+            max_position_embeddings (:obj:`int`, optional, defaults to 512):
+                The maximum sequence length that this model might
+                ever be used with. Typically set this to something large just in case
+                (e.g., 512 or 1024 or 2048).
+            embed_init_std (:obj:`float`, optional, defaults to 2048^-0.5):
+                The standard deviation of the truncated_normal_initializer for
+                initializing the embedding matrices.
+            init_std (:obj:`int`, optional, defaults to 50257):
+                The standard deviation of the truncated_normal_initializer for
+                initializing all weight matrices except the embedding matrices.
+            layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
+                The epsilon used by the layer normalization layers.
+            bos_index (:obj:`int`, optional, defaults to 0):
+                The index of the beginning of sentence token in the vocabulary.
+            eos_index (:obj:`int`, optional, defaults to 1):
+                The index of the end of sentence token in the vocabulary.
+            pad_index (:obj:`int`, optional, defaults to 2):
+                The index of the padding token in the vocabulary.
+            unk_index (:obj:`int`, optional, defaults to 3):
+                The index of the unknown token in the vocabulary.
+            mask_index (:obj:`int`, optional, defaults to 5):
+                The index of the masking token in the vocabulary.
+            is_encoder(:obj:`boolean`, optional, defaults to :obj:`True`):
+                Whether the initialized model should be a transformer encoder or decoder as seen in Vaswani et al.
+            summary_type (:obj:`string`, optional, defaults to "first"):
+                Argument used when doing sequence summary. Used in for the multiple choice head in
+                :class:`~transformers.XLMForSequenceClassification`.
+                Is one of the following options:
+                    - 'last' => take the last token hidden state (like XLNet)
+                    - 'first' => take the first token hidden state (like Bert)
+                    - 'mean' => take the mean of all tokens hidden states
+                    - 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
+                    - 'attn' => Not implemented now, use multi-head attention
+            summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
+                Argument used when doing sequence summary. Used in for the multiple choice head in
+                :class:`~transformers.XLMForSequenceClassification`.
+                Add a projection after the vector extraction
+            summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
+                Argument used when doing sequence summary. Used in for the multiple choice head in
+                :class:`~transformers.XLMForSequenceClassification`.
+                'tanh' => add a tanh activation to the output, Other => no activation.
+            summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
+                Argument used when doing sequence summary. Used in for the multiple choice head in
+                :class:`~transformers.XLMForSequenceClassification`.
+                If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
+            summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
+                Argument used when doing sequence summary. Used in for the multiple choice head in
+                :class:`~transformers.XLMForSequenceClassification`.
+                Add a dropout before the projection and activation
+            start_n_top (:obj:`int`, optional, defaults to 5):
+                Used in the SQuAD evaluation script for XLM and XLNet.
+            end_n_top (:obj:`int`, optional, defaults to 5):
+                Used in the SQuAD evaluation script for XLM and XLNet.
+            mask_token_id (:obj:`int`, optional, defaults to 0):
+                Model agnostic parameter to identify masked tokens when generating text in an MLM context.
+            lang_id (:obj:`int`, optional, defaults to 1):
+                The ID of the language used by the model. This parameter is used when generating
+                text in a given language.
+    """
+
+    pretrained_config_archive_map = FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
+    model_type = "flaubert"
+
+    def __init__(self, layerdrop=0.0, pre_norm=False, **kwargs):
+        """Constructs FlaubertConfig.
+        """
+        super().__init__(**kwargs)
+        self.layerdrop = layerdrop
+        self.pre_norm = pre_norm

src/transformers/configuration_utils.py

@@ -82,6 +82,7 @@ class PretrainedConfig(object):
         self.num_return_sequences = kwargs.pop("num_return_sequences", 1)
 
         # Fine-tuning task arguments
+        self.architectures = kwargs.pop("architectures", None)
         self.finetuning_task = kwargs.pop("finetuning_task", None)
         self.num_labels = kwargs.pop("num_labels", 2)
         self.id2label = kwargs.pop("id2label", {i: "LABEL_{}".format(i) for i in range(self.num_labels)})

src/transformers/convert_pytorch_checkpoint_to_tf2.py

@@ -22,6 +22,7 @@ import os
 from transformers import (
     ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP,
     DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP,
@@ -30,9 +31,11 @@ from transformers import (
     T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
     TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
     XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
     XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
     AlbertConfig,
     BertConfig,
+    CamembertConfig,
     CTRLConfig,
     DistilBertConfig,
     GPT2Config,
@@ -43,6 +46,7 @@ from transformers import (
     TFBertForPreTraining,
     TFBertForQuestionAnswering,
     TFBertForSequenceClassification,
+    TFCamembertForMaskedLM,
     TFCTRLLMHeadModel,
     TFDistilBertForMaskedLM,
     TFDistilBertForQuestionAnswering,
@@ -52,10 +56,12 @@ from transformers import (
     TFRobertaForSequenceClassification,
     TFT5WithLMHeadModel,
     TFTransfoXLLMHeadModel,
+    TFXLMRobertaForMaskedLM,
     TFXLMWithLMHeadModel,
     TFXLNetLMHeadModel,
     TransfoXLConfig,
     XLMConfig,
+    XLMRobertaConfig,
     XLNetConfig,
     cached_path,
     is_torch_available,
@@ -77,6 +83,8 @@ if is_torch_available():
         XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
         XLMWithLMHeadModel,
         XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLMRobertaForMaskedLM,
         TransfoXLLMHeadModel,
         TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
         OpenAIGPTLMHeadModel,
@@ -84,6 +92,9 @@ if is_torch_available():
         RobertaForMaskedLM,
         RobertaForSequenceClassification,
         ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        CamembertForMaskedLM,
+        CamembertForSequenceClassification,
+        CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
         DistilBertForMaskedLM,
         DistilBertForQuestionAnswering,
         DistilBertForSequenceClassification,
@@ -107,6 +118,8 @@ else:
         XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
         XLMWithLMHeadModel,
         XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLMRobertaForMaskedLM,
         TransfoXLLMHeadModel,
         TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
         OpenAIGPTLMHeadModel,
@@ -114,6 +127,9 @@ else:
         RobertaForMaskedLM,
         RobertaForSequenceClassification,
         ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        CamembertForMaskedLM,
+        CamembertForSequenceClassification,
+        CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
         DistilBertForMaskedLM,
         DistilBertForSequenceClassification,
         DistilBertForQuestionAnswering,
@@ -152,6 +168,11 @@ else:
         None,
         None,
         None,
+        None,
+        None,
+        None,
+        None,
+        None,
     )
 
 
@@ -207,6 +228,13 @@ MODEL_CLASSES = {
         XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
         XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
     ),
+    "xlm-roberta": (
+        XLMRobertaConfig,
+        TFXLMRobertaForMaskedLM,
+        XLMRobertaForMaskedLM,
+        XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    ),
     "transfo-xl": (
         TransfoXLConfig,
         TFTransfoXLLMHeadModel,
@@ -235,6 +263,13 @@ MODEL_CLASSES = {
         ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
         ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
     ),
+    "camembert": (
+        CamembertConfig,
+        TFCamembertForMaskedLM,
+        CamembertForMaskedLM,
+        CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+        CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
+    ),
     "distilbert": (
         DistilBertConfig,
         TFDistilBertForMaskedLM,
@@ -249,13 +284,6 @@ MODEL_CLASSES = {
         DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
         DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     ),
-    "distilbert-base-uncased-distilled-squad": (
-        DistilBertConfig,
-        TFDistilBertForQuestionAnswering,
-        DistilBertForQuestionAnswering,
-        DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
-        DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
-    ),
     "ctrl": (
         CTRLConfig,
         TFCTRLLMHeadModel,

src/transformers/data/metrics/squad_metrics.py

@@ -555,10 +555,10 @@ def compute_predictions_logits(
         all_nbest_json[example.qas_id] = nbest_json
 
     with open(output_prediction_file, "w") as writer:
-        writer.write(json.dumps(str(all_predictions), indent=4) + "\n")
+        writer.write(json.dumps(all_predictions, indent=4) + "\n")
 
     with open(output_nbest_file, "w") as writer:
-        writer.write(json.dumps(str(all_nbest_json), indent=4) + "\n")
+        writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
 
     if version_2_with_negative:
         with open(output_null_log_odds_file, "w") as writer:

src/transformers/data/processors/squad.py

@@ -306,15 +306,13 @@ def squad_convert_examples_to_features(
             tqdm(
                 p.imap(annotate_, examples, chunksize=32),
                 total=len(examples),
-                desc="Converting squad examples to features",
+                desc="convert squad examples to features",
             )
         )
-
-    print("Converted {} examples into {} features".format(len(examples), len(features)))
     new_features = []
     unique_id = 1000000000
     example_index = 0
-    for example_features in features:
+    for example_features in tqdm(features, total=len(features), desc="add example index and unique id"):
         if not example_features:
             continue
         for example_feature in example_features:
@@ -378,34 +376,31 @@ def squad_convert_examples_to_features(
                     },
                 )
 
-        return (
-            features,
-            tf.data.Dataset.from_generator(
-                gen,
-                (
-                    {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
-                    {
-                        "start_position": tf.int64,
-                        "end_position": tf.int64,
-                        "cls_index": tf.int64,
-                        "p_mask": tf.int32,
-                        "is_impossible": tf.int32,
-                    },
-                ),
-                (
-                    {
-                        "input_ids": tf.TensorShape([None]),
-                        "attention_mask": tf.TensorShape([None]),
-                        "token_type_ids": tf.TensorShape([None]),
-                    },
-                    {
-                        "start_position": tf.TensorShape([]),
-                        "end_position": tf.TensorShape([]),
-                        "cls_index": tf.TensorShape([]),
-                        "p_mask": tf.TensorShape([None]),
-                        "is_impossible": tf.TensorShape([]),
-                    },
-                ),
+        return tf.data.Dataset.from_generator(
+            gen,
+            (
+                {"input_ids": tf.int32, "attention_mask": tf.int32, "token_type_ids": tf.int32},
+                {
+                    "start_position": tf.int64,
+                    "end_position": tf.int64,
+                    "cls_index": tf.int64,
+                    "p_mask": tf.int32,
+                    "is_impossible": tf.int32,
+                },
+            ),
+            (
+                {
+                    "input_ids": tf.TensorShape([None]),
+                    "attention_mask": tf.TensorShape([None]),
+                    "token_type_ids": tf.TensorShape([None]),
+                },
+                {
+                    "start_position": tf.TensorShape([]),
+                    "end_position": tf.TensorShape([]),
+                    "cls_index": tf.TensorShape([]),
+                    "p_mask": tf.TensorShape([None]),
+                    "is_impossible": tf.TensorShape([]),
+                },
             ),
         )
 

src/transformers/modelcard.py

@@ -152,6 +152,8 @@ class ModelCard(object):
             resolved_model_card_file = cached_path(
                 model_card_file, cache_dir=cache_dir, force_download=True, proxies=proxies, resume_download=False
             )
+            if resolved_model_card_file is None:
+                raise EnvironmentError
             if resolved_model_card_file == model_card_file:
                 logger.info("loading model card file {}".format(model_card_file))
             else:

src/transformers/modeling_albert.py

@@ -698,7 +698,7 @@ class AlbertForSequenceClassification(AlbertPreTrainedModel):
         self.num_labels = config.num_labels
 
         self.albert = AlbertModel(config)
-        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        self.dropout = nn.Dropout(config.classifier_dropout_prob)
         self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)
 
         self.init_weights()

src/transformers/modeling_auto.py

@@ -25,6 +25,7 @@ from .configuration_auto import (
     CamembertConfig,
     CTRLConfig,
     DistilBertConfig,
+    FlaubertConfig,
     GPT2Config,
     OpenAIGPTConfig,
     RobertaConfig,
@@ -67,6 +68,13 @@ from .modeling_distilbert import (
     DistilBertForTokenClassification,
     DistilBertModel,
 )
+from .modeling_flaubert import (
+    FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
+    FlaubertForQuestionAnswering,
+    FlaubertForSequenceClassification,
+    FlaubertModel,
+    FlaubertWithLMHeadModel,
+)
 from .modeling_gpt2 import GPT2_PRETRAINED_MODEL_ARCHIVE_MAP, GPT2LMHeadModel, GPT2Model
 from .modeling_openai import OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP, OpenAIGPTLMHeadModel, OpenAIGPTModel
 from .modeling_roberta import (
@@ -122,6 +130,7 @@ ALL_PRETRAINED_MODEL_ARCHIVE_MAP = dict(
         CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
         T5_PRETRAINED_MODEL_ARCHIVE_MAP,
         XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
+        FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
     ]
     for key, value, in pretrained_map.items()
 )
@@ -141,6 +150,7 @@ MODEL_MAPPING = OrderedDict(
         (XLNetConfig, XLNetModel),
         (XLMConfig, XLMModel),
         (CTRLConfig, CTRLModel),
+        (FlaubertConfig, FlaubertModel),
     ]
 )
 
@@ -159,6 +169,7 @@ MODEL_FOR_PRETRAINING_MAPPING = OrderedDict(
         (XLNetConfig, XLNetLMHeadModel),
         (XLMConfig, XLMWithLMHeadModel),
         (CTRLConfig, CTRLLMHeadModel),
+        (FlaubertConfig, FlaubertWithLMHeadModel),
     ]
 )
 
@@ -177,6 +188,7 @@ MODEL_WITH_LM_HEAD_MAPPING = OrderedDict(
         (XLNetConfig, XLNetLMHeadModel),
         (XLMConfig, XLMWithLMHeadModel),
         (CTRLConfig, CTRLLMHeadModel),
+        (FlaubertConfig, FlaubertWithLMHeadModel),
     ]
 )
 
@@ -190,6 +202,7 @@ MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING = OrderedDict(
         (BertConfig, BertForSequenceClassification),
         (XLNetConfig, XLNetForSequenceClassification),
         (XLMConfig, XLMForSequenceClassification),
+        (FlaubertConfig, FlaubertForSequenceClassification),
     ]
 )
 
@@ -201,6 +214,7 @@ MODEL_FOR_QUESTION_ANSWERING_MAPPING = OrderedDict(
         (BertConfig, BertForQuestionAnswering),
         (XLNetConfig, XLNetForQuestionAnswering),
         (XLMConfig, XLMForQuestionAnswering),
+        (FlaubertConfig, FlaubertForQuestionAnswering),
     ]
 )
 
@@ -251,6 +265,7 @@ class AutoModel(object):
                 - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLModel` (Transformer-XL model)
                 - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModel` (XLNet model)
                 - isInstance of `xlm` configuration class: :class:`~transformers.XLMModel` (XLM model)
+                - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertModel` (XLM model)
 
         Examples::
 
@@ -291,6 +306,7 @@ class AutoModel(object):
             - contains `xlnet`: :class:`~transformers.XLNetModel` (XLNet model)
             - contains `xlm`: :class:`~transformers.XLMModel` (XLM model)
             - contains `ctrl`: :class:`~transformers.CTRLModel` (Salesforce CTRL  model)
+            - contains `flaubert`: :class:`~transformers.Flaubert` (Flaubert  model)
 
             The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
             To train the model, you should first set it back in training mode with `model.train()`
@@ -401,6 +417,7 @@ class AutoModelForPreTraining(object):
                 - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
                 - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
                 - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
+                - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
 
         Examples::
 
@@ -440,6 +457,7 @@ class AutoModelForPreTraining(object):
             - contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
             - contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
             - contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
+            - contains `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
@@ -552,6 +570,7 @@ class AutoModelWithLMHead(object):
                 - isInstance of `transfo-xl` configuration class: :class:`~transformers.TransfoXLLMHeadModel` (Transformer-XL model)
                 - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
                 - isInstance of `xlm` configuration class: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
+                - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
 
         Examples::
 
@@ -592,6 +611,7 @@ class AutoModelWithLMHead(object):
             - contains `xlnet`: :class:`~transformers.XLNetLMHeadModel` (XLNet model)
             - contains `xlm`: :class:`~transformers.XLMWithLMHeadModel` (XLM model)
             - contains `ctrl`: :class:`~transformers.CTRLLMHeadModel` (Salesforce CTRL model)
+            - contains `flaubert`: :class:`~transformers.FlaubertWithLMHeadModel` (Flaubert model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
@@ -703,6 +723,7 @@ class AutoModelForSequenceClassification(object):
                 - isInstance of `bert` configuration class: :class:`~transformers.BertModelForSequenceClassification` (Bert model)
                 - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForSequenceClassification` (XLNet model)
                 - isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForSequenceClassification` (XLM model)
+                - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model)
 
 
         Examples::
@@ -740,7 +761,7 @@ class AutoModelForSequenceClassification(object):
             - contains `roberta`: :class:`~transformers.RobertaForSequenceClassification` (RoBERTa model)
             - contains `bert`: :class:`~transformers.BertForSequenceClassification` (Bert model)
             - contains `xlnet`: :class:`~transformers.XLNetForSequenceClassification` (XLNet model)
-            - contains `xlm`: :class:`~transformers.XLMForSequenceClassification` (XLM model)
+            - contains `flaubert`: :class:`~transformers.FlaubertForSequenceClassification` (Flaubert model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`
@@ -850,6 +871,7 @@ class AutoModelForQuestionAnswering(object):
                 - isInstance of `bert` configuration class: :class:`~transformers.BertModelForQuestionAnswering` (Bert model)
                 - isInstance of `xlnet` configuration class: :class:`~transformers.XLNetModelForQuestionAnswering` (XLNet model)
                 - isInstance of `xlm` configuration class: :class:`~transformers.XLMModelForQuestionAnswering` (XLM model)
+                - isInstance of `flaubert` configuration class: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model)
 
         Examples::
 
@@ -885,6 +907,7 @@ class AutoModelForQuestionAnswering(object):
             - contains `bert`: :class:`~transformers.BertForQuestionAnswering` (Bert model)
             - contains `xlnet`: :class:`~transformers.XLNetForQuestionAnswering` (XLNet model)
             - contains `xlm`: :class:`~transformers.XLMForQuestionAnswering` (XLM model)
+            - contains `flaubert`: :class:`~transformers.FlaubertForQuestionAnswering` (XLM model)
 
         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated)
         To train the model, you should first set it back in training mode with `model.train()`

src/transformers/modeling_camembert.py

@@ -33,6 +33,8 @@ logger = logging.getLogger(__name__)
 
 CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
     "camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-pytorch_model.bin",
+    "umberto-commoncrawl-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-commoncrawl-cased-v1/pytorch_model.bin",
+    "umberto-wikipedia-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-wikipedia-uncased-v1/pytorch_model.bin",
 }
 
 

src/transformers/modeling_flaubert.py

@@ -0,0 +1,385 @@
+# coding=utf-8
+# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" PyTorch Flaubert model, based on XLM. """
+
+
+import logging
+import random
+
+import torch
+from torch.nn import functional as F
+
+from .configuration_flaubert import FlaubertConfig
+from .file_utils import add_start_docstrings, add_start_docstrings_to_callable
+from .modeling_xlm import (
+    XLMForQuestionAnswering,
+    XLMForQuestionAnsweringSimple,
+    XLMForSequenceClassification,
+    XLMModel,
+    XLMWithLMHeadModel,
+    get_masks,
+)
+
+
+logger = logging.getLogger(__name__)
+
+FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {
+    "flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/pytorch_model.bin",
+    "flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/pytorch_model.bin",
+    "flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/pytorch_model.bin",
+    "flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/pytorch_model.bin",
+}
+
+
+FLAUBERT_START_DOCSTRING = r"""
+
+    This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ sub-class.
+    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general
+    usage and behavior.
+
+    Parameters:
+        config (:class:`~transformers.FlaubertConfig`): Model configuration class with all the parameters of the model.
+            Initializing with a config file does not load the weights associated with the model, only the configuration.
+            Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
+"""
+
+FLAUBERT_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using :class:`transformers.BertTokenizer`.
+            See :func:`transformers.PreTrainedTokenizer.encode` and
+            :func:`transformers.PreTrainedTokenizer.encode_plus` for details.
+
+            `What are input IDs? <../glossary.html#input-ids>`__
+        attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
+            Mask to avoid performing attention on padding token indices.
+            Mask values selected in ``[0, 1]``:
+            ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.
+
+            `What are attention masks? <../glossary.html#attention-mask>`__
+        token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
+            Segment token indices to indicate first and second portions of the inputs.
+            Indices are selected in ``[0, 1]``: ``0`` corresponds to a `sentence A` token, ``1``
+            corresponds to a `sentence B` token
+
+            `What are token type IDs? <../glossary.html#token-type-ids>`_
+        position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
+            Indices of positions of each input sequence tokens in the position embeddings.
+            Selected in the range ``[0, config.max_position_embeddings - 1]``.
+
+            `What are position IDs? <../glossary.html#position-ids>`_
+        lengths (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`):
+            Length of each sentence that can be used to avoid performing attention on padding token indices.
+            You can also use `attention_mask` for the same result (see above), kept here for compatbility.
+            Indices selected in ``[0, ..., input_ids.size(-1)]``:
+        cache (:obj:`Dict[str, torch.FloatTensor]`, `optional`, defaults to :obj:`None`):
+            dictionary with ``torch.FloatTensor`` that contains pre-computed
+            hidden-states (key and values in the attention blocks) as computed by the model
+            (see `cache` output below). Can be used to speed up sequential decoding.
+            The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states.
+        head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`, defaults to :obj:`None`):
+            Mask to nullify selected heads of the self-attention modules.
+            Mask values selected in ``[0, 1]``:
+            :obj:`1` indicates the head is **not masked**, :obj:`0` indicates the head is **masked**.
+        input_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`, defaults to :obj:`None`):
+            Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation.
+            This is useful if you want more control over how to convert `input_ids` indices into associated vectors
+            than the model's internal embedding lookup matrix.
+"""
+
+
+@add_start_docstrings(
+    "The bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.",
+    FLAUBERT_START_DOCSTRING,
+)
+class FlaubertModel(XLMModel):
+
+    config_class = FlaubertConfig
+    pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+    def __init__(self, config):  # , dico, is_encoder, with_output):
+        super(FlaubertModel, self).__init__(config)
+        self.layerdrop = getattr(config, "layerdrop", 0.0)
+        self.pre_norm = getattr(config, "pre_norm", False)
+
+    @add_start_docstrings_to_callable(FLAUBERT_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        langs=None,
+        token_type_ids=None,
+        position_ids=None,
+        lengths=None,
+        cache=None,
+        head_mask=None,
+        inputs_embeds=None,
+    ):
+        r"""
+    Return:
+        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.XLMConfig`) and inputs:
+        last_hidden_state (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+            Sequence of hidden-states at the output of the last layer of the model.
+        hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+
+            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+        attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
+            Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
+            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
+
+            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+            heads.
+
+    Examples::
+
+        tokenizer = FlaubertTokenizer.from_pretrained('flaubert-base-cased')
+        model = FlaubertModel.from_pretrained('flaubert-base-cased')
+        input_ids = torch.tensor(tokenizer.encode("Le chat manges une pomme.", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
+        outputs = model(input_ids)
+        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
+
+        """
+        # removed: src_enc=None, src_len=None
+        if input_ids is not None:
+            bs, slen = input_ids.size()
+        else:
+            bs, slen = inputs_embeds.size()[:-1]
+
+        if lengths is None:
+            if input_ids is not None:
+                lengths = (input_ids != self.pad_index).sum(dim=1).long()
+            else:
+                lengths = torch.LongTensor([slen] * bs)
+        # mask = input_ids != self.pad_index
+
+        # check inputs
+        assert lengths.size(0) == bs
+        assert lengths.max().item() <= slen
+        # input_ids = input_ids.transpose(0, 1)  # batch size as dimension 0
+        # assert (src_enc is None) == (src_len is None)
+        # if src_enc is not None:
+        #     assert self.is_decoder
+        #     assert src_enc.size(0) == bs
+
+        # generate masks
+        mask, attn_mask = get_masks(slen, lengths, self.causal, padding_mask=attention_mask)
+        # if self.is_decoder and src_enc is not None:
+        #     src_mask = torch.arange(src_len.max(), dtype=torch.long, device=lengths.device) < src_len[:, None]
+
+        device = input_ids.device if input_ids is not None else inputs_embeds.device
+
+        # position_ids
+        if position_ids is None:
+            position_ids = torch.arange(slen, dtype=torch.long, device=device)
+            position_ids = position_ids.unsqueeze(0).expand((bs, slen))
+        else:
+            assert position_ids.size() == (bs, slen)  # (slen, bs)
+            # position_ids = position_ids.transpose(0, 1)
+
+        # langs
+        if langs is not None:
+            assert langs.size() == (bs, slen)  # (slen, bs)
+            # langs = langs.transpose(0, 1)
+
+        # Prepare head mask if needed
+        # 1.0 in head_mask indicate we keep the head
+        # attention_probs has shape bsz x n_heads x N x N
+        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
+        # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x qlen x klen]
+        if head_mask is not None:
+            if head_mask.dim() == 1:
+                head_mask = head_mask.unsqueeze(0).unsqueeze(0).unsqueeze(-1).unsqueeze(-1)
+                head_mask = head_mask.expand(self.n_layers, -1, -1, -1, -1)
+            elif head_mask.dim() == 2:
+                head_mask = (
+                    head_mask.unsqueeze(1).unsqueeze(-1).unsqueeze(-1)
+                )  # We can specify head_mask for each layer
+            head_mask = head_mask.to(
+                dtype=next(self.parameters()).dtype
+            )  # switch to fload if need + fp16 compatibility
+        else:
+            head_mask = [None] * self.n_layers
+
+        # do not recompute cached elements
+        if cache is not None and input_ids is not None:
+            _slen = slen - cache["slen"]
+            input_ids = input_ids[:, -_slen:]
+            position_ids = position_ids[:, -_slen:]
+            if langs is not None:
+                langs = langs[:, -_slen:]
+            mask = mask[:, -_slen:]
+            attn_mask = attn_mask[:, -_slen:]
+
+        # embeddings
+        if inputs_embeds is None:
+            inputs_embeds = self.embeddings(input_ids)
+
+        tensor = inputs_embeds + self.position_embeddings(position_ids).expand_as(inputs_embeds)
+        if langs is not None and self.use_lang_emb:
+            tensor = tensor + self.lang_embeddings(langs)
+        if token_type_ids is not None:
+            tensor = tensor + self.embeddings(token_type_ids)
+        tensor = self.layer_norm_emb(tensor)
+        tensor = F.dropout(tensor, p=self.dropout, training=self.training)
+        tensor *= mask.unsqueeze(-1).to(tensor.dtype)
+
+        # transformer layers
+        hidden_states = ()
+        attentions = ()
+        for i in range(self.n_layers):
+            # LayerDrop
+            dropout_probability = random.uniform(0, 1)
+            if self.training and (dropout_probability < self.layerdrop):
+                continue
+
+            if self.output_hidden_states:
+                hidden_states = hidden_states + (tensor,)
+
+            # self attention
+            if not self.pre_norm:
+                attn_outputs = self.attentions[i](tensor, attn_mask, cache=cache, head_mask=head_mask[i])
+                attn = attn_outputs[0]
+                if self.output_attentions:
+                    attentions = attentions + (attn_outputs[1],)
+                attn = F.dropout(attn, p=self.dropout, training=self.training)
+                tensor = tensor + attn
+                tensor = self.layer_norm1[i](tensor)
+            else:
+                tensor_normalized = self.layer_norm1[i](tensor)
+                attn_outputs = self.attentions[i](tensor_normalized, attn_mask, cache=cache, head_mask=head_mask[i])
+                attn = attn_outputs[0]
+                if self.output_attentions:
+                    attentions = attentions + (attn_outputs[1],)
+                attn = F.dropout(attn, p=self.dropout, training=self.training)
+                tensor = tensor + attn
+
+            # encoder attention (for decoder only)
+            # if self.is_decoder and src_enc is not None:
+            #     attn = self.encoder_attn[i](tensor, src_mask, kv=src_enc, cache=cache)
+            #     attn = F.dropout(attn, p=self.dropout, training=self.training)
+            #     tensor = tensor + attn
+            #     tensor = self.layer_norm15[i](tensor)
+
+            # FFN
+            if not self.pre_norm:
+                tensor = tensor + self.ffns[i](tensor)
+                tensor = self.layer_norm2[i](tensor)
+            else:
+                tensor_normalized = self.layer_norm2[i](tensor)
+                tensor = tensor + self.ffns[i](tensor_normalized)
+
+            tensor *= mask.unsqueeze(-1).to(tensor.dtype)
+
+        # Add last hidden state
+        if self.output_hidden_states:
+            hidden_states = hidden_states + (tensor,)
+
+        # update cache length
+        if cache is not None:
+            cache["slen"] += tensor.size(1)
+
+        # move back sequence length to dimension 0
+        # tensor = tensor.transpose(0, 1)
+
+        outputs = (tensor,)
+        if self.output_hidden_states:
+            outputs = outputs + (hidden_states,)
+        if self.output_attentions:
+            outputs = outputs + (attentions,)
+        return outputs  # outputs, (hidden_states), (attentions)
+
+
+@add_start_docstrings(
+    """The Flaubert Model transformer with a language modeling head on top
+    (linear layer with weights tied to the input embeddings). """,
+    FLAUBERT_START_DOCSTRING,
+)
+class FlaubertWithLMHeadModel(XLMWithLMHeadModel):
+    """
+    This class overrides :class:`~transformers.XLMWithLMHeadModel`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = FlaubertConfig
+    pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+    def __init__(self, config):
+        super(FlaubertWithLMHeadModel, self).__init__(config)
+        self.transformer = FlaubertModel(config)
+        self.init_weights()
+
+
+@add_start_docstrings(
+    """Flaubert Model with a sequence classification/regression head on top (a linear layer on top of
+    the pooled output) e.g. for GLUE tasks. """,
+    FLAUBERT_START_DOCSTRING,
+)
+class FlaubertForSequenceClassification(XLMForSequenceClassification):
+    """
+    This class overrides :class:`~transformers.XLMForSequenceClassification`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = FlaubertConfig
+    pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+    def __init__(self, config):
+        super(FlaubertForSequenceClassification, self).__init__(config)
+        self.transformer = FlaubertModel(config)
+        self.init_weights()
+
+
+@add_start_docstrings(
+    """Flaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of
+    the hidden-states output to compute `span start logits` and `span end logits`). """,
+    FLAUBERT_START_DOCSTRING,
+)
+class FlaubertForQuestionAnsweringSimple(XLMForQuestionAnsweringSimple):
+    """
+    This class overrides :class:`~transformers.XLMForQuestionAnsweringSimple`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = FlaubertConfig
+    pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+    def __init__(self, config):
+        super(FlaubertForQuestionAnsweringSimple, self).__init__(config)
+        self.transformer = FlaubertModel(config)
+        self.init_weights()
+
+
+@add_start_docstrings(
+    """Flaubert Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of
+    the hidden-states output to compute `span start logits` and `span end logits`). """,
+    FLAUBERT_START_DOCSTRING,
+)
+class FlaubertForQuestionAnswering(XLMForQuestionAnswering):
+    """
+    This class overrides :class:`~transformers.XLMForQuestionAnswering`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = FlaubertConfig
+    pretrained_model_archive_map = FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+    def __init__(self, config):
+        super(FlaubertForQuestionAnswering, self).__init__(config)
+        self.transformer = FlaubertModel(config)
+        self.init_weights()

src/transformers/modeling_tf_camembert.py

@@ -0,0 +1,118 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" TF 2.0 RoBERTa model. """
+
+
+import logging
+
+from .configuration_camembert import CamembertConfig
+from .file_utils import add_start_docstrings
+from .modeling_tf_roberta import (
+    TFRobertaForMaskedLM,
+    TFRobertaForSequenceClassification,
+    TFRobertaForTokenClassification,
+    TFRobertaModel,
+)
+
+
+logger = logging.getLogger(__name__)
+
+TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP = {}
+
+
+CAMEMBERT_START_DOCSTRING = r"""
+
+    .. note::
+
+        TF 2.0 models accepts two formats as inputs:
+
+            - having all inputs as keyword arguments (like PyTorch models), or
+            - having all inputs as a list, tuple or dict in the first positional arguments.
+
+        This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having
+        all the tensors in the first argument of the model call function: :obj:`model(inputs)`.
+
+        If you choose this second option, there are three possibilities you can use to gather all the input Tensors
+        in the first positional argument :
+
+        - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)`
+        - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
+          :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])`
+        - a dictionary with one or several input Tensors associated to the input names given in the docstring:
+          :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})`
+
+    Parameters:
+        config (:class:`~transformers.CamembertConfig`): Model configuration class with all the parameters of the
+            model. Initializing with a config file does not load the weights associated with the model, only the configuration.
+            Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
+"""
+
+
+@add_start_docstrings(
+    "The bare CamemBERT Model transformer outputting raw hidden-states without any specific head on top.",
+    CAMEMBERT_START_DOCSTRING,
+)
+class TFCamembertModel(TFRobertaModel):
+    """
+    This class overrides :class:`~transformers.TFRobertaModel`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = CamembertConfig
+    pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """CamemBERT Model with a `language modeling` head on top. """, CAMEMBERT_START_DOCSTRING,
+)
+class TFCamembertForMaskedLM(TFRobertaForMaskedLM):
+    """
+    This class overrides :class:`~transformers.TFRobertaForMaskedLM`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = CamembertConfig
+    pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """CamemBERT Model transformer with a sequence classification/regression head on top (a linear layer
+    on top of the pooled output) e.g. for GLUE tasks. """,
+    CAMEMBERT_START_DOCSTRING,
+)
+class TFCamembertForSequenceClassification(TFRobertaForSequenceClassification):
+    """
+    This class overrides :class:`~transformers.TFRobertaForSequenceClassification`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = CamembertConfig
+    pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """CamemBERT Model with a token classification head on top (a linear layer on top of
+    the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. """,
+    CAMEMBERT_START_DOCSTRING,
+)
+class TFCamembertForTokenClassification(TFRobertaForTokenClassification):
+    """
+    This class overrides :class:`~transformers.TFRobertaForTokenClassification`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = CamembertConfig
+    pretrained_model_archive_map = TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP

src/transformers/modeling_tf_xlm_roberta.py

@@ -0,0 +1,118 @@
+# coding=utf-8
+# Copyright 2019 Facebook AI Research and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" TF 2.0  XLM-RoBERTa model. """
+
+
+import logging
+
+from .configuration_xlm_roberta import XLMRobertaConfig
+from .file_utils import add_start_docstrings
+from .modeling_tf_roberta import (
+    TFRobertaForMaskedLM,
+    TFRobertaForSequenceClassification,
+    TFRobertaForTokenClassification,
+    TFRobertaModel,
+)
+
+
+logger = logging.getLogger(__name__)
+
+TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP = {}
+
+
+XLM_ROBERTA_START_DOCSTRING = r"""
+
+    .. note::
+
+        TF 2.0 models accepts two formats as inputs:
+
+            - having all inputs as keyword arguments (like PyTorch models), or
+            - having all inputs as a list, tuple or dict in the first positional arguments.
+
+        This second option is useful when using :obj:`tf.keras.Model.fit()` method which currently requires having
+        all the tensors in the first argument of the model call function: :obj:`model(inputs)`.
+
+        If you choose this second option, there are three possibilities you can use to gather all the input Tensors
+        in the first positional argument :
+
+        - a single Tensor with input_ids only and nothing else: :obj:`model(inputs_ids)`
+        - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
+          :obj:`model([input_ids, attention_mask])` or :obj:`model([input_ids, attention_mask, token_type_ids])`
+        - a dictionary with one or several input Tensors associated to the input names given in the docstring:
+          :obj:`model({'input_ids': input_ids, 'token_type_ids': token_type_ids})`
+
+    Parameters:
+        config (:class:`~transformers.XLMRobertaConfig`): Model configuration class with all the parameters of the
+            model. Initializing with a config file does not load the weights associated with the model, only the configuration.
+            Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights.
+"""
+
+
+@add_start_docstrings(
+    "The bare XLM-RoBERTa Model transformer outputting raw hidden-states without any specific head on top.",
+    XLM_ROBERTA_START_DOCSTRING,
+)
+class TFXLMRobertaModel(TFRobertaModel):
+    """
+    This class overrides :class:`~transformers.TFRobertaModel`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = XLMRobertaConfig
+    pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """XLM-RoBERTa Model with a `language modeling` head on top. """, XLM_ROBERTA_START_DOCSTRING,
+)
+class TFXLMRobertaForMaskedLM(TFRobertaForMaskedLM):
+    """
+    This class overrides :class:`~transformers.TFRobertaForMaskedLM`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = XLMRobertaConfig
+    pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """XLM-RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer
+    on top of the pooled output) e.g. for GLUE tasks. """,
+    XLM_ROBERTA_START_DOCSTRING,
+)
+class TFXLMRobertaForSequenceClassification(TFRobertaForSequenceClassification):
+    """
+    This class overrides :class:`~transformers.TFRobertaForSequenceClassification`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = XLMRobertaConfig
+    pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+@add_start_docstrings(
+    """XLM-RoBERTa Model with a token classification head on top (a linear layer on top of
+    the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. """,
+    XLM_ROBERTA_START_DOCSTRING,
+)
+class TFXLMRobertaForTokenClassification(TFRobertaForTokenClassification):
+    """
+    This class overrides :class:`~transformers.TFRobertaForTokenClassification`. Please check the
+    superclass for the appropriate documentation alongside usage examples.
+    """
+
+    config_class = XLMRobertaConfig
+    pretrained_model_archive_map = TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP

src/transformers/modeling_utils.py

@@ -284,6 +284,9 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin):
         # Only save the model itself if we are using distributed training
         model_to_save = self.module if hasattr(self, "module") else self
 
+        # Attach architecture to the config
+        model_to_save.config.architectures = [model_to_save.__class__.__name__]
+
         # Save configuration file
         model_to_save.config.save_pretrained(save_directory)
 

src/transformers/optimization_tf.py

@@ -56,16 +56,7 @@ class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
         }
 
 
-def create_optimizer(
-    init_lr,
-    num_train_steps,
-    num_warmup_steps,
-    weight_decay=0.0,
-    adam_epsilon=1e-6,
-    beta_1=0.9,
-    beta_2=0.999,
-    exclude_from_weight_decay=("layer_norm", "bias"),
-):
+def create_optimizer(init_lr, num_train_steps, num_warmup_steps):
     """Creates an optimizer with learning rate schedule."""
     # Implements linear decay of the learning rate.
     learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(
@@ -77,11 +68,11 @@ def create_optimizer(
         )
     optimizer = AdamWeightDecay(
         learning_rate=learning_rate_fn,
-        weight_decay_rate=weight_decay,
-        beta_1=beta_1,
-        beta_2=beta_2,
-        epsilon=adam_epsilon,
-        exclude_from_weight_decay=exclude_from_weight_decay,
+        weight_decay_rate=0.01,
+        beta_1=0.9,
+        beta_2=0.999,
+        epsilon=1e-6,
+        exclude_from_weight_decay=["layer_norm", "bias"],
     )
     return optimizer
 

src/transformers/pipelines.py

@@ -28,7 +28,10 @@ from typing import Dict, List, Optional, Tuple, Union
 import numpy as np
 
 from .configuration_auto import ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, AutoConfig
+from .configuration_distilbert import DistilBertConfig
+from .configuration_roberta import RobertaConfig
 from .configuration_utils import PretrainedConfig
+from .configuration_xlm import XLMConfig
 from .data import SquadExample, squad_convert_examples_to_features
 from .file_utils import is_tf_available, is_torch_available
 from .modelcard import ModelCard
@@ -44,6 +47,7 @@ if is_tf_available():
         TFAutoModelForSequenceClassification,
         TFAutoModelForQuestionAnswering,
         TFAutoModelForTokenClassification,
+        TFAutoModelWithLMHead,
     )
 
 if is_torch_available():
@@ -53,6 +57,7 @@ if is_torch_available():
         AutoModelForSequenceClassification,
         AutoModelForQuestionAnswering,
         AutoModelForTokenClassification,
+        AutoModelWithLMHead,
     )
 
 
@@ -64,7 +69,7 @@ def get_framework(model=None):
         If both frameworks are installed and no specific model is provided, defaults to using PyTorch.
     """
     if is_tf_available() and is_torch_available() and model is not None and not isinstance(model, str):
-        # Both framework are available but the use supplied a model class instance.
+        # Both framework are available but the user supplied a model class instance.
         # Try to guess which framework to use from the model classname
         framework = "tf" if model.__class__.__name__.startswith("TF") else "pt"
     elif not is_tf_available() and not is_torch_available():
@@ -364,7 +369,6 @@ class Pipeline(_ScikitCompat):
     def predict(self, X):
         """
         Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().
-        Se
         """
         return self(X=X)
 
@@ -406,9 +410,8 @@ class Pipeline(_ScikitCompat):
             dict holding all the required parameters for model's forward
         """
         args = ["input_ids", "attention_mask"]
-        model_type = type(self.model).__name__.lower()
 
-        if "distilbert" not in model_type and "xlm" not in model_type:
+        if not isinstance(self.model.config, (DistilBertConfig, XLMConfig, RobertaConfig)):
             args += ["token_type_ids"]
 
         # PR #1548 (CLI) There is an issue with attention_mask
@@ -420,7 +423,10 @@ class Pipeline(_ScikitCompat):
         else:
             return {k: [feature[k] for feature in features] for k in args}
 
-    def __call__(self, *texts, **kwargs):
+    def _parse_and_tokenize(self, *texts, **kwargs):
+        """
+        Parse arguments and tokenize
+        """
         # Parse arguments
         inputs = self._args_parser(*texts, **kwargs)
         inputs = self.tokenizer.batch_encode_plus(
@@ -429,13 +435,19 @@ class Pipeline(_ScikitCompat):
 
         # Filter out features not available on specific models
         inputs = self.inputs_for_model(inputs)
+
+        return inputs
+
+    def __call__(self, *texts, **kwargs):
+        inputs = self._parse_and_tokenize(*texts, **kwargs)
         return self._forward(inputs)
 
-    def _forward(self, inputs):
+    def _forward(self, inputs, return_tensors=False):
         """
         Internal framework specific forward dispatching.
         Args:
             inputs: dict holding all the keyworded arguments for required by the model forward method.
+            return_tensors: Whether to return native framework (pt/tf) tensors rather than numpy array.
         Returns:
             Numpy array
         """
@@ -449,7 +461,10 @@ class Pipeline(_ScikitCompat):
                     inputs = self.ensure_tensor_on_device(**inputs)
                     predictions = self.model(**inputs)[0].cpu()
 
-        return predictions.numpy()
+        if return_tensors:
+            return predictions
+        else:
+            return predictions.numpy()
 
 
 class FeatureExtractionPipeline(Pipeline):
@@ -491,6 +506,71 @@ class TextClassificationPipeline(Pipeline):
         return [{"label": self.model.config.id2label[item.argmax()], "score": item.max()} for item in scores]
 
 
+class FillMaskPipeline(Pipeline):
+    """
+    Masked language modeling prediction pipeline using ModelWithLMHead head.
+    """
+
+    def __init__(
+        self,
+        model,
+        tokenizer: PreTrainedTokenizer = None,
+        modelcard: ModelCard = None,
+        framework: Optional[str] = None,
+        args_parser: ArgumentHandler = None,
+        device: int = -1,
+        topk=5,
+    ):
+        super().__init__(
+            model=model,
+            tokenizer=tokenizer,
+            modelcard=modelcard,
+            framework=framework,
+            args_parser=args_parser,
+            device=device,
+            binary_output=True,
+        )
+
+        self.topk = topk
+
+    def __call__(self, *args, **kwargs):
+        inputs = self._parse_and_tokenize(*args, **kwargs)
+        outputs = self._forward(inputs, return_tensors=True)
+
+        results = []
+        batch_size = outputs.shape[0] if self.framework == "tf" else outputs.size(0)
+
+        for i in range(batch_size):
+            input_ids = inputs["input_ids"][i]
+            result = []
+
+            if self.framework == "tf":
+                masked_index = tf.where(input_ids == self.tokenizer.mask_token_id).numpy().item()
+                logits = outputs[i, masked_index, :]
+                probs = tf.nn.softmax(logits)
+                topk = tf.math.top_k(probs, k=self.topk)
+                values, predictions = topk.values.numpy(), topk.indices.numpy()
+            else:
+                masked_index = (input_ids == self.tokenizer.mask_token_id).nonzero().item()
+                logits = outputs[i, masked_index, :]
+                probs = logits.softmax(dim=0)
+                values, predictions = probs.topk(self.topk)
+
+            for v, p in zip(values.tolist(), predictions.tolist()):
+                tokens = input_ids.numpy()
+                tokens[masked_index] = p
+                # Filter padding out:
+                tokens = tokens[np.where(tokens != self.tokenizer.pad_token_id)]
+                result.append({"sequence": self.tokenizer.decode(tokens), "score": v, "token": p})
+
+            # Append
+            results += [result]
+
+        if len(results) == 1:
+            return results[0]
+        return results
+
+
 class NerPipeline(Pipeline):
     """
     Named Entity Recognition pipeline using ModelForTokenClassification head.
@@ -523,7 +603,8 @@ class NerPipeline(Pipeline):
         self.ignore_labels = ignore_labels
 
     def __call__(self, *texts, **kwargs):
-        inputs, answers = self._args_parser(*texts, **kwargs), []
+        inputs = self._args_parser(*texts, **kwargs)
+        answers = []
         for sentence in inputs:
 
             # Manage correct placement of the tensors
@@ -903,6 +984,16 @@ SUPPORTED_TASKS = {
             "tokenizer": "distilbert-base-uncased",
         },
     },
+    "fill-mask": {
+        "impl": FillMaskPipeline,
+        "tf": TFAutoModelWithLMHead if is_tf_available() else None,
+        "pt": AutoModelWithLMHead if is_torch_available() else None,
+        "default": {
+            "model": {"pt": "distilroberta-base", "tf": "distilroberta-base"},
+            "config": None,
+            "tokenizer": "distilroberta-base",
+        },
+    },
 }
 
 

src/transformers/tokenization_auto.py

@@ -73,7 +73,7 @@ TOKENIZER_MAPPING = OrderedDict(
 )
 
 
-class AutoTokenizer(object):
+class AutoTokenizer:
     r""":class:`~transformers.AutoTokenizer` is a generic tokenizer class
         that will be instantiated as one of the tokenizer classes of the library
         when created with the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)`

src/transformers/tokenization_camembert.py

@@ -40,6 +40,13 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
     "camembert-base": None,
 }
 
+SHARED_MODEL_IDENTIFIERS = [
+    # Load with
+    # `tokenizer = AutoTokenizer.from_pretrained("username/pretrained_model")`
+    "Musixmatch/umberto-commoncrawl-cased-v1",
+    "Musixmatch/umberto-wikipedia-uncased-v1",
+]
+
 
 class CamembertTokenizer(PreTrainedTokenizer):
     """

src/transformers/tokenization_flaubert.py

@@ -0,0 +1,145 @@
+# coding=utf-8
+# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes for Flaubert, based on XLM."""
+
+
+import logging
+import unicodedata
+
+import six
+
+from .tokenization_xlm import XLMTokenizer
+
+
+logger = logging.getLogger(__name__)
+
+VOCAB_FILES_NAMES = {
+    "vocab_file": "vocab.json",
+    "merges_file": "merges.txt",
+}
+
+PRETRAINED_VOCAB_FILES_MAP = {
+    "vocab_file": {
+        "flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/vocab.json",
+        "flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/vocab.json",
+        "flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/vocab.json",
+        "flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/vocab.json",
+    },
+    "merges_file": {
+        "flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/merges.txt",
+        "flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/merges.txt",
+        "flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/merges.txt",
+        "flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/merges.txt",
+    },
+}
+
+PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
+    "flaubert-small-cased": 512,
+    "flaubert-base-uncased": 512,
+    "flaubert-base-cased": 512,
+    "flaubert-large-cased": 512,
+}
+
+PRETRAINED_INIT_CONFIGURATION = {
+    "flaubert-small-cased": {"do_lowercase": False},
+    "flaubert-base-uncased": {"do_lowercase": True},
+    "flaubert-base-cased": {"do_lowercase": False},
+    "flaubert-large-cased": {"do_lowercase": False},
+}
+
+
+def convert_to_unicode(text):
+    """
+    Converts `text` to Unicode (if it's not already), assuming UTF-8 input.
+    """
+    # six_ensure_text is copied from https://github.com/benjaminp/six
+    def six_ensure_text(s, encoding="utf-8", errors="strict"):
+        if isinstance(s, six.binary_type):
+            return s.decode(encoding, errors)
+        elif isinstance(s, six.text_type):
+            return s
+        else:
+            raise TypeError("not expecting type '%s'" % type(s))
+
+    return six_ensure_text(text, encoding="utf-8", errors="ignore")
+
+
+class FlaubertTokenizer(XLMTokenizer):
+    """
+    BPE tokenizer for Flaubert
+
+        - Moses preprocessing & tokenization
+
+        - Normalize all inputs text
+
+        - argument ``special_tokens`` and function ``set_special_tokens``, can be used to add additional symbols \
+        (ex: "__classify__") to a vocabulary
+
+        - `do_lowercase` controle lower casing (automatically set for pretrained vocabularies)
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
+    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+
+    def __init__(self, do_lowercase=False, **kwargs):
+        super().__init__(**kwargs)
+        self.do_lowercase = do_lowercase
+        self.do_lowercase_and_remove_accent = False
+
+    def preprocess_text(self, text):
+        text = text.replace("``", '"').replace("''", '"')
+        text = convert_to_unicode(text)
+        text = unicodedata.normalize("NFC", text)
+
+        if self.do_lowercase:
+            text = text.lower()
+
+        return text
+
+    def _tokenize(self, text, bypass_tokenizer=False):
+        """
+        Tokenize a string given language code using Moses.
+
+        Details of tokenization:
+        - [sacremoses](https://github.com/alvations/sacremoses): port of Moses
+            - Install with `pip install sacremoses`
+
+        Args:
+            - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False)  (bool). If True, we only apply BPE.
+
+        Returns:
+            List of tokens.
+        """
+        lang = "fr"
+        if lang and self.lang2id and lang not in self.lang2id:
+            logger.error(
+                "Supplied language code not found in lang2id mapping. Please check that your language is supported by the loaded pretrained model."
+            )
+
+        if bypass_tokenizer:
+            text = text.split()
+        else:
+            text = self.preprocess_text(text)
+            text = self.moses_pipeline(text, lang=lang)
+            text = self.moses_tokenize(text, lang=lang)
+
+        split_tokens = []
+        for token in text:
+            if token:
+                split_tokens.extend([t for t in self.bpe(token).split(" ")])
+
+        return split_tokens

src/transformers/tokenization_utils.py

@@ -326,7 +326,7 @@ class PreTrainedTokenizer(object):
                 cls.pretrained_init_configuration
                 and pretrained_model_name_or_path in cls.pretrained_init_configuration
             ):
-                init_configuration = cls.pretrained_init_configuration[pretrained_model_name_or_path]
+                init_configuration = cls.pretrained_init_configuration[pretrained_model_name_or_path].copy()
         else:
             # Get the vocabulary from local files
             logger.info(
@@ -998,6 +998,7 @@ class PreTrainedTokenizer(object):
             for key, value in batch_outputs.items():
 
                 padded_value = value
+                # verify that the tokenizer has a pad_token_id
                 if key != "input_len" and self._pad_token is not None:
                     # Padding handle
                     padded_value = [

templates/adding_a_new_example_script/run_xxx.py

@@ -333,7 +333,8 @@ def evaluate(args, model, tokenizer, prefix=""):
 
 def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
     if args.local_rank not in [-1, 0] and not evaluate:
-        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache
+        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset,
+        # and the others will use the cache
 
     # Load data features from cache or dataset file
     input_file = args.predict_file if evaluate else args.train_file
@@ -366,7 +367,8 @@ def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=Fal
             torch.save(features, cached_features_file)
 
     if args.local_rank == 0 and not evaluate:
-        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache
+        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset,
+        # and the others will use the cache
 
     # Convert to Tensors and build dataset
     all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
@@ -620,7 +622,8 @@ def main():
 
     # Load pretrained model and tokenizer
     if args.local_rank not in [-1, 0]:
-        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
+        torch.distributed.barrier()  # Make sure only the first process in distributed training will
+        # download model & vocab
 
     args.model_type = args.model_type.lower()
     config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
@@ -641,15 +644,16 @@ def main():
     )
 
     if args.local_rank == 0:
-        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
+        torch.distributed.barrier()  # Make sure only the first process in distributed training will
+        # download model & vocab
 
     model.to(args.device)
 
     logger.info("Training/evaluation parameters %s", args)
 
-    # Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum if args.fp16 is set.
-    # Otherwise it'll default to "promote" mode, and we'll get fp32 operations. Note that running `--fp16_opt_level="O2"` will
-    # remove the need for this code, but it is still valid.
+    # Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum
+    # if args.fp16 is set. Otherwise it'll default to "promote" mode, and we'll get fp32 operations.
+    # Note that running `--fp16_opt_level="O2"` will remove the need for this code, but it is still valid.
     if args.fp16:
         try:
             import apex

tests/test_pipelines.py

@@ -1,7 +1,8 @@
 import unittest
-from typing import Iterable
+from typing import Iterable, List, Optional
 
 from transformers import pipeline
+from transformers.pipelines import Pipeline
 
 from .utils import require_tf, require_torch
 
@@ -62,9 +63,25 @@ TEXT_CLASSIF_FINETUNED_MODELS = {
     )
 }
 
+FILL_MASK_FINETUNED_MODELS = {
+    ("distilroberta-base", "distilroberta-base", None),
+}
+
+TF_FILL_MASK_FINETUNED_MODELS = {
+    ("distilroberta-base", "distilroberta-base", None),
+}
+
 
 class MonoColumnInputTestCase(unittest.TestCase):
-    def _test_mono_column_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):
+    def _test_mono_column_pipeline(
+        self,
+        nlp: Pipeline,
+        valid_inputs: List,
+        invalid_inputs: List,
+        output_keys: Iterable[str],
+        expected_multi_result: Optional[List] = None,
+        expected_check_keys: Optional[List[str]] = None,
+    ):
         self.assertIsNotNone(nlp)
 
         mono_result = nlp(valid_inputs[0])
@@ -81,6 +98,13 @@ class MonoColumnInputTestCase(unittest.TestCase):
         self.assertIsInstance(multi_result, list)
         self.assertIsInstance(multi_result[0], (dict, list))
 
+        if expected_multi_result is not None:
+            for result, expect in zip(multi_result, expected_multi_result):
+                for key in expected_check_keys or []:
+                    self.assertEqual(
+                        set([o[key] for o in result]), set([o[key] for o in expect]),
+                    )
+
         if isinstance(multi_result[0], list):
             multi_result = multi_result[0]
 
@@ -110,7 +134,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
 
     @require_torch
     def test_sentiment_analysis(self):
-        mandatory_keys = {"label"}
+        mandatory_keys = {"label", "score"}
         valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
         invalid_inputs = [None]
         for tokenizer, model, config in TEXT_CLASSIF_FINETUNED_MODELS:
@@ -119,7 +143,7 @@ class MonoColumnInputTestCase(unittest.TestCase):
 
     @require_tf
     def test_tf_sentiment_analysis(self):
-        mandatory_keys = {"label"}
+        mandatory_keys = {"label", "score"}
         valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
         invalid_inputs = [None]
         for tokenizer, model, config in TF_TEXT_CLASSIF_FINETUNED_MODELS:
@@ -127,21 +151,87 @@ class MonoColumnInputTestCase(unittest.TestCase):
             self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, mandatory_keys)
 
     @require_torch
-    def test_features_extraction(self):
+    def test_feature_extraction(self):
         valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
         invalid_inputs = [None]
         for tokenizer, model, config in FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
+            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
             self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
 
     @require_tf
-    def test_tf_features_extraction(self):
+    def test_tf_feature_extraction(self):
         valid_inputs = ["HuggingFace is solving NLP one commit at a time.", "HuggingFace is based in New-York & Paris"]
         invalid_inputs = [None]
         for tokenizer, model, config in TF_FEATURE_EXTRACT_FINETUNED_MODELS:
-            nlp = pipeline(task="sentiment-analysis", model=model, config=config, tokenizer=tokenizer)
+            nlp = pipeline(task="feature-extraction", model=model, config=config, tokenizer=tokenizer)
             self._test_mono_column_pipeline(nlp, valid_inputs, invalid_inputs, {})
 
+    @require_torch
+    def test_fill_mask(self):
+        mandatory_keys = {"sequence", "score", "token"}
+        valid_inputs = [
+            "My name is <mask>",
+            "The largest city in France is <mask>",
+        ]
+        invalid_inputs = [None]
+        expected_multi_result = [
+            [
+                {"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
+                {"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
+            ],
+            [
+                {"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
+                {
+                    "score": 0.19764970242977142,
+                    "sequence": "<s>The largest city in France is Lyon</s>",
+                    "token": 12790,
+                },
+            ],
+        ]
+        for tokenizer, model, config in FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
+            self._test_mono_column_pipeline(
+                nlp,
+                valid_inputs,
+                invalid_inputs,
+                mandatory_keys,
+                expected_multi_result=expected_multi_result,
+                expected_check_keys=["sequence"],
+            )
+
+    @require_tf
+    def test_tf_fill_mask(self):
+        mandatory_keys = {"sequence", "score", "token"}
+        valid_inputs = [
+            "My name is <mask>",
+            "The largest city in France is <mask>",
+        ]
+        invalid_inputs = [None]
+        expected_multi_result = [
+            [
+                {"score": 0.008698059245944023, "sequence": "<s>My name is John</s>", "token": 610},
+                {"score": 0.007750614080578089, "sequence": "<s>My name is Chris</s>", "token": 1573},
+            ],
+            [
+                {"score": 0.2721288502216339, "sequence": "<s>The largest city in France is Paris</s>", "token": 2201},
+                {
+                    "score": 0.19764970242977142,
+                    "sequence": "<s>The largest city in France is Lyon</s>",
+                    "token": 12790,
+                },
+            ],
+        ]
+        for tokenizer, model, config in TF_FILL_MASK_FINETUNED_MODELS:
+            nlp = pipeline(task="fill-mask", model=model, config=config, tokenizer=tokenizer, topk=2)
+            self._test_mono_column_pipeline(
+                nlp,
+                valid_inputs,
+                invalid_inputs,
+                mandatory_keys,
+                expected_multi_result=expected_multi_result,
+                expected_check_keys=["sequence"],
+            )
+
 
 class MultiColumnInputTestCase(unittest.TestCase):
     def _test_multicolumn_pipeline(self, nlp, valid_inputs: list, invalid_inputs: list, output_keys: Iterable[str]):

tests/test_tokenization_common.py

@@ -495,3 +495,16 @@ class TokenizerTesterMixin:
         assert [token_type_padding_idx] * padding_size + token_type_ids == padded_token_type_ids
         assert [0] * padding_size + attention_mask == padded_attention_mask
         assert [1] * padding_size + special_tokens_mask == padded_special_tokens_mask
+
+    def test_separate_tokenizers(self):
+        # This tests that tokenizers don't impact others. Unfortunately the case where it fails is when
+        # we're loading an S3 configuration from a pre-trained identifier, and we have no way of testing those today.
+
+        tokenizer = self.get_tokenizer(random_argument=True)
+        print(tokenizer.init_kwargs)
+        assert tokenizer.init_kwargs["random_argument"] is True
+        new_tokenizer = self.get_tokenizer(random_argument=False)
+        print(tokenizer.init_kwargs)
+        print(new_tokenizer.init_kwargs)
+        assert tokenizer.init_kwargs["random_argument"] is True
+        assert new_tokenizer.init_kwargs["random_argument"] is False